The other answer deals with the classical view, so as an extension, I will give you a more QM explanation.
The materials you talk about are mostly solids, like the wall, car's metal etc. These solids have certain lattice structures, where the molecules and atoms are in certain relative positions, and these relative positions can hardly change.
The bond that bonds the atoms into the molecular structure of the lattice in the metal and the wall is called covalent bond (some residual effect of the EM force). This bond means that certain electrons, will exist around the whole molecule rather then the single atoms.
And yes, it is this effect of the EM force that keeps the atoms in the molecule, and the lattice. Now if you want to brake the lattice (brake the wall), then you need to apply enough energy to brake the covalent bond.
When the car hits the wall, the kinetic energy of the car's matal's molecules and atoms is enough to brake the wall's atoms out of the covalent bond of the wall. The metal of the car has a lattice that is even more tough, strong. The metal atoms will stay in the lattice more likely, and together they make sure that the kinetic energy is transferred onto the wall's molecules and atoms, and since this level of kinetic energy reaches the limit, where the wall's lattice's atoms will lose the covalent bond, the wall will fall apart.
So there are two effects:
the kinetic energy needs to be enough to brake the atoms out of the covalent bond from the wall's lattic
the lattice's covalent bond of the car's metal is just tougher, stronger then the wall's lattice's covalent bond
You can see this if you try to brake the wall with let's say another material, a wooden car or a plastic car or a glass car or a ceramic car. That will never brake the wall (completely), however hard you hit the wall with it.