When I put my hand on a hot solid why don't the particles transfering heat to my hand exert a force on it? When I put my hand on a hot metal (say) solid, I can feel my hand heating up. I suspect this is caused mostly by particles (electrons, atoms, ...?) from the solid colliding with the particles that make up my hand thereby transferring kinetic energy to it. But why does this lead to my hand heating up and not it (also?) being pushed?
 A: The important point is the question: What kind of energy is transferred to your hand.
Heat in solids is caused by vibration of the molecules and similar effects (see e.g. this question). An object can be very hot without exerting any macroscopic force, because the molecules only wiggle around instead of moving together in one direction.
When you touch a hot surface, the molecules in your hand start to wiggle as well, because the vibrations of the solid transfer to your hand. The important thing to understand here is that these vibrations cause similar vibrations in your hand, but cannot turn into macroscopic mechanical energy. This is why your hand heats up, but is not pushed.
In short: Mechanical energy and heat are both energy, but not easily turned into each other. A hot surface transfers heat to your hand, but no push. Similarly, pushing your hand would not cause it to heat up.

EDIT:
Let me elaborate on the difference between "mechanical energy" and "heat".
The over-simplified physicist model of a solid is a lattice. The atoms/molecules are connected by electric bonds which we see as elastic springs connected points next to each other:

Mechanical energy here would be the whole lattice moving in one direction. This needs a lot of energy as you need to move all atoms.
Heat in this picture corresponds to the points vibrating, i.e., moving back and forth relative to each other. Each vibration has a much lower energy because it only involves the motion of a single atom. Also, the movement is very chaotic because the atoms move in different directions.
This lattice responds very differently to different kinds of energy transfer.
To increase heat, you have to hit single atoms with a relatively high amount of energy per atom. This will cause them to vibrate.
To increase mechanical energy, you have to hit many atoms with a low amount of energy per atom.
Now lets relate this to real situations. A hot solid has strongly vibrating atoms. Putting your hand on it will transfer heat because your hand is hit by small but fast vibrating atoms.
If you close off a gas container with a piston, you'd get two situations:
With high pressure, but low temperature, the piston is hit by many gas atoms of low energy and you get mechanical energy.
With low pressure, but high energy, the piston is hit by few gas atoms of high energy and will heat up.
A: Here is another scenario where the thing that you describe does happen:
A tube is filled with a gas, for example plain air. The tube fits nicely around a finger. The fit is so precise that there is a sufficient seal, so the air cannot escape, but there is only just enough friction between the tube wall and your finger to prevent the tube from sliding off just like that.
Gently increase the temperature of the gas. Now the molecules of the gas have a higher average velocity. The effect of that higher average velocity is that your finger is pushed out of the tube. The force on your finger arises from the accumulative effect of gas molecules bouncing against your skin.
A gas doesn't have internal cohesion. When you give a gas opportunity to expand it will.

Now consider a solid. A solid has internal cohesion. A solid does not expand like a gas at room temperature, and neither does it expand like a gas when you heat it up. (A solid will expand a little, but that's not visible to the naked eye.)
When you heat a solid the molecules of the solid move back and forth faster than at colder temperature. Let's say a particular molecule has - just for an instant - a velocity away from the bulk of the solid. So the molecule is on its way to ascend out of the solid. But as that molecule ascends the forces of cohesion from the neighbouring molecules increase. As a consequence the ascending molecule is pulled back into the solid.
The molecule now acquires a velocity back towards the bulk of the solid. This molecule will overshoot, and will very briefly create a local indentation of the solid.

The motion of the molecules of the solid do transfer heat to your skin as you are touching the heated solid.
And it's not just the outward punches that transfer heat. There is also an effect of interaction with the transient indentations from molecules overshooting on their way back into the bulk of the solid. You can think of that as a suction effect, if you will.
As to your skin being pushed one way or the other: the combined effect of the "punches" and the "suctions" adds to zero. What remains is the transfer of heat. For that transfer the effect of the "punches" and the "suctions" do add up; that is the transfer of heat from a solid to your skin.
A: Your hand is subject to a force, but one that operates over a microscopic distance, so its effect is not transferred to you appreciably.
Contrast the effect to that of steam escaping into a cylinder of a steam engine. The energetic water molecules impact the face of the piston within the cylinder causing the piston to move. The steam remains in contact with the face of the piston as the piston moves, continuing to exert a force on it.
By contrast, the energetic articles that comprise the hot solid are confined within the solid. Unlike the water molecules in steam they are unable to exert a force beyond a microscopic distance from the surface of the solid. Your hand would be 'pushed away' to that extent, but the distance would be a million times smaller than anything you could feel.
The force due to thermal expansion of a solid can be very significant, but it operates over a very short range.
A: It is because the Boltzmann's constant is very small.  You would need temperatures of the scale of $10^{20}$K to feel the push.
A: Your hand is being pushed. If you put a hand onto the body and then suddenly heat the body up in a fraction of a second by 100 degrees, you would feel a mechanical kick into your hand. If you squeeze a body between two plates to prevent it from moving, and then heat the body up, the pressure onto the plates will increase. If you do not let the body expand upon heating, it is going to push you with all it's thermal might.
A: Kinetic energy and thermal energy are fundamentally different phenomena.  We imagine the energetic behavior of particles as essentially similar to the motion of condensed objects because that's the only form of motion we see.  But no one has ever seen an atom and we can only speculate about the details of its behavior through inferences drawn from observations made on a scale that is larger by several orders of magnitude. What's really going on can't be described by analogy to anything familiar.
Something to consider when contemplating the essential difference between thermal motion and kinetic motion is the relativistic invariance of temperature.  If we were to look at the surface of another earth-like planet passing by us at very high speed, we would notice that familiar processes appeared to be happening more slowly.   Objects and people would appear to be moving in slow motion from our perspective.  But if we were able to zoom in further and examine the behavior of the particles making up the mass of that planet, we would observe them to be moving at the same "speed" as would be expected for particles on our planet. If we were somehow able to extend a thermometer and measure the temperature of the matter on the planet, it would perfectly match that of our own.  We could be certain that these temperatures were truly invariant by examining the qualities of the states of particular substances such as water at its triple point.  We could push this to a further extreme and imagine a spaceship moving at 99.99% the speed of light, powered by a fusion reactor with a plasma temperature of 100 million K.  The temperature will remain invariant in all frames even though the "speed" of certain particles would be required to exceed the speed of light.  One is forced to question whether "speed" is the appropriate term.
The nature of heat and thermal energy and how they translate to the familiar scale of motion is very mysterious and confronts us with the limitations inherent in our models of the physical universe.
A: Solids can't exert pressure on you (as long as you don't put them on your body in a gravity field or let yourself be hit by someone throwing them at you). So when you put your hand on them your hand will not be pushed away from it. If you place your hand on the top of a test tube filled with a gas and you suddenly increase the pressure inside the tube, your hand will receive a push too. The same holds if you fill the tube with a liquid (your hand will heat up or cool down too if the temperature of the gas or liquid differs from your hand). If you fill the tube with a solid and increase the pressure inside the solid, the solid will obviously not give you this push. It can transfer part of its internal energy to your hand though, thereby heating it (if its temperature is higher than that of your hand). This shows that only a small part of the total internal energy of a solid is transferred to your hand in comparison to the energy transferred by an expanding gas (as in the test tube example).
Likewise, you can't make a solid move simply by putting your hand on it (without pushing it) while you can stop a gas or a liquid from flowing outside the container it's in.  The key difference is the deformability of gas and liquid, in contrast to that of a solid.
