Thanks for all good comments and answers! After thinking again about the problem I'm posting an additional answer because of one major reason that wasn't clear to me and which is explaining why you see a faint "visual-like" image in thermal images: Diffuse illumination, pretty much like in normal visual images.
In normal images you see things because of both, direct illumnation from the sun and diffuse illumination resulting from photons which are Rayleigh- or Mie-scattered in the atmosphere or reflected by other objects. So the photons hit the scene you observe from every direction. For our everyday experience of seeing things this is essential (if we would only have direct illumination we couldn't see anything inside the shadow or the backside of things not oriented towards the sun).
Now, for thermal radiation I guess there is also diffuse illumination. The air itself has some heat and will emit thermal photons, other objects will emit thermal radiation as well, and most likely there will still be a few photons at thermal wavelengths in the solar spectrum as well. This diffuse illumination is of course much smaller than in the visible, but apparently enough to see some faint "visual-like" image that is a result of the orientation of surfaces, reflectivity in the $\mu$m-range etc., which determine the amount of reflected photons. This is just the analogue to an image in the Visible, produced by diffuse illumination (e.g. in the shadow), but much fainter as there is less light. And this (familiar looking) image is then dominated by the much stronger signal of thermal emissions caused by the heat of the observed objects. (The analoge in the Visible would be to observe a lamp - you would hardly see the lamp itself because it is shining).