Just like different materials have varying reflectivity that leads to some of the variations we can see in visible images. Different materials have varying emissivity that affects how much they radiate. You end up with less information offrom the scene than you would have forin the corresponding visible image because you dont get all the shadows that your brain uses (in part) to perceive depth. Absorption also varies with emissivity and because of this different materials may actually have different temperatures.
Also, not all materials are Lambertian and therefore you will see different radiation depending on angle.
Finally, thermal cameras are quite sensitive. Without a lot of effort in establishing uniform temperature in a scene different things and materials will naturally be at different temperatures. This may occur if things are heated/cooled unevenly or if there are differences in absorptivity.