Just like different materials have varying reflectivity that leads to some of the variations we can see in visible images. Different materials have varying emissivity that affects how much they radiate. You end up with less information of the scene than you would have for the corresponding visible image because you dont get all the shadows that your brain uses (in part) to perceive depth. Absorption also varies with emissivity and because of this different materials may actually have different temperatures.
Also, not all materials are Lambertian and therefore you will see different radiation depending on angle.