Imagine an alien drops a transparent artifact nearby. Not knowing what it is, or whether it's dangerous, you decide not to try to walk up to it and touch it. All you know is that you can see through it, and thus, you don't really know what its outline looks like. However, you have several children and lots of plastic balls. So you give each child a bucket, and ask them to walk in a big circle around the artifact, and throw balls at it. As they do so, you observe the balls bouncing off the artifact in a particular pattern which gives away its overall shape, more or less.
Now, this is a rather imperfect analogy, because the balls respond to gravity quite a bit, and you can see balls that don't bounce towards your eyeball. And yet, this is, on a very gross level, how vision works. The plastic balls are a "poor man's photon", and the children are crude "light sources".
Imagine that the alien drops multiple artifacts, some of which are actually close to together. Now, if a child throws a ball that bounces off one artifact, hits another, and bounces again, you will get information about both bounces. But if the child happens to be at an angle where you don't notice the first bounce because the ball is moving directly towards or away from you, then all you will really see is the final trajectory of the ball. More importantly, if you could freeze time and look at all the balls in flight, along with a short momentum vector, would you be able to answer the question: "Which artifact did this ball bounce off of?" you could most likely guess by looking at the vectors and seeing where they converge. But if you had to answer the question: "What other artifacts did this ball bounce off of?" I think you would be hard-pressed to answer this question at all, because the final direction of the ball doesn't give you this information.
In the same way, when a photon comes into your eyeball and is detected by your retina, there is no "history" of its path encoded into the photon. But think about what the world would look like if you could see the "photonic history". Whenever you look at a scene, you would not only see the traditional image, but you would also see all shiny objects in the scene superimposed on everything else which is visible, all the way back to the light sources. So if there's a lamp nearby which illuminates most of the scene, you would see the lamp in every part of the scene, which would be pretty confusing at best.