Why do images not appear inverted when looking directly through a pinhole camera? I understand that the way light takes through a pinhole creates an inverted image on a surface behind the pinhole. I remember this effect from school experiments, it's also described in this wikipedia article. I punctured a piece of paper and looked through it (instead of watching the reflection), the image appeared as normal to me. Why is that? Why doesn't the scene appear upside down when looking through the hole?
 A: The totality of light that fills space, even in a geometrical optics approximation, is a complicated light field that is difficult to visualize in full. Each point in space generally receives rays of various colors and from various directions simultaneously. We do not perceive or measure the light field unless something is placed in it to absorb or scatter the light -- ultimately our retinas.
Many surfaces that interact with light are not strongly sensitive to the direction from which it comes. These include matte (diffuse) screens, photographic film/CCDs, and our retinas (but not mirrors). At each point, the response effectively integrates rays over all incoming directions.
At a random point in space, this is likely to work out to a general ambient illumination that does not select for where the light originally came from -- i.e., does not form an image. Forming an image requires a specially manipulated light field (often, as in the eye, using a lens) that counteracts the tendency of bundles of rays to diverge and mix.

Now, we can clarify the pinhole situation (diagram inspired by seeking to clarify and correct pwf's answer). A pinhole is designed to form an image on a surface (screen or film) without the need for a lens. This is accomplished by restricting the light field without bending any rays. The image is, of course, inverted (relative to the actual objects).
When your eye is looking directly through the pinhole, an image will form on the retina. The lens of the eye has makes relatively little difference in this case, assuming your focus is relaxed enough to look "through" the hole rather than "at" the hole. The image on the retina is inverted (relative to the actual objects) just as if you were looking at the same scene without any barrier. Therefore, your perceived view of the objects is non-inverted.
If you place a screen (not shown in diagram) facing the pinhole, it will show the inverted image. However, this is visually observed via diffusely reflected/transmitted light from the screen that forms its own light field. This light field is inverted relative to the original light field because its source (the pattern on the screen) is inverted relative to the original objects.
You perceive what is on the screen when your eye forms an image from this new light field. Therefore, the image of the screen that is formed on your retina is non-inverted (relative to the actual objects), and perceived as inverted.
A: You don't see an inverted image by looking through a pin hole because the pin hole is acting as a beam limiting collimator, not a lens - which is to say pinholes do not bend light direction.

*

*light rays from points in the scene only reach points on a screen (placed behind the pinhole) if they are very nearly parallel (accounting for pinholes having a non zero diameter) and traveling along a line drawn from the scenic point to the pinhole.

*Such parallel rays from scenic points pass directly through the pin hole and continue to travel in a straight line before reaching the screen without being bent as they would in a lens.

*Geometrically, because rays from high low, left and right areas of the scene will pass through the pinhole and arrive at low, high, right and left areas of the screen respectively, the resulting screen image is inverted.

If you use an eye to look through the pinhole however, you just see a nearly parallel set of rays arriving from points across a small area of the scene that are reaching the pupil - everything else has just been blocked out by the paper surrounding the pinhole.
The beam collimation effect of the pinhole is to limit light to a notional tunnel formed between the circumference of the pinhole at one end, and the circumference of the pupil (of an eye) at the other. Light which enters the pinhole but leaves the tunnel goes unseen.
The closer the pupil is to the pinhole, the less collimation takes place - rays don't have to be so parallel to reach parts of the pupil's surface and you see more of the scene. The brain interprets rays arriving from the unblocked area of a scene in normal fashion: upright and non inverted - just as it would the same photons arriving without a pinhole in place.
I'll leave details of perception and operation of the visual cortex to another time :)
A: Has the OP just reduced the size of his pupil by adding another aperture (the pinhole) in front of his eye?
A smaller aperture of the will make the scene seem dimmer and if the OP requires corrective lenses their vision will improve as the "stopped down" aperture improves depth of field.
A: The answer is rather simple, when you look through the hole, the hole disappears!
A hole in an opaque object is called a $hole$ precisely because to look through the object, the only place you are gonna get any light is when you are looking through the hole. But when you look through the hole, light from objects behind can reach you just fine - in effect there is no opaque object left that light rays need circumvent or be affected by.$^1$
This scenario is exactly the same as if there was no pierced sheet of paper to begin with.
$\ldots$
The effect occurs only when you look through the hole and not when you look at a screen placed some distance away in front catching an image. This is because the size of the hole is relative to the size of the screen.

*

*if you (your eyes are the screen) or a screen are brought increasingly closer to the hole, relatively speaking the hole is getting larger and larger. Near the hole, the hole is relatively infinite$^{1.5}$. All of light can reach you here without undergoing inversion.


*if instead one uses a telescope from afar to peer through the hole, one once again sees an erect$^2$ image. This is because nice parallel rays are able to reach the telescope exactly like when there was no screen.


*Imagine a tiny little screen, comparable in size to the hole, placed very close to it $\ldots$imagine yourself to be a microscopic point observer gliding on wind currents through the hole. Do you expect to see the object behind inverted?
$\ldots$
When a screen is placed at some distance from a hole, the only way light can reach the  now relatively large regions of the screen is if it angles in. The only part where it reaches straight is, well to reiterate, comparable to the hole's size.

$^1$ignoring diffractive effects
$^{1.5}$ This idea that the closer you get, the larger (or effectively $\infty$) the thing becomes, is a common modelling theme in physics$\ldots$ $multipole$ and what not
$^2$telescope's optics notwithstanding

A1. The model of the observer (be it the eye or its lens or some DSLR or a piece of cardboard) has nothing to do with the physics here. As long as the observer is large enough when far away and small enough when close by, your observations hold.
A: Ignoring diffraction, the pinhole can't change the orientation of what you see because it doesn't change the position or direction of any light rays. It just blocks some of them.
When you put a screen in front of an illuminated object, every point on the object emits light in every direction, which hits the screen at every point. You can imagine that the screen contains superimposed images of the object with every possible position and orientation, but you can't see them because they average out to a uniform blur. Putting a punctured sheet between the object and the screen blocks out most of those images, making the remaining one visible. It doesn't create the image, as such. Blocking the light differently would leave different images visible. For example, if you replace the punctured sheet with a laptop privacy filter, you ought to get a blurry upright image (I haven't actually tried this). A filter that creates a sideways image is also possible in principle.
But no matter what filter you put between the object and your eye, you can't change which image orientation your eye sees, because it has its own mechanism for extracting an image from the mess of light rays in the environment, and you are only giving it less to work with.
A: I assume that what you are describing is peering through a pinhole such that the field-of-view you get is smaller than normal. That is, I assume you describe a looking through the pinhole that comes with "tunnel-vision".
So, I assume that what you are describing is peering through a pinhole such that for the light to traverse the distance from the pinhole to the point where the light transits into the eye is like traversing the length of a short tube.
The distance from your eye to the pinhole is very short. The closer your eye is to the pinhole, the larger the field-of-view you obtain (but some reduction of field-of-view is inevitable).
So: think of peering through the pinhole directly as peering through a tube. When you are peering through a tube you are not surprised that your field-of-view is smaller, and you are not surprised that what you see is still the right side up.
A: The directly-through-the-pinhole image is upside down on the retina of your eye. But all images on the retina are upside down. When the lens in your eye forms a real image on the retina it is inverted. It only looks the right way up to you because you brain post-processes the retina image  in the visual cortex.
By looking with your eye at   the image formed on  the screen  at the back of the camera you see  a triply inverted image – inverted once by the pinhole, once by the lens in your eye, and once by you brain – so you see an upside down image.
A: Looking through a pinhole is like looking through a lens which is close to the eye.  You are seeing an erect virtual image.
A: Consider the following situation:

*

*a flamingo is standing behind a screen with a pinhole

*you look at the flamingo through the pinhole from a position above the pinhole

*you will now see the bottom, i.e. the feet, of the flamingo

*if you move your eyes down to below the pinhole, you will see the head of the flamingo

If you were to put photosensitive paper in the plane where you moved your eyes, you could capture an inverted picture of the flamingo, but basically your eyes are doing the same.
The paper is capturing at once what your eyes would see if they scanned over the entire surface of the paper. The paper sees the whole flamingo at once, whilst your eyes only see a tiny bit of the flamingo at once (if not, your pinhole is not really a pinhole).
What your eyes see:

What a pinhole camera would capture:

A: The answer is one about simple geometry and has nothing to do with what happens on the retina or in the brain: you are not looking through a pinhole camera when you are looking through a small hole in the paper.
Either you are relatively far away from the hole. Then it acts simply as a block. All rays from whatever is on the other side that do not go straight from the source through the hole into the lens of your eye will be blocked. Since the hole is relatively far away, basically everything is blocked, and you will see the hole simply as a little twinkling dot of light.
If you are relatively close to the hole, then the fact that there is paper around the hole simply doesn't matter, and you are seeing the unobstructed view of what you would have seen anyways.
In both cases, the pinhole changes nothing about the route the rays travel to your eye, it simply blocks a more or less significant amount.
The actual pinhole experience happens if you put a white wall behind the hole. Now the rays go from the source through the hole, and hit that wall (inverted, as you expected). Then those rays are bounced off the wall into your eyes - and remarkably from the wrong vertical part, leading to the illusion of being upside down. It is this two-stage operation which leads to your eyes being able to actually see an inverted image. By looking through the pinhole directly, you are removing this aspect.
A: Let's say you're looking at a tree for simplicity, and the hole is at ground level. The light scatters off the tip of the tree in many directions, but only some of it goes towards the hole.
When you look in the hole, the light is coming from above, so you see it pointing up. When the light has to go to the screen, the rays from the tip go below the hole, so you see it pointing down.
A: Put simply, there are two factors:
Pinholes don't flip the image, but filter light rays.
A point on the subject/source will cast rays of light in many directions, but only one will make it through the pinhole. The image is mirrored about the pinhole because the remaining rays are those that cross the point (e.g., from top to bottom), but that ray would have still been there if the "pinhole" was just a "hole".
Your pupil acts as a second pinhole.
Imagine two pinholes in sequence. The first one filters out some light, and the second one does as well. But the image is not "flipped" twice! When you stick your eye up to a pinhole, you are simply re-filtering the light (and bending it, but that's not the cause of the "flip" effect).
