Take a look at this picture - and ask yourself why the operator of the camera has the cloth over his head (the cloth is black on the inside) as he is looking at the back of his camera - which has a piece of ground glass where the image from his pinhole camera is forming:
This used to be how photography was done:
Align the camera to the subject, focus (if you had a lens - pinholes don't need focusing). Cover the aperture. Insert photographic plate. Tell subject to stand still and stop breathing. Remove protection from photographic plate. Open aperture. Light magnesium to produce lots of light. Close aperture. Put protective cover back on photographic plate. Take plate to darkroom. Develop. Fix. Rinse. Dry.
The pinhole is tiny - not a lot of light comes through. All the other light from the environment will drown out the image. You need to make sure that the only light you see is from the object.
The simplest "pinhole camera" is formed by the leaves on a tree. Did you ever notice how the sunlight coming through the leaves makes circles? Those are "pinhole camera images" of the sun. And when there is a partial eclipse of the sun, those circles turn into little crescent shapes:
Which only goes to show that a pinhole camera works really well when there is more light coming from the subject (in this case, the sun) than from any other object.
Incidentally, you can sometimes get a similar effect in a dark church when a single beam of (sun) light comes through a hole in a dark window: when clouds pass near the sun, you will see their image on the floor/wall of the church.