I'm not an expert in peepholes specifically, but as far as I know, there are a couple of factors that give a peephole its peculiar properties.
First, according to Wikipedia, the hole is fitted with a fisheye lens. This is a lens that, in one direction, takes a very wide field of view (up to a full half-sphere) and focuses it into a small area. This diagram should give you a rough idea (although this is certainly not exactly accurate to reality; maybe I'll do a better one later):
The lens is mounted such that it gathers light rays from all angles on the outside of the door, and focuses them on a small pupil-sized region on the inside of the door, so that you (being on the inside) can see whatever is on the outside, through a wide range of angles. This also has the effect of concentrating the light from a large area on the outside into a small area on the inside, thus increasing the intensity (power per unit area), thus making the image brighter.
However, the same process also works in reverse, and that's why the peephole makes it difficult for people to see into your room. If your eye, on the inside, can see a wide range of angles from the outside, a person on the outside can only see a small range of angles from the inside. So they wind up getting a magnified view of a small part of your eye, basically. Also, just as the peephole makes the image outside appear brighter to you, it makes you on the inside appear dimmer to a person on the outside, by spreading the light coming out over a large angular area.
One other factor that tends to help, at least for houses during the daytime, is that there's a lot more light outside than there is inside, so someone attempting to peer in through your peephole also has to deal with reflection of the exterior light getting in the way. (This particular effect also works with windows)