Basically the title. Light can go into my room through some lenses so that I can see it, but light can't escape my room so that people outside? The article about peepholes on Wikipedia doesn't elaborate at all about how this works.
Peepholes are usually wide angle lenses. This allows the home owner to see all corners of their front porch, or for hotel room guests to see practically the entire hallway. Wide angle lenses capture more light than a flat lens because light coming from the sides can still impact it and be refracted. The peephole lens then condenses all of the light it collects down into a tiny stream that is then fed into the eye of whoever is looking through it.
When you look through the peephole from the outside, however, this effect is reversed. The eye tries to collect a wide angle of light coming from a small stream from inside the room. The effect is that whatever you are seeing looks tiny, so you cannot clearly see the face of whoever is inside the house.