Our brain is quite a mighty signal processor. It fuzes many many clues together into one cohesive scene.
As has been mentioned in other answers, the folds of your ear generate interference patterns. The effect of this is to "color" the sound, raising the volume of some frequencies and damping others. These interference patterns are different from different directions, so sounds from different directions get colored differently. Our shoulders also have a similar effect, which is a major player in our understanding of whether a sound comes from above us or below us.
Of course this frequency based approach only works if your brain has some clue what the frequencies should be. Most of the time we can make great guesses, based on a lifetime of listening to things. Sometimes it can be tough.
I did an experiment a while back with my wife to test this. I only did the experiment once, so I cannot claim scientific riggor, but the results were far too amazing to not share. I grabbed a pair of spoons to clack together and told my wife to close her eyes. I told her I'd clack them in different places, and then she needed to reach out and try to point where she thought the spoons were. I did this several times and she reliably got the left-right portion of the direction right (the part that can be determined by delays between the ears), but her up and down sense was way off.
Then I asked her to open her eyes and repeat the experiment. No surprise, its really easy to point at spoons when you can see them! The interesting part came next: I had her close her eyes again. No matter where I clacked the spoons, she almost immediately pointed at them with an eerie precision.
My hypothesis (and I must call it a hypothesis because it is insufficiently tested to be called anything more) is that when she started the experiment, she did not know the frequency spectra that the clacking spoons would have. Thus she could only make vague guesses about how her ears had colored the sounds. When she opened her eyes and saw the spoons clack, the story changed. Now her eyes told her exactly what direction the sound was coming from, so her brain could un-color the sound and start remembering what the "true" sound of the spoons was. When she closed her eyes again, she now knew enough to figure out what her ears were doing to the sound, and thus the direction it came from.
Arguably, she heard the spoons with her eyes.
I encourage others to try this experiment... I want to know if it works just that well for others!