If sound is a longitudinal wave, why can we hear it if our ears aren't aligned with the propagation direction? If a sound wave travels to the right, then the air molecules inside only vibrate  left and right, because sound is a longitudinal wave. This is only a one-dimensional motion. If our ears are oriented perpendicular to this oscillation, e.g. if they are pointing straight up, how can we hear it?
 A: Sound travels outwards from a source in all directions. The waves that are set in motion are spherical.
A: Re. from one of your comments: "But when the air molecule from the centre keeps moving away ,won't there be a vacuum created at the centre" and also this one: "But if the sound wave is emitted for long periods, wouldn't there be a complete vacuum and the sound wave would stop"
I think part of you confusion comes from this:  Even with a longitudinal wave where the particle motion is parallel to the waves propagation direction, the particles do not travel with the wave.  They only move back and forth along the direction of wave propagation.  So the particles are not carried along with the wave.  (It is obvious that this is true for a transverse wave.)
Referring to your original question, unless sound is focused into a beam it generally propagates equally in all directions.  If it is focused into a beam and you were off to one side anything you hear would be due to sidelobes which are lower in amplitude than the main lobe and could be near zero.   
A: The revised question, as I understand it, amounts to asking how it is possible for a sound wave propagating along (instead of towards) a wall with a small hole in it to generate any sound waves on the other side of the hole. What happens in this case is easiest to explain with a diagram:

Whenever the air pressure on the upper side of the hole is different from the pressure on the lower side of the hole, the air on the lower side of the hole sees a net force, and so a new pressure wave is generated on the lower side of the hole. This is an example of diffraction.
As long as the wavelength of the wave on the upper side of the hole is much bigger than the diameter of the hole, at any moment, the air pressure on the upper side of the hole will be almost constant over the entire diameter of the hole, no matter which direction the wave is propagating in.  This is why it doesn't matter if the wave is moving along the wall.
Acoustic wavelengths in air (for the frequency range audible by humans) are roughly 1.5mm–17 m, and the ear canal is maybe 5mm in diameter, so, for all but the highest frequencies, the wavelength will indeed be much bigger than the hole.  (The diagram doesn't illustrate this.  Sorry about that.)
(N.B. A human's external ear has a much more complicated shape, which has evolved to efficiently gather sound waves passing the head in any direction and direct them into the ear canal, but it still does this by diffracting the wave.)
A: You could use an explosion as a metaphor. The shockwaves "push" the air around in a spherical pattern, which then gets "sucked" back due to the low pressure left behind.
In a sense, soundwaves are just very slow and small shockwaves.
This video shows it really well.
A: 
vibration is only a one dimensional motion

This is not generally true.  As a trivial example, one could the movements of water in a pond where a few small rocks have been tossed.  The motion is definitely a wave behavior, and could even be called vibration, but it is most definitely not one dimensional.
Another potential example would be the vibrator on your phone, which vibrates in a circular manner.
But in the end, the key is that atoms in a sound wave don't vibrate "left and right."  They are a longitudinal wave, in which particles move in the direction of the wave's motion and back.
So when something causes a sound, the waves propagate outward from the object creating the sound, as molecules of gas move away from the source and towards the source.  This is typically a 3 dimensional pattern
A: Sound wave is not a transverse wave, as you thought. That means the vibration and the direction of propagation for sound wave are parallel. And the vibration is caused by difference in air pressure at different places. To the question "how I can listen to it" thats because the pressure difference propagates toward your ear and force your eardrum to vibrate.
A: There is more than one way to describe the amplitude of a sound wave. You can describe it as a displacement, in which case it's a vector with units of meters. On the other hand, you can also describe it as a pressure, which is a scalar with SI units of pascals.
It's possible to have a sound sensor whose response is proportional to the displacement, or one whose response is proportional to the pressure. The ear acts like the latter, because the eardrum is a membrane, and the membrane distorts in response to the pressure difference between the inner ear and the outside air. Therefore the ear is not sensitive to the direction in which the wave was propagating (although there are other cues that allow us to infer this for some frequencies, because we have binaural hearing).
A: While the mean air motion of the wave is in one direction (assuming a plane wave), the air molecules actually move in all directions. They are in local thermal equilibrium (due to frequent randomizing collisions), which is what gives meaning to pressure as the basis for modeling acoustics. This random molecular motion in all directions is at speeds of order the speed of sound, hundreds of meters per second.
The mean motion (longitudinal) is an oscillating displacement of micrometers or less for typical sounds, at kilohertz frequencies, corresponding to a speed of millimeters per second at most. It can be much less for faint sounds. The ear is a remarkably sensitive detector!
The ear canal is smaller than the wavelengths of audible sound. Thus, as sound passes by in any direction, the ear mainly responds to the pressure oscillations without regard to the direction of the wave. When a pressure peak surrounds the ear, air is (slightly) pumped into the ear, due to the random motions that equilibrate pressure. When a trough surrounds the ear, air is (slightly) sucked out of the ear. This happens at the frequency of the sound (say a thousand times per second), vibrating the eardrum.
Zwol's answer correctly notes that this can be seen as an instance of diffraction. It is a limit in which the hole is so small that the pressure at any instant is nearly uniform over the hole, so diffraction through the hole is nearly independent of the incident angle.
A: Re. If our ears are oriented perpendicular to this oscillation, e.g. if they are pointing straight up, how can we hear it?
The eardrums react to the pressure difference from one side to the other side. Since the sound waves have a long wavelength compared to the diameter of the eardrum, the ears are not that sensitive to the incoming direction of the sound wave.  If the incoming direction is perpendicular or parallel the pressure difference varying with time across the eardrum will be the same.  It makes no difference whether the waves are longitudinal or transverse.
See:
https://en.wikipedia.org/wiki/Wavelength
"The wavelengths of sound frequencies audible to the human ear (20 Hz–20 kHz) are thus between approximately 17 m and 17 mm, respectively." 
