By blowing at pencil, a piece of paper, or another object up to fifty centimeters away, I can cause it to move away from me significantly. But I can't move an object toward myself by inhaling sharply from that distance, even if it is extremely light. Why is that?
I think the reason is that when you are blowing on an object, you are making lots of air particles collide with it perpendicularly in one direction thus transferring a lot of momentum to the object. When you are sucking air in, the only force that's acting on the object is by the air particles that rush in to fill up the gap that you just created. These particles come in from all kinds of different directions failing to transfer momentum in a consistent way.
I think the reason is twofold.
First, when you are inhaling the air goes from an unbounded domain, to a bounded domain, for exhaling this is the other way around. So for exhaling the air gets accelerated in the hole between the two domains (your mouth), because of continuity (contraction). When you inhale, there is no such requirement, there is only deceleration.
Second (also mentioned by Vinayak), is the direction. When you exhale, you can consider the flow of air as a turbulent jet. It is known for turbulent jets that the spreading angle is about $20^\circ$ (this has to do with turbulent diffusion of momentum) which is generally true for all fluids. This is the reason that after this $50cm$ you cannot blow anything away: the jet is wider, and thus weaker/lower velocity.
Physiology probably makes up for another good part of the effect: The lung is more easily compressed than expanded because muscles primarily work by contraction (and the skeleton´s muscles are more or less wrapped around the chest).
Also, compressing an enclosed volume (e.g. a paperbag) is easier than it´s expansion because the volume´s surface is under tension when compressed (which stabilizes the process) but becomes unstable when expanded (you have to pull in a lot of directions, while compression can be applied at one point).
You can also answer this by just considering the pressure difference between your mouth and the outside-- without even thinking about fluid dynamics!
The absolute max limit of pressure difference you can ever get by sucking-in is simply atmospheric pressure. In other words, the best vacuum generator in the world will, at best, give zero psi abs pressure whilst the outside is ~15 psi.
The absolute max pressure difference you can get when blowing air can be quite high-- several atmospheres (consider inflating a balloon, it must have a higher pressure than atmosphere to even be a recognizable balloon).
This applies even if you're considering the case of blowing or sucking through a straw.