Watch water going down a drain. It has a rotational symmetry and goes into a vortex, whose boundary conditions perpendicular to the flow cover an area much larger than the hole of the drain.
Watch water coming out of a hose. The boundary conditions defining the vortex are to start with the area of the hose perpendicular to the motion.
It is the same with the air, blowing out the mouth is the bound of the vortex thus the energy per unit area carried by the flow is much larger than by inhaling. In inhaling the vortex boundaries are variable but certainly much larger than the hole of the mouth, thus the energy carried by the induced flow of air per unit area is much smaller.
It is the impact of air that blows out the candle, and the area of the flame is small so that a directional blow from the mouth can turn it off. In inhaling the area covered by the flow cannot be controlled and the energy per unit area impacting the flame is much smaller.