The following may be useful to consider.
It is true that Huygen's principle states that every point on a wavefront acts as a source of secondary waves which travel in all possible directions. However, those individual waves are not seen as separate, but only as a superposition.
In other words, say you are looking at a source that is very far away, it will arrive at your eyes essentially as plane waves. If one was to construct Huygen's wavelets at the halfway point between the source and your eyes, those waves would again combine to become plane waves when they reach your eyes. So, if you are looking at the perpendicular direction, the wavefront will not enter your eyes.
However, if you imagine placing a barrier with a small hole half way between the source and your eyes, it might be possible to see the source if you are not looking directly at the it due to diffraction since only part of the plane wavefront is allowed to pass through the hole.
The below figure shows the two cases.
I hope this helps.