Using Huygens to draw pictures is good for illustrating diffraction but you have to be cautious when interpreting the diagrams.
The diagram below shows the Huygens construction for wavelets separated by one wavelength and there being 11 secondary sources in the slit.
Lines $XX'$ and $YY'$ demarcate the region where one would expect to have light if one was considering light to travel in straight lines - geometrical optics.
What you must note is the envelope of the Huygens wavelets between those two lines is a straight line and its length stays constant as the waves travel further and further from the slit.
So by the time you get to 40 wavelength from the slit you almost get what you expected - a semicircular wavefront centred on the slit.
For clarity in the second diagram I have just shown three 3 secondary sources in the slit.
This drawing of the wavelets can be avoided using Mathematics and refinements to the secondary wavelets.
One refinement is to have the amplitude of a secondary wavelet dependent on the direction of its travel being a maximum for the part of the wavelet which is travelling in the same direction as the incident light and getting less an less as the direction of travel moves away from the direction of the incident light.
Also as the wavelets move further from the slit their amplitude is reduced because a certain amount of energy now has to be spread over a longer wavelet.
So the diagrams which you draw give you an idea as to what happens but for the detail you must resort to methods which do not rely on drawing semicircles on paper.