You are right in that there is only one set of physical things going on in diffraction. The reason people talk about two different kinds, is because there are two natural limits in a diffraction problem.
The intensity of light you see at any point is the contribution from all of the points at the aperture, where the contribution from any point decreases as the distance, and every contribution accumulates phase given its path. It is the differences in the path length from the various parts of our aperture to a point of interest that lead to the interesting interference phenomenon associated with diffraction.
Consider an aperture with a characteristic size $a$, and imagine trying to figure out the diffraction at a point roughly in line with the aperture at some distance $d$ from the point at the aperture's center. We can estimate the relative phase difference from the point at the aperture's center and a point near its edge, namely
$$ \Delta \phi = k ( d_{\text{edge}} - d_{\text{center}}) $$
where $k$ is the wavenumber of our light. We can estimate this difference in length using some simple trig.
$$ \Delta \phi = k \left( \sqrt{ d^2 + a^2 } - d \right) \sim \frac{ k a^2 }{ 2 d } $$
So, there is a natural trade off in our problem, between the size of our aperture and the distance we are from it. In particular, we can separate the problem into two limits, one where $ d \ll ka^2 $ where we expect large differences in phase contribution ($ \Delta \phi \gg 1 $) and $ d \gg k a^2 $ where we expect little difference in phase contribution ($ \Delta \phi \ll 1 $). These are the Fresnel and Fraunhofer regions respectively. I've included a little picture for illustration.

As you can imagine, these two limits have very different qualitative phenomenon,
and so that's why people talk about them as two different kinds of diffraction. In the Fresnel limit you have mostly geometric optics type cast shadows, with perhaps some wiggly bits near the edges of your shadow, whereas in the Fraunhofer region, our wave has spread out over a large region and starts interfering with different parts of the cast image. This leads to the observed behavior of Fraunhofer diffraction corresponding to a Fourier transform of the aperture.
In the case of visible light, this characteristic distance is quite large, $$ ka^2 \sim \left( \frac{ a }{ 1 \text{ cm}} \right)^2 \frac{ 2 \pi }{ 550 \text{ nm} } \sim 1 \text{ km } \left( \frac{ a }{ 1 \text{ cm}} \right)^2 $$
so that the Fraunhofer diffraction cannot be seen directly. This is why you commonly see Fraunhofer diffraction associated with the use of a lens, as a converging lens allows you to view this far field pattern much more practically.
Reference
Applications of Classical Physics by Roger D. Blandford and Kip S. Thorne - Chapter 8 - Diffraction