Other answers have suggested that the diffracted photons are receiving momentum through interaction with the slit filter, but I find this hypothesis untenable, as the portion of light that interacts with the wall of the slits bounces back and decoheres, not contributing to the diffraction pattern, so we can fully ignore their contribution here.
Quantum amplitudes are linear, hence removing or filtering a portion of a wavefront does not count as an interaction, hence it cannot transfer momentum to the diffracted amplitude.
The correct answer lies elsewhere: as you pointed the transversal section of the beam wavepacket is assumed to have net momentum drift of zero, and a small Gaussian spread. We need to think about the evolving 2D cross-section of the beam as the system of interest.
As you might know, Gaussian beams are the optimal cross-sectional shape for light to keep itself from spreading. We can think of it as a semi-rigid phase of light.
After the diffraction, it loses the Gaussian transversal section that keeps the beam transversally "packed", and it "dissolves" as if it was transitioning from our conceptual semi-rigid phase to a liquid-like phase which a much higher beam divergence (we certainly can measure beam divergence in rate of increase of beam spread over longitudinal distance)
The Gaussian shape is special because it "saturates" the uncertainty principle inequality such that:
$$ \Delta x \Delta p \propto \hbar $$
This gives its special property of optimal shape conservation over time. Just that, in this case, the "time" variable of the 2D packet is the longitudinal axis of propagation, and instead of actual spatial coordinates, the spread happens in angular coordinates.
When our Gaussian 2D beam crosses the diffraction slit, the Gaussian pattern is "cut" in two slices. What happens to these individual slices of our beam is that their transversal position are resolved better than before, so their transversal momentum must spread out (otherwise they would violate Heisenberg's uncertainty inequality)
An increase in the uncertainty of a variable after a measurement seems to contradict the conservation of the variable, but this comes from the idea that photons existed along the trajectory, and everything suggests this picture of a photon like a small dot carrying momentum along a trajectory is misleading and incorrect