The viscosity of the fluid actually determines if the fluid could flow "fast enough" or not.
Viscosity,$\mu$, is a property of fluid determined by the ratio of shear stress and velocity gradient:
\begin{equation}
\tau(\mathbf{u})=\mu\nabla\mathbf{u}
\end{equation}
which is just the shear stress version of the familiar pressure, except the force is now parallel to the surface rather than perpendicular.
We can see that given a fixed shear stress, the acceleration,$\nabla\mathbf{u}$, decreases if the viscosity increases, meaning the fluid would not be able to flow "fast enough" if a large enough viscosity is given.
Imagine something with super high viscosity, say, tar road. Punching tar road is more painful than punching a body of water because that tar can't deform "fast enough" and hence the impulse of your punch is very high, and hence the reaction of your force during the process of your "punching" is high as well.
To prove that the impulse of your punch is very high, imagine now that you are a perfectly spherical ball with a brain and nervous system and all that, and someone throw you into a pool of water for some reason. Stokes derived from the definition of the shear stress and told us that the reaction force(we usually call it the drag force) that you will experience is actually:
\begin{equation}
F_d=6\pi\mu r\mathbf{u}
\end{equation}
where r is the radius of the ball.
From Newton's second law of motion,$\mathbf{F}=m\mathbf{a}$:
\begin{equation}
m\frac{d\mathbf{u}}{dt}=-6\pi\mu r\mathbf{u}
\end{equation}
Now solve the equation, we can actually get the velocity as a function of time:
\begin{equation}
\mathbf{u}=\mathbf{u}_0 \mathrm{exp}(\frac{-6\pi\mu r}{m}t)
\end{equation}
where $\mathbf{u}_0$ is the velocity when you hit the water.
And by the definition of impulse, we have:
\begin{equation}
\mathbf{J}=\int^{t_f}_0 6\pi\mu r\mathbf{u}dt=\int^{t_f}_0 \mathbf{u}_0 6\pi\mu r \mathrm{exp}(\frac{-6\pi\mu r}{m}t)dt=\mathbf{u}_0 m(1-\mathrm{exp}(\frac{-6\pi\mu r}{m}t_f))
\end{equation}
where $t_f$ is the time you stop moving, theoretically $t_f\rightarrow\infty$, but realistically, you cannot be in the water forever.
We see that if $\mu$ is high, the impulse is high, and vice versa. That's why punching tar road hurt much more than punching a body of water.
So yes, it's related to impulse and momentum. And it's not that water can't flow fast when the impact speeds up(well, that is to assume that water is a perfect Newtonian liquid, but that's another story), but that impact is large when the fluid can't flow fast.
Inertia is part of the answer, yes. But inertia is caused only by gravity, whereas drag force actually takes the electromagnetic property of the material into account.
