Working through the math:
$$
\dot{m} = \frac{dm}{dt} = \iiint_{CV} \frac{\partial \rho}{\partial t} dV + \iint_{CS} \rho \vec{V}\cdot dA
$$
If the flow is steady, all time derivatives are zero and you are left with:
$$
0 = \iint_{CS} \rho \vec{V} \cdot dA
$$
If we consider a 1D flow just as an example, this will give you:
$$
0 = -u_{\text{left}} \rho_{\text{left}} A_{\text{left}} + u_{\text{right}} \rho_{\text{right}} A_{\text{right}}
$$
or
$$
u_{\text{left}} \rho_{\text{left}} A_{\text{left}} = u_{\text{right}} \rho_{\text{right}} A_{\text{right}}
$$
which is an expression that is probably familiar. So, that all shows that $\dot{m} = 0$.
Now to your other equation, $\dot{m}_{\text{system}} = \rho V A$. That doesn't hold for steady flow. So the part you are missing is that your second statement isn't true, and there's no way to get this expression from the equation we started with if we assume the flow is steady. This expression will sometimes show up when specifying boundary conditions, as in $\dot{m}_{\text{in}} = \rho V A$ -- but if it is steady, that means that $\dot{m}_{\text{out}} = \rho V A$ also and the total $\dot{m}$ of the system is zero.
In other words, if you see something that says $\dot{m} = \rho V A$ and it also says the flow is steady, it means they are specifying the $\dot{m}$ across one of the control surfaces. Not all of the control surfaces.
