For any pure substance, the internal energy is a thermodynamic function of state, which means its equilibrium value (per unit mass) is determined once we specify any two intensive properties. This is irrespective of how the material arrived at that state.
For an ideal gas, the equilibrium value per unit mass does not depend on two intensive properties; it depends only on temperature, but not specific volume. So the change in internal energy between two thermodynamic equilibrium states depends only on the temperatures at these two states. If we change from state A to state B by first changing the temperature at constant volume, the difference in internal energy per unit mass will be $C_v(T_B-T_A)$. If we then change the specific volume from $V_A$ to $V_B$, there will be no change in internal energy. Since the internal energy is a function of state, the change in internal energy in going from $(T_A,V_A)$ to $(T_B,V_B)$ will be just $C_v(T_B-T_A)$, irrespective of whether the process is at constant volume.