There are two criteria. First, $c_V=nC_V$ must be constant; second, we must have that $P = f(V) T$ for some function $f$. Both criteria hold in the specific case of an ideal gas, but neither holds for a general thermodynamical system. I'll give the mathematical explanation first, and then the physical explanation second.
Starting from the perspective that $U=U(S,V)$ and $T= \left(\frac{\partial U}{\partial S}\right)_V (S,V)$, note that a small change in $S$ and $V$ will cause small changes
$$dU=\left(\frac{\partial U}{\partial S}\right)_V dS + \left(\frac{\partial U}{\partial V}\right)_S dV \equiv TdS - PdV$$
and
$$dT = \left(\frac{\partial T}{\partial S}\right)_V dS + \left(\frac{\partial T}{\partial V}\right)_SdV$$
Solving the second equation for $dS$ and substituting it in the first equation yields
$$dU = T \frac{1}{\left(\frac{\partial T}{\partial S}\right)_V}dT- \left[T\frac{\left(\frac{\partial T}{\partial V}\right)_S}{\left(\frac{\partial T}{\partial S}\right)_V} + P\right]dV$$
$$= c_V dT+ \left[T\frac{\left(\frac{\partial P}{\partial S}\right)_V}{\left(\frac{\partial T}{\partial S}\right)_V} - P\right]dV$$
Where we've used that $\left(\frac{\partial T}{\partial V}\right)_S = \frac{\partial^2 U}{\partial V\partial S} = -\left(\frac{\partial P}{\partial S}\right)_V$, and that the definition of the specific heat at constant volume is $c_V \equiv T \left(\frac{\partial S}{\partial T}\right)_V$. Finally, note that
$$\frac{\left(\frac{\partial P}{\partial S}\right)_V}{\left(\frac{\partial T}{\partial S}\right)_V} \equiv \left(\frac{\partial P}{\partial T}\right)_V$$
so finally
$$ dU = c_V dT + \left[T\left(\frac{\partial P}{\partial T}\right)_V - P \right]dV$$
Assuming that we are not dealing with variable numbers of particles, what has been written here is completely general, so your question boils down to asking when the second term is zero. The answer is that
$$\left(\frac{\partial P}{\partial T}\right)_V = \frac{P}{T} \implies P = f(V) T$$
for some function $V$. If this is the case, then the second term on the right of the preceding equation vanishes, and we have
$$dU = c_V dT \implies \Delta U = \int c_V dT = \int nC_V dT$$
since $C_V$ is the specific heat per mole. If $C_V$ is constant, then this just becomes
$$\Delta U = nC_V \Delta T$$
From a physical standpoint, the answer is that the energy of an ideal gas is purely kinetic - the gas particles do not have any long-range interactions with each other at all. As a result, since the temperature can be shown to be a measure of the average kinetic energy of the ideal gas particles, the internal energy of the system is unaffected by changes in volume, as long as the temperature is fixed.
This would not be the case if the particles attracted each other, for example. Putting such a system in a larger box with the same amount of kinetic energy would result in a larger average spacing, and therefore a less negative potential energy (remember that attractive potential energies are negative). Therefore, larger box $\implies$ more energy, even if the kinetic energy didn't change.
As I showed,
$$ dU = c_V dT + \left[T\left(\frac{\partial P}{\partial T}\right)_V - P \right]dV$$
The first term on the right describes the change in energy due to change in temperature while holding the volume fixed; the second describes the change in energy due to a change in volume while holding temperature fixed. Because of the lack of interaction between gas particles, the second term goes away, leaving only the first, and so
$$dU = c_V dT = nC_V dT$$