As long as your system is thermally isolated from the environment so that the expansion is adiabatic, the internal energy must stay constant during the expansion. This follows directly from the first law of thermodynamics.
But thinking in terms of the gas particles accelerating into the vacuum due to the pressure gradient, isn't the gas particles acceleration increasing the total KE of the particles in the system.
Here you are making two mistakes. First, you mix up the macroscopic picture of the gas (pressure gradient) and the microscopic picture (molecular motion). In a dilute (ideal) gas approximation, there are no forces acting on the molecules. They simply fill whatever volume is available to them by their random motion. This explains why in an ideal gas, the velocity distribution of molecules does not change upon free expansion, hence the internal energy and temperature remain constant.
The second mistake is the automatic identification of the kinetic energy of molecules with the internal energy of the gas. This is only valid in an ideal gas. In a real gas, the internal energy consists of the kinetic energy of the molecules and the energy of their interaction. It is the total internal energy that remains constant, not just the kinetic energy. In dilute gases, the molecular interactions are typically attractive, which means that upon free expansion, the molecules slow down: the kinetic energy decreases while the interaction energy increases by the same amount. As a consequence, the temperature (not the internal energy) of dilute gases typically decreases upon free expansion. This is the cooling effect you feel when you take a can of spray and let all the gas escape at once.