In your example it is relatively to choose a system with constant mass and so Newton’s laws can be used.
You start off with a pure collision type problem involving two separate masses, one moving and one not and those two masses are the system.
À collision between the masses occurs over a relatively short period of time so any external impulse (force) due to the spring on the system of two masses is negligible compared to the internal impulses (forces) the masses suffer during their collision.
It becomes a collision with linear momentum being conserved and the final velocity of the two masses being half the original velocity of the mass which was moving before the collision.
Once the velocity of the two masses has been found the next phase of the analysis can be done he system now being the 2 masses and the spring with the kinetic energy of the 2 masses plus the spring potential energy being constant.
The maximum compression of the spring occurs when the velocity of the two masses is zero.