Imagine I am standing on Earth, and pushing a tennis ball away from me. The ball moves. If it is very heavy, I will move back instead of the ball. Now consider the same scenario in outer space, where both the ball and I are hanging without any third body to support/assert gravity on us. What happens in this case? Will the ball move or will it make me move backwards, despite being very small? Or will both of us move equally away from each other? How exactly does inertia come into picture here?
When you push on the ball, it pushes back on you with exactly the same force for the same amount of time. Both of you therefore always experience the same impulse. However, the speed imparted to a object by a specific impulse is inversely proportional to its mass. Both you and the tennis ball will move away from the point of contact, but the tennis ball much faster than you.
For example, let's say you and your spacesuite are 100 kg and the tennis ball 200 g. The tennis ball will end up moving away from your original combined rest position 500 times faster than you will, but you will both always move.
Another way to look at this is that total momentum is always conserved. Assuming you both started by floating together and we observe in this same inertial reference frame, the combined momentum of you and the ball must always remain 0 unless some external force acts on either of you. Since you pushing the ball around are all internal forces, your momentum plus the ball's must always sum to zero. If the ball is moving one way, then you must be moving the other way, but at a lower speed due to your higher mass.
This is how rocket engines work. They throw stuff out the back really really fast to give it as much momentum as possible, which gives the same momentum to the rocket but going the other way. The center of mass of the rocket and all its exhaust stays in the same place.
"imagine a spherical cow, in vacuum...."
Newton's Third Law explains the question.
Regarding the follow-up in the comment, the trajectory of your spaceship will not be altered, same reason. You cannot push against the back wall without reacting against something else, and that something else is part of the same solid body (or will be when it hits the front wall). Sum of all forces: zero.
In both cases, the net forces remain in the same place - the barycenter of the system.