Yes, the amount of force applied affects the outcome. What's happening is that any force that you apply comes in a force-pair by Newton's third law, so if you push on him with a force $\vec F$ then the force $-\vec F$ is imposed on you to impose a property called "conservation of momentum," which in a very deep sense comes from the fact that the laws of physics are the same if you move one millimeter over from where you presently are. If you perform this experiment in a frictionless setting you will both move inversely proportional to your masses.
There is a force of friction, however, due to the chair on the ground: when rolling this force is relatively negligible but when stopped it can be comparatively stronger. This force scales like the normal force which in this case is going to be proportional to your weights, as well.
When you move your arms to push your colleague there is generally some springy interface which responds to some of this force. Try to push yourself from your desk softly, then firmer and firmer, until you finally start moving: you will notice that you "move" at first but it is only you pushing yourself "into the back of your seat." If you apply the force too gently, then this springy force will build up between two people in chairs until one of the chairs (with lower friction) starts moving, and then it will relax that compression and the force will suddenly drop a little on your colleague's wheels. This is the basic reason why "when I'm standing he moves, when I'm in my chair I move." If you give a really good shove you should be able to impart momentum to the both of you before this force relaxes -- if your upper body is not strong enough you can probably do it with your feet against the back of their chair and giving a very sharp kick.