According to Newton's third law of motion, to every action there is an equal and opposite reaction. When I'm applying the force on the body which is kept against the wall, so I'm not just applying that force over the body but also on the wall, as a result it reacts by applying opposite force on me which is equal in magnitude to that of my force. Therefore, both the forces cancel each other out and the net force is 0. However, if I apply a force on the wall directly then the wall will exert an equal amount of force over me. These forces are not cancelling each other out. The reaction force is acting on me, whereas the action force is acting on the wall. Inertia is the property of an object which tends to keep the state unchanged, and mass is the measure of inertia of an object. Therefore, I'm more likely to be pushed by the wall backward and accelerate because my mass is very much less than the wall, whereas the wall wouldn't move from its place because it has a greater mass and hence greater inertia.