There is a question, Contradiction between law of conservation of energy and law of conservation of momentum? The answers say that earth does gain some velocity that is negligible. So I ask, what if we all push a wall in the same direction? And if we have been pushing walls since so many years, has the velocity of earth increased?
As hinted at in the comment to your question, one must ask: What are you pushing against? That is, whatever force your hands apply to the wall, your feet must apply to the floor in the opposite direction. The end result is that your efforts have no effect. (Well, if you push hard enough, you can knock the wall down, or perhaps just slide across the floor, depending on the coefficient of friction, but you won't cause any net motion of the Earth.)