I have been thinking and searching about this a lot and have still got all my doubts.
My main hypothesis is that, since our body is actually rotating (and not moving) during push ups, torque must be the main factor. When we push up, we have to produce a torque which should cancel the torque due to gravity. When we do flat push up we are rotating directly against the direction which gravitational torque is acting. However, when we do inclined push ups, the direction in which we are rotating is not same as the gravitational torque, a smaller component of gravitational torque is along the direction of our rotation. So, we have to produce lesser torque to rotate at an incline.
In other words, our body is being pulled down by same force still(at an incline), but we aren't moving directly against that pull. Its like standing against a wall and doing push ups on the wall. There is no Torque against the direction we rotate (i.e perpendicular to the wall).