What is the height in the potential energy formulation? Most examples I've seen describe the (gravitational) potential energy e.g. with an example of a ball thrown upwards, and explain how the sum of the ball's kinetic and potential energy is constant at all times. This is clear to me.
Based on the formula $U=mgh$ my understanding is that an object resting on the ground has a potential energy of 0 (as $h=0$).
What I can't understand is the following setting. Imagine there are three platforms of various heights (see the attached picture). I'm standing on the middle one holding a ball. As the ball is above the "ground", it has some potential energy that is dependent on the height $h_0$. If I would move the ball to the right, so that it is completely over the rightmost platform, its potential energy would be different now, as the distance to the ground has changed (it is now $h_1$). But now, if I would move the ball the same distance but in the other direction, its potential energy would be yet different, as it would be proportional to the  $h_2$. Importantly, in both cases I would need to exercise exactly the same amount of work to move the ball. This, of course, is absurd, as the total energy of the system should be constant, and in this case I could end up with different energies based on whether I move the ball left or right. So, what am I missing?
While writing this question I realized that one possible solution to this problem is to measure object's height relative not to the "ground", but to the gravitational center of the system. Is this how the potential energy is actually defined, and the "height above the ground" is just a simplification to explain the concept?

 A: Actually, gravitational potential energy is constant with reference to a certain point. For example, in the above query, you found out that energy is suddenly increasing when you move it over to the right. This is not the case. The P.E of the body would be the same even if you hadn't moved it to the right
In fact, there is no correct definition for Potential energy. It is only change in potential energy which matters and which we can find.
To find change in potential energy, you just subtract P.E of the two positions.
Hope it helps!
A: There are two parts to your question.

*

*Potential energy, is always defined with respect to something. In three dimensional space, this means a reference value should be specified on a two-dimensional surface. So, when we write $$U = mgh$$, we are actually saying that the potential energy at the surface of the earth is zero. You are of course free to choose whatever reference you feel like. Also, note that the above is an approximation to the potential energy, and in general the potential energy at infinity is taken to be zero. (Note, that you could have also resolved this issue, by thinking about the curved nature of the earth's surface.)

*Now, I urge you to consider all the forces acting on the system. In work-energy theorem, one considers even the work done due to internal forces(In contrast to the change in momentum and its equality with total external force)

