What you're referring to is "gravitational" potential energy.. which is one of many forms of potential energy (gravitational, electrical, chemical, nuclear, etc.)
The idea behind potential energy is that it is energy that hasn't been yet released in an "obvious" form like movement, heat or radiation.
Think of it like money in the bank.. you don't say that it will potentially be money.. IT IS money! It is just money in a less liquefied form.
Same thing applies to energy. Potential energy - in all its forms - is not potentially energy - IT IS energy - the word "potential" comes from the fact that it will "potentially" be released as WORK (movement, heat, radiation, etc.) - or some form of "visible" energy - if you'd like.
Again with the bank analogy - potential energy is like "energy credit" that is stored within a particle due to its position in space. A book lying flat on the floor has zero potential energy when your "y = 0" is set to be the floor. If, by any chance, your book moves and ends up below the floor - it will have negative energy - which is not "negative energy" per se.. it is just an indication that your "zero" or "ground" wasn't deep enough.. which is not necessarily a bad thing.. When applying physics to real life situations.. it is often helpful to choose your "zero" of energy to be in such a place that is convenient to measure with respect to - this all points to concepts in Galilean relativity and how "frames of reference" relate to each other.
Now I am going to answer your question. From what I understand - you're referring to the "ABSOLUTE" energy an object can possess. In the sense that it is measured with respect to the "lowest zero" that can ever be.. and while that's a valid question.. It is very difficult and perhaps not even possible or even useful to quantify that "absolute zero".. There are just too many boxes energy can be hidden in.. Even if you drive that book to the center of the earth.. The book will still have gravitational potential energy with respect to the sun, moon, other planets and distant galaxies!
And that's not even talking about the electrical potential energy between the protons and electrons inside each atom of the book. And the nuclear potential energy that will be released if that book was to undergo nuclear fission, or the chemical potential energy released as heat and light if you were to set that book on fire, etc etc etc.
Potential energy is related to the arrangement of things in space - the arrangement of bodies within gravitational and electrical (and nuclear) fields giving rise to all the forms of "potential energy" or "energy credit" I've talked about. So in order to "find" the "absolute energy" you need to take into consideration all the spatial arrangements of your book with respect to other objects. In the sub-case of gravitational energy due to earth (your system is composed of the book and the earth only).. you'd need to measure your book's energy from the center of mass of the earth - that's ignoring other gravitational effects from the sun and moon, etc.
And that's calculable by integrating the gravitational force in spherical coordinates from the r = 0 (the center of the earth) to r = h (the height of the book from the center of the earth).
But even if you find that quantity - it will be quite practically useless unless you have a system that involves having your book dive deep into the center of the earth and you need to calculate the "potential energy released".
So that's why the idea of "frames of reference" is important in physics to "abstract away" the information that is relevant to us.
Hope this answers your question.