But im struggling to understand where exactly the potential energy is stored in mass.
It is largely in the binding energy of the protons and neutrons in the matter. This represents the vast majority of the rest mass of the nucleons. However, the fundamental particles also have a non-negligible rest mass themselves. This is true for the quarks that make up the nucleons, and also true for electrons. They are small compared to the total mass, but on the rough order of 0.1 or 1%.
Also, the nucleus itself presents significant adjustment in this number, due to the relative energy differences of various atoms. Light elements release energy by making larger atoms (fusion) and very heavy atoms release energy by splitting into smaller ones (fission). These all affect the total "mass" as we commonly refer to it.
Matter is extraordinarily stable. Even the chemical bonds around us are mostly unchanging, or else you would never buy anything from the store! Because of this arbitrary (some might say "anthropic") property of the universe, we have a very well defined configuration of particle with which we can reference "rest mass", which is easily measurable by various methods.
I can conceptualize the classical example of a ball on a platform in the air representing gravitational potential, because if the platform is removed, the ball falls.
You should know that the transition is negligible. If the ball falls, the energy is converted to thermal energy, so aliens will would not measure a change in Earth's mass due to this process, even if they had the absurdly accurate measurements to do so. Over time, thermal energy is radiated out into space.
Jupiter's thermal radiation is, in large part, due to gravitational collapse. Another significant part comes from the absorption of light from the sun. Earth is much closer to the sun, so most of the dynamic energy here comes from solar energy. A rogue gas planet without a nearby star will only have a temperature due to the slow process of gravitational energy release, atomic decay, and the incredibly low temperature of space.
But i dont understand how one could measure or define force on an atomic level.
Is it a function of the force vectors being applied by fundamental forces on all particles in the system?
For example, if it werent for the strong nuclear force, electrons would go crashing into the nucleus because an attractive electromagnetic force exists between protons and electrons. So does the force vector of the electron being pulled towards the nucleus count as potential energy that an atom has?
Up until this point, you've not said a thing about force. It's distracting you here.
Potential is more important. In the hypothetical of gravitational energy, separation distance maps (1-to-1) to gravitational potential. Gravitational potential times mass equals energy, in the same way that electric potential times charge equals energy. That's the utility of the "potential" concept. Potential (in the gravitational example) is force x distance, which might help disposition you regarding force.
The force vector is mostly irrelevant for the example you choose, and for the one I'm talking about. You could say that force x distance mostly equals the rest mass. This is because 1) rest mass is mostly binding energy and 2) potential is force times distance. But it gets complicated with quarks.
In fact, I've tried my best to avoid talking about quarks, because they are muddled by conservation of color charge. That makes it difficult to define what a proton is a combination of to begin with, because you can't just take its constituent quarks and isolate them! That's tricky. Nevertheless, the concept of rest mass can still be communicated to beginners in an obvious way. The universe has fundamental particles, for which quarks and electrons are included. Those have rest mass due to field excitations, which are described in detail by quantum field theory. However, the rest mass of atomic matter is mostly due to the proton/neutron binding energy.
So in a 5kg object there are 450,000,000,000 joules.
Oh, let me return to answer this. We can't currently convert this mass into energy. But in the cosmic sense, there is no energy or mass, just matter-energy.
Small black holes could theoretically be a power plant by being fed atoms, and radiating out Hawking radiation. Depending on the specific parameters, this radiation might be mostly massless photons. Photons still carry energy, but they have 0 rest mass. By this process, we could (theoretically) convert atomic matter into particles with no rest mass. This could (theoretically) all be used for useful work.
This isn't remotely relevant for the present level of development of Earth civilization today because creating a black hole is difficult, but I think it's instructive.