Your reasoning based on the ‘rest mass energy’ is not really relevant to the analysis. On earth, a nucleus at higher altitude would hold more gravitational potential energy. But I don’t think you would intuitively imagine that this would make it less stable. My point is, you could define many different kind of energies/potentials in which you put your nucleus, that doesn’t make it relevant to its stability.
Instead, just recall a very basic idea of the physics of what you are studying to guide your choice of a relevant quantity . The protons and neutrons are held together by the nuclear force. So what you want to minimize is the potential due to that force. Gravitational fields would be irrelevant to describe how strongly the nuclear force holds the nucleus together, right? Similarly, the mass energy doesn’t really help to describe it.
So the energy that needs to be smallest is the one due to all interactions between nucleons. If this energy of interaction is smaller when the nucleons are packed together vs when they are apart, then they wanna stick together. The more negative that difference is, the more stable the nucleus (i.e. the more energy needed to take it apart)