Potential energy does belong to the system. In particular, the potential energy of "an" object cannot be defined without reference to another. When we calculate, say, the gravitational potential energy "of a satellite", we compute
$$U = -G\frac{Mm}{r}$$
and two masses enter in: that of the Earth and that of the satellite. Moreover, both are interchangeable by commutativity, so it is just as valid to take the "focus" from the Earth - it just feels "weird" to us to imagine this, psychologically, since we think the active agent is the one that is dominant, but in truth the satellite is acting just as much - with just as much force - on the Earth as the Earth is acting on the satellite. The real difference is that the Earth is much more "lazy", or "inert", hence the term "inertia", than the satellite: it is vastly (on the order of $10^{21}$ if we take a 1 Mg satellite) less responsive to that force, but the truth is it feels just as strong a pull from the satellite itself.
Because both partners are equal participants, but not equally responsive to each other's mutual influence, and both enter into the calculation of potential energy, we consider it as shared between them. It would not exist with one partner missing - either one.
This becomes more evident when you get to more than two bodies: if you have three, you have to add up the potential from all 3 ways you can choose separate pairs of bodies. For $n$ bodies, you need to add up $\binom{n}{2}$ energies, not $n$. This number grows quadratically. This is also why the total potential energy of a solid sphere is proportional to the square of its mass (or charge, for electrostatics, which has the same mathematical form).