I know that conservative forces can exist outside a system, but I thought that potential energy only exists for internal conservative forces.
It does. But for an object to have gravitational potential energy the "system" has to be the object together with the earth. That makes the force of gravity an internal (to the system) conservative force.
Regarding your comment to @alephzero about the ball in free fall, the ball alone does not have gravitational potential energy. It is the ball/earth system that possesses gravitational potential energy. For this reason you can't say the ball alone has potential energy. If it were not in a gravitational field, its acceleration could just as well been due to any external force such that $F=mg$. This, as I understand it, is what led Einstein to the equivalency principal and eventually to the General Theory of Relativity.
Regarding the two points you would like clarified:
I posted this on physicsforums as well and the people there are saying that in more advanced physics, the internal/external distinction is not used. Is this true, or is what you said correct throughout physics, ie if the system doesn't include the earth then there is no GPE.
I think it is true, as discussed in response to your second point. However, I didn't mean to suggest the object has no GPE. I meant it makes no sense to refer to an object alone as having GPE. It makes no sense for someone to just say "that ball sitting on that table has GPE". I would immediately ask how much and with respect to what? Is it with respect to floor supporting the table?. With respect to the ground outside the room where the table is located? How about with respect to the reference frame of the surface of the table itself? (where it has no GPE!). The bottom line is potential energy is the energy a system has due to its position, shape, or configuration. To me, at least, without specifying the system it makes no sense to talk about GPE.
If what you said is true, then it seems that in more advanced physics, the distinction is indeed not used, because in another book I am using by Kleppner and Kolenkow their definition of PE just assumes that the force is "conservative" without at all mentioning "internal"
I must confess I have not taken any advanced physics classes. I switched from physics to engineering in my junior year. But it's true, as @Aaron Stevens said, internal and external forces are just subjective labels, because it depends on how the system is defined. I think the key is if something has potential energy (any kind, not just GPE) it is due to work by conservative forces because a conservative force is one for which the work done depends only on the initial and final end points of the motion and not on the path taken. As far as I know, potential energy only involves work done by conservative forces. So in that respect there is no need to distinguish between an internal and external force if’s a conservative force.
Hope this helps.