So remember that we're talking about orbitals and filled shells and intentionally inserting impurities and so forth, there's this big complicated microscopic picture going on, and we want to understand it. That's the context of your question.
Minimum energy principle
Fortunately in physics we have a very useful crude tool to understand such things: the pivotal observation that friction forces always oppose velocity, so the power exerted through them $\vec F \cdot \vec v$ is always negative, and so things are always "trying to" get to a minimum energy, until you get down to the thermal chaos that lives at an energy scale of $k_\text B~T$ above that minimum. (Never forget these effects; the minimum energy principle literally predicts that the air molecules of our atmosphere should fall to the floor, so this thermal chaos is literally the only reason you're breathing right now...)
In your sophomore kinematics classes you probably derived the principle of buoyancy from the minimum energy principle; this is another opportunity to use it; in a semiconductor you have all of these atoms next to each other and they generally have their own separate electron levels (shared electron levels are known as a "conduction band" and semiconductors have a "band gap" between this conduction band and these separate electron levels).
Now we can "dope" these semiconductors, placing in their crystal lattices the occasional "wrong" atom, called an "impurity" -- though this is not a bad thing! And those electron levels are shifted somewhat by the added nuclear charge, but also they are more or less filled than usual. We usually start to think of levels of the whole system rather than being very localized here or there.
In N-type doping, we use impurity atoms which are closer to the halogens, so the lattices have "more electrons" than the substrate would normally have. In P-type doping, we use impurity atoms which are closer to the alkali metals, so the lattices have "fewer electrons" (hence more holes!) than the substrate would normally have. Usually either sort of doping makes the semiconductor more conductive; in N-doped semiconductors the electrons carry currents; in P-doped semiconductors the electron-holes carry currents.
Now here is the whole crucial thing: when you put these two dopants next to each other, there is a minimization of energy when the "extra" electrons fall onto these atoms which "don't have enough." In other words: the minimum-energy state features charge separation. You might think, "but the charges attract each other, don't they?" and that's true -- and this balances out against the above tendency. But it does not erase that tendency: there is a net positive charge on the N-type semiconductor and a net negative charge on the P-type semiconductor as the charges pretend that the entire world looks like the substrate lattice that it finds itself in.
A voltage between them?
This middle region where the electrons have "fallen into" the holes is a "depletion region" for charge carriers, a sort of dynamic break in the circuit. But the depletion region grows or shrinks depending on applied voltage--that equilibrium charge separation can be intensified or else made to vanish completely. So under a "forwards bias" of a certain voltage the PN-junction becomes a very good conductor, but under "reverse bias" by the same voltage it stays an insulator, until the bias gets so strong that the diode experiences "breakdown" and the semiconductor conducts regardless.
This "forwards bias" voltage is this "potential difference" that you're talking about here, the 0.6V-0.7V "diode drop" that one uses in practical electrical engineering analysis when travelling across a diode that one assumes is conducting normally. You can indeed interpret it as a fixed voltage occurring due to charge separation, but it's important to understand (and this is where I think your question gets answered): it does not cause the electrons and holes to separate; rather it is the result of that separation.
So the cause of the separation is Pauli exclusion. It's that you have these electron-shells that are more full than they "want to be" over here, and less full than they "want to be" over there; put them in contact and they work together to be only as full as they should be, until another effect (the induced voltage) counteracts this.
Obligatory mention of Fermi levels and chemical potential
A fuller understanding of why they "want to be" in one state or the other involves heavy use of the phrase "Fermi level", but basically you want to understand what a chemical potential is. Just like the laws of entropy maximization imply that two systems will exchange energy until they come to the same "temperature", those same laws imply that two systems will exchange particles until they come to the same "chemical potential." I find that one of the best ways to explain this to students is to imagine vacuuming an entire room by just vacuuming one corner of it. How does this work? You keep one corner extremely well-vacuumed, and then when you walk around the room and kick up dust and step on both the clean and dirty parts of the room, some of this dust falls randomly on the well-vacuumed space where it soon gets sucked up. So the room is at a higher "dust potential" than the well-vacuumed space, so this drives an "entropic current" of dust from the rest of the room into the well-vacuumed space, and the bigger the "potential" difference the greater the current. This "dust potential" is precisely the chemical potential.
Similarly, there is a chemical potential for these electrons known as their "Fermi level", and the dopants adjust not just the aggregate "valence/conduction" band gap and the states inside of it, but also they adjust where those bands are relative to the charge-neutral Fermi level. So then when you have a junction, you draw the Fermi level flat (in equilibrium no electron transport goes either way!), but on the far left side this band gap has the Fermi level close to the valence band so the band gap is positioned "higher" and on the far right side this band gap has the Fermi level close to the conduction band so the band gap is positioned "lower", and you connect the bands continuously to each other in the middle in a normal band diagram.