I have been searching on the Internet but have not found a derivation of the formula for the self gravitational potential energy of a sphere. Can someone show how to do this? I assume it involved 6 nested integrals (3 for each particle $dm$ and 3 for integrating the potential energy for all other particles).
No need to make it that complicated--calculate the potential at the surface of a sphere with a uniform mass distribution with mass $M$. Then calculate the energy required to add a bit of mass with mass $dm$ and radius $dr$ to the top of this sphere. Express $M$ and $dm$ in terms of $r$ and $dr$ and integrate.