As far as I understand it the gravitational binding energy of some distribution of mass is the negative of its gravitational self-potential energy.
I tried to calculate the latter for a solid sphere of radius $R$, mass $M$, and uniform density.
By the shell theorem (or Gauss's law of gravitation), the field strength at a distance $r$ from the center of the sphere is given by
$$\frac{GM_{enc}}{r^2}=\frac{G}{r^2}M\big(\frac{r}{R}\big)^3=\frac{GMr}{R^3}$$
where $M_{enc}=M(r/R)^3$ is the mass enclosed in a sphere of radius $r$.
The gravitational potentiel at a distance $r$ created by this distribution is thus
$$V=-\frac{GMr^2}{2R^3}$$
The self-gravitational potentiel energy is the sum of the gravitational potentiel energies $U \cdot dm$ over all the mass elements $dm$ in the distribution.
Let's proceed by shell integration. The mass contained in the shell of inner radius $r$, outer radius $r+dr$ is simply
$$dm_r=4 \pi r^2\cdot dr\cdot \rho=4 \pi r^2\cdot dr\cdot \frac{M}{4 \pi R^3}=\frac{3Mr^2dr}{R^3}$$
The self-potential energy of the sphere is thus
$$\int^{R}_{0} V(r)dm_r=\int^{R}_{0}\big(\frac{-GMr^2}{2R^3}\big)\big(\frac{3Mr^2dr}{R^3}\big)=\frac{-3GM^2}{2r^6}\int^{R}_{0}r^4dr=-\frac{3GM^2}{10R}$$
which is exactly half of the correct answer.
I checked my work multiple times for simple mistakes but I can't seem to locate the source of the factor of $2$ error. This leads me to believe there is something fundamentally wrong with the way I calculated the energy.
Where is the problem?