The question at hand is:
"Assume that the rest-mass energy $mc^2$ of the electron is equal to its electrostatic self-energy and that the electron charge is distributed uniformly inside a sphere of radius R. What is the value of R (in unit of meter)?"
My question is, what is electrostatic self-energy? Would this be equivalent to integrating $kQ/r$ over the volume of a sphere?
What is electrostatic self-energy?
The self energy of a particle means the energy possessed due to interactions between the particle an the system it is part of. In electrostatics, self energy of a particular charge distribution is the energy of required to assemble the charges from infinity to that particular configuration, without accelerating the charges. It is simply called the electrostatic potential energy stored in the system of charges.
For a simple example, consider an electrostatic field $\vec{E}$ due to some charge $q$. We need to know the electrostatic energy stored in the system of the charges $q$ and some another charge $Q$. Then, we assume we start from infinity (where the electric field due to the charge $q$ is zero). To assemble the charge $Q$ from infinity to a point at a distance $r$ from the charge $q$, we need to do work against the the electric field $\vec{E}$.
We have for electrostatic fields, $\vec{E}=-\nabla V$, where $V$ is a scalar function called the electric potential. The charge $Q$ at any point in the electric field of $q$ experience a force $\vec{F}=Q\vec{E}$ and the work is to be done against this force to assemble the charges in the required configuration. Hence the work done is
$$ \begin{align} W=-\int_\infty^r \vec{F}\cdot d\vec{r} &=-Q\int_\infty^r \vec{E}\cdot d\vec{r}\\ &=-Q\int_\infty^r (-\nabla V)\cdot d\vec{r}\\ &=-Q\int_\infty^r dV\\ &=Q[V(r)-V(\infty)] \end{align} $$
Assuming $V(\infty)=0$, we have the work done
$$ \bbox[5px,border:2px solid green] { W=QV(r) \qquad(1) } $$ This work done is stored as the potential energy of the system of the charges. We can extend the result to any number of charges and to any configuration. For example, we can find the electrostatic self energy or the energy needed to assemble a system of four charges at the corners of a square.
We know that the electric potential $V$ is the potential energy per unit charge, and the electric potential at some distance $r$ from a charge $q$ is
$$V(r)=\frac{1}{4\pi\epsilon_0}\frac{q}{r}\qquad(2)$$
Hence equation $(1)$ becomes
$$ \bbox[5px,border:2px solid red] { W=\frac{1}{4\pi\epsilon_o}\frac{qQ}{r} \qquad(3) } $$
This is the electrostatic self energy of the configuration.
Equation $(3)$ can be generalized to a system of $N$ point charges $q_i$ located at position vectors $r_i$ ($i=1,2,...,N$):
$$ \bbox[5px,border:2px solid blue] { W=\frac{1}{2}\frac{1}{4\pi\epsilon_o}\sum_{i=1}^N\sum_{j\neq i=1}^N\frac{q_iq_j}{r_{ij}} \qquad(4) } $$
where $r_{ij}$ is the separation between $r_i$ and $r_j$
Would this be equivalent to integrating $kQ/r$ over the volume of a sphere?
By definition, the electrostatic self energy in this case would be the work done in assembling the electrons continuously throughout the volume of the sphere of radius $R$. We need to find the electric potential at the surface of the sphere (i.e., at $r=R$). Since we have a continuous system of charges here, we have to replace summation in equation $(4)$ by integration. But it's not equivalent to integrating $kQ/r$ over the volume of a sphere. What we are going to do is as follows:
We build up the sphere by adding subsequent infinitesimal layers of charge (carried from infinite distance). From Gauss’s theorem we know that, for an uniformly charged sphere having charge density $\rho$, radius $r$, and total charge $q=q(r)=\rho(4\pi r^3/3)$, the field and the potential outside the sphere are those of a point charge $q$ located in the center. On building the sphere, we are constructing infinitesimal layers of charge of size $dq=\rho4\pi r^2dr$, thereby increasing the radius of the sphere from $0$ to $R$. Hence the total energy is (from equation ($1$))
$$ \begin{align} W=\int V(r)dQ&=\int_0^R k\frac{q(r)}{r}\rho 4\pi r^2dr\\ &=k\int_0^R\frac{\rho}{r} \left(\frac{4}{3}\pi r^3\right)4\pi r^2dr\\ &=\frac{4\pi\rho^2R^5}{15\epsilon_0}\\ &=\frac{3k}{5}\frac{Q^2}{R} \end{align} $$
where we have substituted for $\rho=Q/(4\pi R^3/3)$. So the electrostatic self energy in your problem is
$$ \bbox[5px,border:2px solid pink] { W=\frac{3k}{5}\frac{Q^2}{R}\qquad(5) } $$
where $Q$ is the total charge enclosed by the sphere. Equation ($5$) gives the energy stored in the system of charges to be assembled continuously throughout a spherical volume of radius $R$. All we have made use is just equation ($1$).
