Since $V=\int_0^R (4\pi r^2) \space dr$ produces the correct formula for the volume of a sphere, the problem is elsewhere. Namely in the $r^2$ factor inside the volume integral.
The problem is that MMOI is measured by the perpendicular distance of a particle to an axis. So the $r^2$ factor inside the integral is incorrect, as it used the radial distance and not the perpendicular distance.
The correct way to integrate is to consider spherical coordinates and use
$$ V = \int_0^R \int_{-\pi/2}^{\pi/2} \int_0^{2 \pi} ( r^2 \cos \psi) \space {\rm d}\varphi \space {\rm d}\psi \space {\rm d} r = \frac{4}{3} \pi R^3$$
and since $m = \rho V$ the MMOI tensor is
$$ \mathrm{I} = \frac{m}{V} \int_0^R \int_{-\pi/2}^{\pi/2} 2 \pi \begin{bmatrix}
\frac{ r^4 \cos \psi (\sin^2 \psi+1)}{2} & & \\ & r^4 \cos^3 \psi & \\ & & \frac{ r^4 \cos \psi (\sin^2 \psi+1)}{2} \end{bmatrix} (2 \pi r^2 \cos \psi)\space {\rm d}\psi \space {\rm d} r $$
$$ \mathrm{I} = \frac{m}{V} \begin{bmatrix}
\frac{8}{15} \pi R^5 & & \\ & \frac{8}{15} \pi R^5 & \\ & & \frac{8}{15} \pi R^5 \end{bmatrix} = \begin{bmatrix}
\frac{2}{5} m R^2 & & \\ & \frac{2}{5} m R^2 & \\ & & \frac{2}{5} m R^2 \end{bmatrix}$$
The definition of the MMOI tensor from the volume integral is
$$ I = \frac{m}{V} \int \begin{bmatrix} y^2+z^2 & -x y & -x z \\ -x y & x^2+z^2 & -y z \\ -x z & -y z & x^2+y^2 \end{bmatrix} {\rm d}V $$
and the position in spherical coordinates is
$$ \pmatrix{x \\ y \\ z} = \pmatrix{ r \cos \psi \cos \varphi \\ r \sin \psi \\ r \cos \psi \sin \varphi} $$