This comes from the representation theory of the rotation group $\mathrm{SO}(3)$.
Quantum mechanics takes place in a vector space, and observables are operators on this space. The total amount of angular momentum is obtained from the angular momenta $L_x,L_y,L_z$ about the three axes of space as
$$ L = \sqrt{L_x^2+L_y^2+L_z^2} $$
since that is the length of the vector $\vec L = (L_x,L_y,L_z)^T$.
The representation theory of the rotation group now tells you that the only possible values for $L$ on so-called irreducible representations, to which the states with $I = \sqrt{l(l+1)}$ belong, are restricted to $L = \sqrt{l(l+1)}$ with $l$ an integer. You cannot make $L_x,L_y,L_z$ behave as angular momentum operators (i.e. as operators which generate the rotations in the sense that they form the Lie algebra belonging to the rotation group) without $L$ taking these integer values. The proof of this is technical and found e.g. on the Wikipedia page I linked in the beginning, or, in another approach, found in my answer here.
The prefactor $\hbar = \frac{h}{2\pi}$ for $\sqrt{l(l+1)}$ is found by dimensional analysis and comparing to experimental results.