The smooth surface is air-liquid interface and the rough surface is let's say a t-shirt.
The image above shows the probability of each reflection, and the $p$ value is the probability of the light rays totally reflected by the air-liquid interface. So this means $p$ represents the fraction of light rays that goes in an angle greater than the critical angle and we are after the intensity (or amount I don't really know how I should call it) of light in the yellow shaded area:
The article I am reading estimates $p$ in the following way.
However, after this estimation it makes more calculations to find the exact value of $p$. It calculates the reflected light incidents which enter with an angle smaller than the critical angle but I do not understand how it does this.
This is the article and it would be very helpful if someone explained all the steps clearly, because I am a high school student and literally have no idea.
Note: I am sorry if the post looks very ugly, I am not sure if I should remove the images. Sorry about that, I will edit according to your warnings.