The smooth surface is air-liquid interface and the rough surface is let's say a t-shirt.
The image above shows the probability of each reflection, and the $p$ value is the probability of the light rays totally reflected by the air-liquid interface. So this means $p$ represents the fraction of light rays that goes in an angle greater than the critical angle and we are after the intensity (or amount I don't really know how I should call it) of light in the yellow shaded area:
The article I am reading finds $p$ in the following way but I do not understand it. Can anyone please explain it in a way that can be clearly understood?