The subscript is simply the charge of the given representation under the $U(1)$, the hypercharge in this case. All irreducible unitary representations of $U(1)$ are one-dimensional and they just map the $e^{i\alpha}$ element of $U(1)$ (a $1\times 1$ matrix) to $e^{iQ\alpha}$, its power, where the exponent contains the factor of the charge $Q$.
One needs to study the representations of Lie groups to be able to produce decompositions such as yours. Take e.g. Lie algebra in particle physics by Howard Georgi, a book by a GUT pioneer and my ex-colleague at Harvard. There are also much more formally, mathematically oriented books, of course. In many of them, getting to the decompositions needed in GUT model building could be hard. But if you actually understand how the representations, tensor products, charges, generators, irreducible representations etc. are defined, it's a matter of mathematical reasoning you may do yourself to derive similar decompositions.
Take this particular one. The $SU(5)$ adjoint representation may be thought of as a Hermitian matrix $M$ and the action of the group element $G\in SU(5)$ is simply
$$ M \mapsto G M G^{-1} $$
Note that there are two copies of $G$ and $G^{-1}=G^\dagger$ because it's unitary.
The Hermitian $5\times 5$ matrix has 25 independent real entries. A general matrix would have 25 complex entries but those below the main diagonal are given by those above the main diagonal. And the diagonal entries are real so they contain "one-half of the real parameters". To summarize, exactly one-half of the parameters survives the Hermiticity condition.
The actual adjoint representation of $SU(5)$ as opposed to $U(5)$ is just 24-dimensional. The Lie group element would have a unit determinant (a priori a complex number with absolute value equal to one). However, $M$ is from the Lie algebra so the determinant condition gets translated to ${\rm Tr}(M)=0$. The matrix is traceless.
Now, embed the Standard Model group to this $SU(5)$. The $SU(3)$ and $SU(2)$ elements are embedded simply by a block-diagonal matrix with a $3\times 3$ block in the left upper corner representing the $SU(3)$ information and another $2\times 2$ block in the right lower corner representing the $SU(2)$ element. This embedding makes it clear that we're splitting the vectors with 5 components to 3+2 components.
Similarly, the 24 or 25 entries of the square matrix is split into the squares and rectangles, $3\times 3$, $2\times 3$ and in the next thick row $3\times 2$ and $2\times 2$. You are supposed to draw a $5\times 5$ square and divide it into these four squares or rectangles.
The 25 of $U(5)$ would simply decompose to these four representations but they wouldn't quite be irreducible. The trace of the $3\times 3$ square block in the left upper corner and the trace of the $2\times 2$ block in the opposite corner may be separated. In the 24-dimensional representation of $SU(5)$, one of these two traces is eliminated so only one is left. It's the $(1,1)$ representation in your list.
Otherwise the remaining four representations exactly correspond to the squares and rectangles. The $3\times 3$ square gives you the $(8,1)$: it's the same square as the original for $SU(5)$ but smaller. It obviously only transforms under the $SU(3)$ transformations that mix the first three rows and first three columns. It's the adjoint of the $SU(3)$ and $8=3^2-1$ much like $24=5^2-1$. Similarly $(1,3)$ comes from the adjoint of $SU(2)$ only affected by the $SU(2)$ transformations and $3=2^2-1$. That's the right lower square.
Then you have the off-block-diagonal elements which have size $3\times 2$ so they obviously have to transform as $(3,2)$, the tensor product of the fundamental representations of the $SU(3)$ and $SU(2)$ groups. There are two such representations – above the diagonal and below the diagonal. They're complex conjugate to each other. Because the representations $3$ and $2$ of the smaller groups are real, the only influence of the complex conjugation is the opposite sign of the $U(1)$ charge.
Finally, I must discuss the subscripts. The rectangle off-diagonal representation I just mentioned has a nonzero charge $Y$, the other one has the same value with the opposite sign. All other representations obviously have to have a vanishing charge under $U(1)$ because those three terms, $(8,1)$, $(1,3)$, and $(1,1)$ are nothing else than the adjoint representation of the Standard Model group and all of this group's generators commute with the $U(1)$, the hypercharge.
So the only number left to explain is the $5/6$ charge for the rectangular representation, and $-5/6$ for the complex conjugate one on the opposite side of the square. The normalization of this charge is a convention. You may rescale the hypercharge to suit your conventions. However, the ratios of the hypercharges are physical. And the hypercharge has to be a multiple of ${\rm diag}(+2,+2,+2,-3,-3)$ because it has to commute with all the matrices of $SU(3)$ and $SU(2)$ in the blocks, so in the blocks, it must be a multiple of the unit matrix, and the hypercharge must be traceless to be a generator of $SU(5)$, as I mentioned, and $2\times 3 - 3\times 2$ really cancels.
The normalization of this $U(1)$ hypercharge generator in physics is chosen in such a way that it agrees with the conventions adopted in electroweak physics. The hypercharge $Y$ is defined as the average electric charge in an $SU(2)$ electroweak multiplet so that $Q=Y+T_3$. Sometimes, the definition $Q=Y/2+T_3$ is used.
In GUT theories, the off-diagonal blocks of the adjoint become hugely massive particles because of some GUT symmetry breaking. These $(3,2)$ states cause proton decay so they better be very heavy. We can't compare them with any known particles.
However, you may take a $5$ of $SU(5)$, the fundamental representation, which should produce the quark fields which are electroweak singlet: note that this $5$ only has components that transform either under $SU(3)$ or $SU(2)$. So it's a right-handed up quark, for example. Its hypercharge is the average of the mutliplet but because it's the anti-down-quark only, actually, you get $Y=1/3$. Similarly, the remaining two components are an electroweak doublet, like the electron and neutrino, whose $Y=((-1)+0)/2=-1/2$.
The states $(3,2)$ in the decomposed $SU(5)$ adjoint come from $5\otimes \bar 5$ and the off-diagonal pieces arise from $3\otimes \bar 2$ so the $Y$ of the first should be added to $-Y$ of the second factor and you get $Y=-1/3-1/2=-5/6$. Note that the convention $Q=Y+T_3$ without the factor of $1/2$ was used. Getting the signs and factors of $2$ right may be messy but I hope it's essentially right.
