Short answers
- Apply the Young calculus (per ACuriousMind's suggestion in the comments). For finding the multiplicity of the trivial representation in a tensor product of representations of $SU(n)$, note that each irreducible representation $D$ of $SU(n)$ has a unique conjugate irreducible representation $\bar D$ such that the Young calculus allows $D\otimes \bar D$ to include a rectangular Young diagram of full height $n$ (which is invariant under $SU(n)$). As suggested by Wooster in the comments, in order for $D\otimes\bar D$ to accommodate such a Young diagram, the joint symmetry types of $D$ and $\bar D$ must be compatible with (i.e. have nonzero overlap with) some tensor power/outer product of the fully antisymmetric tensor $\epsilon_{i_1,\dots,i_n}$. In $SU(n)$, Young diagrams of this type correspond to the invariant or trivial representation.
- If you view the $n$ dimensional representation of $SU(n)$ as a sort of single-particle Hilbert space, then the invariants formed from tensor products of this representation can be thought of as '$SU(n)$-neutral' many-particle states. More abstractly you could interpret representations $SU(n)$ very differently as a sort of gauge theory where 'particle number' is gauged.
- The branching problem has been solved in a number of special cases. For example, there is an explicit formula for the branching of representations for $SU(n)\rightarrow SU(n-1)$. For low-rank representations, the Young calculus is a powerful general-purpose tool for determining branching. One strategy is to decompose the fundamental representation of $SU(5)$ into representations of $H\subset SU(5)$, and then iteratively compare how tensor products decompose. As an example, consider the problem of decomposing rank 2 representations of $SU(5)$ into representations of $SU(2)\subset SU(5)$. The fundamental (vector) representation of $SU(5)$ breaks up as $5_5 \rightarrow 2_2\oplus (1_2 \oplus 1_2 \oplus 1_2)=2_2\oplus 3\times 1_2$. Next, we have $5_5\otimes 5_5=10_5^A\oplus 15_5^S \rightarrow (2_2\oplus 3\times 1_2)\otimes (2_2\oplus 3\times 1_2)=(2_2\otimes 2_2)\oplus 3 \times (2_2\otimes 1_2) \oplus 3\times (1_2\otimes 2_2)\oplus 9\times 1_2=3_2^S\oplus 1_2^A\oplus 3\times 2_2^S\oplus 3\times 2_2^A\oplus 3\times 1_2^A\oplus 6\times 1_2^S$. Grouping terms according to symmetry, we see that $10_5^A\rightarrow 4\times 1_2^A\oplus 3\times 2_2^A$, and $15_5^S\rightarrow 3_2^S\oplus 3\times 2_2^S\oplus 6\times 1_2^S$.
Background on the Young calculus
In physics, irreducible representations are often labeled by their dimension. This notation is compact, but obscures the underlying algebraic structure. Young diagrams provide a more transparent notation based on a deep result, Schur-Weyl duality, that relates irreducible representations of $GL(n)$ to those of the permutation group $S_r$ on $r$ symbols (here $r$ is the rank of a tensor representation). Ultimately, Schur-Weyl duality comes from the fact that finite dimensional representations of $GL(n)$ can all be constructed out of tensor products of a single fundamental representation (this is the analogue of the $\frac{1}{2}$ representation of $SU(2)$ from elementary quantum mechanics). For now, all you need to know is that there is a 1-1 correspondence between representations of $GL(n)$ and the set of all Young diagrams with maximum height $n$. Young diagrams greatly simplify the task of decomposing tensor products of representations of $GL(n)$, as well as many subgroups of $GL(n)$ with 'similar' structure (e.g. $U(n)$, $SL(n)$, $SU(n)$, etc.). They also make it easier to notice certain partial solutions to the branching problem, such as determining how representations of $GL(n)$ decompose into representations of $GL(n-1)$.
Let $r$ be a positive integer. Young diagrams are associated with partitions of $r$: sequences of integers $\lambda_1\geq\lambda_2\geq\cdots\geq\lambda_k\geq 0$ such that $\sum_j \lambda_j = r$. Given a partition $(\lambda_1,\dots,\lambda_k)$, draw a Young diagram as follows: (i) draw a horizontal row of $\lambda_1$ boxes, (ii) draw a horizontal row of $\lambda_{j+1}$ boxes starting from the left below the $j$th row, $1\leq j<k$. For example, the partition $(2,1,1)$ of $r=4$ would correspond to the diagram
As mentioned above, each diagram with at most $n$ rows corresponds to an irreducible representation of $GL(n)$. Again, this fact is useful because $GL(n)$ is closely related to many other groups of interest in physics. A Young diagram can be thought of as an efficient way to keep track of the symmetrization of tensor indices: after placing tensor indices $i_1$ through $i_r$ in the squares of a Young diagram, the corresponding irreducible tensors are symmetric (even) under permutations that preserve rows, and antisymmetric (odd) under permutations that preserve columns.
There is a general formula for the dimension of a $GL(n)$ representation labelled by a Young diagram, but in practice the dimension can be computed more efficiently for low rank using the decomposition rules for tensor products, to be explained now.
The tensor-product decomposition rules for $GL(n)$ follow from a special sort of 'inverse branching' problem for the permutation group $S_r$. In the end, one obtains the following rules:
Let $L=(\lambda_1,\dots,\lambda_k)$ and $M=(\mu_1,\dots,\mu_\ell)$ be two irreducible representations of $GL(n)$, given by their Young diagrams.
- Draw the diagrams corresponding to $L$ and $M$. In the diagram for $M$, choose a distinct symbol for each row (e.g. $a$ for the first row, $b$ for the second, $c$ for the third, etc.), and write the symbol in each box of that symbol's row.
- Find all ways in which $\mu_1$ $a$'s can be added to the Young diagram of $L$ so that no two $a$'s appear in the same column, and the resulting graph is another Young diagram (i.e. the length of rows is nonincreasing).
- For each larger Young diagram obtained above, find all the ways in which $\mu_2$ $b$'s can be placed with no two in the same column, together with an additional constraint: when reading the added symbols from right to left, top to bottom, the number of $a$'s that have been read must match or exceed the number of $b$'s that appear at any step.
- Repeat for the $\mu_3$ $c$'s, then $\mu_4$ $d$'s, etc., except that now when imposing the last constraint mentioned in step 3, the recorded number of $c$'s cannot exceed the number of $b$'s (etc.).
- The tensor product $L\otimes M$ decomposes into a direct sum of all Young diagrams obtained this way.
As an example, consider the following tensor product:
To decompose this, first we label the second diagram with $A$'s and $B$'s:
Next we find all the ways of adding $A$ blocks, and then $B$ blocks, to the Young diagram of $L$ according to the above rules:
$\rightarrow$ 
Note that diagrams like the ones below are not permitted:
The first two diagrams contain two $A$'s in the same column, while the last one isn't allowed because when reading the added symbols right-left top-bottom, we obtain $ABBA$, which has more $B$'s than $A$'s after the third letter (this is from the rule stated in step 3).
Now it turns out that all irreducible representations of $GL(n)$ remain irreducible when restricted to $SU(n)$. However, some representations of $GL(n)$ that were previously distinct become isomorphic. This comes from the fact that it is possible for two irreducible representations of $GL(n)$ to differ from each other only by powers of the determinant homomorphism: $\det(gh)=\det(g)\det(h)$. Once the determinant is set to unity in $SU(n)$ (or $SL(n)$ for that matter) this distinction vanishes, and representations that only differed in their power of $\det(g)$ are isomorphic. Fortunately there is a simple way to account for this redundancy: under $SU(n)$, the representations $(\lambda_1,\dots,\lambda_n)$ and $(\lambda_1+s,\dots,\lambda_n+s)$ are equivalent. To account for the redundancy, we simply choose $s=-\lambda_n$ and label representations of $SU(n)$ with only $n-1$ non-increasing integers instead of $n$. A consequence of this is that if $\lambda_1=\lambda_2=\dots=\lambda_n$, then $[\lambda_1,\dots,\lambda_n]\cong[0,0,\dots,0]$: tensors corresponding to rectangular Young diagrams are invariant under $SU(n)$.
For finding the multiplicity of the trivial representation in tensor products, you can check from the decomposition rules that each irreducible representation $V$ of $SU(n)$ has a unique conjugate $\bar V$ such that $V\otimes\bar V$ includes the trivial representation.
References for further reading:
Group Theory and its Application to Physical Problems (by Morton Hamermesh): chapters 7 and 10.
Theory of Group Representations and Applications (A. Barut & R. Raczka): chapters 7 and 8.
