Gauge theory - definition of the trace I am currently reading Nakahara's book and starting from chapter $10$, some sort of trace constantly shows up in the equations. For example,
\begin{equation}
S=-\frac{1}{2}\int\mathrm{tr}(F\wedge*F)
\end{equation}
(Yang-Mills action, section $10.5.5$) and
\begin{equation}
k=-\frac{1}{8\pi^2}\int_M\mathrm{tr}(F\wedge F),
\end{equation}
(instanton number, Example $12.6$ in section $12.6.2$), where $F\in \Omega^2(U)\otimes\mathfrak g$ is the (local) curvature $2$-form (in general, $U\subset M$, but we can assume $U=M$).
In section $1.8.2$, he mentions that "the trace is over the group matrix", but I'm not sure what he means (I have a guess though) and I haven't found another explanation in the rest of the book.
Here's my guess:
If $T_1,\ldots,T_r$ is a basis of $\mathfrak g$, $F=F^a\otimes T_a$, where $F^a\in\Omega^2(U)$. My guess would be
\begin{equation}
\mathrm{tr}(F\wedge*F):=F^a\wedge*F_a:=F^a\wedge*F^b\langle T_a,T_b\rangle=\langle F^a,F^b\rangle\langle T_a,T_b\rangle\mathrm{d}V\in\Omega^n(M)
\end{equation}
($\mathrm{d}V\in\Omega^n(M)$ is the volume form) and
\begin{equation}
\mathrm{tr}(F\wedge F):=F^a\wedge F_a:=F^a\wedge F^b\langle T_a,T_b\rangle\in\Omega^4(M)
\end{equation}
(we are obviously assuming that an inner product on $\mathfrak{g}$ is given).
 A: I am a little bit late to this question, but since I asked myself the same question some years ago, I would like to give some additonal mathematically insight. This is not an alternative to the answer by @ohneVal above, but should rather be a mathematical comment for all physicists interested in a more mathematical point of view. Feel free to comment :-)
In your question, you took fields defined on a subset $U$ of $\mathcal{M}$ and then said that you assume $U=\mathcal{M}$. This is of course fine, but you can do this much more precisely in the language of mathematical gauge theory, which is the geometry of principal fibre bundles and their associated vector bundles. In this context, I can also show you how to define the trace completely precise in the Yang-Mills action.
To start with, let me fix some notation:

*

*A compact Lie group $G$ with Lie algebra $\mathfrak{g}$ and an $\mathrm{Ad}$-invariant inner product $\langle\cdot,\cdot\rangle_{\mathfrak{g}}$ on $\mathfrak{g}$. Usually, $G$ is compact and simple in the physical context and $\langle\cdot,\cdot\rangle_{\mathfrak{g}}$ is then nothing else then (a multiple of) the Killing form, but in this general discussion it don't has to be.

*Let $(\mathcal{M},g)$ be a pseudo-Riemannian manifold and $\pi:P\to\mathcal{M}$ be a principal $G$-bundle. In physics, $(\mathcal{M},g)$ is of course just Minkowski space, but again we take some more general spacetime.

*A connection $1$-form $A\in\Omega^{1}(P,\mathfrak{g})$ with corresponding curvature $F\in\Omega^{2}(P,\mathfrak{g})$, i.e $F=\mathrm{d}A+\frac{1}{2}[A\wedge A]$. In physical terms, a connection $1$-form corresponds to a gauge field. To be precise, if you take a local section $s:U\to P$ of $P$, which is called a "local gauge", then $s^{\ast}A$ is a $\mathfrak{g}$-valued field defined on some subset $U$ of spacetime.

Now, if you want to define the Yang-Mills action, you have to work on $\mathcal{M}$ instead of $P$. This can be done in the following way: It is a general mathematical fact that there is the following isomorphism:
$$\Omega^{k}_{\mathrm{hor}}(P,\mathfrak{g})^{\mathrm{Ad}}\cong\Omega^{k}(\mathcal{M},\mathrm{Ad}(P)),$$
where $\Omega^{k}_{\mathrm{hor}}(P,\mathfrak{g})^{\mathrm{Ad}}$ is some subset of $\Omega^{k}(P,\mathfrak{g})$ of forms satisfying extra properties which are not so important right now and where $\mathrm{Ad}(P)=P\times_{\mathrm{Ad}}\mathfrak{g}$ denotes the "adjoint bundle", which is a vector bundle defined on some certain quotient of $P\times\mathfrak{g}$. (In mathematics, this is a particular case of so-called "associated vector bundles", which can be defined for every principal bundle and every representation $(\rho,V)$ on $G$). Now, the curvature $F$ has this additional properties and hence, using this isomorphism, you can view $F$ as an element of $\Omega^{2}(\mathcal{M},\mathrm{Ad}(P))$. To be precise, we will denote $F$ viewed as an element in this set by $F_{\mathcal{M}}$, but usually one just uses the same symbol.
Now in order to define the Yang-Mills action, we need two further definitions. First of all, we need to define the "trace-wedge product" of $\mathrm{Ad}(P)$-valued form, i.e. a map of the type
$$\mathrm{Tr}(\cdot\wedge\cdot):\Omega^{k}(\mathcal{M},\mathrm{Ad}(P))\times\Omega^{l}(\mathcal{M},\mathrm{Ad}(P))\to\Omega^{k+l}(\mathcal{M}).$$
This can be done in the following obvious way: Take $\alpha\in\Omega^{k}(\mathcal{M},\mathrm{Ad}(P))$ and $\beta\in\Omega^{l}(\mathcal{M},\mathrm{Ad}(P))$. If we take some local frame $\{e_{a}\}_{a=1}^{\mathrm{dim}(G)}\subset\Gamma(U,\mathrm{Ad}(P))$, which is a family of local sections of $\mathrm{Ad}(P)$, such that form at each point a basis, then we can write
$$\alpha\vert_{U}=\sum_{a=1}^{\mathrm{dim}(G)}\alpha^{a}\otimes e_{a}\hspace{1cm}\text{and}\hspace{1cm}\beta\vert_{U}=\sum_{a=1}^{\mathrm{dim}(G)}\beta^{a}\otimes e_{a}$$
where $\alpha^{a}\in\Omega^{k}(U)$ and $\beta^{a}\in\Omega^{l}(U)$ are real-valued forms. We define the trace then locally by
$$\mathrm{Tr}(\alpha\wedge\beta)\vert_{U}:=\sum_{a,b=1}^{\mathrm{dim}(G)}(\alpha^{a}\wedge\beta^{b})\langle e_{a},e_{b}\rangle_{\mathrm{Ad}(P)}$$
Here $\langle\cdot,\cdot\rangle_{\mathrm{Ad}(P)}\in\Gamma(\mathrm{Ad}(P)^{\ast}\otimes\mathrm{Ad}(P)^{\ast})$ is a bundle metric on $\mathrm{Ad}(P)$, i.e. a family of inner products on the fibres for each point, which is naturally induced by our given inner product $\langle\cdot,\cdot\rangle_{\mathfrak{g}}$ via
$$\langle [p,v],[p,w]\rangle_{\mathrm{Ad}(P)}:=\langle v,w\rangle_{\mathfrak{g}}$$
for all $[p,v],[p,w]\in\mathrm{Ad}(P)$ and for all $p\in P$. This also the reason why we use the symbol "Tr", because as mentioned above, if $G$ is compact and simple, then $\langle\cdot,\cdot\rangle_{\mathfrak{g}}$ is necessarily a negative multiple of the Killing form. One cane easily check that the above definition is independent of the chosen local frame and well-defined globally. As a second ingredient, we need the Hodge-star operator for bundle-valued forms. This can be defined using coordinates:
$$\ast\alpha\vert_{U}:=\sum_{a=1}^{\mathrm{dim}(G)}(\ast\alpha^{a})\otimes e_{a}$$
Again, this is well-defined and independent of choices.
After all this preliminaries, you see that the following action makes perfectly sense:
$$\mathcal{S}_{\mathrm{YM}}[A]:=-\frac{1}{2}\int_{\mathcal{M}}\mathrm{Tr}(F_{\mathcal{M}}\wedge\ast F_{\mathcal{M}})$$
Strictly speaking, $\mathcal{M}$ should be a compact manifold, since otherwise the integral might be not well-defined, but let us ignore this at this point.
This is the way as it is usually done in mathematics.

Let me also briefly discuss an alternative form of the Yang-Mills action which can often be found in the mathematical literature. For this, one introduces the L2-inner product of bundle-valued forms: First of all, we extend the bundle metric $\langle\cdot,\cdot\rangle_{\mathrm{Ad}(P)}$ to $\mathrm{Ad}(P)$-valued forms, i.e.
$$\langle\cdot,\cdot\rangle_{\mathrm{Ad}(P)}:\Omega^{k}(\mathcal{M},\mathrm{Ad}(P))\times\Omega^{k}(\mathcal{M},\mathrm{Ad}(P))\to C^{\infty}(\mathcal{M})$$
via
$$\langle\alpha,\beta\rangle_{\mathrm{Ad}(P)}\vert_{U}:=\sum_{a,b=1}^{\mathrm{dim}(G)}\langle\alpha^{a},\beta^{b}\rangle\langle e_{a},e_{b}\rangle_{\mathrm{Ad}(P)}$$
Here $\langle\alpha^{a},\beta^{b}\rangle$ denotes just the usual inner product of real-valued forms, which definition can be found in probably all books on differential geometry. Using this, we define
$$\langle\cdot,\cdot\rangle_{\mathrm{Ad}(P),L^{2}}:\Omega^{k}(\mathcal{M},\mathrm{Ad}(P))\times\Omega^{k}(\mathcal{M},\mathrm{Ad}(P))\to \mathbb{R}$$
via
$$\langle\alpha,\beta\rangle_{\mathrm{Ad}(P),L^{2}}:=\int_{\mathcal{M}}\langle\alpha,\beta\rangle_{\mathrm{Ad}(P)}\mathrm{vol}_{g}$$
where $\mathrm{vol}_{g}$ denotes the volume form on $(\mathcal{M},g)$. You can easily check that we then have that
$$\mathcal{S}_{\mathrm{YM}}[A]=-\frac{1}{2}\langle F_{\mathcal{M}},F_{\mathcal{M}}\rangle_{\mathrm{Ad}(P),L^{2}}=-\frac{1}{2}\Vert F_{\mathcal{M}}\Vert^{2}_{\mathrm{Ad}(P),L^{2}}$$
This is basically a consequence of the well-known formula $\alpha\wedge\ast\beta=\langle\alpha,\beta\rangle\mathrm{vol}_{g}$ for real-valued forms applied to the local expressions above.
Btw, this expression of the Yang-Mills action also manifestly shows gauge-invariance of the action. In mathematics, a gauge-transformation is a principal bundle automorphism $f:P\to P$, i.e. a diffeomorphism, which is $G$-equivariant and which maps fibres to fibres. Now one can check that
$$F[f^{\ast}A]=\mathrm{Ad}_{\sigma_{f}^{-1}}\circ F[A]$$
where $\sigma_{f}\in C^{\infty}(P,G)$ is the map defined by $f(p)=p\cdot\sigma_{f}(p)$ for all $p\in P$. Then it is clear that we have that
$$\mathcal{S}_{\mathrm{YM}}[A]=\mathcal{S}_{\mathrm{YM}}[f^{\ast}A]$$
precisely because of $\mathrm{Ad}$-invariance of $\langle\cdot,\cdot\rangle_{\mathfrak{g}}$.

General references for all the things I mentioned above are the excellent texbook "Mathematical Gauge Theory" by M. Hamilton from Springer as well as "Differential Geometry and Mathematical Physics" from Springer, especially volume 2, by G. Rudolph and M. Schmidt. If you speak german, then there is also a very good book by H. Baum with the title "Eichfeldtheorie", also from Springer.
A: The trace in those formulae is simply the trace over the group indices of the field strength tensor $F$, you are free to compute it in whichever basis you want, since the trace is an invariant, however the extra structure of an inner-product is not needed only matricial algebra. So once you have a representation $\rho$ and a basis, $\{T_a\}$ for the vector space in which $\rho$ acts, in this case $\mathfrak{g}$ itself, you can write your field strength tensor as
$$F = \sum_a F^a T_a $$
which you can think as a matrix of forms (since it belongs to $\Omega^2(U)\otimes \mathfrak{g}$). When the base is taken such that the structure constants of the algebra are fully antisymmetric
$${\rm tr}( T_a T_b ) = \frac{1}{2}N\delta_{ab}$$
and then you have the formula
$${\rm tr}(F\wedge F) = \sum_{a,b} F^a \wedge F^b \; {\rm tr}(\rho(T_a)\rho(T_b)) = \sum_{a} \frac{N}{2} F^a \wedge F^a$$
where $\rho(T_a)\rho(T_b)$ is matrix multiplication, no inner product. $\rho(\mathfrak{g})$ has in principle only the structure of a matrix group under multiplication thus, no inner product. You can think of $\rho(T_a)$'s as being a basis for $\mathfrak{su}(2)$ or $\mathfrak{su}(3)$, which are instrumental in physics. Other than that your statements are ok.
I believe Nakahara does not care about different representations too much, but that is how it is formally done. The gauge field $A$ we know transforms always in the adjoint and is therefore itself always written in terms of some "traditional" matrices for the fundamental representation (since that is where the adjoint representation acts on). Perhaps reading in this direction clarifies the operation above.
