The elements of the gauge transformations belong to a gauge group. In physics, it's most typically $SU(N)$ (both the electroweak theory, with its $SU(2)$, and the QCD for quarks, $SU(3)$, use these $SU(N)$ groups; $U(1)$ we first learn in electromagnetism – but we must reinterpret the charge as the "hypercharge" when we study the electroweak theory – is the only extra addition we need for the Standard Model). It's a group of all complex $N\times N$ matrices $M$ that obey
$$MM^\dagger=1, \quad \det M = 1$$
Note that $M^\dagger=(M^*)^T$ is the Hermitian conjugate; the first condition makes the matrix "unitary", therefore $U$. The determinant of a unitary matrix could be any complex number whose absolute value equals one. The second condition says that the determinant must be one and nothing else, that's the "special" or $S$ condition in $SU(N)$.
The gauge field transforms as
$$ A_\mu \to M(A_\mu+ie\partial_\mu) M^\dagger$$
up to different conventions. That's needed for the covariant derivative $D_\mu$ to transform nicely. Forget about the complicated formula above. The point is that $A_\mu$ takes values in the Lie algebra of the Lie group.
In other words, you may imagine an infinitesimal transformation – infinitely close to the identity – in the gauge group, e.g. $SU(N)$. Assume
$$ M = 1+i\epsilon G $$
The factor $\epsilon$ makes it infinitesimal, the factor of $i$ is a convention popular among physicists but omitted by mathematicians (physicists like things to be Hermitian, without $i$, they would have to be anti-Hermitian).
Here, $G$ is the kind of $N\times N$ matrix that the gauge field can have as a value.
Now, substitute this Ansatz for $M$ into the conditions $MM^\dagger=1,\det M=1$. You may neglect $\epsilon^2$ "very small" terms and the conditions become
$$1+i\epsilon G - i\epsilon G^\dagger = 1, \quad \det(1+i\epsilon G) = 1$$
Mathematics implies that these conditions are equivalent to
$$ G = G^\dagger, \quad {\rm Tr}(G) =0 .$$
To get the first one, I just subtracted $1$ from both sides and cancelled $i\epsilon$. To get the latter, I used the "sum of products over permutations" formula for the determinant and noticed that only the product of the diagonal entries contributes $O(\epsilon)$ terms and they're proportional to the sum of the diagonal entries, the trace.
At any rate, you should try to understand this maths and its conclusion is that the Hermiticity of the generator $G$ – matrices that are combined with various real coeffcients to get $A_\mu$ – is equivalent to the gauge group's being unitary; and the tracelessness is equivalent to the group's being "special" i.e. requiring the unit determinant.
It's perhaps useful to mention why $SU(N)$ is considered the "simplest" class of gauge groups. The $S$ has to be there because $U(N)$ isn't simple – it's pretty much isomorphic to $SU(N)\times U(1)$ where the two factors could be treated separately and we want to work with the smallest allowed pieces of gauge groups which are $SU(N)$ and $U(1)$. And $SU(N)$ is more "elementary" than $SO(N)$ or $USp(2N)$ because complex numbers are more fundamental in group theory (and physics) than real numbers or quaternions. In fact, the groups $SO(N)$ and $USp(2N)$ may be defined as $SU(N)$ with some "extra structure" (orientifolds) added which makes some natural group-theoretical analyses somewhat more convoluted than those for $SU(N)$. But one may still say that the Lie algebra for $SO(N)$ would be composed of antisymmetric real (or antisymmetric pure imaginary, depending on the conventions concerning factors of $i$) matrices, in analogy with the Hermitian matrices above; they're automatically traceless.