Part of your confusion here probably comes from your notation. Usually we reserve the index $\mu$ for spacetime. The generators of $SU(N)$ are more commonly labelled with Latin indices $t^a$. See for example here.
We can split the amplitude into two parts, according to whether they concern color or kinematics. You are just interested in the color part. Each quark comes with a color "polarization", i.e. a normalized basis vector $c$ in the fundamental representation of $SU(N)$. The quark-gluon vertices just give color factors of $t^a$.
Now the cross section is given by the modulus squared of the amplitude so we have
$$ \sigma \propto {1\over N^2} \sum_{initial} \sum_{final} [c^\dagger _{\overline d} ~t^a ~c_d][c^\dagger _{t} ~t^a ~c_{\overline t}][c^\dagger _{\overline d} ~t^b ~c_d]^*[c^\dagger _{t} ~t^b ~c_{\overline t}]^* $$
Now using that the color polarizations are normalized we can reinterpret the sum as a double trace term
$$ \sigma \propto {1\over N^2}Tr(t^a t^b)Tr(t^a t^b) $$
Now we can use the color algebra relation
$$Tr(t^a t^b) = \frac{1}{2}\delta^{ab}$$
to conclude that
$$\sigma \propto \frac{(N^2 - 1)}{N^2}$$
As for a reference, you could do worse than read Part III of Peskin and Schroeder's book. It's quite a big step up from Griffiths though. But if you want to understand what's really going on then you'll need to do QFT at some stage!
Where Does the Trace Come From?
Observe that all the square bracket factors in my first expression for $\sigma$ are scalars, so we may commute them. We can thus write
$$ \sigma \propto {1\over N^2} \sum_{initial} \sum_{final} [c^\dagger _{\overline d} ~t^a ~c_d][c^\dagger _{\overline d} ~t^b ~c_d]^*[c^\dagger _{t} ~t^b ~c_{\overline t}]^*[c^\dagger _{t} ~t^a ~c_{\overline t}] $$
Now expanding out the complex conjugation and using that the $t^a$ are Hermitian we find
$$ \sigma \propto {1\over N^2} \sum_{initial} \sum_{final} [c^\dagger _{\overline d} ~t^a ~c_d~c_d^\dagger ~t^b c_{\overline d} ][~c^\dagger_{\overline t}~t^b c _{t} c^\dagger _{t} ~t^a ~c_{\overline t}] $$
Now the color polarizations are normalized, so the central terms in each factor just give $1$. This leaves
$$ \sigma \propto {1\over N^2} \sum_{initial} \sum_{final} [c^\dagger _{\overline d} ~t^a ~t^b c_{\overline d} ][~c^\dagger_{\overline t}~t^b ~t^a ~c_{\overline t}] $$
But now the sum over initial colors gives exactly the definition of $Tr(t^at^b)$. Similarly for the final colors.