Traces in 't Hooft-Veltman scheme I'm currently looking at the 't Hooft-Veltman regularization scheme and I'm a bit confused on how exactly one calculates traces in this scheme. As far as I understand one has to divide the $D$-dimensional subspace into a $4$-dim. one and a $(D-4)$-dim. one, i.e.
$$
\begin{align*}
  g_{\mu\nu} =: \bar g_{\mu\nu} + \hat g_{\mu\nu},\quad
  k^{\mu} =: \bar k^{\mu} + \hat k^{\mu},\quad\text{and}\quad
  \gamma^{\mu} =: \bar \gamma^{\mu} + \hat \gamma^{\mu},   
\end{align*}
$$
where the bar's indicate the $4$-dim. one and the hats the $(D-4)$-dim. one.
The HVBM scheme is now defined by
\begin{align}
  \{\bar \gamma^\mu, \gamma^5\} = 0,\\
  [\hat \gamma^\mu, \gamma^5] =0,\\
  \quad\text{and}\quad
  \{\bar\gamma^\mu,\hat\gamma^\nu \} = 0.
\end{align}
If one now considers for example a trace of this form
$$\mathrm{tr}[\gamma^a \gamma^5\gamma^b\gamma^c\gamma^d\gamma^5],$$
my approach would be to expand each $\gamma$-matrix into the bar and hat and then use the linearity of the trace to look at all the individual terms. Let's pick one of the terms that gives me trouble, e.g.
$$
\mathrm{tr}[\bar\gamma^a\gamma^5\hat\gamma^b\bar\gamma^c\hat\gamma^d\gamma^5].
$$
I can now with the above relations move one of the $\gamma^5$'s to the other and eliminate it with $(\gamma^5)^2=1$, so I will end up with something that is proportional to
$$\mathrm{tr}[\bar\gamma^a\hat\gamma^b\bar\gamma^c\hat\gamma^d]$$
but how exactly do I handle a trace that mixes hats with bars?
I used a computer algebra system (FeynCalc) to compute the result and I get
$$
\mathrm{tr}[\gamma^a \gamma^5\gamma^b\gamma^c\gamma^d\gamma^5]=
8 \hat{g}^{ad} g^{bc}-4 g^{ad} g^{bc}-8 \hat{g}^{ac} g^{bd}+4 g^{ac} g^{bd}+8 \hat{g}^{ab} g^{cd}-4 g^{ab} g^{cd}.
$$
How exactly do the $\hat g$ come into play here?
 A: From the HVBM prescription one can easily derive that
$$
\{ \gamma^\mu, \gamma^5 \} = \{ \bar{\gamma}^\mu, \gamma^5 \} + \{ \hat{\gamma}^\mu, \gamma^5 \} =  \{ \hat{\gamma}^\mu, \gamma^5 \} = 2 \hat{\gamma}^\mu \gamma^5 = 2 \gamma^5 \hat{\gamma}^\mu
$$
So whenever you want to anticommute $\gamma^5$ past a $\gamma^\mu$, you pick up a $(D-4)$-dimensional contribution, since
$$
\{ \gamma^\mu, \hat{\gamma}^\nu \} = \{ \bar{\gamma}^\mu, \hat{\gamma}^\nu \} + \{ \hat{\gamma}^\mu, \hat{\gamma}^\nu \} = \{ \hat{\gamma}^\mu, \hat{\gamma}^\nu \} = 2 \hat{g}^{\mu \nu}
$$
Computer algebra tools usually employ such relations to bring each trace containing $\gamma^5$ matrices to a form, where there is only a single $\gamma^5$ remaining in the very end of the chain.
In your example one can actually eliminate all $\gamma^5$ in the trace, cf. e.g. DiracSimplify[GAD[a, 5, b, c, d, 5]] in FeynCalc.
Regarding the question of calculating traces with $\gamma^5$ in this scheme,
the simplest, but also the most inefficient approach, would be to use the relation
$$
\gamma^5 = \frac{i}{4!} \bar{\varepsilon}^{\mu \nu \rho \sigma} \bar{\gamma}_\mu \bar{\gamma}_\nu \bar{\gamma}_\rho \bar{\gamma}_\sigma,
$$
which eliminates $\gamma^5$  for the price of adding 4 more matrices to the trace.
One can optimize it a bit by exploiting that
$$
[ \gamma^\alpha, \gamma^5 ] = \frac{i}{3} \bar{\varepsilon}^{\alpha \mu \nu \rho} \bar{\gamma}_\mu \bar{\gamma}_\nu \bar{\gamma}_\rho = \frac{i}{3} \bar{\varepsilon}^{\alpha \mu \nu \rho} {\gamma}_\mu {\gamma}_\nu {\gamma}_\rho,
$$
where $\bar{\varepsilon}$ is always understood to be 4-dimensional. Then one arrives at
$$
\textrm{Tr} (\gamma^{\nu_1} \ldots \gamma^{\nu_n} \gamma^5)=  \sum_{j=2}^n (-1)^j g^{\nu_1 \nu_j} \textrm{Tr}(\gamma^{\nu_2} \ldots \gamma^{\nu_{j-1}} \gamma^{\nu_{j+1}} \ldots \gamma^{\nu_n} \gamma^5) - \frac{i}{6} \bar{\varepsilon}^{\nu_1 \nu \rho \sigma } \textrm{Tr} (\gamma^{\nu_2} \ldots \gamma^{\nu_n} \gamma_\nu \gamma_\rho \gamma_\sigma)
$$
which is IMHO what is used in the Mathematica package TRACER.
To my knowledge, the most elegant and also the most efficient (for automatic calculations) approach is to use the so-called West's formula from https://doi.org/10.1016/0010-4655(93)90011-Z
$$
\textrm{Tr}  (\gamma^{\nu_1} \ldots \gamma^{\nu_n} \gamma^5) = \frac{2}{D-4} \sum_{i=2}^n \sum_{j=1}^{i-1} (-1)^{i+j+1} g^{\nu_{i} \nu_{j}} \,
\textrm{Tr}  \left ( \prod_{k=1,k \neq i,j}^n \gamma^{n_k} \gamma^5 \right ) \quad \textrm{for} \quad n \geq 6
$$
This is precisely what FeynCalc does when you ask it to calculate $\gamma^5$ traces in HVBM.
Notice that there is also a "simplified" version of HVBM for some particular types of QCD calculations, where you get the same results as with HVBM but the algebra is much simpler. In particular, there is no need to introduce Dirac matrices that live in 4 and $(D-4)$-dimensions. This prescription is known under the name of Larin's scheme, see e.g. 1506.04517
