Traces in different representation I am actually working with Green-Schwarz anomaly cancellation mechanism in which I have came across a strange formula which relates trace in the adjoint representation (Tr) to trace in fundamental representation (tr).For the special case of $SO(n)$, the relation is 
$$Tr(e^{iF})= \frac{1}{2}(tre^{iF})^2-\frac{1}{2}(tre^{2iF}).$$
This relation can be found in 'String theory and M theory' of Becker, Becker and Schwarz Chapter 5. They say that this is a result which follows from Chern character factorization property (I guess something similar to how Chern character can be represented by a product of Chern characters defined on two vector bundles, when the whole character is evaluated on the product of vector bundles but I am not sure because I have never came across such term).
Can any one tell me how to relate the traces in one representation to the other because I have never seen it before in any Group theoretic context.
Chapter 13 of GSW also contains some information on it but that is not much useful.
 A: Good luck. To check the cancellation for particular groups like $E_8\times E_8$ and $SO(32)$, you will indeed have to get through similar group-theoretical tasks. Similar trace formula for the traces of $E_8$ transformations are especially yummy, including the factor of $1/30$.
The orthogonal case is easier even if one is not an intimate friend of all "Chern things", and I am not.
In your formula, $\exp(iF)$ may be understood to be a general element of the $SO(n)$ group while $F$ is an element of the Lie algebra. The latter is antisymmetric in the fundamental representation, OK? So it may be "diagonalized", and the eigenvalues are paired into pairs $\pm\lambda_i$. 
Similarly, $\exp(iF)$ may be diagonalized and the eigenvalues are coming in pairs $\exp(\pm i\lambda_j)$, OK? Alternatively, the $2\times 2$ block with these eigenvalues may be written as the matrix $((\cos\lambda_j,\sin\lambda_j),(-\sin\lambda_j,\cos\lambda_j))$.
Now, it's important to realize how the matrix elements look in the adjoint representation. The adjoint representation of $SO(n)$ is $n(n-1)/2$ dimensional, OK? Let me assume that $n$ is even - embed $SO(2k+1)$ to $SO(2k+2)$ if necessary. It is the antisymmetric part of the tensor product $n\times n$. So if the matrix elements of a transformation $\exp(iF)$ in the fundamental representation are $\exp(i\lambda_j)$, the matrix elements in the adjoint (antisymmetric tensor) representation are combinations of products of two such things, i.e. combinations of $\exp(i\lambda_j+i\lambda_k)$, OK?
It's easy to see that if we diagonalize $\exp(iF)$ in the fundamental representation, with eigenvalues $\exp(\pm \lambda_j)$, the trace over the fundamental representation is just the sum of these phases which is
$$ {\rm tr} (\exp(iF)) = \sum_{\pm} \sum_j \exp(\pm i\lambda_j) = 2\sum_j \cos\lambda_j $$
In the naturally associated basis of the adjoint representation, the transformation $\exp(iF)$ is also diagonalized. Because the $n(n-1)/2$ basis vectors are just pairs of basis vectors of the fundamental representation, the diagonal entries of $\exp(iF)$ are just products of two diagonal entries in the fundamental representation, so they are
$$\exp(\pm i\lambda_j\pm i\lambda_k), \quad j \lt k$$
where the two $\pm$ signs are independent. The trace is the sum of all these numbers which is 
$${\rm Tr}(\exp(iF)) = \sum_{j\lt k} \left[ 2\cos(\lambda_j+\lambda_k)+2\cos(\lambda_j-\lambda_k) \right] $$
This is the left hand side of your identity. The first term on the RHS is
$$ \frac 12 2^2\left(\sum_j \cos\lambda_j \right)^2$$
where I just squared a previous result while the second term is
$$ - \frac 12 \sum_j 2\cos 2\lambda_j $$
where I just doubled the argument of the cosine. Now, both sides may be simplified to
$$ 4\sum_{j\lt k} \cos \lambda_j \cos \lambda_k $$
In the case of the left hand side, it's because the $\sin\cdot\sin$ terms cancel when one sums the two terms with equal or opposing signs. You use $\cos(a+b)=\cos a\cos b - \sin a \sin b$ – perhaps all things would be simpler using the complex exponential notation, anyway. In the case of the right hand side, the $j\neq k$ terms from the first term are right, including the right factor of four, while the $j=k$ terms should be subtracted from the first term by the second term. Well, there is a constant term left:
$$2\cos^2 \lambda_j - \cos 2\lambda_j = 1$$
But the constant terms match because they're verified when $\lambda_j=0$ for all $j$. The trace of the identity operator ($F=0$) on the left hand side is $n(n-1)/2$, simply the dimension of the rep, while the first term on the RHS produces $n^2/2$ and the second one gives you $-n/2$ so things are OK.
I am sure you fill in the details and ask if something really needs some extra assistance.
Of course that there exist proofs that avoid the diagonalization and proofs that are conceptually linked to more esoteric parts of maths. But an explicit calculation using the explicit matrix elements of the transformations relatively to both representations might be helpful to go through at least once in a lifetime.
Incidentally, you may have noticed that the basis in which the transformations were diagonalized were "complex" – the coordinates of the eigenvectors relatively to the "usual real basis" of the $n$-dimensional space were complex. But that's not a problem. I just solved a more general problem for the traces in the whole $SO(n,C)$ group, and the result for $SO(n,R)$ may be considered a special case of it. Quite generally, in physics and "deep enough" maths, one should never protest against complex numbers (complex coordinates of eigenvectors and perhaps complex eigenvalues of unitary matrices etc.) while diagonalizing things.
A: $\newcommand{\tr}{\mathrm{tr}}$For $\mathrm{SO}(N)$, the adjoint representation can be seen as the antisymmetric part of the tensor product of the fundamental representation with itself.
In general, we have the following property for such an antisymmetric representation:

Given a representation $(V,\rho)$, the representation $V\otimes V$ decomposes as $S^2 V \oplus \Lambda^2 V$ where $S^2 V$ are the symmetric tensors and $\Lambda^2 V$ are the antisymmetric tensors, and we have
  $$ \chi_{\Lambda^2 V}(g) = \frac{1}{2}(\chi_V(g)^2 - \chi_V(g^2))$$
  where $\chi(g)$ is the trace of $\rho(g)$ in the corresponding representation.

From this, it follows directly that
$$\chi_{\mathfrak{so}(N)}(g) = \frac{1}{2}(\chi_\text{fund}(g)^2 - \chi_\text{fund}(g^2))$$
which yields the desired relation for $g = \mathrm{e}^{\mathrm{i}F}$. So, let us prove the general claim about the traces:
Let $v_1,\dots,v_n$ be a basis of $V$. Then, $a_{ij} := v_i \otimes v_j - v_j \otimes v_i$,  $j > i$ are a basis of the antisymmetric tensors $\Lambda^2 V$. The matrix element of the representation $\sigma$ of $g$ on $\Lambda^2 V$ in this basis are given by:
$$ \sigma(g)_{kl,ij} = \rho(g)_{ki}\rho(g)_{lj} - \rho(g)_{kj}\rho(g)_{li}$$
since $\sigma(g)a_{ij} = (\rho(g)v_i) \otimes (\rho(g)v_j) - (\rho(g)v_j) \otimes (\rho(g)v_i)$ by definition.
Thus, 
$$ \chi_{\Lambda^2 V}(g) = \tr(\sigma(g)) = \sum_{i < j} \sigma(g)_{ij,ij} = \frac{1}{2} \sum_i \sum_j \sigma(g)_{ij,ij}$$
and inserting the matrix elements gives
$$ \chi_{\Lambda^2 V}(g) = \frac{1}{2}\sum_i\sum_j ( \rho(g)_{ii}\rho(g)_{jj} - \rho(g)_{ij}\rho(g)_{ji}) = \frac{1}{2}(\chi_V(g)^2 - \chi_V(g^2))$$
which is what we wanted to show.
