Group theory: The energy splitting of the five-fold $d$ orbitals in a $D_{3h}$ crystal field

Question

I am trying to figure out how the degerenate 5-fold $d$ orbitals is split in a $D_{3h}$ crystal field. A practical case would be a subtitutional $\rm Fe$ impurity in the hexagonal graphene lattice. What I am supposed to do is:

1. find the character table of $D_{3h}$ group and the symmetry operations;

2. calculate the character of these symmetry operations for the full rotational group; and

3. use the decomposition formula to work out how many times each irreducible representation of group $D_{3h}$ is contained in the reducible representation of the full rotational group.

step (1): The first step is pretty easy to do

$D_{3h}$" />

step (2): The character expressions of these symmetry operations of the full rotational group are as follows: $\chi^{(l)}(C_n)=sin[(l+1/2)\alpha]/sin[\alpha/2]$ for rotation $C_n$ with $\alpha=2\pi/n$
$\chi^{(l)}(\sigma_h)=\chi^{(l)}(\sigma_v)=\chi^{(l)}(C_2\bigotimes i)=(-1)^l\times\chi^{(l)}(C_2)$
$\chi^{(l)}(S_n)=\chi^{(l)}(C_{n/2}\bigotimes i)=(-1)^l\times\chi^{(l)}(C_{n/2})$ for improper rotation $S_n$
In the present case,
$\chi^{(2) \mathrm{reducible}}(E)=lim_{n\rightarrow\infty}\chi^{(2)}(C_n)=[l+1/2]\alpha/[\alpha/2]=2l+1=5$
$\chi^{(2) \mathrm{reducible}}(\sigma_h)=\chi^{(2) \mathrm{reducible}}(\sigma_v)=(-1)^{2}\times\chi^{(2)}(C_2)=sin[5/2\times\pi]/sin[\pi/2]=1$
$\chi^{(2) \mathrm{reducible}}(C_3)=sin[5/2\times2\pi/3]/sin[\pi/3]=sin[5\pi/3]/sin[\pi/3]=-1$
$\chi^{(2) \mathrm{reducible}}(S_3)=(-1)^2\times\chi^{(2)}(C_{3/2})=sin[5/2\times2\pi/(3/2)]/sin[1/2\times2\pi/(3/2)]=sin[10\pi/3]/sin[2\pi/3]=-1$
$\chi^{(2) \mathrm{reducible}}(C_2^\prime)=sin[5/2\times\pi]/sin[\pi/2]=1$

step (3): The decomposition formula for reducible representation:
$a_j=1/h\sum_k{N_k\chi^{\Gamma_j}(C_k)^*\chi^{\mathrm{reducible}}(C_k)}$
where $h=12$ is the order of group $D_{3h}$, $\Gamma_j$ the irreducible representation of $D_{3h}$, $C_k$ the class $k$ with $N_k$ elements.
By appying the decomposition formula to the irreducible representation $A^\prime_1$ of $D_{3h}$, one has
$a_{A^\prime_1}=1/12\times[\chi^{A^\prime_1}(E)^*\times\chi^{(2) \mathrm{reducible}}(E) + \chi^{A^\prime_1}(\sigma_h)^*\times\chi^{(2) \mathrm{reducible}}(\sigma_h)+ N_{C_3}\times\chi^{A^\prime_1}(C_3)^*\times\chi^{(2) \mathrm{reducible}}(C_3)+ N_{S_3}\times\chi^{A^\prime_1}(S_3)^*\times\chi^{(2) \mathrm{reducible}}(S_3) +N_{C_2^{\prime}}\times\chi^{A^\prime_1}(C_2^{\prime})^*\times\chi^{(2) \mathrm{reducible}}(C_2^{\prime})+ N_{\sigma_v}\times\chi^{A^\prime_1}(\sigma_v)^*\times\chi^{(2) \mathrm{reducible}}(\sigma_v)]=1/12\times[5+1-2-2+3+3]=2/3$
In principle, $a_{A^\prime_1}$ should be either zero or an positive integer. But I got a fractional number. I repeated the calculations several times but got the same result. I appreciate it if anyone can help me out with this puzzle.

Answer 1

I think I spotted your error. You assumed that $S_3$ is a rotation and inversion, which is not the case in the $D_{3h}$ group. This leads to the wrong character for $\chi(S_3)$. The correct $S_3$ operation is a combination of the $C_3$ rotation and the $\sigma_h$ reflection, which leads to a character $\chi(S_3)=1$