EDIT4: I think I was now able to track down where this dogma originally came from. Howard Georgi wrote in TOWARDS A GRAND UNIFIED THEORY OF FLAVOR
There is a deeper reason to require the fermion representation to be complex with respect to SU(3) × SU(2) × U(1). I am assuming that the grand unifying symmetry is broken all the way down to SU(3) × SU(2) × U(1) at a momentum scale of $10^{15}$ GeV. I would therefore expect any subset of the LH fermion representation which is real with respect to SU(3) X SU(2) X U(1) to get a mass of the order of $10^{15}$ GeV from the interactions which cause the spontaneous breakdown. As a trivial example of this, consider an SU(5) theory in which the LH fermions are a 10, a 5 and two $\bar 5$'s. In this theory there will be SU(3) × SU(2) X U(1) invariant mass terms connecting the 5 to some linear combination of the two $\bar 5$-'s. These ten (chiral) states will therefore correspond to 5 four-component fermions with masses of order 10 as GeV. The 10 and the orthogonal linear combination of the two 5-'s will be left over as ordinary mass particles because they carry chiral SU(2) X U(1).
Unfortunately I'm not able to put this argument in mathemtical terms. How exactly does the new, invariant mass term, combining the $5$ and the $\bar 5$ look like?
EDIT3: My current experience with this topic is summarized in chapter 5.1 of this thesis:
Furthermore the group should have complex representations necessary to accommodate the SU(3) complex triplet and the complex doublet fermion representation. [...] the next five do not have complex representations, and so, are ruled out as candidates for the GUT group. [...] It should be pointed out that it is possible to construct GUT's with fermions in the real representation provided we allow extra mirror fermions in the theory.
What? Groups without complex representations are ruled out. And a few sentences later everything seems okay with such groups, as long as we allow some extra particles called mirror fermions.
In almost every document about GUTs it is claimed that we need complex representations (=chiral representations) in order to be able to reproduce the standard model. Unfortunately almost everyone seems to have a different reason for this and none seems fully satisfactory to me. For example:
Witten says:
Of the five exceptional Lie groups, four ( G 2 , F 4 , E 7 , and E 8 ) only have real or pseu-doreal representations. A four-dimensional GUT model based on such a group will not give the observed chiral structure of weak interactions. The one exceptional group that does have complex or chiral representations is E6
This author writes:
Since they do not have complex representations. That we must have complex representations for fermions, because in the S.M. the fermions are not equivalent to their complex conjugates.
Another author writes:
Secondly, the representations must allow for the correct reproduction of the particle content of the observed fermion spectrum, at least for one generation of fermions. This requirement implies that G gut must possess complex representations as well as it must be free from anomalies in order not to spoil the renormalizability of the grand unified theory by an incompatibility of regularization and gauge invariance. The requirement of complex fermion representations is based on the fact that embedding the known fermions in real representations leads to diculties: Mirror fermions must be added which must be very heavy . But then the conventional fermions would in general get masses of order M gut . Hence all light fermions should be components of a complex representation of G gut .
And Lubos has an answer that does not make any sense to me:
However, there is a key condition here. The groups must admit complex representations - representations in which the generic elements of the group cannot be written as real matrices. Why? It's because the 2-component spinors of the Lorentz group are a complex representation, too. If we tensor-multiply it by a real representation of the Yang-Mills group, we would still obtain a complex representation but the number of its components would be doubled. Because of the real factor, such multiplets would always automatically include the left-handed and right-handed fermions with the same Yang-Mills charges!
So... what is the problem with real representations? Unobserved mirror fermions? The difference of particles and antiparticles? Or the chiral structure of the standard model?
EDIT:
I just learned that there are serious GUT models that use groups that do not have complex representations. For example, this review by Langacker mentions several models based on $E_8$. This confuses me even more. On the one hand, almost everyone seems to agree that we need complex representations and on the other hand there are models that work with real representations. If there is a really good why we need complex representations, wouldn't an expert like Langacker regard models that start with some real representation as non-sense?
EDIT2:
Here Stech presents another argument
The groups E7 and E8 also give rise to vector-like models with $\sin^2 \theta = 3/4$. The mathematical reason is that these groups have, like G and F4, only real (pseudoreal) representations. The only exceptional group with complex... [...] Since E7 and Es give rise to vector-like theories, as was mentioned above, at least half of the corresponding states must be removed or shifted to very high energies by some unknown mechanism