Very basic question on group representation theory

Question

I'm sorry for this naive question. I feel that since I was first introduced to the idea of group representation, I did not correctly grasp the idea. Unfortunately, therefore, several confusions keep bothering me. The definition of representation that I know (from Maggiore's QFT book, and Howard Georgi's Lie Algebras in Particle Physics) is highlighted below.

A representation of a group $\mathcal{G}$ on a vector space $\mathbb{V}$ is a mapping $\mathcal{D}: g\rightarrow \mathcal{D}(g)$ (sometimes also written as $\mathcal{D}: \mathcal{G}\to GL(\mathbb{V})$) onto a set of invertible linear operators $GL(\mathbb{V})$ such that $$\mathcal{D}(e)=\mathbb{1}\tag{1}$$ where $\mathbb{1}$ is the identity operator in the space $GL(\mathbb{V})$. For any two elements $g_1,g_2\in\mathcal{G}$,$$\mathcal{D}(g_1)\mathcal{D}(g_2)=\mathcal{D}(g_1*g_2).\tag{2}$$

Now I have a few questions regarding this.

Question 1 The definition of representation that I stated above (i) doesn't require the associativity $$\Big(\mathcal{D}(g_1)\mathcal{D}(g_2)\Big)\mathcal{D}(g_3)=\mathcal{D}(g_1)\Big(\mathcal{D}(g_2)\mathcal{D}(g_3)\Big)$$ has to be satisfied, and (ii) doesn't require that $$\mathcal{D}(g^{-1})=\mathcal{D}^{-1}(g)$$ have to be satisfied. Does it mean that $GL(\mathbb{V})$ does not form a group? That cannot be true. In that case, what am I missing?

Question 2 For concreteness, suppose I want to understand the representation of the group $\mathcal{G}={\rm SO(3)}$. What is $\mathbb{V}$ in this case? And what is $GL(\mathbb{V})$ in this case? To be even more concrete, if $\mathbb{V}=\mathbb{R}^3$, $GL(\mathbb{V})$ has to be to a set of $3\times 3$ matrices. Should these representation matrices need to be orthogonal and unimodular? Does it follow from the definition of representation? Let us consider a different representation. Say a 5-dimensional representation of SO(3). What is $\mathbb{V}$ and $GL(\mathbb{V})$ in this case? Should these representation matrices again need to be orthogonal and unimodular?

Answer 1

1. You are wrong about associativity and compatibility of inverses not being implied by eq. (1) and (2). We have: \begin{align} \left(D(g_1)D(g_2)\right)D(g_3) & \overset{(1)}{=} D(g_1g_2)D(g_3) \overset{(1)}{=} D\left( (g_1g_2)g_3\right) = D\left( g_1 (g_2 g_3) \right) \overset{(1)}{=} D(g_1) D(g_2g_3) \\ & \overset{(1)}{=} D(g_1)\left( D(g_2) D(g_3)\right),\end{align} where the only equality that does not hold because of eq. (1) is just associativity in the original group. Likewise, eq. (2) implies that $D(g^{-1}) = D(g)^{-1}$, try to figure that one out yourself.

2. There is no "the" representation of $\mathrm{SO}(3)$, there are countably many different ones up to isomorphism, which we physicists usually label by their half-integral spin $s\in\frac{1}{2}\mathbb{Z}$. For each of these representations, the representation space is $\mathbb{V} = \mathbb{C}^{2s + 1}$. I don't understand what you mean by "What is $\mathrm{GL}(\mathbb{V})$?" - this is always the group of invertible matrices on $\mathbb{V}$, i.e. all matrices with non-zero determinant, and the image of $G$ under the representation map $D$ is some subgroup of it. (It is a subgroup precisely because of point 1.)

   Since you seem to actually want to ask whether e.g. a three-dimensional representation of $\mathrm{SO}(3)$ always has $D(\mathrm{SO}(3)$ as being unimodular orthogonal matrices, the answer is no. For instance, you could very well choose the trivial representation $$ D: \mathrm{SO}(3) \to \mathrm{GL}(3), A\mapsto \mathbf{1}_3,$$ where $\mathbf{1}_3$ is the three-dimensional identity.

   If you require the representation to be three-dimensional and irreducible, then $D(\mathrm{SO}(3)$ is always isomorphic as a group to the group of unimodular orthogonal matrices, but it need not be equal to that group as a subgroup of $\mathrm{GL}(3)$. For instance, any conjugate subgroup to the subgroup of unimodular orthogonal matrices would also be an irreducible representation.

Answer 2

Associativity of composition is a general property of maps between sets, it is independent of any group representation. ${D(g)}^{-1}=D(g^{-1})$ is a simple consequence of your properties (1), (2).