I'm sorry for this naive question. I feel that since I was first introduced to the idea of group representation, I did not correctly grasp the idea. Unfortunately, therefore, several confusions keep bothering me. The definition of representation that I know (from Maggiore's QFT book, and Howard Georgi's Lie Algebras in Particle Physics) is highlighted below.
A representation of a group $\mathcal{G}$ on a vector space $\mathbb{V}$ is a mapping $\mathcal{D}: g\rightarrow \mathcal{D}(g)$ (sometimes also written as $\mathcal{D}: \mathcal{G}\to GL(\mathbb{V})$) onto a set of invertible linear operators $GL(\mathbb{V})$ such that $$\mathcal{D}(e)=\mathbb{1}\tag{1}$$ where $\mathbb{1}$ is the identity operator in the space $GL(\mathbb{V})$. For any two elements $g_1,g_2\in\mathcal{G}$,$$\mathcal{D}(g_1)\mathcal{D}(g_2)=\mathcal{D}(g_1*g_2).\tag{2}$$
Now I have a few questions regarding this.
Question 1 The definition of representation that I stated above (i) doesn't require the associativity $$\Big(\mathcal{D}(g_1)\mathcal{D}(g_2)\Big)\mathcal{D}(g_3)=\mathcal{D}(g_1)\Big(\mathcal{D}(g_2)\mathcal{D}(g_3)\Big)$$ has to be satisfied, and (ii) doesn't require that $$\mathcal{D}(g^{-1})=\mathcal{D}^{-1}(g)$$ have to be satisfied. Does it mean that $GL(\mathbb{V})$ does not form a group? That cannot be true. In that case, what am I missing?
Question 2 For concreteness, suppose I want to understand the representation of the group $\mathcal{G}={\rm SO(3)}$. What is $\mathbb{V}$ in this case? And what is $GL(\mathbb{V})$ in this case? To be even more concrete, if $\mathbb{V}=\mathbb{R}^3$, $GL(\mathbb{V})$ has to be to a set of $3\times 3$ matrices. Should these representation matrices need to be orthogonal and unimodular? Does it follow from the definition of representation? Let us consider a different representation. Say a 5-dimensional representation of SO(3). What is $\mathbb{V}$ and $GL(\mathbb{V})$ in this case? Should these representation matrices again need to be orthogonal and unimodular?