Why does group representation theory look linear? I'm reading first a few chapters of a physicist's group theory book and one naive question comes into my mind. I feel I probably missed something very basic and got bogged down in the details.
My impression is that the theory of (finite) group representation is largely built upon matrices and linear algebra. For instance, those matrix/character orthogonality relations of irreps, the application of a symmetry operator $\hat{P}_g$ as such $\hat{P}_g\vert\alpha^{(\Gamma_n)}\rangle=\sum_\beta D^{(\Gamma_n)}_{\beta\alpha}(g)\vert\beta^{(\Gamma_n)}\rangle$, where $g$ is an element of group $G$, $D$ and $\vert\alpha\rangle$ are the matrix and basis vector of irrep $\Gamma_n$.
All these naturally fit into quantum mechanics, which is also linear. But I don't immediately see a connection to linearity or linear space in the definition of a group. Why can it be so or does it have to be like this? And a follow-up question is: Can a linear (representation) theory capture all the information of something not necessarily linear, i.e., a group?
 A: Concerning OP's title question Why does group representation theory look linear? it should be mentioned that there exist in physics (e.g. in the areas of spontaneous symmetry breaking and supersymmetric field theory) important so-called nonlinear realizations of group actions via an induced representation.
A: A group in itself is not necessarily related to linear spaces. Representation theory (which is a subfield of group theory) is the study of the way in which groups act linearly on vector spaces : a representation of a group $G$ on a vector space $V$ is  a group morphism $\rho:G\to \mathbf{GL}(V)$ into the group of invertible linear maps of $V$.
In physics, groups are almost always relevant in the way by which they act on other things. These group actions are not always representations, because the objects acted upon are not always linear spaces. For example, in general relativity, space-time has not linear structure but it can have an action of a group of isometries.
In quantum mechanics, we are working with a Hilbert space $\mathcal H$ and its linear structure is very important to the theory. Therefore, the only way a group $G$ can act on $\mathcal H$ is linearly, in which case $\mathcal H$ carries a representation of $G$.
To recap :

*

*the connection to linearity is not in the definition of a group, but in the definition of a representation of a group

*because quantum mechanics relies on linearity, symmetries need to act linearly. This is where representations appear

A: A group $G$ is an algebraic structure which encodes one pattern of composition of objects. This makes abstract the notion of transformations which may be composed to give new ones. The whole point, though, is that it is abstract. As I said, it captures the pattern of how these transformations combine.
Once you have a group you may start trying to realize these transformations concretely on sets of objects. This is what we call a group action. Mathematically it is a map $\rho:G\times S\to S$, where $S$ is the set on which you will act with these transformations, where we demand the map to obey $$\rho(gh,x)=\rho(g,\rho(h,x)),\quad \rho(e,x)=x,\tag{1}$$
where $e\in G$ is the identity. Notice that (1) just means that you make sure you are implementing the composition pattern that $G$ encodes!
When the set $S$ is a vector space $V$ you might try to realize the composition pattern with linear transformations. Then you just demand that these maps $x\mapsto \rho(g,x)$ be linear operators for each $g\in G$. This is a representation of $G$.
So, in summary, group representations do not look linear, they are linear by definition. Group representations are linear group actions, which tell you how the pattern of composition encoded by the group can be made concrete with linear maps on a vector space.
A: Linearity is no restriction in the following sense: Every compact Lie group $G$ (in particular every finite group) has a faithful finite-dimensional linear representation, see the Peter-Weyl-theorem. In other words, every such $G$ is isomorphic to a subgroup of the group of complex invertible $n \times n$ matrices, for some $n$.
Remark: There is a simple construction that associates to every nonlinear representation a linear representation: Given a nonlinear representation, by which I mean a group action of $G$ on a set $X$, then you can associate to it a linear representation on the vector space of all functions $f: X \to \mathbb{C}$ by setting $$(gf)(x) = f(g^{-1}x)$$ for all $g \in G$ and all $x \in X$. If the nonlinear representation is faithful then so is the associated linear representation.
One can apply this construction to $X = G$, the group acting on itself by left multiplication, in which case the associated linear representation is the so-called regular representation. This is faithful, and in the special case of a finite $G$ it produces a faithful finite-dimensional linear representation.
A more abstract answer to your question is Tannaka-Krein-duality.
A: This is based on a similar answer I gave in the Math Stack Exchange site:
https://math.stackexchange.com/questions/4027076/intuition-behind-pontryagin-duality/4027123#4027123
The fundamental notion behind the theory of groups is the concept of symmetries and the best place to visualize symmetries is Euclidean space and, by extension, Hilbert's space.
By a symmetry on a Hilbert space $H$ one means any map
$$
  U:H\to H
  $$
that preserves distance, and sends the origin to itself.  In fact any such map is necessarily a unitary operator.
The group $\mathscr U(H)$, formed by all unitary operators on Hilbert's space  is, according to this, the archetype of symmetry!
Given any group $G$, it therefore makes a lot of sense  to try to  model $G$ via  $\mathscr U(H)$,  and this is usually done by considering
group homomorphisms
$$
  u:G\to \mathscr U(H),
  $$
often called  group representations.
