I thought that $\mathrm{SO}(3)$ was a group of matrices
This isn't quite right ─ $\mathrm{SO}(3)$ is a group, period. It can be instantiated as a set of matrices with the usual matrix product, but the group itself is ultimately an abstract concept, and any two isomorphic copies of the group can be thought of as "being" the group.
What you're dealing with here isn't the group itself but a representation of the group. A representation $\rho$ of a group $G$ consists of two things:
- a vector space $V$, and
- a linear group action of $G$ on $V$, i.e., an assignment of a linear transformation $\rho(g):V\to V$ to every group element $g\in G$, which respects the group operation (i.e. $\rho(g_1g_2) = \rho(g_1)\rho(g_2)$).
The easiest way to think of this is when the representation is faithful (i.e., when $\rho:G \to \mathrm{End}(V)$ is injective), in which case you're basically formulating a copy of $G$ as a subgroup of the group $\mathrm{End}(V)$ of linear transformations in $V$. Not all group representations are faithful, though (starting with the trivial representation $\rho(g) \equiv \mathbb I$), so you may only be capturing some partial aspects of $G$ within $\mathrm{End}(V)$.
In your case, the vector space in question is
$$
V = \left\{\vec U : \mathbb R^3 \to \mathbb R^3 \right\}
$$
(possibly with some regularity conditions on $\vec U$), and you're finding one specific representation for each $g=R$ in $G=\mathrm{SO}(3)$: specifically, $\hat P = \rho(R)$ is the transformation that takes an arbitrary vector-valued function $\vec U$ to a second vector-valued function $\rho(R)\vec U$ which is given by
$$
\left[\rho(R)\vec U\right](\vec x)
=
R\vec U (R^{-1} \vec x).
$$
The nontrivial work then lies in verifying that $\rho(R_1R_2) = \rho(R_1)\rho(R_2)$, which you should check yourself.
If you want to read more about group representations, then any book about group theory for quantum mechanics will be pretty much completely dedicated to it. (Basically, what quantum physicists call "group theory" is what mathematicians call "group representation theory", and what mathematicians call "group theory" is what physicists call "abstract group theory".)
