Group representations and active/passive transformations Suppose we are in Euclidean 3-space with coordinates $x$ and a scalar function $\phi(x)$ defined on it, and consider the group of rotations $SO(3)$ for simplicity. Take a rotation matrix $R \in SO(3)$; then the usual explanation for the difference between active and passive transformation is:


*

*Active: If we picture the function $\phi(x)$ as having a "bump" somewhere, an active transformation moves this bump around. We now have a new function $\phi'$ of the same coordinates $x$, given by $\phi'(x) = \phi(R^{-1}x)$.

*Passive: We leave the bump fixed and instead make $R$ rotate the coordinate axes to get a new set of coordinates $x'$ on the same physical space. In terms of the new coordinates, the field is expressed as a function $\phi'(x') = \phi(Rx')$ or $\phi'(x) = \phi(Rx)$ if you prefer, since the name of the coordinates doesn't make any difference.
This makes perfect sense but there's a problem: the passive transformation is not a representation of the group of rotations. If we are to think of transformations as an action of $SO(3)$ (or the Lorentz group or whatever your favorite group is) on the space of functions, we can only use active transformations, since only they are actually a group action. Does this imply that if we study fields as representations of some group we are restricting ourselves to active transformations only? Shouldn't the active and passive viewpoints be equivalent?
Edit: the passive transformation is not a group representation because if we define $(\rho_R \phi)(x) = \phi(Rx)$, we get $\rho_{R_1} \rho_{R_2} = \rho_{R_2 R_1}$ instead of the other way around.
 A: Active vs. Passive.
An active transformation is a map $T\colon \mathrm O(n)\to \mathrm{End}(\mathscr C(\mathbb R^n))$ defined as
$$
(T_R\phi)(x):=\phi(R^{-1}x)\tag1
$$
A passive transformation is a map $f\colon\mathrm O(n)\to \mathrm{End}(\mathbb R^n)$ defined as
$$
f_R(x):=Rx\tag2
$$
These two view points are related through
$$
T_R\phi\equiv\phi\circ f_R^{-1}\tag3
$$
They both furnish representations of $\mathrm O(n)$:
$$
T_R\circ T_{R'}=T_{RR'},\qquad f_R\circ f_{R'}=f_{RR'}\tag4
$$
as is trivially checked:
$$
\begin{aligned}
(T_R(T_{R'}\phi))(x)&\overset{(1)}=(T_{R'}\phi)(R^{-1}x)\overset{(1)}=\phi(R'^{-1}R^{-1}x)=\phi((RR')^{-1}x)\\
f_R(f_{R'}x)&\overset{(2)}=f_R(R'x)\overset{(2)}=RR'x
\end{aligned}\tag5
$$
as required.
Needless to say, and because of equation $(3)$, establishing the group structure of $T$ automatically implies that of $f$ and vice-versa. We included both calculations to illustrate both viewpoints.
Left vs. Right.
OP is actually asking about a non-standard notion of active vs. passive, that I will call left vs. right for lack of a better term. I will focus on $T$, but the discussion in terms of $f$ is analogous.
A left (active) transformation is a map $T\colon \mathrm O(n)\to \mathrm{End}(\mathscr C(\mathbb R^n))$ defined as
$$
(T_R\phi)(x):=\phi(R^{-1}x)\tag6
$$
A right (active) transformation is a map $\tilde T\colon \mathrm O(n)\to \mathrm{End}(\mathscr C(\mathbb R^n))$ defined as
$$
(\tilde T_R\phi)(x):=\phi(Rx)\tag7
$$
Unlike before, these two transformations are two different transformations, not two different points of view of the same transformation. That being said, they are related through
$$
T=\tilde T\circ \mathrm{inv}\tag8
$$
where $\mathrm{inv}\colon R\mapsto R^{-1}$ is the inverse map.
As discussed before, $T$ is a $G$-homomorphism, where $G$ is the group $G=(\mathrm{End}(\mathscr C(\mathbb R^n)),\circ)$. In other words, $T$ is a representation of $\mathrm O(\mathbb R^n)$, with $\mathrm{End}(\mathscr C(\mathbb R^n))$ being the representation space. The group product is just composition.
On the other hand, $\tilde T$ is a $G$-anti-homomorphism (indeed, $\tilde T$ is the composition of a homomorphism $T$ and the standard (involutive) anti-homomorphism $\mathrm{inv}$, cf. $(8)$). In other words, $\tilde G$ it is not quite a representation, in accordance with OP's claims. One should point out that $\tilde T$ is a $\tilde G$-homomorphism, where $\tilde G$ is the group $\tilde G=(\mathrm{End}(\mathscr C(\mathbb R^n)),\star)$, where the group product is defined as $\star=\circ\otimes\tau$, where $\tau\colon A\otimes B\mapsto B\otimes A$ is the swap map. In other words, $\star$ is the product 
$$
A\star B:=B\circ A\tag9
$$
That $\tilde G$ is indeed a group is an easy exercise that is left to the reader. This group is typically known as the opposite group of $G$, and is denoted by $G^\mathrm{op}$.
The conclusion is now immediate: $\tilde T$ is also a representation of $\mathrm O(n)$, but the group product is not $\circ$, but $\star$ instead.
A: I think an "active" transformation is a left action, and a "passive" transformation is a right action.
Active transformation:
$$(R_1 \cdot (R_2 \cdot \phi)) (x) = (R_2 \cdot \phi) (R_1^{-1} x) = \phi(R_2^{-1} R_1^{-1} x) = ((R_1 R_2) \cdot \phi) (x)$$
Passive transformation:
$$(R_1 \cdot (R_2 \cdot \phi)) (x) = (R_2 \cdot \phi) (R_1 x) = \phi(R_2 R_1 x) = ((R_2 R_1) \cdot \phi) (x)$$
Usually physicists don't ever think about right actions, but mathematically there's no reason not to.
