I think the answers already present give a good idea of what rotation matrices do. Because you expressed specifically confusion about this business of which matrix acts on the basis and which on the components, here is a quick intro to what transfromations do to bases and the components of vectors.
Vectors, Bases and Components
Choose any basis of a vector space ${\cal B} = \{\mathbf{e}_1, \mathbf{e}_2, \mathbf{e}_3\}$. Any vector in this vector space can be expressed by its coordinates,
$$ \mathbf{v} = \sum_{i=1}^3 v_i \mathbf{e}_i.$$
You can, but this is purely a convention, express the vector by its components in this basis by writing it in a column,
$$ [\mathbf{v}]_{\cal B} := \begin{pmatrix}v_1 \\ v_2 \\ v_3\end{pmatrix} := \sum_{i=1}^3 v_i \mathbf{e}_i.$$
The second definition is the way that this notation should be understood. This is what components are. Components are always given with relation to some basis, they are meaningless without this basis.
But choosing a basis is an act of personal convenience. There is no experiment that can tell you if nature views a system in cylindrical coordinates or spherical or what it may be. Therefore, the vector $\mathbf{v}$ should be the same no matter in what basis it is expanded. This brings us to the basis change.
Basis Change
Consider a different basis ${\cal B}' = \{\mathbf{e}_1', \mathbf{e}_2', \mathbf{e}_3'\}$. Since ${\cal B}$ and ${\cal B}'$ are both bases, the three basis vectors in $\cal B'$ can be constructed from the basis vectors in $\cal B$. We can write this as
$$\mathbf{e}_1' = \sum_{j=1}^3 A_{1j}\mathbf{e}_j,\qquad
\mathbf{e}_2' = \sum_{j=1}^3 A_{1j}\mathbf{e}_j,\qquad
\mathbf{e}_3' = \sum_{j=1}^3 A_{1j}\mathbf{e}_j,$$
or more conveniently expressed as
$$\mathbf{e}_i' = \sum_{j=1}^3 A_{ij}\mathbf{e}_j, \qquad i \in\{1,2,3\}.$$
Again, the information stored in all the $A_{ij}$ can, by convention, be grouped in a matrix, which is the arrangement
$$
[\matrix{A}]_{\cal BB'}
:= \begin{pmatrix}
A_{11} & A_{12} & A_{13}\\
A_{21} & A_{22} & A_{23}\\
A_{31} & A_{32} & A_{33}
\end{pmatrix}.$$
Now, this is the way the basis transforms. What will happen to the coordinates? Obviously, a vector $\mathbb{v}$ can just as well be written in the new coordinates as it can be writte in the old ones,
$$ \mathbf{v} = \sum_{i=1}^3 v_i' \mathbf{e}_i' \stackrel{!}{=} \sum_{i=1}^3 v_i \mathbf{e}_i.$$
The question is, what are the $v_i'$? Well, we can just insert:
$$ \mathbf{v}
= \sum_{i=1}^3 v_i' \mathbf{e}_i'
= \sum_{i=1}^3 v_i' \sum_{j=1}^3 A_{ij}\mathbf{e}_j
= \sum_{j=1}^3 \left( \sum_{i=1}^3 v_i' A_{ij}\right) \mathbf{e}_j
\stackrel{!}{=} \sum_{j=1}^3 v_j \mathbf{e}_j$$
This is equivalent to
$$\sum_{i=1}^3 v_i' A_{ij} = v_j.$$
For the final step, we use that the matrix $[\matrix{A}]_{\cal BB'}$ is invertible (which is a necessary condition for the basis change to take one basis into another, otherwise the new "basis" wouldn't have rank 3 and there would be vectors that could not be expressed in it, so it can't be a basis). Therefore, there exists $(\matrix{A}^{-1})_{ij}$ with
$$\sum_{j=1}^{3} (\matrix{A}^{-1})_{ij} A_{jk} = \delta_{ik}.$$
Therefore, we can equivalently transform the equation for the components to be
$$\sum_{i,j=1}^3 v_i' A_{ij}(\matrix{A}^{-1})_{jk} = \sum_{i=1}^3 \delta_{ik} v_i' = v_k' \stackrel{!}{=} \sum_{j=1}^3 v_j(\matrix{A}^{-1})_{jk}.$$