Whoah, there's actually so much algebra behind this. It's all about covariant and contravariant indices.
I'll try to make it easy: to transform a basis, you use a certain matrix $R$, so that $u_2= R\cdot u_1$, which means "each vector of the new basis is obtained applying $R$ to the corresponding vector of the old basis". In keywords, new basis = R · old basis
.
However, given a vector, the coordinates transform inversely, that is, they use the inverse matrix $R^{-1}$.
$v_2 = R^{-1} v_1$, new coordinates = R^-1 old coordinates
.
This said, the particular case of rotations is that a rotation is a unitary orthogonal matrix, so that
$$ R^{-1}=R^t $$
Which you can easily check: since $\sin(-\alpha)=-\sin (\alpha)$ (and cosines are the same), putting the minus sign in the other place means turning a "negative angle", or rotating it backwards, so it is the inverse matrix. If you multiply both matrices, you'll see that you get the identity matrix.
Thus we can conclude that
- The minus sign is upside if you are transforming the basis vectors.
- The minus sign is below if you are transforming coordinates of vectors.
I'm assuming this is rotation around the z axis. It works the same for the x axis, but ratoations around the y axis have the sign flipped, so be careful.