Wigner-Eckart theorem and vectors Let's consider a system in state $^3$D$_1$:
$$\vec{L}^2=L(L+1)=6 $$
$$\vec{S}^2=S(S+1)=2$$
$$\vec{J}^2=J(J+1)=2$$
According to Wigner-Eckart theorem, if this is an irreducible representation, all vectors are proportional, so for example $\vec{S}=a\vec{J}$.
$$\vec{S}\cdot\vec{J}=a\vec{J}^2$$
$$\vec{J}-\vec{S}=\vec{L}$$
$$\vec{J}^2+\vec{S}^2-2\vec{S}\cdot\vec{J}=\vec{L}^2$$
$$\vec{S}\cdot\vec{J}=-1$$
Therefore we find $a=-\frac{1}{2}$, however:
$$\vec{S}^2=2\neq\frac{1}{2}=a^2\vec{J}^2$$
Why doesn't $\vec{S}^2=a^2\vec{J}^2$ hold true, when we have $\vec{S}=a\vec{J}$?
 A: You are misinterpreting W-E theorem. Let me review it for your
example. You are given two irreps, the one for $\vec S$, of dimension 3, and the one for $\vec L$, of dimension 5. Then you build up their direct product, which is of dimension 15 and reducible. Reduction is effected via $\vec J$ and gives three irreps, of dimensions 3, 5, 7 ($3+5+7=15$). Then you consider the first, of dimension 3.
W-E theorem says that any vector operator will have - within that
irrep - matrix elements proportional to those of $\vec J$. Up to now
it's all ok.
But then you operate - within that representation - using algebraic
relations between $\vec S$, $\vec L$, $\vec J$, and this isn't
justified on ground of W-E theorem. Actually it's wrong, because 
$\vec S$ as well as $\vec L$ have matrix elements from that irrep to
the other two. (Only $\vec J$ is reduced into blocks and has zero
matrix elements among different eigenvalues of $J^2$.) Only in the
full 15D rep you'd be permitted to write your equations.
A: $\let\l=\lambda
\def\bJ{{\bf J}} \def\bL{{\bf L}} \def\bS{{\bf S}} 
\def\cD{{\cal D}} \def\cG{{\cal G}} \def\cV{{\cal V}} 
\def\ket#1{|#1\rangle}
\def\frac#1#2{{\textstyle{#1\over#2}}} \def\half{\frac12}
\def\a{^{(3)}} \def\b{^{(5)}} \def\c{^{(7)}} \def\d{^{(15)}} 
\def\Da{\cD\a} \def\Db{\cD\b}\def\Dc{\cD\c} \def\Dd{\cD\d}  
\def\Va{\cV\a} \def\Vb{\cV\b}\def\Vc{\cV\c} \def\Vd{\cV\d}  
\def\Pa{\Pi\a} \def\Pb{\Pi\b}\def\Pc{\Pi\c} \def\Pd{\Pi\d}  
\def\bVa{\bar V\a} \def\bVb{\bar V\b}$
Let's review the problem from its beginning. (I prefer to use boldface capital letters for vector operators, ordinary capital for scalar operators. Lowercase letters always mean ordinary numbers. In
particular, $j$, $l$, $s$ denote quantum numbers.)
We start with a set of states with definite $l$ and $s$: $l=2$, $s=1$. The corresponding kets span a 15-dimensional vector space $\Vd$, tensor product of a 5D space $\bVb$ having a base formed by $L_z$ eigenvectors and a 3-dimensional one $\bVa$, with base of $S_z$ eigenvectors. We could denote the corresponding base in $\Vd$ by 
$$\ket{2,1;m_l,m_s}$$ 
with $m_l$ ranging from $-2$ to $+2$ and $m_s$ from $-1$ to $+1$, but little use will be made of this notation. It's important however to note that in that representation $L_z$, $S_z$ are diagonal and
$|\bL|^2$, $|\bS|^2$ are not only diagonal but multiple of the unity:
$$|\bL|^2 = 6\,I \qquad |\bS|^2 = 2\,I.$$
In the following I'll let $I$ understood when this causes no
ambiguity.
As to $\bJ$, it's defined by
$$\bJ = \bL + \bS \tag1$$
then $J_z=L_z+S_z$ is also diagonal but this can't be said of
$|\bJ|^2$. In fact $|\bJ|^2$ doesn't commute with $L_z$ and $S_z$.
Instead of the $L_z$, $S_z$ base another is of much use: the one where $|\bJ|^2$ and $J_z$ are diagonal. Of course eigenvalues of $|\bJ|^2$ are $j(j+1)$ with $|l-s|\le j\le l+s$ while 
$-j\le m_j\le j$. Note that $|\bJ|^2$ in $\Vd$ isnt't a multiple of unity - allowed values for $j$ are $1,2,3$. The base kets are labelled
$$\ket{j;m_j}.$$ 
Transition from the $m_l,m_s$ representation to the $j,m_j$ is
accomplished by a unitary matrix. Finding that matrix is the
Clebsch--Gordan problem.
There's a deeper way of looking at this problem. Let me summarize it
briefly. We have two representations of the rotation group SO(3) (or better SU(2), but here we don't need to worry about the difference). By representation is meant


*

*A vector space $\cV$.

*A group $\cG$ of linear operators over $\cV$, homomorphic to SO(3).
(By homomorphic is meant that every rotation has an image in $\cG$,
respecting multiplication properties. The correspondence needs not to be one-one.)
In our case we have two such representations: 


*

*One, called $\Db$, acting on $\bVb$ and generated by operators $\bL$

*Another, called $\Da$, acting on $\bVa$ and generated by operators
$\bS$.
Actually in our discussion the group $\cG$ has never appeared - only
angular momentum operators $\bL$, $\bS$, $\bJ$ are playing the game.
Maybe you know why it's so: angular momentum operators are the Lie
algebra of SO(3) and what we are dealing with are representations of that Lie algebra. Without deepening the relationship between a Lie group and its algebra, I want only to recall that in a sense the Lie algebra operators describe the infinitesimal transformations of the group and that almost all properties of that group can be deduced from those of its Lie algebra.
So we start with two representations (physically orbital and spin
angular momenta) whose base vectors are
$$\ket{2;m_l} \quad {\rm and}\quad \ket{1;m_s}$$
respectively belonging to $\bVb$ and to $\bVa$. What is $\Vd\,$? It's their tensor product:
$$\Vd = \bVb\!\otimes \bVa.$$
One base in $\Vd$ is given by the set of pairs
$$\ket{2;m_l}\,\ket{1;m_s}$$
usually contracted (as I did above) to $\ket{2,1;m_l,m_s}$. Of  course $\Vd$ too hosts a representation $\Dd$ of SO(3) and of its Lie algebra: we get it simply adding $\bL$ and $\bS$. To be precise, we had to extend $\bL$ from $\bVb$ to $\Vd$ and $\bS$ from $\bVa$ to
$\Vd$ too:
$$\bL \otimes I\b \qquad I\a\!\otimes \bS$$
but that is usually left understood.
Now the concept of invariant subspace and of minimal invariant
subspace must be introduced. Let's begin with $\Vd$. All operators
like $\bL$, $\bS$ and their linear combinations when applied to a
generic ket $\ket u\in\Vd$ give a result still belonging to $\Vd$ -
that's trivial. The question arises whether proper subspaces $\cV$
of $\Vd$ exist sharing the same property:
$$\Dd \cV \subseteq \cV.$$ 
Any such $\cV$ is called an invariant subspace of $\Dd$. If no other invariant subspace $\cV'\subset\cV$ exists, then $\cV$ is called a minimal invariant subspace. We don't yet know if proper invariant subspaces of $\Vd$ do exist (well, actually we know, but let's pretend we're learning all this matter anew). If $\Vd$ had no proper invariant subspaces, we'd say $\Dd$ is irreducible - otherwise $\Dd$ is reducible. In the latter case we are interested in reducing $\Dd$, i.e. in finding all minimal invariant subspaces of $\Vd$, such that their direct sum equals $\Vd$.
In physical terms, that's the problem of composition of angular
momenta, and we know the answer:
$$\Vd = \Va \oplus \Vb \oplus \Vc.\tag2$$
The meaning of eq. (2) is that every ket of $\Vd$ is the unique sum of three kets, one belonging to $\Va$, another to $\Vb$ and a latter to $\Vc$. Note that $\Va$, $\Vb$, $\Vc$ are mutually orthogonal (that's a theorem for SO(3) and generally for compact or finite groups).
So a complete reduction of $\Dd$ is equivalent to changing to the base $\ket{j;m_j}$. In that base all matrices representing SO(3) or its Lie algebra are reduced to blocks: only matrix elements internal to component subspaces are non zero. Note however that this holds for $J_x$, $J_y$, $J_z$ and their linear combinations - but not for the components of $\bL$ and $\bS$. As to $|\bL|^2$ and $|\bS|^2$, they are always $5\,I$ and $3\,I$ - as they are multiple of the unity this holds in any orthonormal base. $|\bJ|^2$ is slightly more complicated:it's a multiple of unity in each invariant subspace, but with different coefficients:
$$|\bJ|^2 = 2\,I\a + 6\,I\b + 12\,I\c. \tag3$$
Shortly we'll see another way of expressing that.
Now consider your first passages. From eq. (1) we get
$$\bL = \bJ - \bS$$
and squaring
$$|\bL|^2 = |\bJ|^2 + |\bS|^2 - 2\,\bJ\cdot\bS$$
$$2\,\bJ\cdot\bS = |\bJ|^2 + |\bS|^2 - |\bL|^2 \tag4$$
All these equations, in particular (4), hold in $\Vd$. We must be
careful when trying to apply it to subspaces like $\Va$, which is what you did. In order to operate safely I'll now introduce the technique of projection operators.
Maybe you know them, but let me write a short summary, using our space $\Vd$ and its subspaces as an example. Recall eq. (2) and its
interpretation. If $\ket u\in\Vd$ then there are three kets
$\ket{u\a}\in\Va$, $\ket{u\b}\in\Vb$, $\ket{u\c}\in\Vc$, uniquely
determined and such that 
$$\ket u = \ket{u\a} + \ket{u\b} + \ket{u\c}.$$
$\ket{u\a}$, $\ket{u\b}$ and $\ket{u\c}$ are mutually orthogonal.
Now the projection operator $\Pi\a$ is so defined:
$$\Pa\,\Va = \Va \qquad \Pa\,\Vb = 0 \qquad \Pa\,\Vc = 0 \tag5$$
(please note the notation abuse: I've written "0" meaning the
"zero ket" which is an element of $\Vd$, not a number). A parallel
definition is given for $\Pb$ and $\Pc$. The following properties are implied:
$$\Pa + \Pb + \Pc = I \tag6$$
$$\Pi^{(i)}\,\Pi^{(k)} = \Pi^{(i)} \delta_{ik} \quad (i,j = 3,5,7)$$
For a generic operator $A$ acting on $\Vd$ I'll define
$$A^{(i)} = \Pi^{(i)} A\,\Pi^{(i)}.$$
For instance, what is ${(|\bJ|^2)}\a\,$? From eq. (3) you can easily
verify that
$${(|\bJ|^2)}\a = 2\,\Pa.\tag7$$
And what about $(\bJ\cdot\bS)\a\,$? From (4), (6)
$$2\,(\bJ\cdot\bS) = 2\,\Pa + 6\,\Pb + 12\,\Pc - 4\,I = 
-2\,\Pa + 2\,\Pb + 8\,\Pc$$
$$(\bJ\cdot\bS)\a = -\Pa.$$
On the other hand
$$\eqalign{
(\bJ\cdot\bS)\a &= \Pa(\bJ\cdot\bS)\,\Pa \cr 
&= \bJ\a\cdot\bS\a + \Pa\bJ\,\Pb\cdot\Pb\bS\,\Pa +
\Pa\bJ\,\Pc\cdot\Pc\bS\,\Pa \cr
&= \bJ\a\cdot\bS\a \cr}.$$
I've used (3) which implies $\Pa\bJ\,\Pb=\Pa\bJ\,\Pc=0$. Then
$$\bJ\a\cdot\bS\a = -\Pa \tag8$$
And at last Wigner and Eckart came! Applied to our example the theorem
says, as you wrote
$$\bS\a = a\,\bJ\a \tag9$$
for some $a$ to be determined. Multiplying by $\bJ\a$
$$\bJ\a\cdot \bS\a = a\,\bJ\a\cdot\bJ\a = 2\,a\,\Pa$$
from (7). Using (8) in LHS
$$a = -\half.$$
On the other hand, from (7) and (9)
$$\bS\a\cdot\bS\a = a^2 {(|\bJ|^2)}\a = \half\,\Pa.$$
So you see where your error was. You'd taken $|\bS|^2=2$, which is
true in $\Vd$ but not in $\Va$. This is because $\bS$ isn't reduced in
blocks, like $\bJ$. Would you try to check that $\Pa\bS\,\Pb$,
$\Pa\bS\,\Pc$, $\Pb\bS\,\Pc$ don't vanish?
