If you calculated $$\boxed{||{\vec{u}'}||^2 = {u'}_x^2+ {u'}_y^2}$$ you would obtain $${u'}^2 = \frac{1}{1-\dfrac{uv\cos\theta}{c}} \left(u^2+v^2-2uv\cos\theta-\frac{u^2v^2\sin^2\theta}{c^2}\right)$$ Then one can notice that $uv\cos\theta = \vec{u}\cdot\vec{v}$ using the definition of the scalar product and $uv\sin\theta = |\vec{u}\times\vec{v}|$ from the definition of the cross product. Finally $$ (\vec{u}-\vec{v})^2 = (\vec{u}-\vec{v}) \cdot (\vec{u}-\vec{v}) = \vec{u} \cdot \vec{u} + \vec{v} \cdot \vec{v} - 2 \vec{u} \cdot \vec{v} = u^2 + v^2 - 2 u v\cos\theta $$
If You wonder, why there is scalar and cross product in the formula, notice that the scalar product is just a number and the cross product is squared, so it is just the squared length of product vector.
