I can't check your error , but here is how you can obtain the angular velocity

the angular velocity is:

$$ \vec \omega=\frac {\vec R\times \vec V}{\vec R\cdot\vec R}\tag 1 $$

with :
$$\vec R=\left[ \begin {array}{c} x\\y\\0
\end {array} \right] 
\quad,\vec V=\left[ \begin {array}{c} v_{{x}}\\ v_{{y}}
\\ 0\end {array} \right]\quad\Rightarrow \quad
\vec\omega_{xy}= \left[ \begin {array}{c} 0\\0\\ 
{\frac {-yv_{{x}}+xv_{{y}}}{{x}^{2}+{y}^{2}}}\end {array} \right] 
$$

and with 

$$ \vec R'=S\,\vec R\quad,\vec V'=S\,\vec V\quad,
S= \left[ \begin {array}{ccc} \cos \left( \phi \right) &\sin \left( \phi
 \right) &0\\ -\sin \left( \phi \right) &\cos
 \left( \phi \right) &0\\ 0&0&1\end {array} \right]$$

you obtain from equation (1)

$$ \vec \omega'=\frac {\vec{S\,R} \times \vec{S\,V}}{\vec{S\,R}^T\,\vec{S R}}= \frac {\vec R\times \vec V}{\vec R\cdot\vec R}=\vec \omega$$

the transformation matrix $~\mathbf S~$ has no effect on the results