I can't check your error , but here is how you can obtain the angular velocity vector
the angular velocity vector is:
$$ \vec \omega=\frac {\vec R\times \vec V}{\vec R\cdot\vec R}\tag 1 $$
with : $$\vec R=\left[ \begin {array}{c} x\\y\\0 \end {array} \right] \quad,\vec V=\left[ \begin {array}{c} v_{{x}}\\ v_{{y}} \\ 0\end {array} \right]\quad\Rightarrow \quad \vec\omega_{xy}= \left[ \begin {array}{c} 0\\0\\ {\frac {-yv_{{x}}+xv_{{y}}}{{x}^{2}+{y}^{2}}}\end {array} \right] $$
and with
$$ \vec R'=S\,\vec R\quad,\vec V'=S\,\vec V\quad, S= \left[ \begin {array}{ccc} \cos \left( \phi \right) &\sin \left( \phi \right) &0\\ -\sin \left( \phi \right) &\cos \left( \phi \right) &0\\ 0&0&1\end {array} \right]$$
you obtain from equation (1)
$$ \vec \omega'=\frac {\vec{S\,R} \times \vec{S\,V}}{\vec{S\,R}^T\,\vec{S R}}= \frac {\vec R\times \vec V}{\vec R\cdot\vec R}=\vec \omega$$$$ \vec \omega'=\frac {\left(\vec{S\,R}\right) \times \left(\vec{S\,V}\right)}{\left(\vec{S\,R}\right)^T\,\left(\vec{S R}\right)}= \frac {\vec R\times \vec V}{\vec R\cdot\vec R}=\vec \omega$$
if you have one rotation angle ,the transformation matrix $~\mathbf S~$ has no effect on the results
with $$\vec R=\begin{bmatrix} 0 \\ y \\ z\\ \end{bmatrix}\quad,\vec V=\begin{bmatrix} 0 \\ v_y \\ v_z\\ \end{bmatrix}\quad, S= \left[ \begin {array}{ccc} 1&0&0\\ 0&\cos \left( \phi \right) &\sin \left( \phi \right) \\ 0&-\sin \left( \phi \right) &\cos \left( \phi \right) \end {array} \right] \\ \vec \omega_{yz}= \left[ \begin {array}{c} {\frac {-zv_{{y}}+yv_{{z}}}{{y}^{2}+{z}^{2}} }\\ 0\\ 0\end {array} \right] $$