If you are doing this in a simulation framework, instead of working with components on a rotated coordinate system and the problems that arise from measuring angles and having to decide which sign to use where, I suggest you work with vectors all of which need to be expressed on a common coordinate system.
So here is the typical algorithm for handling collision between point masses (or spheres with no rotation) using vector algebra. With bold are vector quantities and with italics are scalar values. These work the same in 2D as in 3D.
- At the time frame just before the collision, the two masses $m_1$ and $m_2$ have velocity vectors $\boldsymbol{v}_1$ and $\boldsymbol{v}_2$.
- Find the contact point $\boldsymbol{r}_A$ and the direction of the contact normal $$\boldsymbol{n} = {\rm unitvector}(\boldsymbol{r}_2-\boldsymbol{r}_1) \tag{1}$$
- Find the relative impact velocity $$ v_{\rm imp} = \boldsymbol{n} \cdot (\boldsymbol{v}_2 - \boldsymbol{v}_1) \tag{2} $$ Where $\cdot$ is the vector inner product (dot product). The relative velocity should be negative for approaching objects.
- Find the reduced mass of the system $$ m_{\rm eff} = \frac{m_1 m_2}{m_1 + m_2} \tag{3} $$
- Find the impulse $J$ needed for a coefficient of restitution $\epsilon$ $$ J = -(1+\epsilon)\, m_{\rm eff}\, v_{\rm imp} \tag{4}$$
- Apply the impulse to the objects as a velocity step
$$ \begin{aligned} \Delta \boldsymbol{v}_1 & = -\frac{J}{m_1} \boldsymbol{n} & \Delta \boldsymbol{v}_2 & = +\frac{J}{m_2} \boldsymbol{n} \end{aligned} \tag{5}$$
Proof that the above obeys the conservation of linear momentum
Momentum before the impact is $\boldsymbol{p} = m_1 \boldsymbol{v}_1 + m_2 \boldsymbol{v}_2$. Momentum after the impact is
$$ \require{cancel} \begin{aligned}\boldsymbol{p} & =m_{1}\left(\boldsymbol{v}_{1}+\Delta\boldsymbol{v}_{1}\right)+m_{2}\left(\boldsymbol{v}_{2}+\Delta\boldsymbol{v}_{2}\right)\\
& =m_{1}\left(\boldsymbol{v}_{1}-\tfrac{J}{m_{1}}\boldsymbol{n}\right)+m_{2}\left(\boldsymbol{v}_{2}+\tfrac{J}{m_{2}}\boldsymbol{n}\right)\\
& =m_{1}\boldsymbol{v}_{1}-\cancel{J\boldsymbol{n}}+m_{2}\boldsymbol{v}_{2}+\cancel{J\boldsymbol{n}}\;\;\;\checkmark
\end{aligned} \tag{6} $$
Proof that the above obeys and the law of collisions
The relative velocity before the impact is $v_{\rm imp} = \boldsymbol{n} \cdot ( \boldsymbol{v}_2 - \boldsymbol{v}_1 )$. Similarly the relative velocity after the impact is $v_{\rm bounce} = \boldsymbol{n} \cdot \left( (\boldsymbol{v}_2 + \Delta \boldsymbol{v}_2 ) - (\boldsymbol{v}_1 + \Delta \boldsymbol{v}_1) \right)$. The law of impact states that $\boxed{v_{\rm bounce} = -\epsilon\, v_{\rm imp}}$
Expanded out the law of collision is used to find the impulse $J$
$$\begin{aligned}\boldsymbol{n}\cdot\left(\left(\boldsymbol{v}_{2}+\Delta\boldsymbol{v}_{2}\right)-\left(\boldsymbol{v}_{1}+\Delta\boldsymbol{v}_{1}\right)\right) & =-\epsilon\;\boldsymbol{n}\cdot\left(\boldsymbol{v}_{2}-\boldsymbol{v}_{1}\right)\\
\boldsymbol{n}\cdot\left(\left(\boldsymbol{v}_{2}+\frac{J}{m_{2}}\boldsymbol{n}\right)-\left(\boldsymbol{v}_{1}-\frac{J}{m_{1}}\boldsymbol{n}\right)\right) & =-\epsilon\;\boldsymbol{n}\cdot\left(\boldsymbol{v}_{2}-\boldsymbol{v}_{1}\right)\\
\boldsymbol{n}\cdot\left(\frac{J}{m_{2}}\boldsymbol{n}+\frac{J}{m_{1}}\boldsymbol{n}\right) & =-\epsilon\;\boldsymbol{n}\cdot\left(\boldsymbol{v}_{2}-\boldsymbol{v}_{1}\right)-\boldsymbol{n}\cdot\left(\boldsymbol{v}_{2}-\boldsymbol{v}_{1}\right)\\
\boldsymbol{n}\cdot\left(\frac{1}{m_{2}}+\frac{1}{m_{1}}\right)\boldsymbol{n}J & =-(1+\epsilon)\,\boldsymbol{n}\cdot\left(\boldsymbol{v}_{2}-\boldsymbol{v}_{1}\right)\\
J & =-(1+\epsilon)\,\frac{\boldsymbol{n}\cdot\left(\boldsymbol{v}_{2}-\boldsymbol{v}_{1}\right)}{\frac{1}{m_{2}}+\frac{1}{m_{1}}}\\
J & =-(1+\epsilon)\,m_{{\rm eff}}\,v_{{\rm imp}} \;\;\;\checkmark
\end{aligned} \tag{7}$$