you can use those equations
\begin{align*}
&m_1\,(\mathbf v_1-\mathbf u_1)=-\lambda\,\mathbf n\\
&m_2\,(\mathbf v_2-\mathbf u_2)=\lambda\,\mathbf n\\
&\left[(\mathbf v_2-\mathbf{v}_1)+\epsilon\,(\mathbf u_2-\mathbf u_1)\right]\cdot\mathbf n=0
\end{align*}
you have 3 equations for the 5 unknowns; the 4 components of the vectors $~\mathbf v_i~$ and $~\lambda$
theory
starting with Newton equation immediately after the collision
\begin{align*}
&m_1\,\frac{d\mathbf v}{dt}= -F_c\,\mathbf n\quad\Rightarrow\\
&m_1\,\int_{\mathbf u_1}^{\mathbf v_1}=-\int F_c\,\mathbf n\,dt=-\lambda\mathbf{n}\\
&m_1\,(\mathbf v_1-\mathbf u_1)=-\lambda\,\mathbf n
\end{align*}
the conservation of the energy
\begin{align*}
&E=\frac{1}{2}\left(m_1\,(\mathbf{v}_1)^2+m_2\,(\mathbf{v}_2)^2-
m_1\,(\mathbf{u}_1)^2-m_2\,(\mathbf{u}_2)^2\right)=0\\
&2\,E=\left(m_1\,\left [(\mathbf{v}_1)^2- (\mathbf{u}_1)^2\right]
+m_2\,\left[(\mathbf{v}_2)^2-
(\mathbf{u}_2)^2\right]\right)=0\\
&2\,E=\left(m_1\,\left [\mathbf{v}_1- \mathbf{u}_1\right]\cdot
\left [\mathbf{v}_1+ \mathbf{u}_1\right]
+m_2\,\left[\mathbf{v}_2-\mathbf{u}_2\right]
\cdot \left[\mathbf{v}_2+\mathbf{u}_2\right]\right)=0\\
&\text{with}\quad \mathbf{v}_1- \mathbf{u}_1=-\frac{\lambda}{m_1}\,\mathbf n
\quad, \mathbf{v}_2- \mathbf{u}_2=\frac{\lambda}{m_2}\,\mathbf n\\
&\left[(\mathbf v_2-\mathbf{v}_1)+(\mathbf u_2-\mathbf u_1)\right]\cdot\mathbf n=0
\end{align*}
and with the coefficient of restitution $~\epsilon~$
\begin{align*}
&\left[(\mathbf v_2-\mathbf{v}_1)+\epsilon\,(\mathbf u_2-\mathbf u_1)\right]\cdot\mathbf n=0
\end{align*}
thus for $~\epsilon=1~$ you obtain the conservation of the energy
the solution
with
\begin{align*}
&\mathbf{v}_1+\frac{\lambda}{m_1}\,\mathbf n=\mathbf u_1\\
&\mathbf{v}_2-\frac{\lambda}{m_2}\,\mathbf n=\mathbf u_2\\
&(\mathbf{v}_1-\mathbf{v}_1)\cdot \mathbf n=-\epsilon\,(\mathbf u_2-\mathbf u_1)\,\cdot\mathbf n
\end{align*}
you obtain linear equation system $~\mathbf{A}\mathbf x=\mathbf b~$ with
\begin{align*}
&\mathbf{A}=\begin{bmatrix}
E_2 & 0 & \frac{1}{m_1}\,\mathbf{n} \\
0 & E_2 & -\frac{1}{m_2}\,\mathbf{n} \\
(\mathbf{n})^T & -(\mathbf{n})^T & 0 \\
\end{bmatrix}_{5\times 5}\quad,
\mathbf x=\begin{bmatrix}
\mathbf{v}_1 \\
\mathbf{v_2} \\
\lambda \\
\end{bmatrix}\quad,
\mathbf b=\begin{bmatrix}
\mathbf{u}_1 \\
\mathbf{u}_2 \\
-\epsilon\,(\mathbf u_2-\mathbf u_1)\,\cdot\mathbf n \\
\end{bmatrix}
\end{align*}
- $~\mathbf v_i~$ final after the collision
- $~\mathbf u_i~$ initial velocities
- $~\mathbf n~$ unit normal vector
- $~F_c~$ constraint force towards the normal vector
- $~\epsilon~$ coefficient of restitution