How to determine the result of 2D elastic collision 
As shown by the image, a disk of radius $R_1$ mass $M_1$ and initial velocity $V_0$ collides with another still disk of radius $R_2$ mass $M_2$.  Both disks has no rotation initially.  The direction of $V_0$ is indicated by $\theta$. For three situations there are unique solutions:

*

*When $\theta = 0$, the problem becomes 1D, and both disks has no rotation afterword.

*When there is no friction, both disks has no rotation afterward, and the still disk gain a speed $V_2 = 2 V_0 \frac{\cos \theta}{1 + \frac{M_2}{M_1}}$ along N.

*When $\theta$ is sufficiently large so that $f = \mu_0 N$, in which $\mu_0$ is the static frictional coefficient.  In this case, the momentum transfer along N is $\mu_0$-fold of the momentum transfer along $f$.  Both disks rotate but in opposite direction afterward.  The solution for $V_2$ in the $N$ direction afterward is $$V_2 = 2 V_0 \frac{\cos \theta + \mu_0 \sin \theta}{(1 + \frac{M_2}{M_1})(1 + 3 \mu_0^2)}$$
In the case when $\theta$ is small, how to find a unique solution? Newton mechanics should have unique solution in all cases. And experimentally the outcome should not be random.  So what constrain did I miss?
 A: Friction acts only so long as there is relative slipping between the surfaces of the sphere. For smaller angles, $\mu$ is sufficient for the relative slipping to cease by the end of the collision. For larger angles $f=\mu N$ may not be sufficient to reconcile the tangential velocities of the surfaces by the end of the collision.
For smaller angles, after the collision, the relative tangential velocity of the surfaces of the two spheres is zero, thanks to friction. This should give you your final constraint.
Do the math assuming friction is sufficient to prevent slipping by the end of the collision, ie the angle is small enough. If this value of friction turns out greater than $\mu N$, solve it with $f=\mu N$ (your initial assumption of a small angle was wrong). The words "smaller" and "larger" are qualitative.
A: I will post an answer myself, and we can discuss if it is correct.
First, I argue that this problem has a unique and deterministic answer:

*

*If you play pool, you will know that the balls after collision do not have random movement.  Two identical hits will produce two identical results.

*This is also required by Newton mechanism.

The normal force $N$ causes a change of linear momentum $\Delta P_N = N \Delta t$ along $N$, and results in a speed $v_2$ along $N$.  Without friction, the speed perpendicular to $N$ which is $v_0  \cos \theta$ is not affected, so that the collision is 1D without causing either disk to rotate, or $v_2= \frac{2 v_0 \cos \theta}{1 + m_2/m_1}$.
The touching points between the two disks has relative speed difference at $\theta$, so that the effect of friction can be captured by $\gamma=max(\tan \theta,μ_0)$, in which $μ_0$ is the static frictional coefficient.  It is assumed that the touching points between the two disks have no relative motions during collision.
For disk 2, the frictional force $f$ causes a change of linear momentum $\gamma N \Delta t = \gamma ∆P_N$ perpendicular to $N$, and results in a speed $\gamma v_2$ perpendicular to $N$.  The frictional force also causes an angular momentum $\gamma N r_2 ∆t = \gamma \Delta P_N r_2 = \frac{1}{2} m_2 r_2^2 \omega_2$, and results in an angular speed of $\omega_2 = 2 \gamma \frac{v_2}{r_2}$.
For disk 1, the friction force introduces an angular speed of $\omega_1 = \frac{m_2}{m_1} 2 \gamma \frac{v_2}{r_1}$, and linear speed of $v_0 \cos⁡ \theta - \frac{m_2}{m_1} v_2$ along $N$ and $v_0 \sin \theta - \gamma \frac{m_2}{m_1} v_2$ perpendicular to $N$.
For elastic collision: $v_2 = 2 v_0 \frac{\cos \theta + \gamma \sin \theta}{(1 + \frac{m_2}{m_1})(1 + 3 \gamma^2)}$, which is independent of $r_1$ and $r_2$.
If the two disks has arbitrary initial angular momentum $\Omega_1$ and $\Omega_2$, the initial relative speed at the touching point is $v_0 \cos \theta$ along $N$, and $v_0 \sin \theta - \Omega_1 r_1 + \Omega_2 r_2$ perpendicular to $N$.  Thus, $\gamma = \frac{\sin \theta - \frac{\Omega_1 r_1}{v_0} + \frac{\Omega_2 r_2}{v_0}}{\cos \theta}$:

*

*If $\gamma > 0$, it is capped by $+\mu$.

*If $\gamma < 0$, it is floored by $-\mu$.

