Sigh, I wrote this answer when there was more details in the question... I agree with JAlex that the simplest way to account for conversation of momentum is to use a single impulse $J$ to represent the collision.

$$\hat{v_t} = \frac{J_t}{m_2} + v_t$$ $$\hat{v_n} = \frac{J_n}{m_2} + v_n$$ $$\hat{\omega_S} = -\frac{J_t}{L} + \omega_S$$ $$\hat{\omega} = \frac{J \bullet r_1}{m_1(r_1 \bullet r_1)} + \omega$$

As a slight change in notation, $L$ is the angular inertia of the sphere.

To ensure that the collision doesn't result in penetration:

$$\hat{v_n} \leq -\hat{\omega} \, \sin(\alpha)$$

This results in a deformed half plane (topologically) constraint on the impulse.

To ensure that energy is conserved:

$$m_2(\hat{v_t}^2+\hat{v_n}^2)+L\hat{\omega_S}^2 + m_1 r_1^2 \hat{\omega}^2 \leq m_2(v_t^2+v_n^2)+L\,\omega_S^2 + m_1 r_1^2 \omega^2 $$

This is a deformed disk constraint.

The Intersection of these two constraints define the area of valid collisions. All the stuff about slip velocity and friction is really just to get a better estimate of what the tangential impulse will be, but unusual internal geometry / structure or external constraints can violate those rules.

In particular, because of the constraint on the point, you can have the relative slip-velocity double, or double in the opposite direction, even with no tangential impulse (frictionless).

If you want to model very high friction but elastic (aka super ball) collisions, then you need to define the deformation model you're going to use in order to get a single defined answer, rather than a valid range.

If you want to assume that the slip/sliding velocity will reduce from a non-zero value to zero during the collision then it doesn't make sense to try to conserve energy, as there must have been rubbing to reduce that slip velocity which must result in frictional losses.

Deformation Modeling

On possible deformation model is fully elastic:

$$ F_t = -k_t \, x_t $$ $$ F_n = -k_n \, x_n $$

Where $x$ is the contact displacement, and $k$ represents the stiffness of the material.

$$\frac{d \, x_n}{d\,t} = v_n + \omega\, r_1 \, \sin(\alpha) $$ $$\frac{d \, x_t}{d\,t} = v_t - \omega\, r_1 \, \cos(\alpha) - r_2 \, \omega_S $$

$$\frac{d \, v_t}{d\,t} = \frac{F_n}{m_2} $$ $$\frac{d \, v_n}{d\,t} = \frac{F_t}{m_2} $$ $$\frac{d \, \omega_S}{d\,t} = \frac{F_t}{L} $$

Then if we initialize $x$ to zero we can integrate until $x_n$ is once again zero, and at that point we'll have our new velocities. Note that even though there's no damping in these equations it still doesn't guarantee that there isn't energy lost. If $x_t$ doesn't reach zero at the same time that $x_n$ does then there will be energy "lost" that's stored in the tangential stiffness when the collision ends.

This same integration could be done with damping terms added to the force equations to model less elastic collisions.

Sigh, I wrote this answer when there was more details in the question... I agree with JAlex that the simplest way to account for conversation of momentum is to use a single impulse $J$ to represent the collision.

$$\hat{v_t} = \frac{J_t}{m_2} + v_t$$ $$\hat{v_n} = \frac{J_n}{m_2} + v_n$$ $$\hat{\omega_S} = -\frac{J_t}{L} + \omega_S$$ $$\hat{\omega} = \frac{J \bullet r_1}{m_1(r_1 \bullet r_1)} + \omega$$

As a slight change in notation, $L$ is the angular inertia of the sphere.

To ensure that the collision doesn't result in penetration:

$$\hat{v_n} \leq -\hat{\omega} \, \sin(\alpha)$$

This results in a deformed half plane (topologically) constraint on the impulse.

To ensure that energy is conserved:

$$m_2(\hat{v_t}^2+\hat{v_n}^2)+L\hat{\omega_S}^2 + m_1 r_1^2 \hat{\omega}^2 \leq m_2(v_t^2+v_n^2)+L\,\omega_S^2 + m_1 r_1^2 \omega^2 $$

This is a deformed disk constraint.

The Intersection of these two constraints define the area of valid collisions. All the stuff about slip velocity and friction is really just to get a better estimate of what the tangential impulse will be, but unusual internal geometry / structure or external constraints can violate those rules.

In particular, because of the constraint on the point, you can have the relative slip-velocity double, or double in the opposite direction, even with no tangential impulse (frictionless).

If you want to model very high friction but elastic (aka super ball) collisions, then you need to define the deformation model you're going to use in order to get a single defined answer, rather than a valid range.

If you want to assume that the slip/sliding velocity will reduce from a non-zero value to zero during the collision then it doesn't make sense to try to conserve energy, as there must have been rubbing to reduce that slip velocity which must result in frictional losses.

Deformation Modeling

On possible deformation model is fully elastic:

$$ F_t = -k_t \, x_t $$ $$ F_n = -k_n \, x_n $$

Where $x$ is the contact displacement, and $k$ represents the stiffness of the material.

Geometric constraints: $$\frac{d \, x_n}{d\,t} = v_n + \omega\, r_1 \, \sin(\alpha) $$ $$\frac{d \, x_t}{d\,t} = v_t - \omega\, r_1 \, \cos(\alpha) - r_2 \, \omega_S $$

Equations of motion:

$$\frac{d \, v_t}{d\,t} = \frac{F_n}{m_2} $$ $$\frac{d \, v_n}{d\,t} = \frac{F_t}{m_2} $$ $$\frac{d \, \omega_S}{d\,t} = \frac{F_t}{L} $$ $$\frac{d \, \omega}{d \, t} = \frac{-F_n \sin(\alpha)-F_t \cos(\alpha)}{m_1 \, r_1}$$

Then if we initialize $x$ to zero we can integrate until $x_n$ is once again zero, and at that point we'll have our new velocities. Note that even though there's no damping in these equations it still doesn't guarantee that there isn't energy lost. If $x_t$ doesn't reach zero at the same time that $x_n$ does then there will be energy "lost" that's stored in the tangential stiffness when the collision ends.

This same integration could be done with damping terms added to the force equations to model less elastic collisions.

Sigh, I wrote this answer when there was more details in the question... I agree with JAlex that the simplest way to account for conversation of momentum is to use a single impulse $J$ to represent the collision.

$$\hat{v_t} = \frac{J_t}{m_2} + v_t$$ $$\hat{v_n} = \frac{J_n}{m_2} + v_n$$ $$\hat{\omega_S} = \frac{J_t}{L} + \omega_S$$ $$\hat{\omega} = \frac{J \bullet r_1}{m_1(r_1 \bullet r_1)} + \omega$$

As a slight change in notation, $L$ is the angular inertia of the sphere.

To ensure that the collision doesn't result in penetration:

$$\hat{v_n} \leq -\hat{\omega} \, \sin(\alpha)$$

This results in a deformed half plane (topologically) constraint on the impulse.

To ensure that energy is conserved:

$$m_2(\hat{v_t}^2+\hat{v_n}^2)+L\hat{\omega_S}^2 + m_1 r_1^2 \hat{\omega}^2 \leq m_2(v_t^2+v_n^2)+L\,\omega_S^2 + m_1 r_1^2 \omega^2 $$

This is a deformed disk constraint.

The Intersection of these two constraints define the area of valid collisions. All the stuff about slip velocity and friction is really just to get a better estimate of what the tangential impulse will be, but unusual internal geometry / structure or external constraints can violate those rules.

In particular, because of the constraint on the point, you can have the relative slip-velocity double, or double in the opposite direction, even with no tangential impulse (frictionless).

If you want to model very high friction but elastic (aka super ball) collisions, then you need to define the deformation model you're going to use in order to get a single defined answer, rather than a valid range.

If you want to assume that the slip/sliding velocity will reduce from a non-zero value to zero during the collision then it doesn't make sense to try to conserve energy, as there must have been rubbing to reduce that slip velocity which must result in frictional losses.

Sigh, I wrote this answer when there was more details in the question... I agree with JAlex that the simplest way to account for conversation of momentum is to use a single impulse $J$ to represent the collision.

$$\hat{v_t} = \frac{J_t}{m_2} + v_t$$ $$\hat{v_n} = \frac{J_n}{m_2} + v_n$$ $$\hat{\omega_S} = -\frac{J_t}{L} + \omega_S$$ $$\hat{\omega} = \frac{J \bullet r_1}{m_1(r_1 \bullet r_1)} + \omega$$

As a slight change in notation, $L$ is the angular inertia of the sphere.

To ensure that the collision doesn't result in penetration:

$$\hat{v_n} \leq -\hat{\omega} \, \sin(\alpha)$$

This results in a deformed half plane (topologically) constraint on the impulse.

To ensure that energy is conserved:

$$m_2(\hat{v_t}^2+\hat{v_n}^2)+L\hat{\omega_S}^2 + m_1 r_1^2 \hat{\omega}^2 \leq m_2(v_t^2+v_n^2)+L\,\omega_S^2 + m_1 r_1^2 \omega^2 $$

This is a deformed disk constraint.

The Intersection of these two constraints define the area of valid collisions. All the stuff about slip velocity and friction is really just to get a better estimate of what the tangential impulse will be, but unusual internal geometry / structure or external constraints can violate those rules.

In particular, because of the constraint on the point, you can have the relative slip-velocity double, or double in the opposite direction, even with no tangential impulse (frictionless).

If you want to model very high friction but elastic (aka super ball) collisions, then you need to define the deformation model you're going to use in order to get a single defined answer, rather than a valid range.

If you want to assume that the slip/sliding velocity will reduce from a non-zero value to zero during the collision then it doesn't make sense to try to conserve energy, as there must have been rubbing to reduce that slip velocity which must result in frictional losses.

Deformation Modeling

On possible deformation model is fully elastic:

$$ F_t = -k_t \, x_t $$ $$ F_n = -k_n \, x_n $$

Where $x$ is the contact displacement, and $k$ represents the stiffness of the material.

$$\frac{d \, x_n}{d\,t} = v_n + \omega\, r_1 \, \sin(\alpha) $$ $$\frac{d \, x_t}{d\,t} = v_t - \omega\, r_1 \, \cos(\alpha) - r_2 \, \omega_S $$

$$\frac{d \, v_t}{d\,t} = \frac{F_n}{m_2} $$ $$\frac{d \, v_n}{d\,t} = \frac{F_t}{m_2} $$ $$\frac{d \, \omega_S}{d\,t} = \frac{F_t}{L} $$

Then if we initialize $x$ to zero we can integrate until $x_n$ is once again zero, and at that point we'll have our new velocities. Note that even though there's no damping in these equations it still doesn't guarantee that there isn't energy lost. If $x_t$ doesn't reach zero at the same time that $x_n$ does then there will be energy "lost" that's stored in the tangential stiffness when the collision ends.

This same integration could be done with damping terms added to the force equations to model less elastic collisions.