a) Inelastic, rough collision of free spheres
In such a collision, two coefficients are used. The coefficient of restitution in the normal direction (the ratio $c_N$ of relative normal velocities before and after, between $0$ and $1$) determines elasticity, with $1$ being perfectly elastic and $0$ perfectly inelastic. The coefficient of restitution in the tangential direction ($c_T$, the same for tangential velocities) determines smoothness, with $1$ being perfectly smooth and $-1$ perfectly rough.
Both may be somewhat dependent on the impact parameters (and not intrinsic to the spheres), but they (together with momentum and angular momentum conservation laws) define the collision result.
A further discussion can be found in this PDF from page 14 onwards.
b) Elastic, smooth collision with constraints
One of the spheres is constrained to move along a circle and simplified to a point mass. The other sphere is free and impacts it at an oblique angle. We have three degrees of freedom and three conservation laws (two for angular momentum and one for energy). From these, we can determine the post-collision state of the system.
Generalizing situation b) to inelasticity and roughness cannot be done through coefficients of restitution. A perfectly smooth collision need not always have $c_T = 1$ and a perfectly elastic collision need not have $c_N = -1$. A simple counterexample is a perfectly elastic and perfectly smooth collision of a very light orbiting point mass and a stationary free sphere. The point mass will bounce back with little change to the speed of the free sphere, effectively a $c_T$ of $-1$, even though the collision is smooth.
How can such a collision be characterised instead?
There are two effects at hand: The normal bounce and the tangent bounce. If we disregard one, the other behaves in line with coefficients of restitution (i.e. a COR of $-1$ is a perfect bounce and a COR of $1$ is nothing changing).
In the general situation, we have four unknowns: The two speeds of the free marble (the most convenient coordinate system here being the speed in the normal direction and the speed in the tangent direction), the angular velocity of the point, and the rotational angular velocity of the marble. Angular momentum around the center of orbit (of the constrained point) is conserved and all forces act through the contact point, so the angular momentum of just the free marble around the contact point is also conserved. This gives us two equations.
There are three known solutions that conserve energy: A "collision" where all parameters stay the same (total pierce), a perfectly smooth collision, and a perfectly rough collision where the normal velocity of the free marble stays the same (just the tangent component of the collision).
By analogy with nonconstrained collisions, there should be a perfectly rough, perfectly elastic collision that conserves energy, but I cannot find it. Simply finding the differences of pre- and post-collision speeds and rotation speeds for both $c_T = -1$ and $c_N = -1$ and adding them together with the original values (adding the impulses) leads to the total energy changing (in either direction depending on the setup).
$c_T$ and $c_N$ are also the subject of a Q&A I posted with a simpler setup and more details on the results.