There is no answer that conserves energy. First, from elasticity, we know that the normal component of the velocity stays constant. We then have two unknowns: the new tangent velocity and the new rotation speed. Conservation of angular momentum around the point of contact (all forces act through it, so the net torque around it will be zero) provides one equation. If we use conservation of kinetic energy as the other equation, there are two solutions, one perfectly smooth, and the other perfectly rough, as demonstrated here in detail.
The more general model assumes two coefficients of restitution. A COR in the tangential direction $c_T$ determines roughness, while a COR in the normal direction $c_N$ determines the elasticity of a collision.
These coefficients are defined using the original and new velocities of the contact point. $v$ is the speed of the sphere's center of mass. $\omega$ is its rotation speed. Values after collision are represented by a hat. Tangential velocity is positive to the right, rotation is positive counterclockwise, $r$ is the sphere's radius:
$$c_N = \frac{\hat{v_N}}{v_N}$$
$$c_T = \frac{\hat{v_T} + \hat{\omega} r}{v_T + \omega r}$$
$c_N = -1$ is a perectly elastic collision, $c_N = 0$ a perfectly inelastic one. $c_T = 1$ is perfectly smooth, while $c_T = -1$ is perfectly rough.
The conservation of angular momentum dictates (the moment of inertia of the sphere being $Jmr^2$, $J$ being $\frac{2}{5}$ for a full sphere, $\frac{2}{3}$ for a hollow one, and $1$ for a hollow cylinder):
$$Jmr^2\hat{\omega} - mr\hat{v_T} = Jmr^2\omega - mrv_T$$
Combining this with the definition of $c_T$ and $c_N$, the results are:
$$\hat{\omega} = \frac{v_T(c_T-1) + r\omega(c_T+J)}{r(J+1)}$$
$$\hat{v_T} = \frac{JR\omega(c_T-1) + v_T(Jc_T+1)}{J+1}$$
$$\hat{v_N} = v_Nc_N$$
As noted here, $c_T$ may vary for the same sphere depending on the impact angle, and can be computed fro the coefficient of friction between the sphere and the wall. The more general problem of sphere-sphere collisions is studied here (starting around page 15 of the PDF).
