$\let\a=\alpha \let\om=\omega \let\tm=\times \def\D#1#2{{d#1\over d#2}} \def\ba{\mathbf a} \def\br{\mathbf r} \def\bv{\mathbf v} \def\bF{\mathbf F} \def\bL{\mathbf L} \def\bM{\mathbf M} \def\bP{\mathbf P} \def\bR{\mathbf R} \def\cM{{\cal M}} \def\frac#1#2{{\textstyle {#1 \over #2}}}$
I think that friction is the only force that can affect the
rotational motion of the sphere, as only it exerts a torque on the
sphere..
As you'll see, this statement isn't true, and is also ill-defined.
Let's discuss your question not in terms of energy but in terms of
forces acting. To do so I find expedient to recall a more general form
equation of moments can take.
I'm referring to
$$\D\bL t = \bM \tag1$$
where $\bL$ is angular momentum, $\bM$ resultant moment of external
forces (commonly called "torque" in english mechanics literature, but
I don't like that term, for reasons I can't pause to explain here).
There are some points I have to clarify, however. The first is that
eq. (1) is only meaningful if a reference frame K is assumed. This I
will take for granted and never change in the following. The first
reason why a frame is needed is to give a meaning to points'
velocities. For the present problem K is most conveniently taken as
the frame where the slope is stationary.
In order to define a moment of forces we must also choose a reference
point. This is not to be confused with the reference frame. It can be
a fixed point in K (often the origin of coordinates) but any
other point, fixed or moving, is allowed and occasionally useful.
This is why I defined your final paragraph "ill-defined". You didn't
say which reference point for moments you were assuming. Very likely
you were thinking of ball's c.o.m. But in this case too your statement
isn't correct, as the normal reaction of the slope has a (negative)
moment wrt to ball's centre. As you will presently see, it's possibile
(and convenient) to choose a different reference point, giving rise to
a wholly different analysis of force moments.
Assume our system consists of mass points of masses $m_i$, position
vectors $\br_i$ (taken from coordinates origin O), velocities $\bv_i$.
If the reference point is O, we have
$$\bL = \sum m_i\,\br_i \tm \bv_i.$$
If the net external force acting on $i$-th point is $\bF_i$, then
$$\bM= \sum \br_i \tm \bF_i.$$
We have
$$\D\bL t = \sum m_i\,(\bv_i \tm \bv_i + \br_i \tm \ba_i) =
\sum m_i\,\br_i \tm \ba_i = \sum \br_i \tm \bF_i = \bM$$
and eq. (1) is proven.
Let's now change the reference point to O$'$ (position $\br'$). (Note
che $\br'$ may also depend on time.) Define
$$\bL' = \sum m_i\,(\br_i - \br') \tm \bv_i$$
$$\bM' = \sum (\br_i - \br') \tm \bF_i$$
and compute
$$\eqalign{
\D{\bL'}t &= \sum m_i\,(\bv_i - \bv') \tm \bv_i +
\sum m_i\,(\br_i - \br') \tm \ba_i \cr
&= -\sum m_i\,\bv' \tm \bv_i + \sum (\br_i - \br') \tm \bF_i \cr
&= -\bv' \tm \bP + \bM' \tag2 \cr}$$
where $\bP$ is the system's total momentum.
Eq. (2) simplifies if $\bv'$ is parallel to $\bP$ i.e. to c.o.m. velocity ($\bv'=0$ included):
$$\D{\bL'}t = \bM'.\tag3$$
This is the only case we'll have to consider.
Let's now come to our problem. I'll take $x$-axis along the slope,
positive upwards, $y$-axis positive downwards. As a consequence
$z$-axis is horizontal, parallel to slope plane. Origin O is on the
slope. It's assumed that the ball's center is moving in $(x,y)$ plane.
As a reference point O$'$ for moments the best choice is the contact
point between ball and slope. Then O$'$ is moving, but its velocity is
the same as that of the ball's c.o.m., so that eq. (3) applies. A bonus
of this choice for O$'$ is that moment of the unknown force the slope
exerts on the ball (friction included) vanishes. So as external force
we are left with weight alone.
Weight can be reduced to just one force applied to ball's centre. Its
moment is parallel to $z$. Then if $\cM$ is ball's mass, $r$ its
radius we have
$$M'_z = -\cM\,g\,r \sin\a.\tag4$$
As to $\bL'$, it's also parallel to $z$ and is written
$$L'_z = I \om.\tag5$$
$\om>0$ if the ball is rolling up. $I$ is the moment of inertia wrt to
an axis parallel to $z$ through O$'$. For a homogeneous ball its value is
$$I = \frac75\,\cM\,r^2.\tag6$$
From (3--6) we find
$$\D \om t = -{5 g \over 7 r}\,\sin\a$$
$$\om(t) = \om(0) - \left(\!{5g \over 7r}\,\sin\a\!\right) t.$$
Since $v(t) = \om(t)\,r$, we have also
$$v(t) = v(0) - \frac57\,g\,\sin\a\>t.$$
As you can see, there is no trace of friction in the final result. It
is required, of course, in order to force pure roll. But it does no
work, nor contributes to slowing and halting the ball. (Incidentally, not even mass and radius of the ball have any effect as far as a homogeneous ball is considered.)