Case 1: Ball rolls up the plane. Gravity retards the ball, so its linear velocity will decrease. In order to prevent slipping, angular velocity should also decrease. So friction should provide a torque opposite to the rotation of the ball. This direction turns out to be up the plane at the lower most point of the ball. The diagram should make it understandable.
Case 2 : Ball rolls down the plane. In this case gravity accelerates the particle. So its linear velocity increases. In order to keep up, its angular velocity must also increase. Friction provides torque to increase angular velocity, which turns out to be up the plane as well.