I was wondering, if the force of friction with the ground does not make any work on the ball and just give it the necessary torque to rotate (hence the consideration of static friction coefficient in the movement of the wheel of a car for example) thus there are no dissipative forces doing any work in the system. By this reasoning the ball should never stop (taking an ideal ground with friction for example), or at least it wouldn't stop so fast like it does in the real life.
The reasoning you have stated is correct, as the ball maintains a constant velocity in those conditions. However, in reality, the ball squishes a little bit in the direction which it is moving, so a greater frictional force acts (opposing linear motion instead of "assisting" rotary motion), causing a deceleration and hence the ball stops.
It is also affected by other environmental causes such as drag due to motion with respect to the air.