The issue here is that you are assuming if something is rolling without slipping it must be that friction is acting on the body. This is just not the case in the ideal situation without rolling friction.
If you have an object rolling without slipping with no other torques acting upon the body, then friction is not acting on the object and it continues rolling with the same linear and angular velocities. It is similar to a book sitting on a table. No horizontal net force is acting on this book, so there is no static friction force acting on it.
Now let's say we have our rolling object and we try to apply an external torque. Then since this torque is "trying" to cause the object to slip, static friction comes into play and must be considered with the net torque. This is similar to our book when we try to push it. Static friction will oppose this. The difference though is that our applied torque can influence the rotation of the object, whereas our book will remain at rest. The similarity though is that static friction only comes into play when we try to make the two surfaces slide relative to each other.
In the real world though there is rolling friction. This is what usually causes things to slow down even when we aren't supplying our own torque.