The idea of a center of mass even works for a "disconnected" system of particles. You can take any set of particles, call it your system and talk about its center of mass. The equation of the center of mass would still be valid, even if all the particles are moving chaotically and there's no axis to talk about for the entire system.
Even for rigid bodies, the "axis of mass" wouldn't be a fixed line. The axis of rotation could keep changing depending on how the external torques add up vectorially. So the "axis of rotation" wouldn't be a property of the rigid body, but rather something which arises in a particular situation. The axis of mass could literally be any line passing through the center of mass.
EDIT-
Responding to your edit, I think the whole point of your post was that, since any point on the axis of rotation can be seen as following a simple path throughout the motion of the ball, so all the axis points are as special as the CoM.
However, this is not true. The other points on the axis are NOT following a simple path in general. In the most general case of a direction changing torque (and hence a continuously changing axis of rotation), any other point (other than the CoM) on any of the instantaneous axes of rotation, is not following a simple path throughout the motion of the sphere. More importantly, the equation $a_{point}=\frac{1}{M}\vec{F_{ext}}$ is only valid when the point is the CoM. This equation is what's responsible for the CoM to follow a simple path. The motion of any other point is affected by the internal as well as the external forces in general.
As for "why don't we define a general diameter of the sphere as the set of all possible axes of rotations?", that'd be useless as any line passing through the CoM of a rigid body can behave as an axis of rotation. The axis of rotation at an instant is entirely dependent on the direction of the instantaneous angular momentum, which is again dependent on the external torques. The CoM, on the other hand, is a fixed property of the rigid body. This is why you study things like: CoM of a circular disc, CoM of a uniform cylinder, etc, instead of the axis of rotation of a circular disc.
I think you might see some sentences in your physics book like "The axis of rotation is always some line passing through the center of mass (unless there's a rotation about a forced axis)". That is all there is to the idea of "axis of mass".