In classical mechanics we simply have quantities which are simply scalar like momentum, energy of system but when we transit to QM it's an ad-hoc principle, at least to me, that we'll be dealing with quantities which are operator and hence commutativity can't be assumed a priori. Now I want to know if there exist anything which are kind of extension of operator since scalar in mechanics are scaled identity operator. By extension I have analogy of Real, Complex, Quaternion, Octonion in mind, the preceding thing is a substructure of the next.
All operators met in textbook QM are linear. So, if one rephrases slightly your question
Does there exist anything more general than linear operators in QM?,
then the answer suggests itself: non-linear operators.
As it happens, non-linear operators do play an important role in quantum field theory, where interactions can change the number of particles.
A possible extension of operators will be operator-valued distributions.
Functions contain scalars as the constant functions, distributions are a generalization of functions, but there is a wide class of distributions whose codomains are not a field, it is an algebra of operators instead.
The context for this generalization is vector bundles over a manifold.
This article will be helpful for you.