how you would distinguish between rotational operators and inverse
operators, would a sufficient definition be that rotational operators
is a transformation with one fixed point?
One can define a rotation as an operation that maps an arbitrary vector $\vec v$ to $\vec v'$ through an infinite sequence of infinitesimal operations which leaves the length of the vector invariant.
To exemplify, let us consider rotations in the plane. From the figure bellow
we see that the only infinitesimal operation we can do on $\vec v$ that leaves its length invariant is
$$x\rightarrow x'=x-\epsilon y,\quad y\rightarrow y'=y+\epsilon x.$$
Such infinitesimal operation can be written as
$$\vec v'=(I+\epsilon T)\vec v,$$
where $I$ is the identity matrix and
$$T=\begin{bmatrix}0&-1\\1&0\end{bmatrix}.$$
Now do infinitely many such operations in sequence such that $n\epsilon=\theta$ where $n$ in an integer going to infinity and $\theta$ is a finite real,
$$\vec v'=\left(I+\frac{\theta}{n} T\right)^n\vec v=\exp(\theta T)\vec v .$$
The last equal sign is an identity. The above equation defines the rotation by an angle $\theta$, $R(\theta)=\exp(\theta T)$. One can compute this exponential by Taylor expanding and we obtain
$$R(\theta)=\begin{bmatrix}\cos\theta&-\sin\theta\\ \sin\theta&\cos\theta\end{bmatrix}.$$
We say that a matrix $M$ represents a rotation if and only if it can be written in the above form. Note that a matrix such as
$$S(y)=\begin{bmatrix}-1&0\\0&1\end{bmatrix},$$
which simply maps $(x,y)$ to $(-x,y)$ is not a rotation. It is called a reflection (what you are strangely calling inverse operator).
You can easily verify that rotations matrices are orthogonal (O), $RR^T=I$, and special (S), $\det R=1$. They form the group $SO(2)$ (or $SO(3)$ in three dimensions). Reflection matrices have determinant $-1$ but are also orthogonal. Together with the rotation matrices the form the group $O(2)$ (or $O(3)$ in three dimensions).
Am I right to assume that the $\mathrm{SO}(3)$ rotation group does not
have much of application in quantum mechanics but is rather used more
in classical mechanics whereas $\mathrm{SU}(2)$ is used more in
quantum mechanics, particularly for $s =\frac{1}{2}$ spin systems
where we work in a two dimensional Hilbert space?
In three dimensions the infinitesimal rotations are generated by three generators, $T_1,T_2,T_3$ which play the role of $T$ above. They satisfy the commutation relations
$$[T_a,T_b]=i\epsilon_{abc}T_c,$$
and form a Lie algebra namely $\mathfrak{su}(2)$. The point is that both groups $SO(3)$ and $SU(3)$ have the same Lie algebra. The infinitesimal operations are the same. Moreover in general one can represent these generators with square matrices of different size. Once we choose the size of these matrices (the choice is not arbitrary), we obtain the group associated to the Lie algebra. For example, if we start with the algebra $\mathfrak{su}(2)$ and choose to represent it by $2\times 2$ matrices, then the group obtained is $SU(2)$. On the other hand if we represent it by $3\times 3$ matrices we obtain the group $SO(3)$. This latter group is indeed important in quantum mechanics. It relates to spin $1$.
how does it follow that there are four independent parameters for the
general unitary matrix, the way I see it there are three independent
parameters, namely, $a$, $b$ and $\gamma$?
As already mentioned by jc315's comment, the six real parameter are subjected to two constraints which leaves four real independent parameter.
