Is angular displacement a vector? How is work defined in rotation if it isn't? Is angular displacement has a direction as well ie is it a vector because angular velocity has a direction and it is defined as rate of change of angular displacement hence it must also be a vector .
Eg consider rotation about a fixed axis it has many similarity with linear one dimensional motion. Like work done by force is force times dx similarly here work done by torque is torque times angular displacenment and must be dot product of two vectors so am I correct is angular displacement a vector as well along the axis of rotation
It isn't according to the answer so how can we define work for rotation about a fixed axis ? Work by definition is scalar product of two vectors then shouldn't it be not defined or is conventionally Work defined in some other way in this case ?
 A: Rotations along different axes do not commute, but vector addition commutes, therefore, angular displacement cannot be a vector.
Angular displacements form a group however, the rotation group $SO(3)$, consisting of orthogonal matrices satisfying $O^{-1}=O^T$ and $\det O=1$.
If you consider a one-parameter family of rotations, indexed by, say, time, so a family $O_t$, such that $O_t\in SO(3)$ for all $t$ and such that $O_0=1$ the identity matrix, then the angular velocity at $t=0$ is given by $$ \omega(t=0)=\frac{d}{dt}O_t|_{t=0}. $$
To evaluate this derivative we consider that $O_tO_t^T=1$ for all $t$ and as such $$ \frac{d}{dt}(O_tO_t^T)|_{t=0}=\frac{d}{dt}O_t|_{t=0}+\frac{d}{dt}O_t^T|_{t=0}=0 $$ thus $$ \omega(0)+\omega^T(0)=0\Longrightarrow\omega(0)=-\omega^T(0), $$ so $\omega(0)$ is skew-symmetric. In the above derivation, during the product rule the second factor in the first term and the first factor in the second term got eliminated because both $O$ and $O^T$ are the identity matrix at $t=0$.
Now, orthogonal matrices form a group, because they are closed under matrix multiplication and inversion, but do not form a linear space, because they are not closed under addition and scalar multiplication.
Antisymmetric matrices on the other hand, are closed under addition and scalar multiplication, and are also closed under commutators, so they do form a vector space, and also a Lie-algebra. In fact, if you have learned any Lie theory, you know that given any Lie-group (which $SO(3)$ is), the Lie algebra of a group is generated by the derivatives of one-parameter subgroups of the Lie group at the identity.
Moreover, in 3 dimensions and only in 3 dimensions, antisymmetric matrices have three independent components, same amount as vectors, so they can be mapped to vectors. This map is given by $\omega_{ij}\mapsto\omega_i=\frac{1}{2}\sum_{j,k=1}^3\epsilon_{ijk}\omega_{jk}$ (the factor of 1/2 might not be needed, but am too lazy to check now).
You can also check that this map turns commutators into cross products, so if $A,B$ are antisymmetric matrices, and $\star$ is the map from antisymmetric matrices to vectors, then $$ \star[A,B]=\star A\times\star B $$ (an additional constant factor like the 1/2 before might enter the picture here, once again, am too lazy to check).
This is why angular velocity is a vector but angular displacements aren't. Angular displacements are elements of a Lie-group, which can be differentiated, but this derivative will leave the Lie-group and will become Lie algebra elements. And Lie algebra elements do form a vector space.
