How to get result $3 \otimes 3 = 6 \oplus \bar{3}$ for $SU(3)$ irreducible representations?

Question

Let's have $SU(3)$ irreducible representations $3, \bar{3}$. How to get result that $$ 3\otimes 3 =6 \oplus \bar{3}~? $$ I'm interested in $\bar{3}$ part. It's clear that for $3 \otimes 3$ we can use tensor rules by expanding corresponding matrix on symmetric $6$ and antisymmetric parts. But why we have $\bar{3}$, not $3$, for antisymmetric part?

Answer 1

Section A : The connection of the transformations of complex $\:3\times 3\:$ antisymmetric tensors and their representative complex $\:3$-vectors.

---

Let $\:U\:$ be a special unitary transformation in $\:SU(3)\:$ represented by the $\:3\times 3\:$ complex matrix \begin{equation} U= \begin{bmatrix} u_{11} & u_{12} & u_{13} \\ u_{21} & u_{22} & u_{23} \\ u_{31} & u_{32} & u_{33} \end{bmatrix} \tag{A-01} \end{equation} Since $\:UU^{\boldsymbol{*}}=I\:$ we have $\:U^{\boldsymbol{*}}=U^{-1}$, so \begin{equation} U^{\boldsymbol{*}}=\left(\overline{U}\right)^{\mathsf{T}}=\overline{U^{\mathsf{T}}} \begin{bmatrix} \overline{u}_{11} & \overline{u}_{21} & \overline{u}_{31} \\ \overline{u}_{12} & \overline{u}_{22} & \overline{u}_{32} \\ \overline{u}_{13} & \overline{u}_{23} & \overline{u}_{33} \end{bmatrix} = U^{-1} \tag{A-02} \end{equation} where $\:\overline{u}\:$ = the complex conjugate of $\:u\:$ and $\:U^{\mathsf{T}}\:$ the transpose matrix of $\:U$. By $\:\det\left(U\right)=1$ we have \begin{equation} U^{-1}= \begin{bmatrix} (u_{22}u_{33}-u_{23}u_{32}) & (u_{13}u_{32}-u_{12}u_{33}) & (u_{12}u_{23}-u_{13}u_{22}) \\ (u_{23}u_{31}-u_{21}u_{33}) & (u_{11}u_{33}-u_{13}u_{31}) & (u_{13}u_{21}-u_{11}u_{23}) \\ (u_{21}u_{32}-u_{22}u_{31}) & (u_{12}u_{31}-u_{11}u_{32}) & (u_{11}u_{22}-u_{12}u_{21}) \end{bmatrix} \tag{A-03} \end{equation}

\begin{equation} U^{-1}= \begin{bmatrix} + \begin{vmatrix} u_{22} & u_{23}\\ u_{32} & u_{33} \end{vmatrix} & - \begin{vmatrix} u_{12} & u_{13}\\ u_{32} & u_{33} \end{vmatrix} & +\begin{vmatrix} u_{12} & u_{13}\\ u_{22} & u_{23} \end{vmatrix} \\ &&\\ - \begin{vmatrix} u_{21} & u_{23}\\ u_{31} & u_{33} \end{vmatrix} & + \begin{vmatrix} u_{11} & u_{13}\\ u_{31} & u_{33} \end{vmatrix} & - \begin{vmatrix} u_{11} & u_{13}\\ u_{21} & u_{23} \end{vmatrix} \\ &&\\ + \begin{vmatrix} u_{21} & u_{22}\\ u_{31} & u_{32} \end{vmatrix} & - \begin{vmatrix} u_{11} & u_{12}\\ u_{31} & u_{32} \end{vmatrix} & + \begin{vmatrix} u_{11} & u_{12}\\ u_{21} & u_{22} \end{vmatrix} \end{bmatrix} \tag{A-03$^{\prime}$} \end{equation}

By equations (A-02) and (A-03) the complex conjugate of the elements $\:U\:$ are expressed in terms of the elements themselves \begin{equation} \begin{bmatrix} \overline{u}_{11} & \overline{u}_{21} & \overline{u}_{31} \\ \overline{u}_{12} & \overline{u}_{22} & \overline{u}_{32} \\ \overline{u}_{13} & \overline{u}_{23} & \overline{u}_{33} \end{bmatrix} \\= \begin{bmatrix} (u_{22}u_{33}-u_{23}u_{32}) & (u_{32}u_{13}-u_{33}u_{12}) & (u_{12}u_{23}-u_{13}u_{22}) \\ (u_{23}u_{31}-u_{21}u_{33}) & (u_{33}u_{11}-u_{31}u_{13}) & (u_{13}u_{21}-u_{11}u_{23}) \\ (u_{21}u_{32}-u_{22}u_{31}) & (u_{31}u_{12}-u_{32}u_{11}) & (u_{11}u_{22}-u_{12}u_{21}) \end{bmatrix} \tag{A-04} \end{equation} that is $\:\overline{u}_{11}=u_{22}u_{33}-u_{23}u_{32}\:$,$\:\overline{u}_{21}=u_{32}u_{13}-u_{33}u_{12}\:$... etc.

Now let $\:\boldsymbol{\omega}=\left(\omega_{1},\omega_{2}, \omega_{3}\right)\:$ a complex $\:3$-vector in $\:\mathbb{C}^{3}\:$ and $\:\mathrm{\Omega}\:$ the antisymmetric matrix representing the operation $\:\boldsymbol{\omega}\boldsymbol{\times}\:$

\begin{equation} \mathrm{\Omega}= \begin{bmatrix} 0 & -\omega_{3} & \omega_{2} \\ \omega_{3} & 0 & -\omega_{1} \\ -\omega_{2} & \omega_{1} & 0 \end{bmatrix} =\boldsymbol{\omega}\boldsymbol{\times} \tag{A-05} \end{equation} Suppose that $\:\boldsymbol{\omega}\:$ is transformed to $\:\boldsymbol{\omega}^{\prime}\:$ under a special unitary transformation $\:U \in SU(3)\:$
\begin{equation} \boldsymbol{\omega}^{\prime}=U\boldsymbol{\omega} \tag{A-06} \end{equation} and $\:\mathrm{\Omega}^{\prime}\:$ the antisymmetric matrix representing the operation $\:\boldsymbol{\omega}^{\prime}\boldsymbol{\times}\:$ \begin{equation} \mathrm{\Omega}^{\prime}= \begin{bmatrix} 0 & -\omega^{\prime}_{3} & \omega^{\prime}_{2} \\ \omega^{\prime}_{3} & 0 & -\omega^{\prime}_{1} \\ -\omega^{\prime}_{2} & \omega^{\prime}_{1} & 0 \end{bmatrix} =\boldsymbol{\omega}^{\prime}\boldsymbol{\times} \tag{A-07} \end{equation} We'll determine now the relation between the antisymmetric matrices $\:\mathrm{\Omega}^{\prime}\:$ and $\:\mathrm{\Omega}$. For any $\:\mathbf{z} \in \mathbb{C}^{3}\:$
\begin{equation} \mathrm{\Omega}^{\prime}\mathbf{z}=\boldsymbol{\omega}^{\prime}\boldsymbol{\times}\mathbf{z} =U\boldsymbol{\omega}\boldsymbol{\times}\mathbf{z}=U\boldsymbol{\omega}\boldsymbol{\times} UU^{\boldsymbol{*}}\mathbf{z}=\left[\;\det\left(U\right)\cdot\left(U^{-1}\right)^{\mathsf{T}}\; \right]\left(\boldsymbol{\omega}\boldsymbol{\times}U^{\boldsymbol{*}}\mathbf{z}\right) \tag{A-08} \end{equation} For the last to the right equality in (A-08) we make use of the identity

\begin{equation} \bbox[#E6E6E6,8px]{\boldsymbol{\mathrm{M}}\mathbf{a} \boldsymbol{\times} \boldsymbol{\mathrm{M}}\mathbf{b} = \left[\;\det\left(\boldsymbol{\mathrm{M}}\right)\cdot\left(\boldsymbol{\mathrm{M}}^{-1}\right)^{\mathsf{T}}\; \right]\left(\mathbf{a} \boldsymbol{\times}\mathbf{b}\right)} \tag{B-02} \end{equation} exposed and proved in section B.

Since $\:\det\left(U\right)=1\:$ and $\:U^{-1}=U^{\boldsymbol{*}}\:$ \begin{equation*} \left[\det\left(U\right)\cdot\left(U^{-1}\right)^{\mathsf{T}}\right] \left(\boldsymbol{\omega}\boldsymbol{\times}U^{\boldsymbol{*}}\mathbf{z}\right) =\left[\left(U^{\boldsymbol{*}}\right)^{\mathsf{T}}\left(\boldsymbol{\omega}\boldsymbol{\times}\right) U^{\boldsymbol{*}}\right]\mathbf{z}=\left[\left(U^{\boldsymbol{*}}\right)^{\mathsf{T}}\mathrm{\Omega} U^{\boldsymbol{*}}\right]\mathbf{z}=\left[\overline{U}\mathrm{\Omega} \left(\overline{U}\right)^{\mathsf{T}}\right]\mathbf{z} \end{equation*} so finaly \begin{equation} \boldsymbol{\omega}^{\prime}=U\boldsymbol{\omega}\qquad\Longrightarrow\qquad \mathrm{\Omega}^{\prime}=\left(U^{\boldsymbol{*}}\right)^{\mathsf{T}}\mathrm{\Omega} U^{\boldsymbol{*}}=\overline{U}\mathrm{\Omega} \left(\overline{U}\right)^{\mathsf{T}} \tag{A-09} \end{equation} Since $\:\overline{U}\:$ is also a special unitary transformation,$\:\overline{U}\in SU(3)\:$, replacing $\:U\:$ by $\:\overline{U}\:$ in above equation (A-09) we have

\begin{equation} \boldsymbol{\omega}^{\prime}=\overline{U}\boldsymbol{\omega}\quad\Longrightarrow\quad \mathrm{\Omega}^{\prime}=U\mathrm{\Omega}U^{\mathsf{T}} \tag{A-10} \end{equation} Note that the two equations in (A-10) are equivalent in the following sense : If $\:\mathrm{\Omega}\:$ is a $\:3\times 3\:$ antisymmetric matrix, so representing the product $\:\boldsymbol{\omega}\boldsymbol{\times}\:$ where $\:\boldsymbol{\omega} \in \mathbb{C}^{3}$, and $\mathrm{\Omega}^{\prime}=U\mathrm{\Omega}U^{\mathsf{T}}$,where $U \in SU(3)$, then $\:\mathrm{\Omega}^{\prime}\:$ is also a $\:3\times 3\:$ antisymmetric matrix

\begin{equation*} \text{Proof : } \left(\mathrm{\Omega}^{\prime}\right)^{\mathsf{T}} =\left(U\mathrm{\Omega}U^{\mathsf{T}}\right)^{\mathsf{T}} =\left(U^{\mathsf{T}}\right)^{\mathsf{T}}\mathrm{\Omega}^{\mathsf{T}}U^{\mathsf{T}} =U\left(-\mathrm{\Omega}\right)U^{\mathsf{T}}=-U\mathrm{\Omega}U^{\mathsf{T}} =-\mathrm{\Omega}^{\prime} \end{equation*} and represents the product $\:\boldsymbol{\omega}^{\prime}\boldsymbol{\times}\:$, where $\:\boldsymbol{\omega}^{\prime}=\overline{U}\boldsymbol{\omega}\:$.

\begin{equation} \bbox[#FFFF88,8px]{\mathrm{\Omega}^{\prime}=U\mathrm{\Omega}U^{\mathsf{T}}\quad\Longleftrightarrow\quad \boldsymbol{\omega}^{\prime}=\overline{U}\boldsymbol{\omega}} \tag{A-10$^{\prime}$} \end{equation} This is confirmed also by equating by elements in the equation \begin{equation*} \begin{bmatrix} 0 & -\omega^{\prime}_{3} & \omega^{\prime}_{2} \\ \omega^{\prime}_{3} & 0 & -\omega^{\prime}_{1} \\ -\omega^{\prime}_{2} & \omega^{\prime}_{1} & 0 \end{bmatrix} = \begin{bmatrix} u_{11} & u_{12} & u_{13} \\ u_{21} & u_{22} & u_{23} \\ u_{31} & u_{32} & u_{33} \end{bmatrix} \begin{bmatrix} 0 & -\omega_{3} & \omega_{2} \\ \omega_{3} & 0 & -\omega_{1} \\ -\omega_{2} & \omega_{1} & 0 \end{bmatrix} \begin{bmatrix} u_{11} & u_{21} & u_{31} \\ u_{12} & u_{22} & u_{32} \\ u_{13} & u_{23} & u_{33} \end{bmatrix} \end{equation*} yielding \begin{equation*} \begin{bmatrix} \omega^{\prime}_{1} \\ \omega^{\prime}_{2} \\ \omega^{\prime}_{3} \end{bmatrix} = \begin{bmatrix} (u_{22}u_{33}-u_{23}u_{32}) &(u_{23}u_{31}-u_{21}u_{33}) & (u_{21}u_{32}-u_{22}u_{31}) \\ (u_{32}u_{13}-u_{33}u_{12} & (u_{33}u_{11}-u_{31}u_{13}) & (u_{31}u_{12}-u_{32}u_{11}) \\ (u_{12}u_{23}-u_{13}u_{22} & (u_{13}u_{21}-u_{11}u_{23}) & (u_{11}u_{22}-u_{12}u_{21}) \end{bmatrix} \begin{bmatrix} \omega_{1} \\ \omega_{2} \\ \omega_{3} \end{bmatrix} \end{equation*} and so by (A-04) \begin{equation} \boldsymbol{\omega}^{\prime}=\overline{U}\boldsymbol{\omega} \tag{A-11} \end{equation}

Note: This result has to do with a first step for the construction of baryons from 3 quarks \begin{equation} \boldsymbol{3}\boldsymbol{\otimes}\boldsymbol{3}=\boldsymbol{6}\boldsymbol{\oplus}\overline{\boldsymbol{3}} \tag{A-12} \end{equation}

The invariance of ( the complex $\:3\times 3\:$ tensor's ) antisymmetry under $\:U \in SU(3)\:$ is the invariance of the complex $\:3$-dimensional space of their representative $\:3$-vectors $\:\boldsymbol{\omega}\:$, which are transformed under $\:\overline{U}\:$ and not under $\:U\:$. This explains why $\:\overline{\boldsymbol{3}}\:$ and not $\:\boldsymbol{3}$.

If $\:U=\overline{U}=\mathrm{M}$, that is $\:U\:$ is real, then it represents a pure rotation in $\:\mathbb{R}^{3}\:$,$\:\mathrm{M}^{\mathsf{T}}=\mathrm{M}^{-1}\:$ and (A-10$^{\prime}$) yields \begin{equation} \mathrm{\Omega}^{\prime}=\mathrm{M}\mathrm{\Omega}\mathrm{M}^{-1}\quad\Longleftrightarrow\quad \boldsymbol{\omega}^{\prime}=\mathrm{M}\boldsymbol{\omega} \tag{A-13} \end{equation}

---

Section B : A useful identity necessary in section A

---

If $\:\mathbf{a}= \left( \mathrm{a}_{1}, \mathrm{a}_{2}, \mathrm{a}_{3}\right),\:\mathbf{b}= \left( \mathrm{b}_{1}, \mathrm{b}_{2}, \mathrm{b }_{3}\right) $ are complex $\:3$-vectors in $\:\mathbb{C}^{3}\:$ and $\:\boldsymbol{\mathrm{M}}\:$ an invertible linear transformation in this space represented by the $\:3\times 3\:$ complex matrix \begin{equation} \boldsymbol{\mathrm{M}}= \begin{bmatrix} \mathrm{M}_{11} & \mathrm{M}_{12} & \mathrm{M}_{13} \\ \mathrm{M}_{21} & \mathrm{M}_{22} & \mathrm{M}_{23} \\ \mathrm{M}_{31} & \mathrm{M}_{32} & \mathrm{M}_{33} \end{bmatrix} = \begin{bmatrix} \boldsymbol{\rho}_{1} \\ \boldsymbol{\rho}_{2} \\ \boldsymbol{\rho}_{3} \end{bmatrix} \tag{B-01} \end{equation} where $\:\boldsymbol{\rho}_{i}\; (i=1,2,3)\:$ denote its row complex $\:3$-vectors, then \begin{equation} \bbox[#E6E6E6,8px]{\boldsymbol{\mathrm{M}}\mathbf{a} \boldsymbol{\times} \boldsymbol{\mathrm{M}}\mathbf{b} = \left[\;\det\left(\boldsymbol{\mathrm{M}}\right)\cdot\left(\boldsymbol{\mathrm{M}}^{-1}\right)^{\mathsf{T}}\; \right]\left(\mathbf{a} \boldsymbol{\times}\mathbf{b}\right)} \tag{B-02} \end{equation} where \begin{equation} \det \left(\boldsymbol{\mathrm{M}}\right) =\text{the determinant of } \boldsymbol{\mathrm{M}} = \boldsymbol{\rho}_{1}\circ \left(\boldsymbol{\rho}_{2} \boldsymbol{\times}\boldsymbol{\rho}_{3}\right) \tag{B-03} \end{equation}

\begin{equation} \left(\boldsymbol{\mathrm{M}}^{-1}\right)^{\mathsf{T}} = \text{the transposed inverse of} \: \boldsymbol{\mathrm{M}}= \dfrac{1}{\det\left(\boldsymbol{\mathrm{M}}\right)} \begin{bmatrix} \left(\boldsymbol{\rho}_{2} \boldsymbol{\times}\boldsymbol{\rho}_{3}\right) \\ \left(\boldsymbol{\rho}_{3} \boldsymbol{\times}\boldsymbol{\rho}_{1}\right) \\ \left(\boldsymbol{\rho}_{1} \boldsymbol{\times}\boldsymbol{\rho}_{2}\right) \end{bmatrix} \tag{B-04} \end{equation} The expression $\:\mathbf{a}\circ\mathbf{b}\:$ is defined by \begin{equation} \mathbf{a}\circ\mathbf{b}=\mathrm{a}_{1}\mathrm{b}_{1}+\mathrm{a}_{2}\mathrm{b}_{2}+\mathrm{a}_{3}\mathrm{b}_{3} \tag{B-05} \end{equation} not to be confused with the usual inner product in $\:\mathbb{C}^{3}\:$ \begin{equation} \boldsymbol{\langle}\mathbf{a},\mathbf{b}\boldsymbol{\rangle}=\mathrm{a}_{1}\overline{\mathrm{b}}_{1}+\mathrm{a}_{2}\overline{\mathrm{b}}_{2}+\mathrm{a}_{3}\overline{\mathrm{b}}_{3} \tag{B-06} \end{equation} Proof: Let \begin{equation*} \mathbf{h}=\boldsymbol{\mathrm{M}}\mathbf{a} \boldsymbol{\times}\boldsymbol{\mathrm{M}}\mathbf{b} \end{equation*} If $\:\left\lbrace \mathbf{e}_{1}, \mathbf{e}_{2}, \mathbf{e}_{3}\right\rbrace\:$ is an orthonormal base of $\:\mathbb{C}^{3}\:$ then one can formally write, for any two row vectors \begin{equation} \boldsymbol{\rho} \boldsymbol{\times}\boldsymbol{\sigma} = \begin{vmatrix} \mathbf{e}_{1} & \mathbf{e}_{2} & \mathbf{e}_{3} \\ \rho_1 & \rho_2 & \rho_3 \\ \sigma_1 & \sigma_2 & \sigma_3 \end{vmatrix} \end{equation} ("formally" because a determinant is number, here the $\mathbf{e}_i$ are vectors). Hence \begin{equation} \mathbf{h} = \begin{vmatrix} \mathbf{e}_{1} & \mathbf{e}_{2} & \mathbf{e}_{3} \\ \left( \mathrm{M}_{11}\mathrm{a}_{1}+\mathrm{M}_{12}\mathrm{a}_{2} +\mathrm{M}_{13}\mathrm{a}_{3}\right) & \left( \mathrm{M}_{21}\mathrm{a}_{1}+\mathrm{M}_{22}\mathrm{a}_{2} +\mathrm{M}_{23}\mathrm{a}_{3}\right) & \left( \mathrm{M}_{31}\mathrm{a}_{1}+\mathrm{M}_{32}\mathrm{a}_{2} +\mathrm{M}_{33}\mathrm{a}_{3}\right) \\ \left( \mathrm{M}_{11}\mathrm{b}_{1}+\mathrm{M}_{12}\mathrm{b}_{2} +\mathrm{M}_{13}\mathrm{b}_{3}\right) & \left( \mathrm{M}_{21}\mathrm{b}_{1}+\mathrm{M}_{22}\mathrm{b}_{2} +\mathrm{M}_{23}\mathrm{b}_{3}\right) & \left( \mathrm{M}_{31}\mathrm{b}_{1}+\mathrm{M}_{32}\mathrm{b}_{2} +\mathrm{M}_{33}\mathrm{b}_{3}\right) \end{vmatrix} \end{equation} or in more compact form \begin{equation*} \mathbf{h} = \begin{vmatrix} \mathbf{e}_{1} & \mathbf{e}_{2} & \mathbf{e}_{3} \\ \left(\boldsymbol{\rho}_{1}\circ\mathbf{a}\right) & \left(\boldsymbol{\rho}_{2}\circ\mathbf{a}\right) & \left(\boldsymbol{\rho}_{3}\circ\mathbf{a}\right) \\ \left(\boldsymbol{\rho}_{1}\circ\mathbf{b}\right) & \left(\boldsymbol{\rho}_{2}\circ\mathbf{b}\right) & \left(\boldsymbol{\rho}_{3}\circ\mathbf{b}\right) \end{vmatrix} \end{equation*} so \begin{eqnarray*} \mathrm{h}_{1} & = & \left(\boldsymbol{\rho}_{2}\circ \mathbf{a}\right)\left(\boldsymbol{\rho}_{3}\circ\mathbf{b}\right) -\left(\boldsymbol{\rho}_{2}\circ\mathbf{b}\right)\left(\boldsymbol{\rho}_{3}\circ\mathbf{a}\right)\\ & = & \boldsymbol{\rho}_{2}\circ\underbrace{\left[\left(\boldsymbol{\rho}_{3}\circ\mathbf{b}\right)\mathbf{a} -\left(\boldsymbol{\rho}_{3}\circ\mathbf{a}\right)\mathbf{b}\right]} _{\boldsymbol{\rho}_{3}\boldsymbol{\times}\left(\mathbf{a} \boldsymbol{\times}\mathbf{b}\right)} = \boldsymbol{\rho}_{2}\circ\left[\boldsymbol{\rho}_{3}\boldsymbol{\times}\left(\mathbf{a} \boldsymbol{\times}\mathbf{b}\right)\right]\\ & = & \left(\boldsymbol{\rho}_{2} \boldsymbol{\times}\boldsymbol{\rho}_{3}\right) \circ \left(\mathbf{a} \boldsymbol{\times}\mathbf{b}\right) \end{eqnarray*} that is
\begin{equation*} \mathrm{h}_{1}=\left(\boldsymbol{\rho}_{2} \boldsymbol{\times}\boldsymbol{\rho}_{3}\right) \circ \left(\mathbf{a} \boldsymbol{\times}\mathbf{b}\right) \end{equation*} and by cyclic permutation of the indices 1,2,3 we have for the other two components \begin{equation*} \mathrm{h}_{2}=\left(\boldsymbol{\rho}_{3} \boldsymbol{\times}\boldsymbol{\rho}_{1}\right) \circ \left(\mathbf{a} \boldsymbol{\times}\mathbf{b}\right) \end{equation*} \begin{equation*} \mathrm{h}_{3}=\left(\boldsymbol{\rho}_{1} \boldsymbol{\times}\boldsymbol{\rho}_{2}\right) \circ \left(\mathbf{a} \boldsymbol{\times}\mathbf{b}\right) \end{equation*} and finally \begin{equation*} \mathbf{h}=\boldsymbol{\mathrm{M}}\mathbf{a} \boldsymbol{\times}\boldsymbol{\mathrm{M}}\mathbf{b} = \begin{bmatrix} \left(\boldsymbol{\rho}_{2} \boldsymbol{\times}\boldsymbol{\rho}_{3}\right) \\ \left(\boldsymbol{\rho}_{3} \boldsymbol{\times}\boldsymbol{\rho}_{1}\right) \\ \left(\boldsymbol{\rho}_{1} \boldsymbol{\times}\boldsymbol{\rho}_{2}\right) \end{bmatrix} \left( \mathbf{a} \boldsymbol{\times}\mathbf{b} \right) = \left[\;\det\left(\boldsymbol{\mathrm{M}}\right)\cdot\left(\boldsymbol{\mathrm{M}}^{-1}\right)^{\mathsf{T}}\; \right]\left(\mathbf{a} \boldsymbol{\times}\mathbf{b}\right) \end{equation*} Note that for $\:\boldsymbol{\mathrm{M}}\:$ a real orthonormal matrix \begin{equation*} \boldsymbol{\mathrm{M}}\boldsymbol{\mathrm{M}}^{\mathsf{T}}=\boldsymbol{\mathrm{I}} \quad \Longrightarrow \quad \boldsymbol{\mathrm{M}}^{-1}=\boldsymbol{\mathrm{M}}^{\mathsf{T}} \text{ and } \det\left(\boldsymbol{\mathrm{M}}\right)=\pm 1 \end{equation*} and equation (B-02) yields as expected \begin{equation} \left(\boldsymbol{\mathrm{M}}\mathbf{a} \boldsymbol{\times} \boldsymbol{\mathrm{M}}\mathbf{b}\right) =\pm\boldsymbol{\mathrm{M}}\left(\mathbf{a} \boldsymbol{\times}\mathbf{b}\right) \end{equation} The '+' sign is valid for $\:\boldsymbol{\mathrm{M}}\:$ being a pure rotation while the '-' sign is valid for $\:\boldsymbol{\mathrm{M}}\:$ a rotation plus a reflection.