When dealing with an undamped MDOF dynamic system, one can find the natural frequency (eigenvalue) and corresponding modeshape-vector (eigenvector) for mode $n$ by solving the eigenvalue problem:
$$ [K] \{ \phi \}_n = \omega_n^2 [M] \{ \phi \}_n $$
Furthermore, it is stated in a lot of litterature, that for any two modes, $n$ and $r$, their eigenvectors are orthogonal, because they can be shown to fulfill the following:
$$ \{ \phi \}_n [M] \{ \phi \}_r = 0 \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; \{ \phi \}_n [K] \{ \phi \}_r = 0 $$
Two vectors are, however generally, defined to be orthogonal if: $\;\;\; \{ \phi \}_n^T \{ \phi \}_r = 0 \;\;\;$, and this is actually not true for the modeshape-vectors you find by solving the eigenvalue problem for an undamped dynamic system!
Just to be completely clear, i will give an example using a 2DOF system:
For such a system, the stiffness and mass matrices will be the following:
$$K = \begin{bmatrix} k_1 + k_2 & -k_2 \\ -k_2 & k_2 \end{bmatrix} \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; M = \begin{bmatrix} m_1 & 0\\ 0 & m_2 \end{bmatrix} $$
I will just select some arbitrary values:
$$m_1 = 50 \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; m_2 = 20 \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; k_1 = 90 \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; k_2 = 50$$
which gives:
$$K = \begin{bmatrix} 140 & -50 \\ -50 & 50 \end{bmatrix} \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; M = \begin{bmatrix} 50 & 0\\ 0 & 20 \end{bmatrix} $$
By solving the eigenvalue problem using these matrices, i get the following eigenvectors:
$$ \{ \phi \}_1 = \begin{Bmatrix} 0.575\\ 1.000 \end{Bmatrix} \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; \{ \phi \}_2 = \begin{Bmatrix} -0.695\\ 1.000 \end{Bmatrix} $$
which gives:
$$ \{ \phi \}_1^T \{ \phi \}_2 = 0.6 \neq 0 $$
By using the mass or stiffness matrix, sure enough we get:
$$ \{ \phi \}_1 [M] \{ \phi \}_2 = 0 \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; \{ \phi \}_1 [K] \{ \phi \}_2 = 0 $$
but still $\{ \phi \}_1^T \{ \phi \}_2 \neq 0$ so they are not orthogonal! So why do we call them orthogonal?
^2
in the post. $\endgroup$