I was wondering if it is important in Quantum Mechanics to deal with operators that have an orthonormal basis of eigenstates? Imagine that we would have an operator (finite-dimensional) acting on a spin system that has real eigenvalues, but its eigenvectors are not perpendicular to each other. Is there any reason why such an operator cannot describe an actual physical quantity?
If two states are orthogonal, this means that $\langle \psi | \phi \rangle = 0$. Physically this means that if a system is in state $|\psi\rangle$ then there is no possibility that we will find the system in state $|\phi\rangle$ on measurement, and vis versa. In other words the 2 states in some sense mutually exclusive. This is an important property for operators because it means that the results of a measurement are unambiguous. A state with a well defined momentum $p_1$, i.e. an eigenstate of the momentum operator, cannot also have a momentum $p_2 \ne p_1$. Observables having an orthogonal (and complete) set of eigenstates is therefore a requirement in order for the theory to make physical sense (or at least for repeated measurements to give consistent results, as is experimentally observed)
If the eigenstates of your operator is not a orthogonal set, then your operator is not a hermitian operator, or in other words, is not an observable.
Actually, non-hermitian operators "appears" all the time, but if you investigate decoherence mechanism you may note that this operators don't affect directly the classical realm. This is because you can't construct consistent histories upon question about this non-hermitian operators. More concretely, if you have the coherent state $|\alpha \rangle$, i.e.
$$ a|\alpha \rangle=\alpha |\alpha \rangle $$
where $a$ is the annihilation operator, then asking the probability of the system have some value of $\alpha$ don't make sense because we always can represent one $\alpha$ state in superposition of others $\alpha's$.
A measurement extracts information from a quantum system that can be copied. Not all of the information in a quantum system can be copied as a result of the no-cloning theorem. The information that can be copied has to be described by an operator that remains unchanged under the unitary operator representing the measurement. And the only operators that respect this requirement are normal operators.
A similar argument can be given in terms of state vectors showing that only outcomes in orthogonal state vectors can be recorded: