Why do we use Hermitian operators in QM? Position, momentum, energy and other observables yield real-valued measurements. The Hilbert-space formalism accounts for this physical fact by associating observables with Hermitian ('self-adjoint') operators. The eigenvalues of the operator are the allowed values of the observable. Since Hermitian operators have a real spectrum, all is well.
However, there are non-Hermitian operators with real eigenvalues, too. Consider the real triangular matrix:
$$
\left( \begin{array}{ccc}
1 & 0 & 0 \\
8 & 4 & 0 \\
5 & 9 & 3 \end{array} \right) 
$$
Obviously this matrix isn't Hermitian, but it does have real eigenvalues, as can be easily verified.
Why can't this matrix represent an observable in QM? What other properties do Hermitian matrices have, which (for example) triangular matrices lack, that makes them desirable for this purpose?
 A: One problem with the given $3\times 3$ matrix example is that the eigenspaces are not orthogonal. 
Thus it doesn't make sense to say that one has with 100% certainty measured the system to be in some eigenspace but not in the others, because there may be a non-zero overlap to a different eigenspace.
One may prove$^{1}$ that an operator is Hermitian if  and only if it is diagonalizable in an orthonormal basis with real eigenvalues. 
See also this Phys.SE post.

$^{1}$We will ignore subtleties with unbounded operators, domains, selfadjoint extensions, etc., in this answer.
A: To give an answer that is a little more general than what you're asking I can think of three reasons for having hermitian operators in quantum theory:

*

*Quantum theory relies on unitary transforms, for symmetries, basis changes or time evolution. Unitary transforms are generated by hermitian operators as in $U=\exp(\mathrm iHt)$. And unitary Lie group representations come with a lie algebra of hermitian operators.


*Outcomes of measurements are taken from a set of orthogonal states with real measurement values. This structure if efficiently represented by a hermitian operator that comes with an eigenstructure that matches these requirements precisely.


*State representations of subsystems and ensembles lead to hermitian operators. For ensembles this can be seen from the construction as a convex sum of projectors, which are necessarily hermitian. For subsystem states it comes out of tracing a projector over tensor factor spaces. This is related to point 2) because processes like decoherence connect measurement outcomes with density operators.
A: If you want to see something different, there are actually a few articles by Carl Bender developing quantum mechanics formulated with parity-time symmetric operators. He shows that some Hamiltonians are not Hermitian, yet they have real eigenvalues and seem to represent valid physical systems. If you think about it, the requirement that your operator is parity-time symmetric makes more sense physically than hermiticity. In a later article, his quantum mechanics approach was proven to be equivalent to the standard one where operators are hermitian.
If you are interested, you can read http://arxiv.org/abs/quant-ph/0501052
A: The point of eigenstates and the entire linear algebra of quantum mechanics is that the projections $\langle\phi|\psi\rangle$ of the state $|\psi\rangle$ onto each eigenstate $|\phi\rangle$ represent the probability amplitudes of each eigenstate. In particular, this means:
$$\sum |\langle\phi|\psi\rangle|^2 = 1=|\langle\psi|\psi\rangle|^2$$
Where the summation is taken over all the eigenstates of an operator. As this must be true for all states $|\psi\rangle$, the thing on the left must be a Pythagoran sum, so the $|\phi\rangle$s must form an orthogonal basis. Alternatively, one may just note that we must have $\langle \phi_1|\phi_2\rangle=0$ if the eigenvalues corresponding are distinct, as two distinct observations must be mutually exclusive.

That shows that the matrices must be normal. That they are chosen to be Hermitian is non-essential, but useful, as has already been discussed.
