Math of eigenvalue problem in quantum mechanics I learned the eigenvalue problem in linear algebra before and I just find that the quantum mechanics happen to associate the Schrodinger equation with the eigenvalue problem. In linear algebra, we always gives the matrix of certain size (e.g. 4x4) so to find the eigenvalues is identical to solve the related secular equation. In quantum mechanics, in most case, it seems that the dimension of the matrix is not known or very large so to how to find the eigenvalue in those case? I just saw a problem in a book, if we know $A|a\rangle=a|a\rangle$ and $A|b\rangle=b|b\rangle$ for a Hermitian operator $A$. What's the eigenvalue of a Hamilton operator $H=|a\rangle\langle b| + |b\rangle\langle a|$. It is so confusing to me because if we don't know the matrix and elements, how can we write the secular equation and find the eigenvalues? Sorry I didn't take any class on quantum mechanics before, I learn it all my myself so I might be wrong in some statement above. 
 A: In principle, the operator $H$ is not a matrix. However, you can write down a matrix representation. For that, you need a basis, which should be complete (=very large matrix, probably infinitely large) - for many illustrating cases in quantum mechanics, it's not complete.
If your basis consists of $|a\rangle$ and $|b\rangle$, it's not complete, but you can still describe some effects nicely. 
You gave the example $H=|a\rangle\langle b| + |b\rangle\langle a|$, which means that $H$ is (it's just another way to write it)
$$H=\begin{pmatrix}
0 & 1\\ 
1 & 0
\end{pmatrix},$$
and you can calculate its eigenvalues.
For a general 2x2 Hamiltonian matrix, the formula is
$$H=\sum_{i,j}c_{i,j}|i\rangle\langle j|=\begin{pmatrix}
c_{1,1} & c_{1,2}\\ 
c_{2,1} & c_{2,2}
\end{pmatrix}$$
$i$ and $j$ can take the value $a$ and $b$.
The matrix is a 2x2 matrix because the Hamiltonian only contains two vectors, $a$ and $b$.
How to expand a Hamiltonian operator into a matrix is explained nicely on Wikipedia (first four formulas of the section). 
If it's already in the abstract form $H=|a\rangle\langle b| + |b\rangle\langle a|$, you can just read it off: The prefactor of $|a\rangle\langle a|$ ($=0$) is the top left entry, the prefactor of $|a\rangle\langle b|$ ($=1$) is the top left entry etc.
You don't need to know the explicit form of basis in which the matrix is written. Remember that the eigenvalues of a matrix are invariant to unitary transformations.
A: You need to understand the difference between a linear transformation T and the matrix A which represents it. The entries of A depend on the specific basis you're working in. You should recall from linear algebra that to find the columns of A, you apply T to the basis vectors.
For example, in your problem, the Hamiltonian $\hat{H}$ plays the role of T, and $\{\vert a \rangle, \lvert b \rangle\}$ is your basis. If you apply $\lvert a \rangle$ to $\hat{H}$, you get
$$ \hat{H} \lvert a \rangle = \lvert a \rangle\langle b \vert a \rangle + \lvert b \rangle\langle a \vert a \rangle = \lvert b \rangle.$$
Note that this calculation is entirely independent of the representations of the vectors and the Hamiltonian.
Now relative to our chosen basis, it should be clear to you that the vector $\lvert a \rangle$ is represented by the ordered pair $\begin{pmatrix} 1 \\ 0\end{pmatrix}$ and $\lvert b \rangle$ is represented by the ordered pair $\begin{pmatrix} 0 \\ 1\end{pmatrix}$. In terms of matrices and the ordered pairs, the relation $\hat{H}\lvert a \rangle = \lvert b \rangle$ would be written
$$A\begin{pmatrix} 1 \\ 0 \end{pmatrix} = \begin{pmatrix} 0 \\ 1 \end{pmatrix},$$ so the first column of A is $\begin{pmatrix} 0 \\ 1 \end{pmatrix}$. You can find the second column of A by applying $\lvert b \rangle$ to $\hat{H}$. Then once you have the matrix, you can find the eigenvalues and eigenvectors the usual way.
A: The stuff about the operator $A$ is just so that you'll know that $|a\rangle$ and $|b\rangle$ are orthogonal (if we assume $a \neq b$). This follows from the properties of Hermitian operators (just evaluate $\langle a | A | b\rangle$ two different ways).
Since $|a\rangle$ and $|b\rangle$ are orthogonal, they are a good choice for a basis, and we can write the matrix of $H$ in this basis. It is the operator that switches $|a\rangle$ and $|b\rangle$ (this is fairly easy to see without the matrix, but you asked for the matrix). The square of this operator is the identity, so its eigenvalues are $\pm 1$.
A: Eigenvalue problems are much more general than what you see in matrices. For example, you can look for solutions to the eigenvalue problem
$ \frac{d }{d x} y = \lambda y$
solve for $\lambda$. The solution is, of course, $y = e^{\lambda x}$ and $\lambda$ can be a continuum. If you impose some other boundary conditions, you may end up with a discrete spectrum of $\lambda$.
If you insist on a matrix representation, you can use orthogonal functions as your "basis vectors" and their orthogonality relations to define an inner product. Read up about the quantum harmonic oscillator, and Hermite polynomials are an orthogonal basis to expand the solution in that also happen to solve the differential equation exactly.
A: This matrix has only 2 states $ |a> $ and $ |b> $
then ti will gbe a 2x2 matrix
with elements
$ a1,1 = <a|a><b|a>+<a|b><a|a> $
$ a1,2=a2,1 = <a|a><b|b>+<a|b><a|b> $
$a2,2 = <b|a><b|b>+ <b|b><a|b> $
From this matrix you can evaluate the eigenvalues
