Representation of operators in quantum mechanics For which systems we represent the Hamiltonian as a differential operator and for which system we represent it by a matrix? Can the momentum be represented by a matrix operator?
 A: If the Hilbert space of the system in question is finite-dimensional, then in a given basis for the Hilbert space, the Hamiltonian (and every other observable for that matter), will be represented by a matrix.
If the Hilbert space is infinite-dimensional, the situation is a bit different.  In Quantum Mechanics, we typically assume that the Hilbert spaces we deal with are separable which means that they admit a countable, orthonormal basis.  The representation of the Hamiltonian in any such basis will be a "matrix" that is "infinite-dimensional."
Consider, for example, the quantum harmonic oscillator.  Let $B = \{|n\rangle\}$ denote the energy eigenbasis where $|n\rangle$ is the energy eigenvector with eigenvalue $E_n = (n+\frac{1}{2})\hbar\omega$, then the Hamiltonian in this basis looks as follows:
\begin{align}
  [\hat H]_B = \begin{pmatrix}
            E_0 &  &  &   \\
             & E_1 &  &  \\
             &  & E_2 &  \\
             &  &  &  \ddots\\
          \end{pmatrix}
\end{align}
Moreover, the momentum operator is also represented as an "infinite-dimensional" matrix in this basis.  Recall that the momentum operator can be written in terms of creation and annihilation operators as follows:
\begin{align}
  \hat p = i\sqrt{\frac{m\omega\hbar}{2}}(\hat a^\dagger - \hat a)
\end{align}
which gives
\begin{align}
  \langle m|\hat p|n\rangle 
  &= i\sqrt{\frac{m\omega\hbar}{2}}(\sqrt{n+1}\langle m|n+1\rangle - \sqrt{n}\langle m|n-1\rangle) \\
  &=  i\sqrt{\frac{m\omega\hbar}{2}}(\sqrt{n+1}\delta_{m,n+1} - \sqrt{n}\delta_{m,n-1})
\end{align}
and we can therefore easily write out the first few entries of the momentum operator written in the energy eigenbasis:
\begin{align}
  [\hat p]_B = i\sqrt{\frac{m\omega\hbar}{2}}\left(
\begin{array}{ccccc}
 0 & -1 &  &  &  \\
 1 & 0 & -\sqrt{2} &  &  \\
  & \sqrt{2} & 0 & -\sqrt{3} &  \\
  &  & \sqrt{3} & 0 & \ddots \\
  &  &  & \ddots & \ddots \\
\end{array}
\right)
\end{align}
On the other hand, every state in the Hilbert space can be written in the so-called position $\{|x\rangle\}$ and momentum $\{|p\rangle\}$ "bases."  These aren't strictly speaking bases of the Hilbert space, but they pretty much work the same way in the sense that a given state can be written as a integral over these "continuous" basis elements as opposed to sums over "discrete" bases;
\begin{align}
  |\psi\rangle &= \int_{\mathbb R}dx\psi(x)|x\rangle, \\
  |\psi\rangle &= \int_{\mathbb R}dp\tilde\psi(p)|p\rangle
\end{align}
In this notation, every physical state corresponds to a square integrable function $\psi$ in the position basis and $\tilde\psi$ in the momentum basis.  It is in these "bases" that the various observables are represented by multiplication/differential operators on a function space.  In particular, for example the momentum operator is represented as follows:
\begin{align}
  (\hat p\psi)(x) &= \langle x|\hat p|\psi\rangle = \frac{\hbar}{i}\frac{d\psi}{dx}(x) \\
  (\hat p\tilde\psi)(p) &= \langle p|\hat p|\psi\rangle = p\tilde \psi(p).
\end{align}
In short, the representation of observables, whether they are matrix or differential operator representations, depends on the basis you decide to represent them in.
