Where does the Density of States come from?

Question

There is an existing question here, which asks about the propagator for a free particle and the difference in its form when expressed as an integral over $p$ or over $E$. The accepted answer points to a function called the Density of States. I am wondering where this Density of States comes from and why it is necessary?

In particular, I am confused as to why the following discrete sum is always true for an orthonormal eigenbasis $|E\rangle$:

$$\sum_{E} \lvert E\rangle\langle E\rvert=I\tag{1}$$

However, the corresponding integral for a continuous $E$ is, in general, not always equal to $I$ (because the Density of States function ($\rho(E)$) is required:

$$\int_{-\infty}^{\infty} \lvert E\rangle\langle E\rvert dE \neq I \tag{2}$$

Presumably, it is because of the addition of the $dE$ term in the integral, which in itself requires normalization (even though $|E\rangle$ is already orthonormal)?

Answer 1

The answer lies in the tricky nature of the continuum. As you probably know, when we normalize our basis and the Hilbert space is compact, then we can choose $\langle n | m \rangle = \delta_{m,n}$ and $|n\rangle$ is dimensionless. However, when during class one introduces a basis which is a continuum of states (let's say the position states) then one normalizes $\langle x | x' \rangle = \delta(x-x')$ which is very different than $\delta_{x,x'}$! One difference is that $|x\rangle$ now has dimensions. Let's say that now we want to perform a 'squeeze' transformation on our space such that $|y\rangle = |a x\rangle$. The new states maintain $\langle y | y' \rangle = \delta(ax-ax') = \delta(x-x')/a$. What happened here? The $a$ factor came our naturally, because the new states are concentrated in a new density with measure $a$ over the original states.

One way to stay always on a safe ground, and for me also to gain intuition, is to start from finite space and then gradually to take the continuum limit. In finite space of length $L$, with periodic boundary conditions, the momentum values are spaced with $2\pi/L$. So we have $I = \sum_n |p=2\pi n / L \rangle \langle p=2\pi n /L|$. And we can write the wave function explicitly $\psi_p(x) = e^{ipx}/\sqrt{L}$. Note that $\sqrt{L}$ is here to make sure that the wave-functions are normalized. Without it, you will not get the identity!

Now we want to take $L\to \infty$. Two things happen - one is that the sum for $I$ becomes 'denser' as the $p$ become closer, but also $\psi_p(x)\to 0$. This is not good as we want a finite wave-function. So we choose $\psi_p(x) = e^{ipx}$, (which is the usual choice for plane waves in an infinite space), and get that $I = 1/L \sum_p |p\rangle \langle p|$. Now we can take the continuum limit, and the sum becomes an integral. What is $dp$? It is the spacing between consecutive values of $p$ i.e. $2\pi/L$. So we get $$I = \frac{1}{2\pi}\int\! dp |p\rangle \langle p|$$ and voila - the $1/2\pi$ is exactly the density-of-states of our $p$ eigenbasis. Now we can do a convenient choice, and incorporate it into our definition of the wave-function, $\psi_p(x) = e^{ipx}/\sqrt{2\pi}$. The d.o.s. is now $1$, and indeed $I=\int\! dp |p\rangle \langle p$. Note that both choices are ok. They just represent different normalizations of our eigenbasis. In the first choice $\langle p | p' \rangle = 2\pi \delta(p-p')$ and in the second one $\langle p | p' \rangle = \delta(p-p')$. In both cases it is derived from a limit of the integral $$\int_{-L/2}^{L/2}\! dx e^{i(p-p')x}$$ when we took $L\to\infty$, where in the second case we just put $2\pi$ by hand.