Value of matrix elements between bound and unbound states How do you assign a numerical value to the matrix element between states of the discrete part $|n\rangle  $and the continuous parts $|\alpha\rangle $of a spectrum of an operator ? The states of the continuous spectrum have no well defined norm so how do you assign a value to a matrix element like
$$
\langle n|\alpha\rangle
$$
where $n$ is the index for the discrete set of states and $\alpha$ is a parameter for the continous states. Is such an expression even defined ? An example could be bound states of an electron in an atom and plane waves for excited unbound states of the electron.
When expressions like this occur, how are they evaluated, or is there a work around ?
 A: If you are talking about overlaps between the discrete and continuous states of the same hermitian operator, they will always vanish (as in the bound vs unbound energy eigenstates that you mentioned).
In general though, it does make sense to ask about the overlap between a normalizable state and a non-normalizable one. This is actually something that we do all the time. When we talk about the "position wavefunction" $\psi(x)$ this is actually
$$ \psi(x) = \langle x|\psi \rangle$$
Here, $|x \rangle$ is an eigenstate of the unbounded operator x, beloning to the continous spectrum. Clearly, this quantitiy is well defined, even though the norm $\langle x|x\rangle = \infty$. The difference between this quantity and the overlap between normalizable wavefunctions is that the latter is a set of numbers while the former is a density. (In this sense, the "dimensionality" of $\langle x|\psi\rangle$ and $\langle x_i |\psi \rangle$ for a regularized set of wavefunctions $|x_i\rangle$ is really different)
If you want a more "computational" example consider the overlap of the eigenfunctions of the harmonic oscillator with the eigenfunctions of the momentum operator, i.e plane waves.
$$ \langle k|E_n\rangle= \frac{1}{\sqrt{2\pi}} \int dx \,e^{-ikx} \psi_n(x) $$
It is just the fourier transform of the energy eigenstate wavefunction.
The probability that the energy eigenstate has a momentum in the intervall $[k_0,k_0+\Delta k]$ is
$$ P = \int_{k_0}^{k_0+\Delta k} dk \, |\langle k |E_n\rangle|^2$$
This can be verified by considering the finite box model (of lenght L), where the momentum eigenstates $|k_n \rangle$ are normalizable and the momentum is quantized $k_n = \frac{2 \pi n}{L}$:
$$ P = \sum_{k_n} |\langle k_n|\psi \rangle |^2 = \sum_{k_n} |\int dx \frac{e^{-ik_n x}}{\sqrt{L}} \psi(x)|^2$$
and the limit $\sum_{k_n} \rightarrow L \int \frac{dk}{2\pi}$.
