Axiomatic structure behind Dirac's formulation of QM? According to the paper Quantum Mechanics Beyond Hilbert Space by J.P. Antoine, several mathematical structures have been devised to make mathematical sense of Dirac's formulation of quantum mechanics (specifically, QM with operators having continuous spectra). I was wondering if there was a general set of axioms that could be specified that we would want any such structure to satisfy for it to be a sufficient mathematical image of quantum mechanics. 
 A: The standard treatment of quantum mechanics proceeds by way of asserting that the physical states of systems correspond to vectors in Hilbert spaces.  There is no problem dealing with continuous spectra in this standard setting, see e.g. the Wikipedia page on decomposition of spectra.  In this treatment, however, there are parts of the spectrum of an operator that do not correspond to eigenvalues of certain eigenvectors in the Hilbert space.  Therefore, there exist points in the spectrum that don't correspond to physical states.  In addition, in this treatment, there are certain "kets" that do not correspond to elements of the Hilbert space.
Some people find this state of affairs unsatisfactory, and as a result came the development of the rigged Hilbert space formulation of quantum mechanics.  In this formulation, all elements of the spectrum of an operator are eigenvalues corresponding to eigenvectors in the rigged Hilbert space.  In addition, all objects that one would like to call "kets," are elements of some rigged Hilbert space.
For example, for a one-dimensional free particle moving on the real line $\mathbb R$, the position "eigenvector" $|x\rangle$ does not correspond to an element of the Hilbert space $L^2(\mathbb R)$; it does not correspond to a square-integrable function.  However, rigged Hilbert spaces are "larger" than Hilbert spaces and the rigged Hilbert space for a free particle moving on the real line does contain such objects.
In this sense, the rigged Hilbert space formulation is perhaps a more natural way of making Dirac notation, for example, rigorous.  However, the decision to use rigged Hilbert spaces is, as far as I am aware, a matter of taste.  The conventional formulation via Hilbert spaces can handle any situation you might come across in quantum mechanics, but it requires one to keep a clear distinction between physical states in the Hilbert space, and unphysical states, which are represented by distributions.
