How to make rigorous the idea of a continuous complete set? In Quantum Mechanics, when using Dirac's formalism one of its features is the expansion of state vectors into continuous basis of eigenvectors of unbounded self-adjoint operators. Let $\mathcal{H}$ be the state space of a quantum system and $A$ some unbounded self-adjoint operator. 
Then, the usual thing that is done is: one assumes that $A$ has a continuous set of eigenvectors $\{|a\rangle : a\in \mathbb{R}\}$ indexed by its eigenvalues $a\in \mathbb{R}$ and assumes that one can write any state vector as a "linear combination" of those eigenvectors in the sense of the following integral:
$$|\psi\rangle = \int_{-\infty}^{\infty} \langle a|\psi\rangle |a\rangle \ da. $$
Another feature is the "completeness relation" which can be written as the following integral
$$\int_{-\infty}^{\infty}|a\rangle \langle a| \ da = I,$$
being $I$ the identity operator. Finally, there is also the orthogonality relation which is:
$$\langle a | a' \rangle =\delta(a-a').$$
Those three features of Dirac's formalism, although very useful are not rigorous. There are some points I've noticed:


*

*I'm unaware of the validity the spectral theorem for unbounded operators with continuous spectra. In that case, I don't know if one could say that the eigenvectors form a basis. In truth it is not even clear what is meant by basis in this context.

*On the expansion equation, we have the integral of a function $f : \mathbb{R}\to \mathcal{H}$ given by $f(a)=\langle a|\psi\rangle |a\rangle$ and it is not clear to me at first how the integral of such function can be defined.

*On the completeness relation, we have yet another strange integral. Now it is of a function $g : \mathbb{R}\to \mathcal{L}(\mathcal{H},\mathcal{H})$ being $\mathcal{L}(\mathcal{H},\mathcal{H})$ the space of operators on $\mathcal{H}$. This function $g$ is defined by $g(a)=|a\rangle \langle a|$ and it is not clear how the integral of such a function is defined again.

*The orthogonality relation seems really strange. It is not usual orthogonality, but involves the Diract delta, which is a distribution. In that case, although $\langle a|a'\rangle$ should be a complex number, it is being set equal to a distribution, which is a functional over a space of test functions.
I've heard that the Rigged Hilbert Space formalism, also known as Gel'fand triple, solves all these problems. But I didn't understand yet how. In truth, what I know about this construction is that we pick a dense subspace $\Omega$ of the Hilbert space $\mathcal{H}$ where all relevant operators can be defined and are invariant. Then we look at the space of antilinear functionals and linear functionals. This gives meaning for the space of kets and bras, but I don't know how it makes sense of all these constructions I talked about above.
In that case, how this part of the Dirac formalism can be made rigorous? How does one solve these four points? How it is possible to make sense of those integrals and the relations involved? And finally, how the Gel'fand triple can be used here to make everything correct?
 A: There is no eigenvector corresponding to continuous spectrum. The formalism of Gel'fand triples does not give much help in solving your doubts either, and it has very few applications in my experience. One reason is that those "generalized eigenvectors" are not in the Hilbert space but on a bigger space, and what you can do with them is not much on a rigorous standpoint (for example, they can act- by topological duality -only on a subset of the vectors of the Hilbert space; they cannot be multiplied; they cannot act on other generalized eigenvectors;...).
The spectral theorem is, instead, valid for any self-adjoint operator. Even more importantly, there is a one-to-one correspondence between $A\in SelfAdjoint(\mathscr{H})$ and special families of projections called spectral families (or projection-valued measures) $\{P_\Omega\}_{\Omega\in Borel(\mathbb{R})}\subset \mathcal{L}(\mathscr{H})$.
These projectors are the generalization of $\lvert \psi_n\rangle\langle\psi_n\rvert$, projecting on the subspace with eigenvalue $n$ (supposed to be of multiplicity one); so they project, roughly speaking, on the subspace of vectors where the operator take values only in the $\Omega$ portion of the spectrum. In fact, if we denote by $dP(\lambda)$ the measure with respect to the spectral family $\{P_\Omega\}_{\Omega\in Borel(\mathbb{R})}$ (that can be rigorously defined), the associated operator can be written as:
$$A=\int_\mathbb{R}\lambda dP(\lambda)\; ;$$
and the associated decomposition of the identity may be written as:
$$1=\int_\mathbb{R} dP(\lambda)=P_{\mathbb{R}}\; .$$
The last equation is the correct mathematical way of writing the second equation of the OP.
The other equations instead, do not have a rigorous counterpart since there are no eigenvectors of operators with continuous spectra (of course you may write $\psi=\int_{\mathbb{R}}dP(\lambda)(\psi)$, but that is not so useful).
