How to deal with eigenvectors which are not square integrable? In Quantum Mechanics there is one type of situation I'm still unsure on how to deal with. First of all, I want to make clear I'm trying to understand how to deal with this rigorously. What I'm talking about is this: from what I understood, in Quantum Mechanics, in order to study a particle we consider a Hilbert space $\mathcal{H}$ whose elements are functions $\psi : \mathbb{R}^3\to \mathbb{C}$ with the property that if $W\subset \mathbb{R}^3$ then the probability of finding the particle inside $W$ is
$$P(W)=\int_{W}|\psi(x)|^2d^3x.$$
In particular, elements of $\mathcal{H}$ are square integrable functions, so that $\mathcal{H}\subset L^2(\mathbb{R}^3)$. The inner product in $\mathcal{H}$ is the standard one from $L^2(\mathbb{R}^3)$ restricted.
Now, consider a particle in one-dimension and let $\hat{p} : \mathcal{H}\to \mathcal{H}$ be the momentum operator
$$\hat{p} = -i\hbar \partial_x$$
In that case the eigenvector equation for $\hat{p}$ is
$$\hat{p}\psi = -i\hbar \partial_x \psi = p \psi \Longrightarrow \psi(x)=Ae^{i\frac{p}{\hbar}x}.$$
Now there is a problem. This is not in $\mathcal{H}$. In fact, $\psi \notin L^2(\mathbb{R})$, since $|\psi(x)|^2=|A|^2$.
The same kind of thing happens when talking about the free particle. The "usual" solution for this is to say: "well, this solution is not normalizable, but a certain kind of linear combination can" and then consider
$$\psi(x) = \int_{-\infty}^\infty g(p)e^{ipx/\hbar}dp.$$
Now, from a rigorous standpoint it is not clear for me what is going on. How to deal with eigenvectors of operators like $\hat{p}$? Those eigenvectors cannot exist on $\mathcal{H}$, so what we should do? Also, why to consider this integral? It might still not be in $\mathcal{H}$.
I've heard the true solution to this is using rigged Hilbert spaces (Gelfand triple), but I don't know how it is done. How the use of this structure can make all of this rigorous and why it does the job?
 A: Indeed using a rigged Hilbert Space is almost certainly what you should use and there are tutorials on it on arxiv (I'll add them when/if/I can find them again).
However it isn't necessarily going to help you understand what an average physicist is doing on the one hand, and on the other hand there were already many things you were doing that not rigorous long before you got to eigenvectors.

First of all, I want to make clear I'm trying to understand how to deal with this rigorously. 

So I'll point out some issues with your early work.

to study a particle we consider a Hilbert space $\mathcal{H}$ whose elements are functions $\psi : \mathbb{R}^3\to \mathbb{R}$ with the property that if $W\subset \mathbb{R}^3$ then the probability of finding the particle inside $W$ is
$$P(W)=\int_{W}|\psi(x)|^2d^3x.$$
In particular, elements of $\mathcal{H}$ are square integrable functions, so that $\mathcal{H}\subset L^2(\mathbb{R}^3)$. 

Actually you need a measure on $\mathbb{R}^{3n}$ then you want for measurable $W\subset \mathbb{R}^{3n}$ then the probability of finding the particle inside $W$ is
$$P(W)=\int_{W}|\psi(x)|^2d^{3n}x=\int_{\mathbb R^{3n}}|\psi(x)|^2\Xi_W(x)d^{3n}x
$$
 Where for the last integral $\Xi_W$ is one if you are in $W$ and zero otherwise and where $d^{3n}x$ is the measure on $\mathbb R^{3n}.$ But also we need to note that $\psi$ isn't function $\psi : \mathbb{R}^3\to \mathbb{R}$ it is a function $\psi : \mathbb{R}^{3n}\to \mathbb{C}$ that is measurable, that is complex valued that has a 3n dimensional domain when you have n particles and whose square gives a finite value when you integrate it with that measure. But wait there's more. When you have such a $\psi : \mathbb{R}^{3n}\to \mathbb{C}$ that still isn't an element of your Hilbert space. Your Hilbert space is rigorously the set set of equivalence classes where $\psi_1 : \mathbb{R}^{3n}\to \mathbb{C}$ and $\psi_2 : \mathbb{R}^{3n}\to \mathbb{C}$ are in the same equivalence class if (and only if) $\int |\psi_1(x)-\psi_2|^2d^{3n}x=0$ and so the inner product requires that you select a representative of the equivalence class and you have to use representatives to do addition and even scalar a multiplication and rigorously you'd have to show that is all well defined (and the word well defined is a technical term in mathematics). If you do all that and you used the Lebesque measure then you have the Hilbert Space. And this is what you need even if you don't care for or about momentum eigenstates (and you can't measure those in the lab anyway).
To be fair there are other ways. But these were issues. For instance your inner product wasn't an inner product if you didn't take those equivalence classes because then nonzero things were orthogonal to themselves and to everything else. And if you didn't restrict to measurable subsets $W$ then you'd either have to abandon the axiom of choice (the axiom of choice doesn't come up in the lab either) or some other axiom mathematicians like to assume so often that they won't bother telling you that they assumed it in which case how can you avoid it. And if you didn't have 3n then you can't handle more than one particle (this you need, multiple particles do appear on the lab). And you need the complex numbers. And you need the square integrability to get inner products that are finite. And so on.

Now, consider a particle in one-dimension and let $\hat{p} : \mathcal{H}\to \mathcal{H}$ be the momentum operator
$$\hat{p} = -i\hbar \partial_x$$

Now you started by defining it to go to the Hilbert space which means lots of work to make sure you defined something well defined. What if it sends something to a function that isn't square integrable? It is a function so it takes an input and gives an output. What if it gives different equivalence classes as outputs when you select different representatives from the input equivalence class. What if no element of the equivalence class gives a differentiable function?
There are so many problems if you want to be rigorous. And its possible that it would have been easier to start with some eigenfunctions (not equivalence classes) of realistic experimentally achievable results and then make a single vector space out of their linear combinations and then get your dynamics and such. 
But then you would not have every fancy theorem you might want. All you'd have is a description of what is realizable on the lab. Who wants that?

How the use of this structure can make all of this rigorous and why it does the job?

A rigged Hilbert Space has more sets. You have some smooth functions that you can have your "operators" act on as much as you want. And you can different sets that act on each other and with each other. And then you can carefully keep track of which thing is from which set and let them act.
So can consider functions that are linear functions of the nice ones. And functions that are conjugate linear functions of the nice functions and functions of those functions. You really have to develop everything.
But hopefully you have a taste of what you need and don't need.
