As you said they name is totally related with the diagram, since the i-variable is simple the sum (+) of the momentum that go to an annihilation minus (-) the momentum that are created from a vertex the i-diagram, as we can easily see from:
But, and here goes what you are asking, once you defined them in the way we said above, they remain the same for all the other cases, and each one gets a physical general intuition that works for any diagram.
Taking this general picture we can see it clearly:
The s-variable will be the total energy of the interaction in the mass center, which means that for any reference frame you compute it with different $p_1$ and $p_2$ the result will be the same (Actually this is the principal motivation to use them, that they are Lorentz invariant), and concretely the result will be the energy of the CM frame, to show it let's start with: \begin{equation} (p_1+p_2)^2 = p_1^2+p_2^2 + 2p_1p_2= E_1^2 - |\vec{p}_1|^2 + E_2^2 - |\vec{p}_2|^2 + 2(E_1E_2 - \vec{p}_1\vec{p}_2) \end{equation} and now since in that reference frame the $\vec{p}_1= -\vec{p}_2$, we get: \begin{equation} (E_1^2+E_2^2+2E_1E_2) -|\vec{p}_1|^2 - |\vec{p}_2|^2 - 2 \vec{p}_1(-\vec{p}_1) = (E_1+E_2)^2 \end{equation} where $E_1+E_2$ is the total energy in the CM frame, you see?
The t and u diagrams are less intuitive, but also you can show, that they represent the four-momentum$^2$ that has been interchanged in the interaction, the momentum that goes from $p_1$ into $p_2$ through the doted lines! (Also the same for any reference frame).