- The notion of the virtual particle originates from the quantum mechanical old-fashioned perturbation theory. In the latter, time is a dedicated quantity. Within this theory, the matrix element of the transition $|i\rangle\to |f\rangle$ behaves as
$$
\mathcal{M}_{i\to f} = \overbrace{V_{if}}^{\text{1st order}} +\overbrace{\sum_{n \neq i}\frac{V_{in}V_{nf}}{E_{n}-E_{i}}}^{\text{2nd order}} +\dots
$$
In the second term, which corresponds to the second-order perturbation theory, we have a sum over intermediate states $|n\rangle$ - physical states (particles with correct dispersion relation, etc), but with energy conservation violation. This is a standard QM story on the time-energy uncertainty relation in perturbation theory: intermediate states exist during times $\Delta t \sim \hbar/\Delta E$, where $\Delta E$ is the energy violation. These states are what you call the ``virtual particles''.
The graphical representation of the matrix element within the old-fashioned perturbation theory is shown below using $e-\mu$ scattering as an example.
We have here two diagrams, each corresponding to different intermediate states (the state between the vertical red lines). In the first case, the muon emits a photon which is later absorbed by the electron. In the second case, the situation is the opposite.
When we are talking about relativistic processes, it is possible to re-interpret the old-fashioned perturbation theory in explicitly Lorentz-covariant fashion, where time is no longer dedicate quantity. After some manipulations, the matrix elements can be given in the form
where the photon line now corresponds to some object which has nothing to do with the physical photon - the propagator $D_{\mu\nu}$. Unlike the field operator of a free photon, which satisfies the Maxwell equation
$$
\partial^{2}A_{\mu} = 0,
$$
the propagator satisfies completely different equation (here I neglect some technical moments)
$$
\partial^{2}_{x}D_{\mu\nu}(x-y) \sim g_{\mu\nu}\delta(x-y)
$$
Few examples of experimental manifestation.
Scattering of particles on energies close to the mass of intermediate states. The cross-section of the process $e^{+}e^{-} \to \tau^{+}\tau^{-}$ is peaked when the invariant mass $s$ of $e^{+}e^{-}$ pair is close to the mass of $Z$ boson. This is interpreted using virtual particles in the following manner. This process goes through the virtual $Z$ boson. Once $s$ approaches $m_{Z}^{2}$, the $Z$ boson becomes more and more real, and at $s = m_{Z}^{2}$ the process goes through the real $Z$ boson: $e^{+}e^{-}\to Z, \quad Z\to \tau^{+}\tau^{-}$.
Zommerfeld enhancement. When the oppositely charged particles collide, their scattering cross-section gets specific enhancement with the lowering of their relative momentum. The reason for this is the existence of the intermediate bounded state for these two particles, whose contribution to the cross-section increases once the relative momentum becomes comparable with the binding energy.
- So, to conclude: the virtual particle comes from the perturbation-based description of interactions in quantum mechanics. When dealing with relativistic processes, instead of the virtual particles it is more convenient to deal with the propagator, which is a kind of the "superposition" of the virtual particles. Some experimental phenomena may be explained using the notion of the virtual particles.