Why does causality lead to different contours when calculating propagators? On the Wikipedia page for propagators they mention three types of Green's functions for the Klein-Gordon equation:

*

*The retarded propagator (taken when $x^0 > y^0$).

*The advanced propagator (taken when $x^0 < y^0$).

*The Feynman propagator.

It is not clear to me why $x^0 > y^0$ suggests we take the contour in the upper half plane, why $x^0 < y^0$ implies the contour is to be taken in the lower half plane, and what is the motivation behind contour in the Feynman propagator and how it is able to handle both cases ($x^0 > y^0$ or $x^0 < y^0$ or $x^0 = y^0$).
Why is this the case?
 A: The integral $$G(x,y)=\int \dfrac{d^4p}{(2\pi)^4}\dfrac{e^{-ip(x-y)}}{p^2+m^2}\tag{1}$$
is undefined because of the fact that the denominator $p^2+m^2$ vanishes when $p$ goes on shell. This corresponds to a whole three-dimensional hyperboloid in the integration domain. This is a special feature of the Lorentzian signature of Minkowski spacetime. Indeed, in Euclidean signature, $p^2+m^2$ can never be zero because $p^2\geq 0$ and then (1) uniquely defines the inverse of the scalar Laplace operator. In Lorentzian signature, we must pick an inverse.
To pick an inverse we introduce a prescription that makes (1) well-defined. Each of the prescriptions defines a different function, with different properties.
First of all, let us write (1) a bit differently by using the definition of $p^2=-(p^0)^2+|\vec{p}|^2$. In that case
$$G(x,y)=\int \dfrac{dp^0d^3\vec{p}}{(2\pi)^4}\dfrac{e^{-ip(x-y)}}{-(p^0)^2+|\vec{p}|^2+m^2}\tag{2}.$$
Now let us imagine that we take the integral over $p^0$ first with $\vec{p}$ held fixed. We can see that the singularity is hit when $(p^0)^2=m^2+|\vec{p}|^2$. This happens at $$p^0 = \pm \sqrt{m^2+|\vec{p}|^2}\tag{3}$$
Since the integral over $p^0$ is over the whole real line, and the two solutions in (3) are both sitting on the line, the integration goes through them as expected. You can circumvent this by defining $G(x,y)$ as
$$\int \dfrac{dp^0d^3\vec{p}}{(2\pi)^4}\dfrac{e^{-ip(x-y)}}{-(p^0\pm i\epsilon)^2+|\vec{p}|^2+m^2},\quad \text{or},\quad \int \dfrac{dp^0d^3\vec{p}}{(2\pi)^4}\dfrac{e^{-ip(x-y)}}{-(p^0)^2+|\vec{p}|^2+m^2\pm i\epsilon} \tag{4}.$$
In the first option you get the singularities at $$p^0=\pm i \epsilon + \sqrt{m^2+|\vec{p}|^2},\quad p^0 = \pm i\epsilon -\sqrt{m^2+|\vec{p}|^2}.\tag{5}$$
The singularities are thus either pushed upward or downwards in the imaginary direction. So suppose we take the sign $+$. The singularities are then pushed upwards. You may close the contour up or down. If you close it upwards you enclose the singularities and get a contribution from the associated residues. If you close it downwards you don't get it. In one case you get a non-zero result, in the other you get zero. So what determines the contour choice?
Well, it is essentially the exponential. Observe you have $e^{-ip(x-y)}$ on the integrand. This is $e^{ip^0(x^0-y^0)}e^{-i\vec{p}\cdot (\vec{x}-\vec{y})}$. Now since we are doing contour integration, we will have both real and imaginary parts of $p^0$ along the contour, so these exponentials are $$e^{i{\rm Re}(p^0)(x^0-y^0)-{\rm Im}(p^0)(x^0-y^0)}e^{-i\vec{p}\cdot (\vec{x}-\vec{y})}\tag{6}.$$
This is the key to the choice of contour. If you close the contour upwards, along the big semicircle you will have ${\rm Im}(p^0)\to \infty$. So if $(x^0-y^0)<0$ you will have one exponential $e^{-{\rm Im}(p^0)(x^0-y^0)}$ which badly diverges, while if $(x^0-y^0)>0$ you have one exponential $e^{-{\rm Im}(p^0)(x^0-y^0)}$ which goes to zero and eliminates the big semicircle contribution. So you must close the contour upwards when $x^0-y^0>0$. By the same argument, you must close the contour downwards when $x^0-y^0<0$.
Now, recall that when you pick the plus sign in (5), the poles are shifted upwards. So you only get a non-zero contribution when the contour is closed upwards, which is the case for $x^0-y^0>0$. So for the plus sign in (5) you get something non-zero for $x^0-y^0>0$ and zero for $x^0-y^0<0$. This is the retarded propagator.
When you pick the minus sign in (5) the opposite happens. The poles are shifted downwards and you only get a non-zero contribution when the contour is closed downwards, which is the case for $x^0-y^0<0$. So for the minus sign, you get a non-vanishing result for $x^0-y^0<0$ and zero for $x^0-y^0>0$. This is the advanced propagator.
Finally, consider the second option in (4). Now the singularities are at $$p^0 
=  \sqrt{m^2+|\vec{p}|^2\pm i\epsilon},\quad p^0 = - \sqrt{m^2+|\vec{p}|^2\pm i\epsilon}\tag{7}.$$
Since $\epsilon$ will be taken to zero, you can expand the square roots and the singularities lie at $$p^0 
=  \sqrt{m^2+|\vec{p}|^2}\pm \frac{i\epsilon}{2\sqrt{m^2+|\vec{p}|^2}},\quad p^0 = - \sqrt{m^2+|\vec{p}|^2}\mp \frac{i\epsilon}{2\sqrt{m^2+|\vec{p}|^2}}\tag{8}.$$
Regardless of the choice of sign in the $i\epsilon$ prescription, the important point is that now one pole will be shifted upwards and the other downwards. The contour analysis now shows that in both cases $x^0-y^0>0$ where the contour must be closed upwards and $x^0-y^0<0$ where the contour must be closed downwards you will get contributions. The choice of the sign will determine the time-ordering of the resulting correlator.
So, in summary, (1) by itself is ill-defined because the wave operator in Lorentzian signature does not have a unique inverse. To make (1) well-defined you can choose several possible $i\epsilon$ prescriptions as in (4), which are essentially boundary conditions on causal behavior of the inverse. The choice of prescription then picks an inverse of the wave operator. For each such choice, you may then evaluate the integral over $p^0$ using contours and you discover that that choice of prescription is tied to some causal behavior of $G(x,y)$ because of its dependence on $x^0-y^0$. This is why causality leads to different contours when calculating propagators.
A: There is already a correct answer from user Gold. The short answer is that the causal structure of spacetime gets encoded in the Fourier transformed momentum space via an $i\epsilon$ prescription. The Fourier transformation is often done via the residue theorem by closing the integration contour in the upper or lower half-plane (choosing whichever makes the integrand exponentially suppressed), cf. OP's title question. Below we list the explicit formulas.

*

*If we consider Greens functions $\Delta(x)$ for the Klein-Gordon equation$^1$
$$-(\Box-m^2)\Delta(x)~=~\delta^d(x),\tag{1a}$$
the Feynman, the retarded, and the advanced Greens functions are$^2$
$$\begin{align} \frac{1}{i}\Delta_F(x)
~=~&\theta_{\eta}(x^0)\Delta_+(x) + \theta_{\eta}(-x^0)\Delta_-(x), \cr
\frac{1}{i}\Delta_{R/A}(x)~=~&\pm \theta_{\eta}(\pm x^0)\left(\Delta_+(x) -\Delta_-(x)\right), 
 \end{align}\tag{1b}$$
respectively; where
$$\Delta_{\pm}(x)\tag{1c}$$ are the Wightman functions [1].


*The spatially Fourier transformed Greens functions
$$\begin{align} \hat{\Delta}(\vec{k},t)
~=~&\int_{\mathbb{R}^d}\! d^{d-1}\vec{x}~e^{-i\vec{k}\cdot \vec{x}}\Delta(x)\cr
~=~&\int_{\mathbb{R}}\! \frac{dk^0}{2\pi}~e^{-ik^0t}\widetilde{\Delta}(k),\cr
(\partial_t^2+\omega_{\vec{k}}^2) \hat{\Delta}(\vec{k},t)~=~&\delta(t),\cr
\omega_{\vec{k}}~:=~\sqrt{\vec{k}^2+m^2},
\end{align}
\tag{2a}$$
are
$$\begin{align} \frac{1}{i}\hat{\Delta}_F(\vec{k},t)
~=~& \theta_{\eta}(t)\hat{\Delta}_+(\vec{k},t) 
+ \theta_{\eta}(-t)\hat{\Delta}_-(\vec{k},t)\cr
~=~&\frac{1}{2\omega_{\vec{k}}}\left(\cos(\omega_{\vec{k}}t)+ \frac{1}{i}{\rm sgn}_{\eta}(t)\sin(\omega_{\vec{k}}t)\right)e^{-\epsilon|t|},\cr
\frac{1}{i}\hat{\Delta}_{R/A}(\vec{k},t)
~=~&\pm \theta_{\eta}(\pm t)\left(\hat{\Delta}_+(\vec{k},t) -\hat{\Delta}_-(\vec{k},t)\right)\cr
~=~&\pm \theta_{\eta}(\pm t)\frac{1}{2i\omega_{\vec{k}}}\sin(\omega_{\vec{k}}t)e^{-\epsilon|t|},
 \end{align}\tag{2b}$$
respectively; where the Fourier transformed Wightman functions are
$$\hat{\Delta}_{\pm}(\vec{k},t) 
~=~\frac{1}{2\omega_{\vec{k}}}e^{\mp i\omega_{\vec{k}}t-\epsilon|t|}.\tag{2c}$$


*The fully Fourier transformed Greens functions
$$\begin{align} \widetilde{\Delta}(k)
~=~&\int_{\mathbb{R}^d}\! d^dx~e^{-ik\cdot x}\Delta(x)\cr
~=~&\int_{\mathbb{R}}\! dt~e^{ik^0 t}\hat{\Delta}(\vec{k},t) ,\cr
(k^2+m^2)\widetilde{\Delta}(k)~=~&1,\end{align}
\tag{3a}$$
are
$$\begin{align} \widetilde{\Delta}_F(k) 
~=~&\frac{1}{2\omega_{\vec{k}}} \left( \frac{1}{k^0-i\epsilon + \omega_{\vec{k}}} -\frac{1}{k^0+i\epsilon -\omega_{\vec{k}}}\right)\cr
~=~&\frac{1}{2\omega_{\vec{k}}} \left( \frac{1}{k^0(1+i\epsilon) + \omega_{\vec{k}}} -\frac{1}{k^0(1+i\epsilon) -\omega_{\vec{k}}}\right)\cr
~=~& \frac{1}{-(k^0(1+i\epsilon))^2+ \omega_{\vec{k}}^2}\cr
~=~& \frac{1}{-(k^0)^2+ \omega_{\vec{k}}^2-i\epsilon}\cr
~=~&\frac{1}{k^2+m^2-i\epsilon} \cr
~=~& \widetilde{\Delta}_E(k^0(\epsilon-i),\vec{k}),\cr
\widetilde{\Delta}_{R/A}(k)
~=~&\frac{1}{2\omega_{\vec{k}}} \left( \frac{1}{k^0\pm i\epsilon+ \omega_{\vec{k}}} -\frac{1}{k^0\pm i\epsilon -\omega_{\vec{k}}}\right)\cr
~=~& \frac{1}{-(k^0\pm i\epsilon)^2+ \omega_{\vec{k}}^2}\cr
~=~&\frac{1}{k^2+m^2\mp i\epsilon{\rm sgn}(k^0)},
 \end{align}\tag{3b}$$
respectively; where the Fourier transformed Wightman functions are
$$\begin{align}\widetilde{\Delta}_{\pm}(k)
~=~&\frac{\pi}{\omega_{\vec{k}}} \delta(k^0\mp \omega_{\vec{k}}) \cr
~=~&\theta(\pm k^0)~2\pi \delta(k^2+m^2).\end{align}\tag{3c}$$
Here
$$ \widetilde{\Delta}_E(k_E)~=~\frac{1}{k_E^2+m^2} \tag{3d}$$
is the Fourier transformed Euclidean Greens function, cf. Wick rotation.
References:

*

*N.D. Birrell & P.C.W. Davies, Quantum fields in curved space, 1984; section 2.7.

--
$^1$ In this answer we use Minkowski signature $(-,+,\ldots,+)$ in $d$ spacetime dimensions and set the speed of light $c=1$ to unity.
$^2$ The Heaviside step function $\theta(t)$ is here regularized as a generalized function
$$\begin{align} \theta_{\eta}(t) 
~=~& \frac{1}{\pi}{\arg}(-t+i\eta)\cr
~=~&\frac{1}{\pi}{\rm arccot}\frac{-t}{\eta}\cr
~=~&\frac{1+{\rm sgn}_{\eta}(t)}{2}, \cr
{\rm sgn}_{\eta}(t)~=~&\frac{2}{\pi}\arctan\frac{t}{\eta},
\end{align}\tag{4a}$$
where $\eta>0$ is an infinitesimal regularization parameter. The regularization (4a) is needed to prove that the Greens functions (2b) satisfy the differential equation (2a).
