Understanding the Static Structure Factor of a Liquid I am trying to understand static structure factors, $S(Q)$, for liquids. This function represents pairwise correlations in reciprocal space and can be measured experimentally using X-ray or neutron diffraction.
My understanding is that coordination shells will be reflected as features in $S(Q)$ at $Q = 2 \pi / r$. The real-space representation of $S(Q)$ is given by
$$g(r)=\frac{1}{(2\pi)^{3}\rho}\int_{0}^{\infty}4\pi Q^{2}(S(Q)-1)\frac{\sin Qr}{Qr}dQ$$
But can these shells be reflected as valleys rather than peaks? I have a clear peak in $g(r)$ at an $r$ corresponding to a valley in $S(Q)$, which surprises me.
Also, what does it mean to have a peak at low $Q$ where $S(Q) < 1$? A corresponding peak in $g(r)$ must be greater than 1, but the feature in $S(Q)$ would suggest the opposite, i.e., that the liquid is anti-correlated with respect to an ideal gas, where $g(r)=1$ for all $r$.
 A: The key concept to understanding the relationship between the structure factor $S(Q)$ and the radial distribution function $g(r)$ in fluid systems is that the two functions contain information about the correlations of related but not coinciding quantities.
Indeed, while the radial correlation function $h(r) = g(r)-1$ can be defined  from the correlation between fluctuations of the instantaneous pair density in real space:
$$
h(|{\bf r}-{\bf r'}|)= \frac{1}{\rho^2}\left< \sum_{ij} \delta({\bf r}- {\bf r}_i)\delta({\bf r'}- {\bf r}_j) -\rho^2\right>,
$$
The structure factor corresponds to the fluctuations of the Fourier components of the same fluctuations:
$$
S({\bf Q})=\frac{1}{N} \left< \hat \rho_{{\bf Q}} \hat\rho_{{\bf -Q}}  \right>
$$
where
$$
\hat \rho_{\bf Q}=\sum_i e^{-i {\bf Q} \cdot {\bf r}_i}
$$
is the Fourier transform of the instantaneous one-particle density
$$
\rho({\bf r})= \sum_i \delta({\bf r}- {\bf r}_i).
$$
The Fourier transform relation between $h(r)$ and $S(Q)$ does not allow the establishment of a local correspondence between the two functions. Actually, a peak in $Q$-space, around a wavenumber $\bar Q$, is not directly related to a single peak in $r$-space. It rather implies the presence of a quasi-periodicity, a repeated length $~\frac{2\pi}{\bar Q}$. And similarly, the other way around. No one-to-one correspondence exists between single peaks and single valleys from $Q$ to $r$ space.
For example, it is easy to verify that the main peak of the structure factor of a liquid close to the melting point is not far from the periodicity of the corresponding crystal, which is different from the nearest neighbor distance for structures with more than one atom per cell. Not only can peaks and quasi-periodicities be related, but other features of the two functions can be related, always in a non-local way. For example, the slope of the rising part of the first peak of $g(r)$ corresponds with the decaying of the large $Q$ oscillations of $S(Q)$. The steeper is the rising part of $g(r)$, the slower oscillations in large $Q$ region of $S(Q)$.
A: 
$$g(r)=\frac{1}{(2\pi)^{3}\rho}\int_{0}^{\infty}4\pi Q^{2}(S(Q)-1)\frac{\sin Qr}{Qr}dQ$$


But can these shells be reflected as valleys rather than peaks? I have a clear peak in $g(r)$ at an $r$ corresponding to a valley in $S(Q)$, which surprises me.

Your claim that the peak in $g(r)$ "corresponds to" a valley in $S(Q)$ does not make sense, the distribution function $g(r)$ is calculated as an integral over all $Q$

Also, what does it mean to have a peak at low $Q$ where $S(Q) < 1$?

A peak in $S(Q)$ indicates that the scattered intensity is relatively significant at that momentum transfer value ($Q$).

Update (per OP's request)
As an example, starting from Wikipedia's Eq (10)
$$
S(Q) = 1 + \rho\int d^3r e^{-i\vec Q\cdot \vec r}g(r)\;,
$$
and inverting the Fourier transform, we have:
$$
g(r)= \frac{2}{\rho(2\pi)^2r}\int_0^{\infty}dQ Q\sin(Qr)\left(S(Q)-1\right)\;.
$$
As an example, let's assume that $(S(Q) - 1)$ is very strongly peaked at $Q_0$, with peak width $\Delta Q$. Then we have:
$$
g(r) \approx 
\frac{2}{\rho(2\pi)^2r}\int_{S_0-\Delta Q/2}^{S_0+\Delta Q/2}dQ Q\sin(Qr)\left(S(Q)-1\right)
\approx
\frac{2}{\rho(2\pi)^2r}\Delta Q Q_0\sin(Q_0 r)\left(S(Q_0)-1\right)
$$
$$
=C_0 \frac{\sin(Q_0 r)}{r}\;,
$$
where $C_0$ is a constant.
The function $C_0\frac{\sin(Q_0 r)}{r}$ approaches a constant value $C_0 Q_0$ at $r=0$ and approaches 0 as $r\to \infty$ and wiggles a bit in between (since the sine function wiggles).

As a second example, let's go the other way... Assume that the radial distribution function has a strong peak at a single radial value $R$ like:
$$
g(r) \approx \delta(r-R)\frac{1}{4\pi r^2}
$$
Then, we have
$$
S(Q) = 1 + 2\pi\rho\int dr r^2 \int_{-1}^{1} d\mu e^{-i Q r \mu}g(r)\;,
$$
$$
= 1 + 4\pi\rho\int dr r^2 \frac{\sin(Q r)}{Qr}g(r)\;,
$$
$$
\approx
1 + \rho \frac{\sin(Q R)}{QR}\;.
$$
So, we see that if $g(r)$ is strongly peaked in $r$ space then $S(Q)$ wiggles in a way where the period depends on the peak value of $g$, and conversely if $S(Q)$ is strongly peaked in $Q$ space then $g(r)$ wiggles in a way where the period depends on the peak value of $S$
