What does it mean for a phase to be unstable due to quantum fluctuations? Generally in the literature on quantum critical phenomena (as opposed to ordinary critical phenomena in statistical mechanics), there is the idea that quantum fluctuations can prevent ordering of a phase. My very basic question is: formally speaking, what does it mean for a phase to be unstable (or melted) due to quantum fluctuations? I want to emphasize that I am looking for a formal definition that isn't tied to a specific model Hamiltonian.
I have seen this idea discussed in the case of the Heisenberg antiferromagnet in 1D, where the classical antiferromagnet state (all spins alternating spin up and down) is unstable towards the creation of domains. One way I hear people describe this is that the "quantum spin fluctuations melte the classical antiferromagnet phase", but that seems odd to me, because I can't really put my finger on how this idea generalizes. Is it the fact that spin is a non-commuting operator that is important here? Otherwise what makes this destruction by "quantum fluctuations" as opposed to a phase destroyed by classical fluctuations? After all, you could say similar things about classical spins not ordering in the 1D Ising model due to spin fluctuations, right?
For example, one may look at the following links where the author(s) all use language suggesting that "quantum fluctuations" prevent ordering of a system in various ways:
Example 1: Antiferromagnetism
Example 2: Quantum Paraelectricity
Example 3: Quantum Pendulum
 A: I will give a partial answer. Assume we have a system with Lagrangian density of
$\mathcal{L}(\phi,\partial_\mu\phi)$
EOM is (this is just Euler-Lagrange equations, there is no quantum mechanics so far)
$$\frac{\delta}{\delta\phi}\mathcal{L}\bigg|_{\phi=\phi_{cl}}=0$$
Where $\phi_{cl}$ is classical solution. Now assume you want to calculate VEV of $\phi$
$$\langle\phi\rangle=Z^{-1}\int\mathcal{D}\phi \phi\exp\bigg(i\int d^dx\mathcal{L}(\phi,\partial_\mu\phi)\bigg)$$
now we want to see the influence of quantum fluctuations to classical solution. We can make the integral substitution
$$\phi\to\bar{\phi}+\phi_{cl}$$
so we have
$$\langle\phi\rangle=Z^{-1}\int\mathcal{D}\bar{\phi} (\phi_{cl}+\bar{\phi})\exp\bigg(i\int d^dx\mathcal{L}(\phi_{cl}+\bar{\phi},\partial_\mu(\phi_{cl}+\bar{\phi}))\bigg)$$
now we can write this as
$$\langle\phi\rangle=\phi_{cl}+ Z^{-1}\int\mathcal{D}\bar{\phi} (\bar{\phi})\exp\bigg(i\int d^dx\mathcal{L}(\phi_{cl}+\bar{\phi},\partial_\mu(\phi_{cl}+\bar{\phi}))\bigg)$$
Ok now the first term is just the solution of the classical equation of motion so it is the classical solution, the second term is a functional average of all possible field configurations so it is a quantum mechanical object which includes the quantum fluctuations.
In other words the first term is the classical solution, and the second is quantum fluctuations.
So this is the general principle behind how quantum fluctuations change the classical solutions. You can play the same game while calculating any kind of average.
And in general including quantum fluctuations on top of classical solution just as I did, may break order. For example in $1+1$ dimensional $XY$ model, quantum fluctuations destroy long range order and reduce it to algebraic order. You can find this calculation on page 93 of Wen's QFT book. He exactly does what I did here, by first writing full theory and expanding it around the classical solution he shows that even if first terms have long range order by adding higher terms this reduces to algebraic order!
A: Yes (in addition to the other answers), the point is that if multiple terms in the Hamiltonian don't commute with one another, the ground state of the total Hamiltonian is no longer the same as the ground state of each individual term; and therefore can no longer qualify as a 'phase' in the traditional sense.
To speak of a 'phase', you (generally) want a many-body state that

*

*Is robust to small decoherence, i.e. you can look at a macroscopic 'order parameter' that makes classical sense.

*Does not manifestly change in time

The clearest example is perhaps crystallization of atoms. The Hamiltonian consists of a Lennart-Jones-like potential $\sum_{ij}V(\hat{r}_i-\hat{r}_j)$ , and a kinetic part $\sum_i \frac{\hat{p}^2}{2m}$. If this were a classical system, we could just treat $r_i$ and $p_i$ as classical numbers and minimize these terms separately. The first one results in the formation of a periodic lattice, the second one confirms that they all stand still, and there you have the ground state.
However, you cannot do this, because $\hat{r_i}$ and $\hat{p}_i$ are in fact conjugate operators, and according to quantum mechanics, they don't share a ground state. Now of course, the total Hamiltonian will still have a ground state, but it is not so clearly interpretable. The result is that atoms 'keep wiggling' according to QM, even at zero temperature (where everything is in the ground state). For some elements, such as Helium, these 'fluctuations' even prevent the proper formation of a solid phase in a natural setting.
A more general way to phrase this is that the quantum harmonic oscillator, unlike the classical one, must have an energy of at least 1/2. The question is then for each system, is this zero-point energy small enough to maintain macroscopic order?
For another perspective, the introductionary chapter of Quantum Noise may be relevant.
Note: as LorenzMayer has reminded me in the comments, also in a classical system can it be that the ground state doesn't minimize all Hamiltonian terms separately (frustrated systems). However, the point here is rather that for a given system, quantumness makes things worse than for the classical limit and as such can let systems reach a tipping point where order is destroyed.
A: The mandatory reference for such issues is Sachdev's Quantum Phase Transitions. There, consulting the 2nd edition, page 7, we read:
"We note that phase transitions in classical models are driven only by thermal fluctuations,
as classical systems usually freeze into a fluctuationless ground state at T = 0. In contrast,
quantum systems have fluctuations driven by the Heisenberg uncertainty principle even in
the ground state, and these can drive interesting phase transitions at T = 0."
Fluctuations are important in critical systems. One of the fundamental aspects of criticality is the null gap, that is, the first excited energy (or energies, as is most common) is accessible by a vanishingly small ammount of energy that usually scales as 1/L, where L is a characteristic length or area or volume, you get it.
At page 8 he gives a really nice example:
"The low-lying magnetic excitations of the insulator LiHoF4 consist of fluctuations of the Ho ions between two spin states that are aligned parallel and antiparallel to a particular crystalline axis. These states can be represented by a two-state “Ising” spin variable on each Ho ion. At T = 0, the magnetic dipolar interactions between the Ho ions cause all the Ising spins to align in the same orientation, and so the ground state is a ferromagnet.Bitko, Rosenbaum, and Aeppli [49] placed this material in a magnetic field transverse to the magnetic axis. Such a field induces quantum tunneling between the two states of each Ho ion, and a sufficiently strong tunneling rate can eventually destroy the long-range magnetic order. Such a quantum phase transition was indeed observed"
The reference is Phys. Rev. Lett. 77, 940.
So what about Goldstone Bosons? Well, remember the Landau idea of spontaneous symmetry breaking? We can reason (approximately) as Goldstone and Nambu themselves did (reversing the arrow of time, if you like, in the previous example of Sachdev):

*

*We have a continuous symmetry (the ground state is not unique)

*Act with the charge operator

*Either we kill the vacuum or not.

*If we don't, then a zero frequency mode lives

*Infinite wavelength, likely means long-range order.

*Here, dimension considerations are essential. Let us assume we are above the critical dimension.

*Long-range correlations are one of the ingredients to set the party on fire

*Let Heisenberg enter the game and just wait (=

