Phase change boundary near Lifshitz Point I am following the text of Chaikin and Lubensky.
They breifly discuss the results of the following equations but I find myself confused by some of the details:
$$
F = \frac12 \int \text{d} \vec{x} (r\phi^2 + 2u\phi^4 + c_\parallel (\nabla_\parallel\phi)^2 + c_\perp (\nabla_\perp\phi)^2 + D (\nabla^2\phi))^2.
$$
They state that $c_\parallel,D, u$ are all positive values and that $c_\perp$ can be positive or negative.
From here they state that the phase diagramss can be shown to be:

They state and I agree, that in the $c_\perp>0$ case the solution should be uniform and so we can ignore the gradient terms. This makes it very easy to find that the phase transition occurs when $r$ changes sign giving us the line to the left of the lifshitz point.
However, there is little detail for the $c_\perp <0$ case which I will now outline my thinking.
We will use the Euler-Poisson equation to find the minimization of the free-energy. We assume that the perpendicular direction is along $z$ and that the solution is still uniform in the parallel direction. I find that this minimization gives us:
$$
2r\phi + 8u\phi^3 - 2 c_\perp\partial_z^2\phi + 2D\partial_z^4\phi=0,
$$
I further assume that the $\phi$ must have the form of a modulation (based on the fact that the region is called the modulation region). If we assume that $\phi = a_0\cos(b_0 z)$ the above solution reduces and removes the pesky derivatives to give:
$$
2r\phi + 8u\phi^3 + 2b_0^2  c_\perp\phi + 2b_0^4 D\phi=0.
$$
From which I clearly see 2 solutions:
$$
\phi=0,
$$
and
$$
\phi = \pm\sqrt{-\frac{r+c_\perp b_0^2 + D b_0^4}{D}}.
$$
From this I can predict that there is phase change when the numerator in the square-root becomes a negative value as this is the first time the two solutions become real valued. So I would get that one of the lines occurs as:
$$
r = -c_\perp b_0^2 - D b_0^4.
$$
At this point I can not spot another possible phase change.  However, I am already a little dubious of the above answer as firstly it is only linearly with $c_\perp$ but in the textbook diagram it appears to be quadratic. Secondly, it does not connect to the the Lifshitz point unless $b_0=0$ which is useless.
Any advice would be greatly appreciated. I'm not sure what I'm doing wrong at this point.
 A: Here is a partial answer that seems to give the right phase boundary in the $r>0, c_\perp<0$ part of the phase diagram.
Following the argument in the post, I will consider the following free energy
$$ F = \frac12\int dz\, [r\phi^2+2u\phi^4+c_\perp(\partial_z\phi)^2+D(\partial_z^2\phi)^2] $$
Assuming $\phi=a_0\cos b_0z$ and plug the ansatz into the expression for $F$, the free energy density is given by
$$ f= \frac{1}{4}[ (r+c_\perp b_0^2+Db_0^4)a_0^2+3ua_0^4]$$
So now the task is to minimize $f$ with respect to both $b_0$ and $a_0$. We simply solve $\frac{\partial f}{\partial b_0}=0, \frac{\partial f}{\partial a_0}=0$:
$$ b_0(c_\perp + 2Db_0^2)=0$$
$$ a_0[R(b_0)+6ua_0^2]=0$$
If $c_\perp>0$, then the only solution for $b_0$ is $b_0=0$, and we are back to the standard Landau case.
If $c_\perp<0$, but $r>0$, the $b_0=0$ solution leads to a paramagnetic state with $a_0=0$. The $b_0^2=-\frac{c_\perp}{2D}$ solution gives $a_0^2=\frac{1}{6u}(c_\perp^2/4D-r)$. If $r>\frac{c_\perp^2}{4D}$, again the only minimum is the paramagnetic one. If $0<r<c_\perp^2/4D$, then the minimum is the modulated solution. So we obtain the quadratic portion of the phase boundary in the $r>0$ half (i.e. $r=c_\perp^2/4D$).
Next we consider $c_\perp<0, r<0$. The $b_0=0$ solution leads to $a_0^2=\frac{-r}{6u}$, and $b_0^2=-\frac{c_\perp}{2D}$ again gives $a_0^2=\frac{1}{6u}(\frac{c_\perp^2}{4D}-r)$. The latter has lower free energy, regardless of the values of $c_\perp$ or $r$ as long as both are negative. As $c_\perp$ goes to 0, $b_0$ also vanishes, continuously connected to the uniform case. So still does not agree with the lower portion of the phase diagram...
A: I recently encountered the same problem in the same book as your question. Although an answer has already been accepted, I think I can solve some of the issues in Meng Cheng's solution.
I will follow the same procedure of evaluating the free energy using the ansatz $\phi(\vec{x})=a \cos(bz) $ for a scalar field in (2+1)-dimensions, where $x,y$ correspond to the parallel ($\parallel$) components and $z$ corresponds to the perpendicular ($\perp$) components.
Then,
$$
F=\frac{1}{2}\int d^3x \left[ ra^2 \cos^2(bz) +2ua^4 \cos^4(bz)+ c_{\perp}(-ab\sin(bz))^2 +D(-ab^2\cos(bz))^2\right]
$$
Integrals of the form $\int d^3x \cos^2(bz)$ don't have a well-defined value, so instead of studying the total free energy, we switch to the free energy density $f=F/V$. Thus we replace $\int d^3x \cos^2(bz)$ by its average value over space.
In other words, we use
$$
\frac{\int d^3x \cos^2(bz)}{V} \implies \lim_{L\to \infty}\frac{\int_{-L}^{L} dz \cos^2(bz)}{2L} = \frac{1}{2}
$$
Similarly
$$
\frac{\int d^3x \sin^2(bz)}{V} \implies \lim_{L\to \infty}\frac{\int_{-L}^{L} dz \sin^2(bz)}{2L} = \frac{1}{2}
$$
$$
\frac{\int d^3x \cos^4(bz)}{V} \implies \lim_{L\to \infty}\frac{\int_{-L}^{L} dz \cos^4(bz)}{2L} = \frac{3}{8}
$$
There is, however, a hidden assumption here and it is at the crux of Meng Cheng's problems. These results are only valid if $b\neq 0$. If $b$ equals zero, then the averages must be 1!!! (or zero) This follows from:
$$
\frac{\int d^3x \cos^2(bz)}{V} \implies_{b \to 0} \frac{\int d^3x 1}{V} = 1
$$
Thus, we have two free energy densities:
$$
f(b \neq 0) = \frac{1}{2} \left[ \frac{1}{2} r a^2 + \frac{3}{4} u a^4 + \frac{1}{2}c_{\perp}a^2b^2 + \frac{1}{2}Da^2b^4\right] = \frac{1}{4} \left[ (r + c_{\perp}b^2 + Db^4)a^2 + \frac{3}{2} u a^4 \right] 
$$
$$
f(b=0)= \frac{1}{2} \left[ r a^2 + 2 u a^4\right]
$$
The next steps are just finding the (a,b) parameters that minimize the densities.
Case 1: $b \neq 0$
We find the extrema:
$$
\frac{\partial f}{\partial a}=0 \iff a(r+4ua^2) = 0 \iff a=0 \quad \lor \quad a^2 = -\frac{r}{4u}
$$
We study their stability by the second derivative criteria. I will skip those steps (a simple calculus exercise). The result is that the solution $a=0$ (Paramagentic) is only stable when $r>0$. The solution $a\neq 0$ (Ferromagnetic) is a local minimum when $r<0$ otherwise, the solution does not exist because of the condition $a^2 = -\frac{r}{4u} \geq 0$.
I illustrate those results in the following figures:


Case 2: $b \neq 0$
Here we have to maximize both $a$ and $b$. We can find the extrema as follows:
$$
\frac{\partial f}{\partial b} = 0 \iff (c_\perp+2Db^2)ba = 0 \iff a=0 \quad \lor \quad b^2 = - \frac{c_\perp}{2D}
$$
Notice that here we are using the fact that this free energy is valid only when $b \neq 0$
$$
\frac{\partial f}{\partial a} = 0 \iff a(r + c_{\perp}b^2 + Db^4 + 3 u a^2) = 0 \iff a=0 \quad \lor \quad a^2 = -\frac{r + c_{\perp}b^2 + Db^4}{3u} 
$$
The solution $a=0$ corresponds to the paramagnetic solution studied before. Thus, we are left with the equations:
$$
b^2 = - \frac{c_\perp}{2D}
$$
$$
a^2 = -\frac{r + c_{\perp}b^2 + Db^4}{3u}
$$
By noticing that $b^2 \geq 0$ we conclude that the modulated solution can only exist when $-c_\perp \geq 0$. With that requirement:
$$
a^2 = -\frac{r + c_{\perp}(- \frac{c_\perp}{2D}) + D(- \frac{c_\perp}{2D})^2}{3u} = - \frac{r-\frac{c_\perp^2}{8D}}{3u}
$$
By noticing that $a^2 \geq 0$ we conclude that $r \leq \frac{c^2_\perp}{8D}$. This explains the upper parabola just as in Meng Cheng's answer.
I illustrate this region in the following figure:

Now, we could study the second derivative to check the limit of metastability of the modulated phase, but I will skip that to go straight into the phase transition boundary.
First order transition:
As you can see from the images, there is a region in which both the ferromagnetic and modulated phases can exist. The region is given by $r<0$ and $-c_\perp > 0$. To determine which phase will exist in equillibrium, we need to determine which of the two phases has lower free energy.
By substituting the values of $a$ and $b$ into their respective free energy densities we get:
$$
f_{\text{Ferro}}= \frac{1}{2} \left[ r \left(-\frac{r}{4u}\right) + 2 u \left(-\frac{r}{4u}\right)^2\right] = - \frac{r^2}{16u}
$$
$$
f_{\text{Mod}} = \frac{1}{4} \left[ \left(r + c_{\perp}\left(- \frac{c_\perp}{2D}\right) + D\left(- \frac{c_\perp}{2D}\right)^2\right) \left(- \frac{r-\frac{c_\perp^2}{8D}}{3u}\right) + \frac{3}{2} u \left(- \frac{r-\frac{c_\perp^2}{8D}}{3u}\right)^2 \right] = -\frac{\left( r- \frac{c_\perp^2}{8D}\right)^2}{24 u}
$$
The ferromagnetic phase will be more stable whenever:
$$
- \frac{r^2}{16u} < -\frac{\left( r- \frac{c_\perp^2}{8D}\right)^2}{24 u}
$$
$$
\iff r^2 > \frac{2}{3} \left(r- \frac{c_\perp^2}{8D}\right)^2 \iff r^2 + 2r \left(\frac{c_\perp^2}{4D}\right) - \frac{1}{2}\left(\frac{c_\perp^2}{4D}\right)^2
$$
By doing some algegraic manipulation keeping in mind that $c_\perp <0$ and $r<0$, it is possible to reduce that inequality to the condition:
$$
-r > \left(\sqrt{\frac{3}{2}}-1\right) \left(\frac{c_\perp^2}{4D}\right)
$$
which yields the lower parabola as shown in the following image:

We can note that in general, both parabolas have different curvatures, just like the one shown by Chaikin and Lubensky, and the values for $a^2$ change discontinuously. Thus a first-order transition.
Final remarks:
The solution and the phase diagram given by Meng Cheng correctly describe the phase diagram for a vector order parameter. I encourage you to try and corroborate that yourself!
Hope my answer adds some clarity :))
