First of all, I should say that everything here is given in Peskin & Schroeder, but it is spread through several chapters. I've focused on the particular question and added some of my own perspectives on the logical progression of the arguments.
You can determine whether a phase transition will be first- versus second-order by studying the effective action, $\Gamma[\phi_{\mathrm{cl}}]$. Recall that this object is a functional of the fields $\phi_{\mathrm{cl}}(x)$ which satisfies
$$
\frac{\delta}{\delta \phi_{\mathrm{cl}}(x)} \Gamma[\phi_{\mathrm{cl}}] = 0 \tag{1}\label{eq1}
$$
in the absence of external sources. Here, the field $\phi_{\mathrm{cl}}$ is given by
$$
\phi_{\mathrm{cl}}(x) = \langle \phi(x) \rangle,
$$
where the expectation value is taken with respect to the exact ground state of the QFT. Therefore, assuming one knows the functional $\Gamma[\phi_{\mathrm{cl}}]$ exactly, one can determine the vacuum expectation value of your fields by solving Eq. (\ref{eq1}). One is usually interested in using $\phi_{\mathrm{cl}}$ as a so-called "order parameter," where the solution $\phi_{\mathrm{cl}} = 0$ is the "disordered phase" while $\phi_{\mathrm{cl}} \neq 0$ is the "ordered phase." The phase transitions between the two phases is called first order or discontinuous if the value of $\phi_{\mathrm{cl}}$ changes discontinuously from zero at the phase transition, while it is called continuous if it changes continuously from zero to its non-zero values.
Of course, it is totally impossible to calculate $\Gamma[\phi_{\mathrm{cl}}]$ exactly in a generic interacting QFT, but we can use perturbation theory to obtain it. Let's consider (Euclidean) O($N$)-symmetric $\phi^4$ theory in four dimensions (the properties of the phase transition are not altered by Wick rotation):
$$
\int d^4 x \left[ \frac{1}{2} \left( \partial_{\mu} \phi_{\alpha} \right)^2 + \frac{s_0}{2} \phi^2 + \frac{\lambda_0}{4} \phi^4 \right].
$$
Here, we will take $\lambda_0$ to be small so we can use perturbation theory, and $s$ is a tuning parameter. The index $\alpha = 1,2,..,N$, and $\phi^2 = \sum_i \phi_{\alpha} \phi_{\alpha}$. The effective action is calculated to leading order in Peskin and Schroeder, and in terms of renormalized parameters it is
$$
\frac{1}{\mathrm{volume}} \Gamma[\phi_{\mathrm{cl}}] = \frac{s}{2} \phi_{\mathrm{cl}}^2 + \frac{\lambda}{4} \phi_{\mathrm{cl}}^4 + \frac{1}{64 \pi^2} \left\{ (N-1) (\lambda \phi_{\mathrm{cl}}^2 + s)^2\left( \log\left[ (\lambda \phi_{\mathrm{cl}}^2 + s)/M^2 \right] - \frac{3}{2} \right) + (3 \lambda \phi_{\mathrm{cl}}^2 + s)^2\left( \log\left[ (3 \lambda \phi_{\mathrm{cl}}^2 + s)/M^2 \right] - \frac{3}{2} \right) \right\}.
$$
Here, $M$ is an arbitrary mass scale introduced to renormalize the theory.
Without interactions, there is clearly a continuous transition at $s=0$, where $\phi_{\mathrm{cl}} = -\sqrt{-s/\lambda}$ for $s<0$ which smoothly goes to $\phi_{\mathrm{cl}} = 0$ for $s>0$. But now including our extra term, if we look at the $s \rightarrow 0$ limit, we find
$$
\frac{1}{\mathrm{volume}} \Gamma[\phi_{\mathrm{cl}}] \approx \lambda\frac{\phi_{\mathrm{cl}}}{4} \left\{ 1 + \frac{\lambda}{16 \pi^2} \left[ (N+8) \left(\log(\lambda \phi_{\mathrm{cl}}^2/M^2) - \frac{3}{2}\right) + 9 \log 3\right] \right\}. \tag{2}\label{eq2}
$$
Here, we notice a major problem - one which generally haunts massless QFTs. Although we have obtained the second term in the brackets as a perturbation of the first, we see that it is proportional to $\lambda \log \phi_{\mathrm{cl}}/M$. Then no matter how small $\lambda$ is, there is still always a small field value of $\phi_{\mathrm{cl}} \sim M e^{-1/\lambda}$ where the second term will be comparable to the first, invalidating perturbation theory. This just gets worse at higher orders, where we expect terms of the form $\lambda^n \log^n \phi_{\mathrm{cl}}/M$. Since we are precisely interested in the small $\phi_{\mathrm{cl}}$ limit, our expression cannot be trusted to determine the phase transition.
But this seems kind of stupid, since the mass scale $M$ was completely arbitrary from the start. If we began with one arbitrary $M$ and find perturbation theory breaks down at $\phi_{\mathrm{cl}} \sim M e^{-1/\lambda}$, what's to stop us from now defining a much smaller scale $M'$ where now perturbation theory works all the way down to $\phi_{\mathrm{cl}} \sim M' e^{-1/\lambda}$? And indeed, interating this process, it seems clear that we should be able to understand our problem all the way down to $\phi_{\mathrm{cl}} = 0$, provided perturbation theory remains valid.
The way to systematically deal with this is using the renormalization group (RG). The idea is to take into account how your observables depend on $M$. This answer isn't the best place to give a thorough introduction to RG, so I will simply use some major results. The effective action turns out to satisfy the following "Callan-Symanzik equation":
$$
\left[ M \frac{\partial}{\partial M} + \left( M \frac{d \lambda}{dM} \right) \frac{\partial}{\partial \lambda} \right] \Gamma[\phi_{\mathrm{cl}}] = 0
$$
(I should mention that this is especially simple because $\phi^4$ theory does not have wave function renormalization at one-loop). The general solution to this differential equation is
$$
\Gamma[\phi_{\mathrm{cl}}] = F(\bar{\lambda}(M,\phi_{\mathrm{cl}})) \phi_{\mathrm{cl}}^4,
$$
where we now have a "running" coupling constant
$$
\bar{\lambda} = \frac{\lambda}{1 - (\lambda/8 \pi^2) (N+8) \log(\phi_{\mathrm{cl}}/M)}.
$$
Comparing this general functional form with Eq. \ref{eq2}, we see that we can write
$$
\frac{1}{\mathrm{volume}} \Gamma[\phi_{\mathrm{cl}}] \approx \bar{\lambda}\frac{\phi_{\mathrm{cl}}}{4} \left\{ 1 + \frac{\bar{\lambda}}{16 \pi^2} \left[ (N+8) \left(\log(\bar{\lambda} \phi_{\mathrm{cl}}^2/M^2) - \frac{3}{2}\right) + 9 \log 3\right] \right\}.
$$
Using the definition of $\bar{\lambda}$ and expanding to leading order in $\lambda$ gives Eq. \ref{eq2}, but now we have used our knowledge of how the theory continuously changes with the scale $M$ to improve perturbation theory.
If we now consider the $\phi_{\mathrm{cl}} \rightarrow 0$ limit, we see that $\bar{\lambda}$ smoothly goes to zero, and the analysis proceeds without issue. The minimum remains at $\phi_{\mathrm{cl}}$ at the transition, and therefore the phase transition is continuous.
Of course, our result is particular to our model. In the four-dimensional abelian Higgs model (scalar QED), where a single scalar field is coupled to a gauge field, it turns out that the system undergoes a first-order phase transition (this is worked out in detail in the "Final Project" in P&S after Chapter 13). Finally, I have not discussed theories in less than four dimensions. This is because massless super-renormalizable theories are plagued by incurable IR divergences in perturbation theory, because these phase transitions are truly strongly coupled and non-perturbative. They are usually studied either by non-perturbative methods, or by a formal expansion in dimensionality close to $d=4$.
