First: you have an incorrect sign in your $\delta_\lambda$: see, for example, David Skinner's AQFT notes. In your notation, you should have that the quartic counterterm is positive
$$
``\delta_\lambda = +\frac{3}{16\pi^2} \frac{1}{\epsilon} \lambda^2 + O(\epsilon^0)"
$$
However, there are deeper problems here. The obvious warning sign is that this expression is manifestly dimensionally incorrect, since from the Lagrangian $\lambda$ and $\delta_\lambda$ have the same dimension. Likewise with your expression $\lambda = \lambda_0 + \frac{3}{16\pi^2 \epsilon} \lambda^2$.
Even though $\epsilon$ is usually taken to be small at the end, you should always make sure that your expressions are dimensionally correct for arbitrary $\epsilon$!
Your confusion lies in not keeping track of where the $\mu$s are. Let's walk through the calculation as it should be done in a Euclidean field theory.
To this end, we switch to always using the dimensionless $g= \mu^{-\epsilon} \lambda$, so that the Feynman rules tell you to put a $g\mu^\epsilon$ or $\delta_g \mu^\epsilon$ at each vertex. Calculating the momentum-space four-point function,
$$
\Gamma^{(4)}(k_1, k_2, k_3, k_4) = -g \mu^\epsilon - \delta_g \mu^\epsilon + (-g\mu^\epsilon)^2 \int \frac{d^d p}{(2\pi)^d} \frac{1}{p^2 + m^2} \frac{1}{(p+ k_1 +k_2)^2 + m^2} + \text{t, u channels} + O(g^3)
$$
Taking $g$ to be finite, we focus on the (momentum independent) divergent part from the three channels:
$$
= \mu^\epsilon\left[-g - \delta_g + 3 \times \frac{g^2}{32\pi^2} \left(\frac{2}{\epsilon} + \log(\mu^2/m^2) + \text{more finite parts} + \text{ divergences at }O(g^3) \right)\right]
$$
where the quantity in square brackets is clearly dimensionless. Let us take as renormalisation condition $\lim_{d\to 4}\Gamma^{(4)}(p_1, p_2,p_3,p_4) = 0.3 \neq \infty$, where the $p_i$s are fixed momenta. This sets the value of $\delta_g$, including finite parts. However, in the relatively unphysical MS scheme, we only bother with eliminating the divergences in $\lim_{d\to 4}\Gamma^{(4)}$. We do that by setting:
$$
\delta_g = \frac{3}{16\pi^2} \frac{1}{\epsilon}g^2 \equiv \frac{z}{\epsilon}g^2, \quad z\equiv \frac{3}{16\pi^2}
$$
Now, let's recall that the bare vertex, $\lambda_0$, which does not depend on $\mu$, is
$$
(g + \delta_g)\mu^\epsilon \phi^4 = \lambda_0 \phi_0^4 \implies \lambda_0 = \frac{(g + \delta_g)}{Z^2}\mu^\epsilon = g(1+ \frac{z g}{\epsilon}) \mu^\epsilon + O(g^3)
$$
where we have used that $Z= 1+O(g^2)$. We know that the bare parameter does not depend on $\mu$, so
$$
\frac{d \lambda_0}{d \log \mu} = 0 = \beta_g (1+ 2 \frac{zg}{\epsilon}) \mu^\epsilon + \epsilon g (1+ \frac{z g}{\epsilon}) \mu^\epsilon + O(g^3)
$$
which rearranges to the correct answer for $\beta_g$ :
$$
\beta_g = -\epsilon g \frac{1+ \frac{z g}{\epsilon}}{1+ 2 \frac{zg}{\epsilon}} + O(g^3) = -\epsilon g + \frac{3}{16\pi^2} g^2 + O(g^3)
$$
which, to the order calculated, is not divergent as $\epsilon \to 0$, as we should expect. There are two points of note here:
- We are doing a formal power series expansion in $g$, which means that we can do the expansion of $1/(1+ x g)$ even though $x$ might be a divergent quantity: see How does a perturbation theory make sense in quantum field theory?
- The $g^3$ term here looks like it would be $\frac{\epsilon}{\epsilon^2}$ divergent, but that is just because we have not calculated the higher order contributions, which would cancel to give something finite.
To answer your questions
- Yes, you can renormalise the massless $\phi^4$ theory using the minimal subtraction scheme? You just need to ensure that you do not try to use any renormalisation conditions at zero momentum, where your loop integrals will diverge.
- Yes, in DREG the $\mu$ appears to ensure that the coupling constant that we use is dimensionless. It is in principle arbitrary - if we could calculate to all loop orders, its value would not matter. In the context of a Lorentzian QFT, higher loop contributions to a process at energy scale$^*$ $E$ will contain factors of $\log(\mu/E)$; picking $\mu\sim E$ makes those terms smaller. We want this because we have to stop calculating loops at some point, so we want to be able to ignore those loops. For more on this, see Intuition for parameter $\mu$ in dimensional regularization and "Running with momentum p" v.s. "running with renormalization scale μ".
Our renormalisation conditions (which are around energy scale $E_R$) set the value of $g$ to be a particular number, so they are of course important. However, because of the "large logs" problem above, when calculating a process at energy scale $E_P$ we require $\mu \sim E_R \sim E_P$ for the perturbative solution to be reliable.
This is the reason that the a priori arbitrary parameter $\mu$, which we introduced simply to make DREG make dimensional sense, is interpreted as a renormalisation scale!
$^*$ An example of such an $E$ would be $\sqrt{s}=\sqrt{(p_1 + p_2)^2}$ in the Lorentzian version of the renormalisation condition above.