Can the cosmological constant problem be solved by imposing that the renormalized value matches the observed value?

Question

My understanding of the cosmological constant problem is that there is a difference of $\sim 120$ orders of magnitude between the value of the cosmological constant predicted by theoretical physics and its actual value observed in cosmology. It seems to me that it is possible to solve this problem with renormalization, therefore I am either misunderstanding the problem, the solution or both.

In particular In the book Introduction to Quantum Effects in Gravity by Mukhanov and Winitzki, in ch. 14.1 - Renormalization of the effective action, I have the impression that they solve the cosmological constant problem by means of renormalization. They do not claim to be solving that problem, and I do not understand why that is not a solution to the cosmological constant problem. In particular, the relevant sections of text are:

We assume that the free gravitational action (without backreaction of quantum fields) contains also the terms quadratic in curvature, $$ S_\mathrm{bare}^\mathrm{grav}\,\left[g_{\mu\nu}\,\right] = \int d^4x\sqrt{-g}\,\left[-\frac{R+2\Lambda_B}{16\pi G_B}\,+\alpha_B\left(\frac{R^2}{120}+\frac{R_{\mu\nu}\,R^{\mu\nu}}{60}\,\right)\right]\tag{14.5} $$ where $\Lambda_B, G_B$, and $\alpha_B$ are called the bare coupling constants; [...]. The modified action for gravity is thus the usm of the free action and the effective action, $$ S_\mathrm{bare}^\mathrm{grav}\,\left[g_{\mu\nu}\,\right] + \Gamma_L\,\left[g_{\mu\nu}\,\right] = \int d^4x\sqrt{-g}\,\left\{\left[-\frac{\Lambda_B}{8\pi G_B}-\frac{A(\tau_0)}{32\pi^2}\,\right]\\- \left[\frac{1}{16\pi G_B}+\frac{B(\tau_0)}{192\pi^2}\right]R \\+\left[\alpha_B-\frac{C(\tau_0)}{32\pi^2}\right]\left[\frac{R^2}{120}+\frac{R_{\mu\nu}\,R^{\mu\nu}}{60}\,\right]\right\} $$ [...]. The renormalization procedure assumes that the bare constants are functions of $\tau_0$ chosen in sucha a way that they cancel the divergences in the effective action. The renormalized coupling constants are then $$ \frac{\Lambda}{8\pi G}=-\frac{\Lambda_B}{8\pi G_B}-\frac{A(\tau_0)}{32\pi^2},\\ \frac{1}{16\pi G}=-\frac{1}{16\pi G_B}-\frac{B(\tau_0)}{192\pi^2}, \\ \alpha = \alpha_B-\frac{C(\tau_0)}{32\pi^2}. $$ After removing the cutoff (setting $\tau_0=0$), the renormalized constants are equal to the observed values of the constant $\alpha$, the cosmological constant $\Lambda$, and Newton's constant $G$

So, if as they claim $\Lambda$ is equal to the observed value of the cosmological constant, why isn't this a solution of the cosmological constant problem by means of renormalization? It seems to me that this is a solution in the sense that we can obtain the observed value using this procedure, in the same way we obtain the observed value of the fine structure constant of QED with renormalization.

Edit: It seems I am misunderstanding the cosmological constant problem itself. If you agree, a clarification of what actually is the cosmological constant problem and why it is so severe, would be greatly appreciated

Answer 1

This is not a problem with cosmological constant in models of universe. The "cosmological constant problem" is a bad name for a particular (bad) interpretation of a certain calculation procedure in QFT that leads to severe contradiction to observation. You can find the usual account of that calculation e.g. in this answer by Nihar Karve: https://physics.stackexchange.com/a/626958/31895

When we canonically quantize e.g. free EM field, we get a field Hamiltonian that is a sum over Fourier expansion modes (where mode $\mathbf k\lambda$ is a multiindex indexing discrete basis for wave vector and polarization), in which there is a contribution $\frac{1}{2}\hbar \omega$ per mode:

$$ H_{cq} = \sum_{\mathbf k,\lambda} \bigg( \hbar\omega_{\mathbf k} a_{\mathbf k,\lambda}^+ a_{\mathbf k,\lambda} + \frac{1}{2}\hbar \omega_{\mathbf k}\bigg). $$

This so-called zero point term $\frac{1}{2}\hbar \omega_{\mathbf k}$, since it is present once for each mode, implies all eigenvalues of $H_{cq}$ are infinite, and expectation value of $H_{cq}$ is infinite for any state.

Having all eigenvalues infinite breaks calculations. The correct conclusion from this is that we have a defective (wrong) Hamiltonian. This is not some serious insurmountable difficulty - we shouldn't expect canonical quantization to always give the correct Hamiltonian. We have a system with infinite number of degrees of freedom, not a finite set of harmonic oscillators, so maybe the Hamiltonian is something non-trivially different from what we have for a finite set of simple harmonic oscillators. So we should at least look for a better Hamiltonian (if not for a completely different formulation of the theory).

However, there are several different ways to improve on the above Hamiltonian, here are two of them:

1. high frequency cutoff in the above sum applied to zero point term, so the Hamiltonian has finite eigenvalues;

2. so-called normally ordered Hamiltonian, which has no zero-point term in any mode, so the Hamiltonian has finite eigenvalues, and zeroth eigenvalue is zero, so ground state has zero eigenvalue;

3. ...

The first way introduces a modification of the Hamiltonian which "cuts off" (suppresses to zero) contribution of terms with frequency that is higher than some arbitrary cutoff frequency $\omega_c$; however, many calculations' results consistent with experiments are affected by choice of value of $\omega_c$ in a negligible way, as long as we don't put it too low. This way of using the cutoff is, albeit ugly, arguably fine in practical sense, since the results are correct and do not depend on the choice strongly.

However, when this cutoff frequency is selected in a particular way, connected to gravity constant $G$ (which is just additional assumption, foreign to QFT), we get a definite but very high value for the zeroth eigenvalue, so-called ground state energy or zero point energy. Notice that in this terminology, the Hamiltonian eigenvalues are identified with possible values of energy of the field. This is not necessary but it is possible, as we can work with that Hamiltonian which defines energy. But we should keep in mind this is QFT energy defined by the choice of Hamiltonian, not necessarily the GR energy consistent with observations of gravity.

Then, some people think:

• maybe the GR stress-energy tensor in vacuum has contribution due to expectation values of stress-energy tensor in QFT, based on the chosen Hamiltonian and momenta (linking the concept of energy in QFT to concept of 00 component of stress-energy tensor in GR);

• maybe we can calculate value of this contribution using the QFT formalism and get a finite value;

• then we use the fact the result of above is non-zero as a basis for a new idea: that the non-zero contribution of the cosmological term in Einstein's equations is actually due to this non-zero Hamiltonian eigenvalue of the ground state.

The cosmological term in Einstein's equations is actually not a part of the stress-energy tensor in GR proper, but formally it can be put into it to get the "effective stress tensor", so with that, the above thought train makes sense.

But of course, it is well-known that this assumption-loaded procedure (the calculation and its interpretation in relation to GR) produces energy density value that is many orders greater than the observed value of the effective energy density in GR (if we adopt the cosmological constant term into the stress-energy tensor). This very bad disagreement should give us pause and make us realize the procedure was obviously quite dubious right from the start, irrespective of the immense disagreement on numerical value. There is no reliable theory involving gravity constant $G$ implying there should be such $G$-dependent cutoff in the Fourier expansions. A theory of gravity on discrete space could conceivably imply it, but we have no indication for such discrete space existing, and there even isn't any compelling relativistic theory for it. Also, there is no necessity to fix the problem with the infinite Hamiltonian using a cutoff or discrete space, there are other ways (below).

Some people talk about the procedure as if it was a QFT prediction of the value of cosmological constant, but it is not. Ground state eigenvalue of Hamiltonian in QFT is not easily argued to be connected to the GR stress-energy tensor. Energy in GR is objective, measurable thing, with effect on geometry; but in quantum theory, system energy value, whether it is particle or field, is arbitrary, it depends on the choice of the Hamiltonian that we use to define energy, it does not affect any measurable thing. So we can't infer value of GR energy from knowledge of QFT energy. These are not necessarily linked.

If we believe there is a link between QFT field energy density and GR energy density (00 component in stress-energy tensor), from observations, we can only extract constraints on the proper Hamiltonian to use when definining energy in QFT; we can require that so defined QFT energy is consistent with GR energy and with observations of its gravity effects. So really what is a correct statement here, assuming the belief about the link, is that the observational constraints on the value of the cosmological constant imply constraints on the choice of energy-defining Hamiltonian in QFT. And these imply immensely lower energy in the ground state than the $G$-based cutoff.

Sabine Hossenfelder put it well: let me ask a rhetorical question: which theory was falsified and rejected as a result of observations showing this calculated value of cosmological constant is completely wrong? None, because it is not a solid prediction based on core principles of QFT; it is obvious the above procedure is full of dubious assumptions. What has been actually falsified is the procedure linking cosmological constant with QFT ground state, in all its different variations, because they all produce wrong result if the parameters are not doctored. Since this procedure, with its dubious assumptions and wrong result, is not an inescapable implication of QFT, nothing about QFT itself has been falsified.

There is still the second way to fix the original problem with infinite Hamiltonian eigenvalues - just use the Hamiltonian without the zero point energy terms:

$$ H_{no} = \sum_{\mathbf k,\lambda} \hbar\omega_{\mathbf k} a_{\mathbf k,\lambda}^+ a_{\mathbf k,\lambda}~~~. $$ Sometimes this is called the normal ordering Hamiltonian, and it is the one used and sufficient in most practical calculations of light-matter interactions. The lowest eigenvalue of this Hamiltonian is zero - the most plausible value ever for the lowest possible energy of EM field in a theory where the concept of energy is based on the Poynting expressions, so this Hamiltonian is also the best, in the class of theories based on Poynting expressions, to define field energy. Expectation values for any non-pathological quantum state are finite, no infinite expectation values appear. There is no arbitrary cutoff, no heuristics with gravity constant; just a "boring" Hamiltonian-based theory of EM field, which correctly describes many experiments involving light-matter interaction, but has nothing to say on effects of quantum field on the macroscopic geometry of spacetime. One can even add the cosmological constant induced energy to it to make QFT energy the same as GR energy, but I see no benefit. This "boring" variant of QFT does not contain any revelation about the cosmological constant, which might seem disappointing to some, but really the topic and prospects have been oversold. Notice and ponder the upside: we have a functional and successful QFT without a dubious procedure to calculate the cosmological constant, and no immense failure in comparison to experiment. One should strive for similar failure-free description for more complicated systems involving several quantum fields, using well-behaved Hamiltonians (or other math devices) with finite eigenvalues, preferably with zero or very small ground state eigenvalue, consistent with observations.

BUT, since in this second way of formulating QFT, the field ground state energy is zero, there is no good physics reason for why it should be connected to the concept of non-zero cosmological term in GR in any direct way. Maybe we want them to be so connected due to some human motivations, but it is not necessary in physics. Not all constants on Nature are connected /produced by/calculable from other constants of Nature. There are many many constants for which we have no idea where their values come from. The cosmological constant can be just another one in the big family.

Non-zero cosmological constant term is really a concept from GR and macroscopic observations; there is no real logical necessity or need in physics to have this concept linked to field ground state Hamiltonian eigenvalue in QFT. I should say many people don't agree on this last point, they think these things should be linked, but I suspect the motivation is problematic. And none of them know how to formulate a functional QFT of gravity where the above procedure produces a reasonable value of cosmological constant close to observed value. All known variations that are free of doctoring the parameters produce wrong results. So I think it very likely the whole enterprise is ill-conceived, and no revelation on the observed value of the cosmological constant will be coming from these QFT-based, or even any microscopic theory-based attempts.

Answer 2

In the approach to renormalizable quantum field theories where we consider the theory as a fundamental theory supposed to be valid at all energy scales without a momentum/energy cutoff, the vacuum energy/cosmological constant is a renormalization parameter, and renormalization parameters can be set to anything we want, including the observed value of the cosmological constant. A cutoff may be introduced for intermediate steps but is always at least theoretically lifted to infinity at the end.

The "prediction" for the vacuum energy comes from viewing a QFT like the Standard Model merely as a Wilsonian effective theory with a meaningful inherent cutoff. If one sets this cutoff around the Planck scale (since that's where we generally expect QFT to break down), one obtains a gigantic value for the vacuum energy (which is now finite without renormalization, since the cutoff prevents UV divergences). See this answer by Nihar Karve for a more detailed exposition.

The exact ways in which this value is a "problem" is a matter of debate, and depends on lots of often unspoken assumptions about the nature of quantum field theories, quantum gravity and the role of renormalization - as becomes obvious here, where the two different viewpoints of QFT result in very different opinions about the significance of the observed value of the cosmological constant.

Answer 3

The term Cosmological Constant Problem is a misnomer. The problem is in QED.

As far as I know it cannot be removed by renormalisation. Anyway, renormalisation is a technique to manage infinities. It is a workaround that does not address the root causes.