# Why does Hartree-Fock work so well?

Why does the Hartree-Fock method for electronic structure work so well for atoms?

More specifically, why is the "correlation energy" a relatively small component of an atom's (ground state) energy? I might also ask why electron-electron interaction appears to For example:

• The true ground state energy of the helium atom is −2.903 hartree
• The energy of the Hartree-Fock solution is -2.862 hartree
• The energy of a solution obtained by ignoring electron-electron interaction altogether is -2.750 hartree

(See F. W. Byron, Jr. and C. J. Joachain, Phys. Rev. 146, 1.) If we quantify the "effect" of different interactions using the interaction's contribution to the ground state, we find

• electron-electron interaction (-2.750 - -2.903 = 0.153) is almost 20x smaller than electron-nucleus interaction (2.903), and
• electron correlation (-2.862 - -2.903 = 0.041) is 70x smaller than electron-nuclear interaction, and almost 4x smaller than electronic mean field interaction (0.153).

Given the basic ingredients for atomic physics (Coulomb interaction, Schrodinger equation, Pauli exclusion), it does not seem like there is any reason a priori that one electron in a Helium atom should be so indifferent to the location of the other. What reasons (other than desperation) did theorists have to expect the mean-field approximation to yield meaningful results? Were they based on experimental observations or some deeper physical argument?

• I think the high ratio of nuclear to electron mass plays a big role. In particular (although I haven't checked), I suspect that if one replaces electrons with muons (i.e. changes the mass parameters in the Schrödinger's equation), the exact ground-state energy will be much more different from the case when muon-muon repulsion is neglected. Sep 26 '20 at 13:47
• The Hartree Fock ground state is the minimal energy state that can be obtained using as a wavefunction a single Slater determinant. The Hamiltonian is "exact" in that all interactions are included. Sep 29 '20 at 4:23

I don't know why (or if) people originally expected Hartree-Fock to work as well as it does, but after thinking about it for a while, I'm personally a little less surprised by it. It seems surprising at first because most wavefunctions are not Slater determinants, but in view of the constraints that I'll derive below, finding any wavefunction that does better than the best Slater determinant seems challenging. "Seems challenging" could be due to my lack of imagination, but it still makes me a little less surprised by how well a single Slater determinant works.

## Conventional formulation of the question


• $$K$$ is the kinetic-energy term,

• $$V$$ is the attractive Coulomb interaction of each electron with the fixed nucleus,

• $$W$$ is the repulsive Coulomb interaction between electrons.

The Hilbert space $$\cH$$ consists of all antisymmetric wavefunctions $$\psi$$, which means that $$\psi$$ changes sign whenever two of the $$N$$ location-and-spin arguments are exchanged. A wavefunction is called a Slater determinant if it can be written as the antisymmetrized product of $$N$$ single-electron wavefunctions. Let $$\cS\subset\cH$$ be the set of Slater determinants.

Let $$\psi(A)$$ denote the expectation value of an operator $$A$$ with respect to a wavefunction $$\psi$$. The true ground state is the wavefunction $$\psi\in\cH$$ that minimizes the quantity $$\psi(H)$$, which is then the energy $$E_0$$ of the ground state: $$E_0\equiv \min_{\psi\in\cH}\psi(H). \tag{2}$$ The Hartree-Fock method uses the minimum of $$\psi(H)$$ among all Slater determinants $$\psi\in\cS\subset\cH$$ as an approximation to $$E_0$$: $$E_\text{HF}\equiv\min_{\psi\in\cS}\psi(H). \tag{3}$$ The subscript HF stands for Hartree-Fock. The question is: why is (3) such a good approximation to (2)?

## Dilating the wavefunction

What properties of the terms $$K$$, $$V$$, and $$W$$ might be important?

• The signs are important. For any state $$\psi$$, the quantities $$\psi(K)$$ and $$\psi(W)$$ are always positive, whereas the quantity $$\psi(V)$$ is always negative.

• The spatial scale is important. Consider the quantities $$\psi(K)$$, $$\psi(V)$$, and $$\psi(W)$$, for any wavefunction $$\psi$$. We can make the magnitudes of all of these quantities smaller by dilating $$\psi$$ in space. This makes $$\psi(K)$$ smaller because it reduces the electrons' momenta (by reducing the magnitude of the wavefunction's gradients), and it makes the magnitudes of $$\psi(V)$$ and $$\psi(W)$$ smaller by moving the electrons farther away from each other and from the nucleus. This assumes that the system is an atom, with one nucleus at the origin, and the dilation (or dilatation?) leaves the origin fixed.

We can make this more specific. Let $$\psi_\lambda$$ be the wavefunction obtained by applying a spatial scale factor $$\lambda>0$$, where $$\lambda<1$$ dilates the wavefunction, and $$\lambda>1$$ compresses the wavefunction. The kinetic term $$K$$ scales like the gradient-squared (because momentum $$\sim$$ gradient), and the Coulomb interaction scales like $$r^{-1}$$ where $$r$$ is the distance between charges. Therefore, \begin{align} \psi_\lambda(K) &= \lambda^2\psi(K) \\ \psi_\lambda(V) &= \lambda\psi(V) \\ \psi_\lambda(W) &= \lambda\psi(W). \tag{5} \end{align} Combine these to get $$\psi_\lambda(H) = \lambda^2\psi(K) + \lambda\big(\psi(V)+\psi(W)\Big). \tag{6}$$ Now, for any given $$\psi$$, let $$\Lambda$$ denote the value of $$\lambda$$ that minimizes the quantity (6). By taking derivative of (6) with respect to $$\lambda$$ and requiring that the result be equal to zero, we get $$\Lambda = -\frac{\psi(V)+\psi(W)}{2\psi(K)}. \tag{7}$$ By construction, $$\psi_\Lambda(H)$$ is the minimum possible expectation value of $$H$$ among all wavefunctions that can be obtained from $$\psi_\Lambda$$ by rescaling in space, so if the original wavefunction happened to already be $$\psi=\psi_\Lambda$$, then we would get $$\Lambda=1$$. This implies $$\psi_\Lambda(V)+\psi_\Lambda(W) = -2\psi_\Lambda(K). \tag{8}$$ This looks like the virial theorem, and it implies $$\psi_\Lambda(H) = -\psi_\Lambda(K) < 0. \tag{9}$$ This is the minimum expectation value of $$H$$ that can be achieved by dilating (or compressing) the wavefunction in space, starting with an arbitrary wavefunction $$\psi$$.

## How good can a single Slater determinant be?

A Slater determinant is an antisymmetrized product of single-electron wavefunctions, which I'll call orbitals. What might the optimal Slater determinant look like?

Start with some generic Slater determinant whose overall scale has already been optimized as described above. To try to reduce the energy further, we could:

• Increase $$|\psi(V)|$$ by squeezing one or more orbitals closer to the nucleus.

• Decrease $$\psi(W)$$ by moving two or more orbitals farther away from each other.

Consider how we might move two orbitals farther away from each other without also moving them farther from the nucleus (which would be counterproductive). One way to do this is to concentrate one of the orbitals on one side of the nucleus and to concentrate the other one on the other side. We can do this without moving either one farther from the nucleus. Therefore, this should decrease $$\psi(W)$$ without changing $$\psi(V)$$. On the other hand, it will increase $$\psi(K)$$, because now each orbital is concentrated in a smaller volume (which forces the momenta to be larger). After making such a change, we might be able to optimize its effect a little more by adjusting the overall scale as explained above. If the net effect of these changes is to reduce the energy overall, then the optimal Slater determinant must already exploit something like this.

Such a configuration seems asymmetric, but that's not necessarily a problem. Even if we expect the true ground state to have some special symmetry, the optimal Slater determinant does not necessarily need to have that same symmetry. It only needs to belong to a family of equally-optimal Slater determinants that collectively have that symmetry. Given one member of that family, we can average over rotations to construct a more symmetric state (which will no longer be a single Slater determinant), which may then be a better approximation to the true ground state, but the intuition described below suggests that this averaging might not change the energy much.

## How much better can a superposition of Slater determinants be?


Since $$|\psi_\text{HF}\ra$$ is already the optimal Slater determinant, any changes we make to the terms in (12) will necessarily make the "diagonal" terms worse, meaning that the contribution of the diagonal terms to the energy will increase. Whatever happens in the cross-terms must overcompensate for the degradation in the diagonal terms. I don't know how to orchestrate different changes in different terms of (12) to accomplish this feat. At the very least, this is yet another factor that limits our ability to improve on $$\psi_\text{HF}$$.

One way to change the cross-terms without changing the diagonal terms is to average over rotations, as described above, assuming that the optimal Slater determinant is not rotationally symmetric. Suppose this helps. How much might it help? Even though the Hilbert space is infinite-dimensional, the set of states below a given energy and localized within a given finite region is essentially finite-dimensional. And in the present case, where $$\psi_\text{HF}$$ is already squeezing things about as tightly as Pauli exclusion allows, finite-dimensional might mean not-very-many-dimensional. This limits the number of different terms in (12) that can be orthogonal to each other. In other words, even though we may consider a large number of terms in (12), many of them won't differ much, so their cross-terms will look essentially like diagonal terms, which means that those cross-terms can't help decrease the overall energy significantly — because we've already chosen $$\psi_\text{HF}$$ to optimize the diagonal terms.

That's nowhere near being a proof that Hartree-Fock is a good approximation, but it makes me less surprised that $$\psi_\text{HF}(H)$$ is already close to optimal for some atoms. A few examples are tabulated here.

As a side note: a wavefunction that gives a good approximation to the energy of the ground state is not necessarily such a good approximation to the true ground state wavefunction itself. The fractional error in the former tends to be of order $$\epsilon^2$$ whenever the error in the latter is of order $$\epsilon$$. This general result is derived in the first section of Goodisman and Klemperer, "On errors in Hartree-Fock calculations," The Journal of Chemical Physics 38, 721 (1963).

• I am having difficulty understanding the sentence beginning with "We might try..." and the sentence following. What do you mean when you say "...many of the cross terms will look essentially like diagonal terms"? As a side note, I wonder how your discussion applies to the hydrogen molecule at large atom separations, where HF is known to perform poorly (too much ionic contribution) and explicitly correlated techniques (valence bond) succeed. Oct 6 '20 at 1:06
• @creillyucla Regarding the sentence you asked about: my loose reasoning was that if there is a way to reduce the energy by manipulating the cross-terms (even though changes in the diagonal terms tend to increase the energy), then maybe having more cross-terms helps. But with a limited energy in a limited volume, we only have an effectively $M$-dimensional Hilbert space for some finite $M$. If the number of terms in the superposition is $J\gg M$, then many of them will be $\approx$ proportional, so those cross-terms are $\approx$ the same as diagonal terms. Oct 6 '20 at 2:49
• one last question for clarification: what is the connection between the cross-terms and the scaling argument you develop earlier? I understand that modifications of that act to scale it spatially will increase the energy, but that does not seem to exclude many other modifications we could envision. Oct 6 '20 at 12:28
• @creillyucla There's not much of a connection, except that it's one more constraint on our ability to improve the energy beyond $\psi_\text{HF}$. You are right that it doesn't exclude many other modifications we could envision. In hindsight, I think a better way to use the rescaling argument is to help optimize a single Slater determinant. I re-organized and modified the text after equation (9) to try to make the reasoning a little more coherent. Oct 6 '20 at 13:36