Did Einstein really invent the cosmological constant to make the universe static in his 1917 paper?

Question

The popular account of Einstein inventing the cosmological constant goes like this:

1. Einstein finds that the Einstein Field Equations predict an expanding universe
2. Unable to accept this, Einstein adds the cosmological constant to his field equations to make the universe static
3. Later, when learning of Hubble's evidence that light from galaxies is redshifted in proportion to their distance from Earth, Einstein accepts the idea of an expanding universe and gets rid of the cosmological constant, calling it his "greatest blunder"

I tried reading Einstein's 1917 paper on the cosmological considerations of general relativity to understand his thought process, but I don't really see the above narrative. My understanding of the paper is more like this:

1. Einstein claims Newtonian gravity cannot handle the case of an infinite universe with constant mass density $\rho$, as this would give a constant potential $\phi$, and Poisson's equation $\nabla^2 \phi = 4\pi G \rho$ paradoxically tells us the mass density must then be zero.
2. Einstein reasons he can "correct" this by adding an additional term $\nabla^2 \phi - \lambda \phi = 4\pi G \rho$, which has a solution of $\phi = \frac{4\pi G \rho}{\lambda}$ that works even out at infinity
3. Einstein has similar concerns for general relativity, and wonders what value the metric $g_{\mu\nu}$ would take in an infinite universe with constant mass density
4. Einstein solves the problem of boundary conditions by making the spatial part of the universe a "closed" sphere, essentially making a universe with no boundary.
5. Using some math I don't understand, he deduces that an additional term $\lambda g_{\mu\nu}$ in the Einstein field equations is required to make the universe closed with constant curvature: $G_{\mu\nu} + \lambda g_{\mu\nu} = \frac{8\pi G}{c^4} T_{\mu\nu}$

To me, it seems like Einstein not at all concerned with the universe expanding, and only concerned with the boundary conditions of the metric $g_{\mu\nu}$ at infinity.

Is the popular narrative about Einstein's cosmological constant wrong? Did he believe the EFE without the cosmological constant predicted an expanding universe? Have I failed to understand something in the paper?

(I would prefer answers avoid referring to the Friedmann Equations, since those came later in 1922, unless you feel those are relevant to the question.)

Answer 1

Some relevant paras from Subtle is the Lord: The Science and the Life of Albert Einstein are quoted at the end. From them, we can see that

1. Einstein's main reason to believe in the cosmological constant was Mach's principle. (He didn't mention properly about Mach's principle in his 1917 paper because he didn't yet give that principle a name which he gave in 1918)
2. Einstein didn't like nonstatic universes (probably due to philosophical reasons) and in 1922 tried to find mistakes in Friedmann's nonstatic solution to Einstein field equations without $\Lambda$ but he later realized that it is a valid solution. But he still believed in his own static model. But this reason to believe in the cosmological constant came after (not before) he introduced the cosmological constant.
3. In 1931 after the experimental evidence he accepted Friedman's metric as the physically correct solution.

Maybe because it is easier to explain the expansion of the universe than Mach's principle popular science books talk more about his objection to the expansion of the universe.

From Subtle is the Lord:

$${\displaystyle R_{\mu \nu }-{\frac {1}{2}}Rg_{\mu \nu }=-\kappa T_{\mu \nu }}\tag{15.20}$$ $${\displaystyle R_{\mu \nu }-{\frac {1}{2}}Rg_{\mu \nu }-\lambda g_{\mu \nu }=-\kappa T_{\mu \nu }}\tag{15.21}$$ So strongly did Einstein believe at that time in the relativity of inertia that in 1918 he stated as being on equal footing three principles on which a satisfactory theory of gravitation should rest [E42]:

1. The principle of relativity as expressed by general covariance

2. The principle of equivalence

3. Mach’s principle (the first time this term entered the literature): ‘The $g_{\mu\nu}$ are completely determined by the mass of bodies, more generally by $T_{\mu\nu}$’. In 1922, Einstein noted that others were satisfied to proceed without this criterion and added, ‘This contentedness will appear incomprehensible to a later generation, however’ [E42a].

In later years, Einstein’s enthusiasm for Mach’s principle waned and finally vanished. I conclude with a brief chronology of his subsequent involvement with cosmology.

1917: Einstein never said so explicitly, but it seems reasonable to assume that he had in mind that the correct equations should have no solutions at all in the absence of matter. However, right after his paper appeared, de Sitter did find a solution of Eq. 15.21 with ρ = 0 [S14, W19]. Thus the cosmological term λgμv does not prevent the occurrence of ‘inertia relative to space.’ Einstein must have been disappointed. In 1918 he looked for ways to rule out the de Sitter solution [E42b], but soon realized that there is nothing wrong with it.

1922: Friedmann shows that Eq. 15.20 admits nonstatic solutions with isotropic, homogeneous matter distributions, corresponding to an expanding universe [F1]. Einstein first believes the reasoning is incorrect [E45], then finds an error in his own objection [E46] and calls the new results ‘clarifying.’

1923: Weyl and Eddington find that test particles recede from each other in the de Sitter world. This leads Einstein to write to Weyl, ‘If there is no quasistatic world, then away with the cosmological term’ [E47].

1931: Referring to the theoretical work by Friedmann, ‘which was not influenced by experimental facts’ and the experimental discoveries of Hubble, ‘which the general theory of relativity can account for in an unforced way, namely, without a λ term’ Einstein formally abandons the cosmological term, which is ‘theoretically unsatisfactory anyway’ [E48]. In 1932, he and de Sitter jointly make a similar statement [E49]. He never uses the λ term again [E50].

1954: Einstein writes to a colleague, ‘Von dem Mach’schen Prinzip sollte man eigentlich überhaupt nicht mehr sprechen,’ As a matter of fact, one should no longer speak of Mach’s principle at all [E51].