What was the motivation to choose the Einstein-Hilbert action?
$$S= \int\sqrt{|\mathrm{det}(g_{\mu\nu})|} \, R\, d^4x $$
where $R$ is the Ricci scalar $ R= R_{ab}g^{ab}$.
What was the motivation to choose the Einstein-Hilbert action?
$$S= \int\sqrt{|\mathrm{det}(g_{\mu\nu})|} \, R\, d^4x $$
where $R$ is the Ricci scalar $ R= R_{ab}g^{ab}$.
A quick and dirty way to see this is to consider the stress-energy $T_{\mu\nu}$ and its relationship to the Lagrangian density $$ T_{\mu\nu}~=~2\frac{\partial{\cal L}}{\partial g^{\mu\nu}}~-~g_{\mu\nu}{\cal L}. $$ Now take the trace of this by multiplying by $g^{\mu\nu}$ so we obtain $$ g^{\mu\nu}T_{\mu\nu}~=~2{\cal L}~+~2g^{\mu\nu}\frac{\partial{\cal L}}{\partial g^{\mu\nu}}. $$ Now compare this to the Einstein field equation $R_{\mu\nu}~-~\frac{1}{2}Rg_{\mu\nu}$ $=~8\pi GT_{\mu\nu}$. Multiply this by $g^{\mu\nu}$ and it is easy to see that ${\cal L}~=~R$.
This is however to consistent in all coordinates, since the Lagrangian density is evaluated as $\int{\cal L}d^4x$. To make this consistent with respect of coordinate transformations this must be multiplied by the power of the Jacobian determinant of the metric tensor. The Jacobian determinant of the metric tensor is $$ det\Big|\frac{\partial x^\mu}{\partial x^\nu}\Big|~=~\sqrt{-g}. $$ This means the full lagrangian density for gravitation is ${\cal L}_g~=~\sqrt{-g}R$.
I believe it comes from the Principle of Least Action. Consider, for the Lagrangian $\mathscr{L}$ the action $$S = \int_{\text{all space}}\mathscr{L}d\,\Omega$$
Consider a small variation in the metric tensor so that $$ g_{\mu \nu} \mapsto g_{\mu \nu}+S g_{\mu \nu} $$ which would naturally lead to a variation in the action itself $S \mapsto S + \delta S$. The action principle implies that $$\delta S = \int_{\text{all space}}\mathscr{L}^{\mu \nu}\delta g_{\mu \nu} d \Omega = 0$$ Where the $(2,0)$ tensor density of weight $1$ is defined as $\mathscr{L}^{\mu \nu} = \frac{\delta \mathscr{L}}{\delta g_{\mu \nu}}$.
The application of the principle of least action confined to gravitational space is $$S_g = \int_{\text{space}}\mathscr{L}_g\,d \Omega$$ The condition required in order to get to the field equations is that $$\delta \int \mathscr{L} d^4 x=0$$ The only scalar density of weight $1$ involving the metric and its derivatives up to second order is $\sqrt{-g}R$. So, if one takes \begin{align} \mathscr{L}_g &= \kappa^{-1}\sqrt{-g}R\\ &= \kappa^{-1}\sqrt{-g}g^{\mu \nu}R_{\mu \nu} \end{align} Then the result follows.