I've read that QM operates in a Hilbert space (where the state functions live). I don't know if its meaningful to ask such a question, what are the answers to an analogous questions on GR and Newtonian gravity?
I interpreted your question differently, more like a mathematics question.
In Quantum Mechanics, we basically have an equation, the Schrödinger equation, which is a differential equation on the space of square-integrable complex valued functions. This space is a Hilbert space, which means that it is a vector space, and it also has a nice topological structure, basically all Cauchy-sequences of vectors converge in that space.
In Newtonian mechanics, the equations are defined on phase space, which is basically a $6N$-dimensional space, $N$ is the total number of particles, on which coordinates for a point consist of the positions and momenta of each particle you want to describe. The solution of the equations induces a flow on this phase space. The structure of phase space is usually that of a symplectic manifold.
In General Relativity, the equations are Einstein's field equations. They link the Riemann tensor to the energy-momentum tensor. They are difficult to solve in the sense that they are nonlinear and you have to specify an energy-momentum tensor, but this tensor will also depend on the geometry of space-time, thus the Riemann tensor. So you have to solve in one go for the geometry and energy-matter distribution. In practice, many simplifying assumptions will be made. But the "space" of solutions is the space of geometries and energy-matter distributions compatible with the field equations.
|show 3 more comments|
First, I'll assume that you're talking about quantization. To understand how to quantize GR it is absolutely necessary to give an account (however sketchy) of the approach used to quantize simpler systems.
This is a procedure whereby one transfers from the classical point of view (Newtonian mechanics or equivalently Lagrangian or Hamiltonian mechanics) to the quantum point of view. Now, there are some general prescriptions how one can quantize classical mechanical systems. The most common one is that one replaces the phase space by Hilbert space, functions on phase space by operators on the Hilbert space and Poisson bracket of functions by commutator of the operators.
The previous paragraph was only dealing with mechanics, i.e. case where there are only few degrees of freedom. But GR is a field theory (of gravitational field) and is actually a kind of gauge theory (but a little special at that). One has to first learn how to quantize classical fields and then gauge fields. To do that you can replace the (infinite-dimensional) phase space of the field by (very large) Hilbert space and produce an analogue of Poisson brackets called Dirac bracket which you then replace by commutators.
(The second very common approach to quantization is via path integral for which you don't need any operators but I won't elaborate on that here because it is a huge area that would take us far away off the topic of your question)
Then to quantize a gauge theory with its own huge gauge symmetry one has to carry out a very nontrivial discussion about the structure of these Dirac brackets.
(There also exist other approaches to this but none of them is particularly easy for a beginner. If you're interested see Faddeev-Popov ghosts in path integral gauge quantization and BRST quantization)
Now, the thing is that GR (as a field theory) is hard to quantize. I.e. if you repeat the above approach for GR, you'll find out that your quantum theory doesn't make sense (because it is not renormalizable).
This suggests that something more than naive approach is needed. And there are actually lots of them. For one thing, one can quantize gravity in certain special dimensions (like 2+1) if one generalizes GR a little (this was done by Witten in '80s). There are also various reformulations that relate quantum gravity and QFT (like AdS/CFT correspondence). There is also matrix string theory that shows duality between matrix quantum mechanics and GR (as pointed out to me by Matt in this question of mine).
In short, quantization of GR is very hard. There are many theories and as of yet there is no experimental evidence that would let us know which one is the correct one.