Can we dispense with the Manifold in General Relativity? I am studying Quantum Gravity  by Rovelli. In chapter 2, the author describes the path that Einstein followed to arrive to General Relativity (GR). At the end of the discussion of the hole argument, Rovelli (and I think also Einstein) arrives to the conclusion that (page 68-69):

There is no meaning in talking about the physical spacetime point

so that the manifold is in fact just gauge, a mathematical construct without physical meaning and that we have to describe physics as a theory of fields on fields, not fields on spacetime. 
This has me confused: as I understand it, in GR the gravitational field is given by the curvature of the spacetime and therefore of the manifold, so it should have a physical meaning after all; I do not understand how the curvature have physical meaning while the underling manifold doesn't. 
How can we dispense with the manifold if its curvature is the gravitational field?
 A: One answer to your question may be found in this paper:
Robert Geroch. "Einstein algebras." Comm. Math. Phys. 26 (4) 271 - 275, 1972.
https://doi.org/10.1007/BF01645521

See also
https://projecteuclid.org/journals/communications-in-mathematical-physics/volume-26/issue-4/Einstein-algebras/cmp/1103858122.full
Some excerpts are quoted below. Refer to the full article for details.

Abstract
An approach to quantization of general relatively using a reformulation of the classical theory in which the events of space-time play essentially no role is discussed.


In Section 2, we write down a set of definitions of tensor fields, the
metric, the Ricci tensor, etc., on a smooth manifold $M$. We then observe,
in Section 3, that the original manifold $M$ is used in these definitions at
just one minor point. This observation is made more explicit by the
introduction of what we call an Einstein algebra.


Section 3: Einsteins Algebras

In the previous section, we stated a sequence of definitions, leading
from a smooth manifold $M$ to a metric on $M$ and finally to all the tensor
operations on $M$. We now make the following observation: the manifold
$M$ entered the discussion of Section 2 at only one point: in the construction
of the ring $\scr I$ of smooth functions on $M$ and the subring $\scr R$ of the constant
functions. All constructions and definitions thereafter were purely
algebraic ones on $\scr (I, R)$.

We are thus led to the following definition. An Einstein algebra
consists of i) a commutative ring $\scr I$,
ii) a subring $\scr R$ of $\scr I$, isomorphic
with the real numbers (and such that the identity $1$ of $\scr R$ is the identity
of $\scr I$ and iii) a metric $g$, such that the contraction property is satisfied
and the Ricci tensor vanishes.
More generally, one could introduce
Einstein algebras with sources (e.g., electromagnetic fields, fluids, etc.)
by introducing smooth tensor fields to represent the sources, and suitably
modifying Einstein's equation. Thus, an Einstein algebra is a purely
algebraic object, making no direct reference to a manifold. Of course,
every space-time which is a solution of Einstein's equation defines an
Einstein algebra. One could just as well take, as the underlying mathematical
framework of general relativity, an Einstein algebra rather
than the usual smooth manifold with smooth metric tensor field.

(I bolded the key idea above. Refer to the full article for details.)
A google search for "Einstein Algebras" (in quotes) will point
to other articles that try to address the Hole Argument, which I have no comments about.
A: 
At the end of the discussion of the hole argument, Rovelli (and I think also Einstein) arrives to the conclusion that (page 68-69): "There is no meaning in talking about the physical spacetime point"

As a matter of history, I don't think that this was Einstein's interpretation. This was the time period when he was stumbling around with the dead-end Entwurf theory. Einstein was at this point working with ideas that didn't include general covariance.
I don't own the Rovelli book, but I would imagine he's saying something like the following. Points in spacetime don't have specific labels. The identity of such a point is not an observable. In classical terms, only if there is an event at that point, e.g., the collision of two world-lines of particles, can we identify the event.

This has me confused: as I understand it, in GR the gravitational field is given by the curvature of the spacetime and therefore of the manifold, so it should have a physical meaning after all; I do not understand how the curvature have physical meaning while the underling manifold doesn't.

For comparison, consider a wavefunction in the Schrodinger picture. The wavefunction has some phase, which is not a physical observable. We find it convenient to work with the wavefunction and its phase, but we should not be misled into thinking that absolute phases are observables in quantum mechanics.
A: 
How can we dispense with the manifold if its curvature is the
  gravitational field?

Your question is pertinent, and quantum gravity must provide an answer to it. Here are some hints for helping you to make up your mind, but quantum gravity is still an open issue.
1.According to today's knowledge and in spite of all attempts, curved spacetime is not quantizable, and thus it is not compatible with quantum mechanics. Spacetime and curved spacetime are mathematical models which have been introduced for the description of the principles of special and general relativity, introduced respectively by Minkowski in 1908 (Minkowski spacetime) and by Einstein and Grossmann some years later (curved spacetime). These models are very useful for the description of special and general relativity, but they are not compatible with quantum mechanics. In this answer I showed that there is a way to express gravity not only as curved spacetime but alternatively also as gravitational time dilation in flat, uncurved R3 space.
2.A second, alternative possibility would be to consider only the curved worldlines of the spacetime manifold, at the exclusion of the vacuum between these worldlines. Vacuum is not defined by theory of gravity of general relativity, vacuum is defined by quantum physics and possibly by cosmology (in the form of dark energy). Already Einstein saw this possibility of the lack of continuity of spacetime, but he did not believe in its feasibility, in a letter to Dällenbach he wrote:

“But you have correctly grasped the drawback that the continuum
  brings. If the molecular view of matter is the correct (appropriate)
  one, i.e., if a part of the universe is to be represented by a finite
  number of moving points, then the continuum of the present theory
  contains too great a manifold of possibilities. I also believe that
  this too great is responsible for the fact that our present means of
  description miscarry with the quantum theory. The problem seems to me
  how one can formulate statements about a discontinuum without calling
  upon a continuum (space-time) as an aid; the latter should be banned
  from the theory as a supplementary construction not justified by the
  essence of the problem, which corresponds to nothing “real”. But we
  still lack the mathematical structure unfortunately. How much have I
  already plagued myself in this way!”

For the purpose of quantum gravity it is important to notice that for fundamental questions we should refer to the fundamental notion of proper time instead of the observer-dependent notion of coordinate time, that means, if we consider worldlines, at my opinion it does not make much sense to consider the parametrization by coordinate time, instead they should be parameterized by proper time. One consequence would be that all lightlike phenomena such as electromagnetic or gravity fields are reduced to zero.
