How can we dispense with the manifold if its curvature is the
gravitational field?
Your question is pertinent, and quantum gravity must provide an answer to it. Here are some hints for helping you to make up your mind, but quantum gravity is still an open issue.
1.According to today's knowledge and in spite of all attempts, curved spacetime is not quantizable, and thus it is not compatible with quantum mechanics. Spacetime and curved spacetime are mathematical models which have been introduced for the description of the principles of special and general relativity, introduced respectively by Minkowski in 1908 (Minkowski spacetime) and by Einstein and Grossmann some years later (curved spacetime). These models are very useful for the description of special and general relativity, but they are not compatible with quantum mechanics. In this answer I showed that there is a way to express gravity not only as curved spacetime but alternatively also as gravitational time dilation in flat, uncurved R3 space.
2.A second, alternative possibility would be to consider only the curved worldlines of the spacetime manifold, at the exclusion of the vacuum between these worldlines. Vacuum is not defined by theory of gravity of general relativity, vacuum is defined by quantum physics and possibly by cosmology (in the form of dark energy). Already Einstein saw this possibility of the lack of continuity of spacetime, but he did not believe in its feasibility, in a letter to Dällenbach he wrote:
“But you have correctly grasped the drawback that the continuum
brings. If the molecular view of matter is the correct (appropriate)
one, i.e., if a part of the universe is to be represented by a finite
number of moving points, then the continuum of the present theory
contains too great a manifold of possibilities. I also believe that
this too great is responsible for the fact that our present means of
description miscarry with the quantum theory. The problem seems to me
how one can formulate statements about a discontinuum without calling
upon a continuum (space-time) as an aid; the latter should be banned
from the theory as a supplementary construction not justified by the
essence of the problem, which corresponds to nothing “real”. But we
still lack the mathematical structure unfortunately. How much have I
already plagued myself in this way!”
For the purpose of quantum gravity it is important to notice that for fundamental questions we should refer to the fundamental notion of proper time instead of the observer-dependent notion of coordinate time, that means, if we consider worldlines, at my opinion it does not make much sense to consider the parametrization by coordinate time, instead they should be parameterized by proper time. One consequence would be that all lightlike phenomena such as electromagnetic or gravity fields are reduced to zero.