What are the local covariant tensors one can form from the metric? Normally in differential geometry, we assume that the only way to produce a tensorial quantity by differentiation is to (1) start with a tensor, and then (2) apply a covariant derivative (not a plain old partial derivative). Applying this to GR, I think one way of stating the equivalence principle is that the only tensorial object that we expect to be "built in" to the vacuum is the metric. Since the covariant derivative is basically defined as a derivative that produces zero when you apply it to the metric, this means that you can't get anything of interest (i.e., local and tensorial) by appying the process described by #1 and #2 to the vacuum. This can be used as a fancy way of arguing that the Newtonian gravitational field $\mathbf{g}$ isn't a tensor, since in the Newtonian limit, it's essentially the gradient of the metric.
However, the process described by #1 and #2 is sufficient but not necessary. In fact, one way of defining curvature is by taking non-covariant derivatives on the metric to form the Christoffel symbols, and then doing further operations involving non-covariant derivatives to get the Riemann curvature tensor -- which surprisingly ends up being a valid tensor.
It seems, then, that the Riemann tensor is a special case. I originally thought that there might be a uniqueness theorem that proves that if we want to produce a local, tensorial quantity from the metric, the only possibilities are the Riemann tensor or curvature polynomials formed from the Riemann tensor and its covariant derivatives.
[EDITS] A comment by joshphysics and the answer by BebopButUnsteady helped me to refine this conjecure as follows.
Joshphysics pointed out that things like $g_{ab}g_{cd}$ might be considered trivial counterexamples. I can think of two possible ways of dealing with this:
(1) BebopButUnsteady's answer shows that this is in some sense not a counterexample at all, since the metric itself can be expressed as a Taylor series in terms of the Riemann tensor and its derivatives. If the metric is analytic, and if we're willing to accept infinite series, then this means that there is no information in the metric that isn't recoverable from the Riemann tensor.
(2) What doesn't seem to exist, apart from curvature polynomials formed from the Riemann tensor and its covariant derivatives, is (a) any varying scalar field, or (b) any vector field. (Part b is basically the equivalence principle.)
 A: The answer to your question is affirmative in the following sense:
In the Riemann normal coordinates at $p$ the coefficients of the Taylor expansion of the metric $g_{ij}(x)$ are  polynomials in the Riemann tensor at $p$ and its covariant derivatives at $p$. [Assuming the proof in this random thing I googled[a]  is correct, starting at (5.1)].
I think this is the correct formalization of your conjecture in the sense that if we are making a tensor out of $g$ the only thing we can use are $g$ and its expansion in normal coordinates. I'll maybe try to write out why I think this is case.
BTW, the local condition is very necessary, since otherwise we could define things like the length of the shortest loop containing $p$ that is in a certain homotopy class, which clearly "depends only on the metric" but is not made out of polynomials the curvature.
Added after this answer was accepted
For those interested, I asked a question on Math SE that contains what I believe to be the correct formalization of the question: "What tensors can I produce from the metric tensor?" “Natural” constructions of tensor fields from tensor fields on a manifold
[a]: Guarrera, D.T., Johnson, N.G., Wolfe, H.F. (2002) The Taylor Expansion of a Riemannian Metric  
$\mbox{                          } $ $\mbox{                          } $ $\mbox{                          } $ $\mbox{                          } $ $\mbox{                          } $ $\mbox{                          } $ $\mbox{                          } $ http://www.rose-hulman.edu/mathjournal/archives/2002/vol3-n2/Wolfe/Rmn_Metric.pdf 
