In the flat spacetime, one can perform normal-ordering to set the energy of the vacuum state to zero. I read in some places that this procedure cannot be consistently performed in the curved spacetimes. I have not found any explanation of this fact in the literature. Why is this a case?
The normal ordering procedure is nothing but the a recurrent machinery to remove vacuum expectation values by means of the so-called Wick's theorem procedure. There, the vacuum state is the unique Poincaré invariant one, the Minkowski vacuum. In curved spacetime there is no such an invariant state. Therefore the normal ordering product must be defined referring to some other fixed states. These states (their two-point functions) should have a short distance behaviour similar to the one of Minkowski vacuum state. Technically they must be Hadamard states. Unfortunately there is no canonical (functorial) way to associate Hadamard states with (globally hyperbolic) spacetimes. This fact prevents one for defining a true generally covariant normal ordering procedure. A way out consists of removing not the expectation value of a state, but the "expectation value" of a local functional defined in terms of the local geometry only (the Hadamard parametrix). This procedure, in view of the fact that the functional is not a (weak) bi-solution of the field equations, has nonetheless annoying shortcomings, in particular the appearance of anomalies (the trace anomaly first of all). This machinery has been developed by various authors in the last 15 years (including myself). For a recent overview see the PhD thesis http://arxiv.org/abs/arXiv:1008.1776 (I was one of the advisors, together with K. Fredenhagen and R.M. Wald).