Is there an explanation for why Hawking radiation causes a black hole to shrink *without* referring to particle/anti-particle pairs? An observer far from a black hole sees thermal radiation at the Hawking temperature. As a consequence, we are told that the black hole loses energy over time: that is, it evaporates. This logical step from radiation to evaporation is usually justified by considering particle pairs near the horizon, and showing that the infalling particle has negative energy. This is ok, but it would be nice if there were an explanation not involving particles at all, i.e. only working at the level of fields moving on curved spacetime.
What I have in mind is some sort of conservation law which says that "Energy of spacetime + expected energy of quantum fields is the same for early- and late-time observers". Then, if the late-time observer sees more energy in the field, the energy of spacetime must decrease to compensate, i.e. the BH must shrink.
Does such a conservation law hold? And can it indeed be used to show the BH shrinks?
 A: You pretty much solved the problem yourself. I'll sketch the argument given on Wald's Quantum Field Theory in Curved Spacetime and Black Hole Thermodynamics, Sec. 7.3. The idea is exactly what you said, so I think I pretty much just need to write down a couple of equations to make it clearer.
The energy properties of the quantum field that gives origin to Hawking radiation on the spacetime are governed by the expectation value of its energy momentum tensor, $\langle \hat{T}_{ab} \rangle$. Since outside the black hole spacetime is stationary (for simplicity, let us assume a Schwarzschild black hole, so that it really is static), we know the four-current $J_a = \langle \hat{T}_{ab} \rangle \xi^b$ is conserved, where $\xi^a$ is the spacetime's Killing vector. This gives us the notion of conservation of energy you mentioned.
On dimensional grounds, the energy flux seen by an observer at infinity should be $F = \frac{\alpha}{M^2}$, where $M$ is the black hole's mass and $\alpha$ is a constant. This equation should hold for any field whose mass is much smaller than the black hole's, so that we can consider the black hole mass as being the relevant one.
At each point of the evaporation process we can approximate the geometry of spacetime as a Schwarzschild black hole with varying mass $M(t)$, at least while we can assume $M \gg \sqrt{\frac{c \hbar}{G}}$, i.e., at least while the black hole's mass is much larger than the Planck mass and hence the local backreaction effects are small. In this case, we can write
$$\frac{\textrm{d} M}{\textrm{d} t} = - F = - \frac{\alpha}{M^2}, \tag{7.3.4}$$
where the tag refers to the equation on Wald's text. Solving this ODE, we get
$$M(t) = \left[M_0^3 - 3 \alpha t\right]^{\frac{1}{3}}, \tag{7.3.5}$$
which implies the black hole evaporates within time $t = \frac{M_0^3}{3 \alpha}$.
Notice, of course, that the semiclassical approximations hold throughout the entire process. It could be that Quantum Gravity effects come into play when the black hole's mass reaches the Planck scale and stops the process, hence stopping the evaporation process. arXiv: 1703.02140 [hep-th], for example, discusses steps on the derivation where things could go wrong and stop the evaporation process.
