In short, they do contribute, but their energy is negative.
As you mentioned, this interpretation (which was presented by Hawking in his original article on the subject) is a bit cartoonish, for we can't really define what we mean by particles in the vicinity of the black hole. However, it provides a more intuitive picture of the process.
In Hawking's original explanation, he mentions that one can see the process by thinking of the creation of pairs of virtual particles (which are the pairs arising from the vacuum that you mentioned), being one with positive energy, one with negative energy. The negative energy particle will at first be outside the black hole, which is classically forbidden (particles can't really have negative energy), but it can tunnel to inside the horizon, where time and space essentially switch places (the coordinate we called $t$ outside the black hole behaves as a spatial coordinate inside it, while the coordinate we called $r$ behaves as a time coordinate). Due to these switching there is no problem with the energy sign anymore and the particle can exist as a real particle inside the black hole. The other particle of the pair, which had positive energy, can exist on the outside without any problem and then escapes to infinity, originating the radiation.
Notice then that energy is conserved in the process: the virtual particle pair is creating conserving energy, the negative energy one is absorbed by the black hole, while the positive energy one escapes. Far away, one can detect the positive energy particle and find out the system has been releasing energy. As a consequence, the black hole is absorbing negative energy and evaporating as a consequence.
Quoting the original paper by Hawking, it is good to remind that
It should be emphasized that these pictures of the mechanism responsible for thermal emission and area decrease are heuristic only and should not be taken too literally.
Hawking radiation is in fact a prediction of Quantum Field Theory in Curved Spacetime, whose fundamental objects are not particles, but quantum fields. Particles can be used to interpret the theory in a few situations, but in general situations (in particular in the vicinity of a black hole undergoing the Hawking effect) they can't even be defined appropriately.