Let's start by considering an extreme: What if we removed all of the vacuum's entanglement?
In relativistic QFT, every state of finite energy is entangled with respect to location. Conversely, any state that is not entangled with respect to location has infinite energy. To get all the entanglement out (if that concept even makes sense at all), we would need to put infinite energy in.
By the way, since you mentioned quantum gravity... The fact that all finite-energy states are entangled in relativistic QFT can be viewed as a faint reflection of something even more profound. According to specific insights from the AdS/CFT correspondence (the Ryu-Takayanagi formula) and more generally from the holographic principle, entanglement is an essential ingredient in the emergence of spacetime itself. If you stray too far from the special pattern of vacuum entanglement that gives ordinary physics in classical spacetime, then spacetime itself becomes disconnected at best, and completely meaningless at worst. This is reviewed in arXiv:1609.00026. You can find lots of papers about it by searching for the keywords Ryu-Takayanagi if you're in the mood for something relatively tame and methodical, or ER=EPR if you're in the mood for some bold extrapolations.
Anyway, the short answer to the question is that in order to harvest entanglement from the vacuum, you have to put something in, like energy. For any reasonable amount of resources (e.g. energy) expended, the effect on the vacuum's entanglement will be negligible. Also note that the degree of entanglement in the vacuum falls off like a power of the distance between the two devices in the entanglement-harvesting pair, so the farther apart they are, the less effective the harvesting will be. I have not actually quantified this myself (I'm speaking from general experience with related things), but you can find more detail in arXiv:1506.03081.
The message here is similar to the message emphasized in Haag 1992, Local Quantum Physics, about another theoretical consequence of vacuum entanglement: Given unlimited resources, it allows us to create any object anywhere in the universe using strictly localized operations right here in our own neighborhood. Of course, we don't have unlimited resources, and when we actually quantify what can be done with reasonable resources, it's nothing very exotic.
Don't get me wrong — entanglement harvesting is still interesting, whether or not it ends up having any practical applications.