Suppose for the sake of argument that you have an object in a vacuum. According to QFT, virtual particle pairs pop in and out of existence constantly. Given enough time, would it be reasonable to expect the object to be broken down by the occasional stray antiparticle? I'm sure it would take billions of years, but would it be possible?
According to QFT, virtual particle pairs pop in and out of existence constantly.
No, they don't. Virtual particles are just lines in a Feynman diagram, not "particles" with actual existence, and the notion does not occur in non-perturbative approaches at all. See this answer of mine, this question and its answers, this question and its answers and their linked question for a host of related discussion that all boils down the fact that the popular claim that particles "pop into existence" is not actually supported by the QFT formalism.
A stable particle in a vacuum will not experience any change at all, QFT or not. However, truly stable particles are rare - even for the proton we are still searching for possible decays, although the lower bound on its average lifetime is rather high by now, and e.g. a neutron decay with a half-life of about 10 minutes. If your particle is anything but an electron - which has no decay channel on its own due to having the lowest mass among all known charged particles - it is conceivable, or even likely, that it will eventually decay.