Your intuition is good, but you're mixing up some quantum and classical phenomena.
In classical (i.e. non-quantum) physics, a vacuum is a region of space with no matter. You can have electromagnetic fields in a vacuum, so long as the charges creating the fields are in a different region. By the same token you can have gravitational fields in a vacuum, generated by masses somewhere else in space. In this classical description of the universe, there are no such things as photons or gravitons, and everything (for the most part) works out.
In quantum physics, the story is not so easy. As you say, now our force fields are particles, too (photons and gravitons), so maybe a "quantum vacuum" shouldn't include them either? Unfortunately, it turns out that in quantum mechanics (as rob pointed out) it is impossible to have a perfect vacuum, a state with no particles in it at all. One way to see this is through the energy-time uncertainty principle: $\Delta E \ \Delta t > \hbar/2$.
A perfect vacuum, a state with no particles at all, must have exactly zero energy. If the energy is exactly zero, then it is completely certain, and $\Delta E = 0$ which violates the uncertainty principle. So the quantum vacuum is not a state with zero particles, it is a state with probably zero particles. And in different situations you may find useful to alter your definition of "probably," so there are a lot of different things physicists will call a "vacuum" in quantum mechanics.
This idea, that quantum mechanically there are always some particles around in any region of space, has some cool consequences that we've verified in the lab! One is the Casimir Effect. This is a force that shows up when you move two objects in a vacuum so close together the pressure from these "virtual" photons causes them to attract. Another is the particle they discovered at the LHC, the Higgs Boson. The Higgs field has a "vacuum expectation value," a perfect quantum vacuum will have a non-zero Higgs field throughout it. Excitations of this field are the Higgs particles found at the LHC!