According to inflation, all the structure that we see like galaxies, etc. originated from quantum fluctuations which were expanded during inflation. But since quantum fluctuations always create particle-anti particle pairs, doesn't that mean that at least the vast majority of matter formed would have it's corresponding anti matter within our visible universe? If this was the case wouldn't we constantly see lots of cosmic explosions from colliding masses of matter and anti-matter? Can someone please tell me where my understanding of these concepts faltering?
The matter-antimatter asymmetry requires the three Sakharov conditions to be satisfied. I'll summarise that link's explanation. Unfortunately, your question isn't completely solved.
The first condition is that some interactions don't conserve baryon number (I.e. baryons minus antibaryons, baryons being three-quark hadrons such as protons and neutrons). How much asymmetry exists is an open question, as it depends on unverified conditions such as supersymmetry and grand unification.
The second condition, CP violation, has been observed since 1964. However, it's still not known how there's enough of it to explain the observed asymmetry. As with the previous condition, the Standard Model doesn't explain all of the effect, but hopefully new physics soon will.
The third condition is out-of-thermal-equilibrium thermal conditions. Early inflation would ensure this.
Adding to J.G. answer, it should be noted that Baryon number violation is possible within the Standard Model. This has been shown by t'Hooft in the 70's and this involves non-perturbative effects. Necessarily so since the Electroweak Lagrangian has a global B symmetry and therefore this carries to any Feynman diagram. But this does not prevent the dynamic of the model to break B. Actually t'Hooft processes break B+L but not B-L, where L is the lepton number. The issue is then quantitative: do we get enough of B-violation? As J.G. explained, this must happen out of thermal equilibrium, so there is the added difficulty of correctly computing the dynamic. Still very much an open question.