My guess is that they find a certain tracks coming from a certain source by "combintaorially" selecting all track pairs and finding their invariant mass. If this is true, of which I am not sure, how is the background defined, i.e how far away from the invariant mass peaks? Any references will help too.
Especially in hadron colliders, often-times you have a lot going on in a single event. There might be multiple jets (which originate from quarks or gluons, which cannot survive alone, cf. confinement etc.).
Many of the particles one would be looking for in such events have a very short life-time, such as a top quark, a W or Z-boson. Thus, as opposed to more obvious things in an event, one has to reconstruct the existence of those particles by correctly identifying their decay product.
For example, a W-boson can decay into two quarks, which will form jets. Now, if you have many jets in an event, which of those two will you pick? Probably the ones whose combined invariant mass is near the W-mass. But still, since unfortunately the objects in the event don't come with labels, we cannot be sure these jets are the correct combination.
Another example would be a top-antitop-event. The top decays, almost to a 100%, into a W boson and a b quark (which will become a jet). Each W can either decay leptonically or hadronically. So in a all-hadronic top-antitop event you have 6(!) jets you have to combine correctly to reconstruct the event. Even if you're looking at a pure sample of top-antitop events (like in a Monte-Carlo simulation), you are bound to get those combinations wrong.
These things are what is called combinatorial background.