My guess is that they find a certain tracks coming from a certain source by "combintaorially" selecting all track pairs and finding their invariant mass. If this is true, of which I am not sure, how is the background defined, i.e how far away from the invariant mass peaks? Any references will help too.
Especially in hadron colliders, often-times you have a lot going on in a single event. There might be multiple jets (which originate from quarks or gluons, which cannot survive alone, cf. confinement etc.).
Many of the particles one would be looking for in such events have a very short life-time, such as a top quark, a W or Z-boson. Thus, as opposed to more obvious things in an event, one has to reconstruct the existence of those particles by correctly identifying their decay product.
For example, a W-boson can decay into two quarks, which will form jets. Now, if you have many jets in an event, which of those two will you pick? Probably the ones whose combined invariant mass is near the W-mass. But still, since unfortunately the objects in the event don't come with labels, we cannot be sure these jets are the correct combination.
Another example would be a top-antitop-event. The top decays, almost to a 100%, into a W boson and a b quark (which will become a jet). Each W can either decay leptonically or hadronically. So in a all-hadronic top-antitop event you have 6(!) jets you have to combine correctly to reconstruct the event. Even if you're looking at a pure sample of top-antitop events (like in a Monte-Carlo simulation), you are bound to get those combinations wrong.
These things are what is called combinatorial background.
In a collision lots of particle are produced . Very often one looks for a particle not directly but through its decay products (look at particle data book for the known particles and their decays). Asan example consider jpsi decay in a muon pair. In each collision one look for a pair of muons. Depending on the collision energy there will be less or more than one muon per event but anywayone will construct all the possible pairs of muons in each event with the muons measured in the event. If the muons come from a jpsi, like fragments of a bomb explosion they will carry the energy of the jpsi, so one calculate the total energy of each pair (its mass). Pions are copiously produced in the collision and they have a probability of decay into muons. This makes another way to produce a pair by random combination of muons produced independantly by pions or other sources. This is a typical combinatorial background. If it leads to the same mass there is no way to separate it from a psi decay. But globally these combinations will lead to a distributions of masses so one will be able to separate the psi peak from the much broader combinatorial contribution. this combiatorial problem is more general: each time one look for a specific pair one will have a certain probability a random combination of other products will look similar