I often hear about a beam of particles on top of a background population in plasma physics. What does exactly beam mean?
Many astrophysical plasmas are, at best, weakly collisional. This means that two populations can stream past each other with few or no collisions. In such a scenario, it is appropriate to define the different populations within each particle species, e.g., electrons in the solar wind are comprised of a core, halo, strahl, and superhalo [e.g., Wilson et al., 2019a].
The most physically meaningful approach is to define the core as the population with the highest phase space density. A beam population is then one that has a lower phase space density and drifts relative to the core.
Is it a subset of particles that belong to a different distribution?
A beam is just a drifting population relative to the core of the same particle species, usually. If there only exists, single, isotropic Maxwellians for both the electrons and ions and they drift relative to each other, those can sometimes be losely referred to as drifting beams but it is not the same phenomena. In this latter scenario, a better description would be to say there exists a current.
What differentiates the background from the beam population?
I am not sure what is meant by background here. If the OP is think about what I would refer to as the core, then see the above comments. If the OP is referring to something like statistical noise, then a beam must have phase space densities above the noise floor to be a beam, must it not?
Side Note
The solar wind core ions are often referred to as a beam by many primarily because the population is cold and fast (i.e., the thermal speed is much less than the bulk flow speed). This type of nomenclature likely derives from lab experiments where the lab is stationary relative to whatever beam is being driving by some device. However, the solar wind core is the core. The drifting alpha-particle population and secondary proton population often found in the solar wind[e.g., Verniero et al., 2020], are examples of beams.