A good question touching the deep physical concepts. I will try to give an answer in several steps.
- Before answer your question let me answer another one: what is a free particle? A simple (but complete) answer is that a «particle» is a long-living elementary excitation of a system (a quantum field). «Long-living» means living long enough to be observable, i.e., weakly interacting with other excitations, «elementary» means that the the total spectrum of the system is (approximately) different sums of energies of such excitations.
You correctly noted that the energy of the bound state is less then the energies of its «free» components. Let’s for example consider a positron-muon bounded state (a muonic atom). As it was correctly noticed by John Rennie you can take these two fermions in a continuous spectrum and they can form the muonic atom with some amount of electromagnetic radiation.
- The interaction of elementary excitations (particles) can be characterized by a some constant usually called a «coupling constant». As I noticed before for a «long-living» excitation the interaction should be week thus the coupling constant should be small.
Our muon and positron can form a bound state because of the interaction with electromagnetic field. The corresponding coupling constant is usually referred as the fine-structure constant and denoted as $\alpha$. If the coupling constant is small enough then the binding energy is proportional to some power the coupling constant, e.g., the binding energy of the muonic atom is approximately $-m_{0}c^2(4\pi\alpha)^2/(2\,n^2)$, where $m_{0}=m_{\mu}m_{e}/(m_{\mu}+m_{e})$ is the muon-positronium reduced mass. For the sake of simplicity let's assume that $m_{\mu}=m_{e}=m$.
Hence your question can be reformulated as follows: what happens if we would increase (by any means) the coupling constant $\alpha$ so that the binding energy becomes to be comparable with (or greater than) $2m$?
- Here I would like to notice that you can produce muonic atoms from the vacuum. For example, an accelerated conductor (or any media) produces photons, e.g., via the dynamical Casimir effect. Photons can interact and produce electron-positron pairs, muon-antimuon pairs and (if there is enough energy) pairs of muonic and anti-muonic atoms. The quantity which characterize the intensity of such production is usually called a «production cross section».
If you start to increase the coupling constant then the production cross section also grows. Since the muonic atoms become lighter and lighter, the energy threshold for the muonic atom production decreases. Thus your system (universe, whatever) becomes very easy excitable, any process leads to the production of the enormous number of very light excitations. And if you will continue to increase the coupling constant so that the binding energy will be greater than $2m$, then the ground state (vacuum) of your system becomes unstable, e.g., all photons in your universe immediately decay into fermions, these fermions form the bound states (atoms), these bound states fill the whole volume of the universe and form Bose-Einstein condensate. Such scenarios are referred as critical phenomena or phase transitions. The main feature of these transitions is that they lead to a total spectrum transformation. After the transition of a critical point you will find a new ground state (vacuum) and new elementary excitations (particles) with new masses and new coupling constants.
