This is kind of a tough question to fully address, since it is fairly broad. But the essence of the answer is that if the conditions are right to allow energy to turn into mass, we can mathematically give the probability that this will happen (these mathematics have, of course, been verified by experimentation).
For starters, as you pointed out, energy and mass are two sides of the same coin. But when jumping from one form to another, energy has to be conserved. This means you'll only get energy turning into a massive particle if there's at least as much energy in a given place as what would be in that particle according to the $E=mc^2$ relation. You can't get something from nothing, so if there's not enough energy, it won't turn into mass.
The second thing to consider is what type of energy is lying around. Different types of energy relate differently to mass. For instance, energy in the form of photons relates to mass very well; that is, it is very compatible with transforming into massive particles (so long as all the conditions are met). On the other hand, as far as we know, there doesn't seem to be any physical mechanism allowing dark energy to transform directly into matter (it's also got a more or less constant energy density which is, quite frankly, too low to produce matter in a small region).
Furthermore, one has to consider all of the conservation laws. For every particle you create, you need to make an anti-particle as well. Momentum has to be conserved within the system, as does charge (electric, colour, whatever type of charge). Momentum conservation is one of the prime reasons you can't have a random photon in free space just spontaneously turning into a couple of particles (you need another photon or charged particle nearby).
Once you satisfy all the conditions of any given situation, then you start having a chance that energy will turn into mass. Of course, the more energy you have around, the higher the chance that some of it turns into matter. But that's all it is, a chance. There's no certainty that at a given point the energy will turn into particles. You can do the math and reasonably expect to see it happen in a certain range of time or around a certain energy level, but it's all random; down to probabilities. Sometimes those probabilities are high, like when you use the potential energy from separating a quark-antiquark pair to create another pair. Other times the probability is low, like bouncing a $1.1MeV$ gamma photon off a gold nucleus; can it create matter? Yes. Will it? Probably not on the first photon.
You need the situation to be just right; there has to be enough energy available of the right type and conditions have to be perfect to allow for all your conservation laws to be upheld. If you have all that, then it boils down to probability. Basically random chance.