As @honeste_vivere said, your question is related to a very broad field in fusion research, that is Magnetohydrodynmic (MHD) stability.
You asked basically two questions:
- What limits the plasma-ß ?
- What is causing instabilities (those which limit the plasma-ß) ?
Let's first look at question 1: in general there are two things limiting the plasma-ß, (a) the equilibrium limit and (b) the stability limit.
As for (a), this requires a bit of explanation: as you probably know, you need twisted magnetic field lines in a torus to get confinement; a simple magnetized torus, where you have a purely toroidal field, has no confinement.
The reason why you need this twist is to cancel the charge accumulation which would arise in a purely toroidal field (due to the diamagnetic current). You cancel those accumulations by twisting the magnetic field lines such that areas of positive and negative charge accumulations are connected and thus currents are flowing to cancel the accumulations. These currents flowing along the magnetic field lines are called Pfirsch-Schlüter-currents (PS-currents) and they lead to a shift of the flux surfaces to the outboard side (due to the accompanying magnetic field generated by the PS-currents).
This shift, called Shafranov-shift, increases with the plasma pressure and thus with the plasma-ß (since higher pressure implies higher diamagnetic currents, thus larger PS-currents, resulting in a larger shift of the flux surfaces). If the shift is equal to the minor plasma radius, the equilibrium limit is reached: no further shift of the central flux surfaces is possible, as the minor plasma radius denotes the boundary of the plasma.
As for (b), you look at instabilities limiting your plasma-ß, which is also your second question. These instabilities are usually distinguished by the drive, (a) current gradient driven, and (b) pressure gradient driven. Obviously, the current driven instabilities are only important for tokamaks where you have a large toroidal current flowing in the plasma, necessary to create part of the confining magnetic field.
When you do the stability analysis for the current driven instabilities, one of the main results is a limit on the plasma current, the so-called Kruskal-Shafranov limit. It is important in a tokamak to not surpass this limit, otherwise you trigger a disruption which is the sudden loss of the confinement resulting in a lot of mechanical stress on the vessel (and all attached components).
Limits on the plasma-ß are mostly from pressure-driven instabilities. One way to analyze stability, is to deform the equilibrium (the flux surfaces) and calculate the resulting change in the potential energy. If the change is negative and the potential energy is thus decreased, this implies instability whereas an increase in potential energy implies stability against that deformation.
Since we are now looking at pressure-driven instabilities, we deform the plasma pressure. We furthermore allow for a poloidal variation of the deformation. Due to the difference of good and bad curvature in a toroidal confinement geometry, the perturbations are stronger on the low-field side and smaller on the high-field side. Those pressure perturbations are called ballooning modes and analyzing their stability gives us combinations of pressure gradients and magnetic shear (change of magnetic field inclination with radius) resulting in stable configurations.
The stability analysis for the current driven instabilities imposed some constraints on plasma current as mentioned earlier. Using these limitations together with the knowledge about stability against ballooning modes, we can look at the profiles of plasma current and pressure that maximizes the plasma-ß. This is then the stability ß-limit.