Well, that is quite a big and important topic in fusion research. One way to categorize all the possible instabilities is their origin, i.e. what is responsible for driving them. We have basically two mechanisms there:
- The plasma current
- The plasma pressure
In both cases it is their gradient which can drive a number of instabilities. (Note that the current driven instabilities are something you usually do not have to worry about in a stellarator.)
In order to identify unstable or stable regimes, you usually deform (via calculations or simulations) the equilibrium by a small quantity and then calculate the resulting change in the potential energy: if the potential energy decreases the system is unstable, if it increases it is stable (like a ball which either sits on top of a mountain or in a valley: a small change in its position will lead to the ball rolling down all the mountain or just returning to its position). Doing such a stability analysis then yields a number of unstable modes ("modes" because you expand the deformation in modes), deformations of the equilibrium against which the plasma is unstable and that therefore grow.
Since you seem to be more interested in tokamaks, the current driven instabilities can be further separated by either assuming a perfectly conducting plasma with no resistivity or taking into account a finite resistivity (ideal modes vs. resistive modes).
The edge localized mode (ELM) that you mentioned is an example where both, current and pressure gradient are responsible for their occurrence.