I know that it must have something to do with (gauge?) theories with N=2 supersymmetry, BPS states and even black holes, but most papers on the subject are too technical for me. So what is wall crossing?
Wall crossing is any discontinuous change of an integer (or at least rational) quantity - or, more generally, any qualitative change of the spectrum etc. - that occurs when one moves to the opposite side from a "wall" in a moduli space or parameter space.
A would-be index may suddenly change discontinuously.
The wall - a codimension one locus in the parameter space - is then referred to as the "wall of marginal stability". One one side, an object may be stable while it is unstable on the other side. It is typically unstable because a decay suddenly becomes plausible because the hypothetical decay product get light enough, if you wish.
The objects are typically BPS objects on the stable side and they can also be BPS black holes or any other BPS objects.
To see objects that may be exactly stable, BPS, but that are also sufficiently diverse, $N=2$ supersymmetry or eight supercharges is an ideal number of supercharges. That's why those considerations played an important role in the Seiberg-Witten insights about $N=2$ gauge theories, among related situations.
Paul Aspinwall liked to say that 8 supercharges is the optimum number for nice physics. Only Nature didn't manage to choose the number 8 for supercharges, but that's not Aspinwall's fault.