BH mass is a conserved quantity and cannot escape the horizon of a BH.
Everything you interact with is outside the horizon. Absolutely everything. What you call the mass is a label you give to a certain kind of curvature that is found outside a black hole. What happened is the matter that fell into the black hole left the outside of it curved as it passed through. That is also how the spacetime outside the earth became curved and how the spacetime outside the sun got curved. The matter falling in leaves the spacetime outside curved. That's actually what matter does.
If you have a spherical shell of mass above a planet you have a solution of type $M+m$ above the shell and a solution of type $M$ between the shell and the surface of the planet. These are two different solutions, and the difference is of type $m$ and connecting together the two solutions is exactly what a mass of mass $m$ does. As the shell falls down the region that has type $M+m$ gets larger (it's the part above the shell and the shell falls down).
Yet in mergers some percent (in GW150914 it was about 5 percent, or 3 solar masses) of the binary's mass escaped either before (one of the two BHs horizons) or after the merger started (one horizon merger, from the merged horizons), and turned into gravitational waves.
That's not correct. Let's add another level of accuracy to that shell model. First we'll note that it isn't the mass in matter that connects the two type of solutions together. It is the energy density (and to a lesser extent the momentum density, the momentum flux, the pressure, and the stress, all five together make the Stress-Energy tensor). As the shell falls the energy increases but the type of solution outside stays the same. That's because more energy close in actually connects the two solutions together than is needed to connect them together farther out.
So you could extract energy from that falling shell and send it out. But then it has the same amount of matter, but less energy. And now it is closer too. So it actually connects different solutions together. So you have the planet, the shell layer and above that you have the energy you extracted and sent up. Outside all three you have type $M+m$ under the energy layer you have $m/2+M$ and under the shell you still have $M.$ So as that energy later expands upwards and eventually gets to you (it is so thin and weak now, from being stretched over a huge area that you barely notice it) then you become inside it and so you see the type $m/2+M$ solution. It looks like some "mass" is gone from the solution but every atom is still there.
What's the point in all that. Basically the gravitational "mass" of an object is not the sum of the masses of all the parts. And indeed it can be less than the sum of the masses of the parts. As in our example.
Even without gravity, the mass of a system can be wildly different than the sum of the masses of the parts. But the thing you called a mass in a gravitational system was probably an energy anyway.
So now the two black holes have a mass that can be less than the mass of the holes. Great. And energy could be extracted from that the same way. If the parts are moving we can slow them down while they are getting closer. Just like we did with the shell. Every atom of the objects might still be there. But we steal their energy and so the system they are a part of can have a smaller $M$ parameter.
The question is how did the BHs mass get reduced?
The system of two black holes has a mass parameter that is not the sum of the parameters of the parts. No real system has a mass parameter that is exactly the sum of the parameters of the parts. The system has some parts that get closer and that move. By slowing them down relative to each other you can reduce the parameter of the system. And that is what happens. Sure, they speed up. But they get slowed down compared to how much they would have sped up without the waves.
Did some virtual negative mass particles fall into the BH
No. And mass isn't the sum of the masses of the parts, so negative mass particles and a conserved additive mass isn't the way to think about anything.
So, if not that, how does the negative potential binding mutual gravitational energy of the two BHs steal mass from the binaries?
Without waves, the two objects orbiting would be a combined system wih it's own rate of rotation $J$ and size $M$ and amount of curvature. It's a type of curvature that could be sourced by fast rotating things that are closer in, or by slower rotating things that are farther out.
They start out as farther away things rotating slower. That's fine. When they move closer they could have moved in a way that produces he same $M$ and $J$ type curvature and that would requiring having some more rotation and some more kinetic energy. To have the exact same $M$ and $J$ they would have to have very specific additional energy and rotation.
But when they emit the waves they end up slower than that target speed. Do the solution outside of them is less. And it's just like the example of sending that layer of energy upwards. It's the solution in between that has a smaller parameter. As the waves go out we see the old $M$ and $J$ solution until the waves get to us, then we are underneath that layer and we see the smaller $M$ solution.
Please, no hand waving. I've looked everywhere and only see hand waving arguments about potential or other energy outside the BHs, but mass is an internal (inside the horizons) and conserved quantity of BHs.
Mass is not an internal quantity (it is a label assigned to spacetime regions outside horizons). And it isn't conserved either. Or even additive. If you insist it is when it isn't then you won't be able to learn correct physics. And energy isn't conserved in general relativity so I'm not sure what you are aiming for anyway.
angular momentum, can be extracted (i.e., escape the EH). Angular momentum through the Penrose process, maybe charge too. Is it the same for mass?
When you extract angular momentum by slowly down the parts then you also extract energy, which is what the parameter $M$ generally correlates with.
