the black hole shrinks due to negative energy of the particle it absorbs when the other particle escapes but it does not matter whether or not it was the matter of the antimatter that was absorbed.
For a virtual pair to become real particles, lets take an e+e- pair, energy must be provided by an energy source. In the simple pair creation by a real gamma ray hitting the field of a nucleus, the energy for the creation of e+e- is provided by the real gamma ray while the virtual from the field of the nucleus balances momentum .
In the case of the black hole the virtual pairs must get the energy from the great potential well that the BH is . When the random quantum mechanical fluctuations at the BH horizon give enough energy for the materialization of an electron or positron and enough energy that it escapes the hole, the energy comes from the potential well of the hole. This energy must be subtracted, that is why it is called negative, from the total potential to keep energy conservation at the horizon locally.
Is the Black Hole losing mass if it absorbs only the normal matter half of a virtual particle and the antimatter half escapes?
The electron and the positron have the same mass, positive, same is true for all particles antiparticles. As in relativistic energies E=mc^**2 it makes no difference if it is a particle or antiparticle that becomes real, energy will be subtracted from the total of the BH.