Does the mass variable $M$ in the Schwarzschild radius equation $R = 2GM/c^2$ refer to rest mass or relativistic mass? Does the mass variable $M$ in the Schwarzschild radius equation $R = 2GM/c^2$ refer to rest mass or relativistic mass?
 A: Relativistic mass is pretty much a deprecated concept.  Essentially any reference to mass in modern physics refers to rest mass, because it is invariant regardless of the situation.
To elaborate a little, perhaps the first statement of "relativistic mass" can be found in Einstein's 1905 paper,* which somewhat naively takes Newton's 2nd law in a stationary reference frame:
$$F_x=ma_x$$
$$F_y=ma_y$$
and transforms it to a reference frame moving with velocity $v$ in the $x$ direction with respect to the first, to obtain:
$$F_x = m \gamma^3 a_{x'}$$
$$F_y = m \gamma^2 a_{y'}$$
where the gamma factor  $\gamma = (1-(v/c)^2)^{-1/2}$.  Using the $F=ma$ formula produces an "x direction mass" of
$$ m (1-(v/c)^2)^{-3/2}$$
and a "y direction mass" of
$$ m (1-(v/c)^2)^{-1}$$
($m$ being the rest mass).
So there is not even really a consistent definition of "relativistic mass," as it is different depending on the direction the force is applied. The difficulty here is resolved by observing that  $F=ma$  cannot be applied to relativistic objects consistently.
Still a third possible definition of relativistic mass, the one most commonly seen, comes from the formula for total energy (rest + kinetic) of a moving particle:
$$E=\gamma mc^2=m_{rel}c^2$$
which is useful for certain calculations, but modern physicists would use the $\gamma mc^2$ formula, which separates out the invariant mass $m$ and the relativistic factor $\gamma$.  This formula shows clearly that $m_{rel}$ is simply the total energy $\frac E {c^2}$ within a constant factor, and so is a redundant concept.
*On the Electrodynamics of Moving Bodies, Sec. 10
A: This is basically covered in the question If a mass moves close to the speed of light, does it turn into a black hole? If $M$ were the relativistic mass we could make things turn into black holes just by making them move fast. Since this isn't the case $M$ has to be the rest mass, sometimes called the invariant mass.
I would echo RC_23's point - no physicist working today would refer to the relativistic mass as it's a misleading concept that was abandoned decades ago. For more on this see Why is there a controversy on whether mass increases with speed?
Just to add further confusion, the Schwarzschild and Kerr black holes actually contain no matter at all. They are both vacuum solutions. In these cases the parameter $M$ is actually the ADM mass and this is a geometrical property of the spacetime.
A: *

*Firstly, the Schwarzschild mass $M$ is a parameter with dimension of mass in the Schwarzschild solution.


*Secondly, the physical meaning of the mass $M$ is that

*

*for a small mass probe $m$ far away the gravitational attraction matches Newton's non-relativistic gravitational formula, cf. e.g. my Phys.SE answer here.


*it is the ADM mass.


*The corresponding Schwarzschild energy $E=Mc^2$ can in principle be given an interpretation as the sum of the total energies of infinitely many, initially well-separated, infinitesimally small masses, infalling from spatial infinity. (More precisely, let us consider infinitely many, initially well-separated, concentric, infinitesimally thin mass shells, infalling from spatial infinity, to preserve spherical symmetry, in keeping with Refs. 1-2. Be aware that building a black hole from scratch is easier said than done, cf. e.g. this Phys.SE post.)




*Usually we describe a Schwarzschild black hole in a coordinate system, where the black hole is at rest. As far as trying to boost the black hole solution, the parameter $M$ should be though as an invariant mass, cf. OP's title question.
References:

*

*Eric Poisson, A Relativist's Toolkit, 2004; Section 3.9 eq. (3.70).


*Eric Poisson, An Advanced course in GR; Section 3.9 eq. (3.9.3).
A: I believe that the answer posted by @Qmechanic is the most accurate answer to this question. I would further like to add that while $M$ is just a parameter in the Schwarzschild metric, its physical interpretation  will depend on the definition of mass one uses to calculate this quantity from the metric itself. The notion of mass in general relativity is not a well understood concept and this due to the fact that the gravitational energy is non-local. For instance, it can be shown that outgoing gravitational radiation from a source will decrease its mass [3]. However, that gravitational radiation has no stress-energy tensor of it's own (i.e. non-local). So, rather than depending on local densities, the parameter $M$ of the source will depend on certain global properties of space-time and this is where ambiguities and non-uniqueness in the definition of mass arises (see [1] for more details).
One could interpret $M$ to be the ADM mass, Komar mass, Bondi mass (in the case of outgoing/incoming radiation) or even as a quasi-local mass , e.g. in Penrose's definition of quasi-local momentum-angular momentum definition. In Penrose's construction, one could write the "mass" in a way that it appears to be a relativistic invariant mass (i.e. of the form $M=P^aP_a$, where $P$ is calculated from definition of conserved charges (see [2])). However, if you use the same definition for a charged black hole metric, you will get $M_{NR}=M-\frac{e^2}{2r}$. On outer horizon this is $M_{NR}=\frac{1}{2}r_+$, i.e. the irreducible mass. For Schwarzschild, $e=0$, thus $r_+=r_s=2M$. So, in this sense, the Schwarzschild mass $M$ can be interpreted as the irreducible mass.
[1] Szabados, L.B. Quasi-Local Energy-Momentum and Angular Momentum in General Relativity. Living Rev. Relativ. 12, 4 (2009). https://doi.org/10.12942/lrr-2009-4
[2]Chapter 13 from https://doi.org/10.1017/CBO9780511624018
[3] https://doi.org/10.1098/rspa.1962.0161
A: In my understanding, the mass $M$ in Schwarzschild's radius definition is the sum of all rest masses of material objects that created black hole. Accordingly, it refers to rest mass.
