The paper by Briel et al uses the formulae in Henry & Henriksen (1986): they start with the spatial electron number density (eq. (2) in Henry & Henriksen)
$$
n_e(r) = n_0\left(1 + \left(\frac{r}{a}\right)^2\right)^{-3\beta/2}.
$$
The cluster gas produces thermal bremsstrahlung, which has an emissivity of the form
$$
\epsilon_\text{br}(r) \sim n^2_e(r)\,g(E,T)\,(kT)^{-1/2}\,e^{-E/kT},
$$
where Henry & Henriksen use
$$
g(E,T) = 0.9(E/kT)^{-0.3}.
$$
The surface brightness is then the total emissivity along the line of sight $z$ and within a certain energy range $[E_1,E_2]$
$$
\begin{align}
S(R) &= \int_{E_1}^{E_2}\text{d}E\int_{-\infty}^{+\infty} \epsilon_\text{br}(r)\,\text{d}z \sim I_1I_2,
\end{align}
$$
with the integrals
$$
\begin{align}
I_1&= (kT)^{-1/2}\int_{E_1}^{E_2}(E/kT)^{-0.3}\,e^{-E/kT}\text{d}E\\
&= \sqrt{kT}\,\bigg[\gamma(0.7,E_2/kT) - \gamma(0.7,E_1/kT)\bigg]
\end{align}
$$
and
$$
\begin{align}
I_2 &= \int_{-\infty}^{+\infty}n^2_0\left(1 + \left(\frac{R}{a}\right)^2+ \left(\frac{z}{a}\right)^2 \right)^{-3\beta}\,\text{d}z\\
&= an^2_0\left(1 + \left(\frac{R}{a}\right)^2\right)^{-3\beta+0.5}
\int_{-\infty}^{+\infty}(1+u^2)^{-3\beta}\,\text{d}u\\
&= \sqrt{\pi}\frac{\Gamma(3\beta-1/2)}{\Gamma(3\beta)}an^2_0\left(1 + \left(\frac{R}{a}\right)^2\right)^{-3\beta+0.5},
\end{align}
$$
where $r^2 = R^2 + z^2$ and $R$ is the projected radius on the plane of the sky; the exact formula for $S(r)$ is given by eq. (3) in Henry & Henriksen. So this is where the formula for the surface brightness comes from in Briel et al:
$$
S(R) = S_0\left(1 + \left(\frac{R}{a}\right)^2\right)^{-3\beta+0.5}.
$$
Unfortunately, in their notation they use $r$ instead of $R$ for the projected radius.
The total mass inside a radius $r_b$ then follows from the spatial electron density:
$$
M(r_b) = 4\pi\,m_\text{A}\int_0^{r_b}n_e(r)\,r^2\text{d}r =
4\pi\,m_\text{A}n_0\int_0^{r_b}\left(1 + \left(\frac{r}{a}\right)^2\right)^{-3\beta/2}\,r^2\text{d}r,
$$
where $m_\text{A}$ is the average mass of an atom in the cluster gas. To calculate $m_\text{A}$, we can assume that the cluster gas is made up entirely of hydrogen and helium atoms. We need to be careful though: each hydrogen atom corresponds with one electron, but each helium atom corresponds with two electrons. Therefore, if $\rho$ is the total mass density,
$$
\rho = n_em_\text{A}= (n_\text{H} + 2n_\text{He})m_\text{A},
$$
where $n_\text{H}$ and $n_\text{He}$ are the number densities of hydrogen and helium atoms, respectively. Now, let's call $X$ the mass fraction of hydrogen atoms. Then
$$
X\rho = n_\text{H}m_\text{H},\qquad (1-X)\rho = n_\text{He}m_\text{He} \approx 4n_\text{He}m_\text{H},
$$
where we used in the last line the fact that the mass of a helium atom is about 4 times the mass of a hydrogen atom. Putting it all together, we find
$$
m_\text{A} = \frac{\rho\, m_\text{H}}{(n_\text{H} + 2n_\text{He})m_\text{H}}\approx \frac{2\rho\, m_\text{H}}{2X\rho + (1-X)\rho} = \frac{2m_\text{H}}{1+X}.
$$
I don't know what value of $X$ is used by Briel et al, but a common value is $X = 0.768$ (see e.g. eqs. (16) & (17) in Wu et al, 1999). Since
$$
m_\text{H} = 1.67\times 10^{-27}\,\text{kg} = 8.42\times 10^{-58}\,\text{M}_\odot,
$$
we obtain
$$
m_\text{A} = 9.52\times 10^{-58}\,\text{M}_\odot.
$$
The other values are listed in Briel et al:
$$
\begin{align}
n_0 &= 2.89\times 10^{-3}\,h_{50}^{1/2}\,\text{cm}^{-3} = 8.49\times 10^{70}\,h_{50}^{1/2}\,\text{Mpc}^{-3},\\
r_b &= 5\,h_{50}^{-1}\,\text{Mpc},\\
a &= 0.42\,h_{50}^{-1}\,\text{Mpc},\\
\beta &= 0.75,
\end{align}
$$
which indeed gives
$$
M(r_b) = 5.1\times 10^{14}\,h_{50}^{-5/2}\,\text{M}_\odot.
$$
Update: the relation between angles and intrinsic size in Coma:
At the time this article was written, its redshift was measured to be $z=0.0235$ (Sarazin et al, 1982). All distances were also given in terms of $h_{50}$, a dimensionless constant defined as
$$
H_0 = 50h_{50}\,\text{km}\,\text{s}^{-1}\,\text{Mpc}^{-1}.
$$
In other words, a value of $h_{50}=1$ corresponds with a Hubble constant of $H_0 = 50\,\text{km}\,\text{s}^{-1}\,\text{Mpc}^{-1}$. For a modern-day value
$H_0 = 68\,\text{km}\,\text{s}^{-1}\,\text{Mpc}^{-1}$, you get $h_{50}=68/50=1.36$.
From Hubble's Law, we get the co-moving distance to Coma:
$$
D_c \approx \frac{cz}{50h_{50}\,\text{km}\,\text{s}^{-1}\,\text{Mpc}^{-1}} = 141h_{50}^{-1}\,\text{Mpc}.
$$
But to convert angles into intrinsic sizes, we need to use the angular diameter distance, which is a small cosmological correction:
$$
D_A = \frac{D_c}{1+z} = 138h_{50}^{-1}\,\text{Mpc}.
$$
Therefore,
$$
a = 10.5' = 0.00305\,\text{rad} \rightarrow aD_A = 0.42h_{50}^{-1}\,\text{Mpc},
$$
which is given in the first paragraph on page L33 in Briel et al.
