The paper by Briel et al uses the formulae in Henry & Henriksen (1986): they start with the spatial electron number density (eq. (2) in Henry & Henriksen) $$ n_e(r) = n_0\left(1 + \left(\frac{r}{a}\right)^2\right)^{-3\beta/2}. $$ The cluster gas produces thermal bremsstrahlung, which has an emissivity of the form $$ \epsilon_\text{br}(r) \sim n^2_e(r)\,g(E,T)\,(kT)^{-1/2}\,e^{-E/kT}, $$ where Henry & Henriksen use $$ g(E,T) = 0.9(E/kT)^{-0.3}. $$ The surface brightness is then the total emissivity along the line of sight $z$ and within a certain energy range $[E_1,E_2]$ $$ \begin{align} S(R) &= \int_{E_1}^{E_2}\text{d}E\int_{-\infty}^{+\infty} \epsilon_\text{br}(r)\,\text{d}z \sim I_1I_2, \end{align} $$ with the integrals $$ \begin{align} I_1&= (kT)^{-1/2}\int_{E_1}^{E_2}(E/kT)^{-0.3}\,e^{-E/kT}\text{d}E\\ &= \sqrt{kT}\,\bigg[\gamma(0.7,E_2/kT) - \gamma(0.7,E_1/kT)\bigg] \end{align} $$ and $$ \begin{align} I_2 &= \int_{-\infty}^{+\infty}n^2_0\left(1 + \left(\frac{R}{a}\right)^2+ \left(\frac{z}{a}\right)^2 \right)^{-3\beta}\,\text{d}z\\ &= an^2_0\left(1 + \left(\frac{R}{a}\right)^2\right)^{-3\beta+0.5} \int_{-\infty}^{+\infty}(1+u^2)^{-3\beta}\,\text{d}u\\ &= \sqrt{\pi}\frac{\Gamma(3\beta-1/2)}{\Gamma(3\beta)}an^2_0\left(1 + \left(\frac{R}{a}\right)^2\right)^{-3\beta+0.5}, \end{align} $$ where $r^2 = R^2 + z^2$ and $R$ is the projected radius on the plane of the sky; the exact formula for $S(r)$ is given by eq. (3) in Henry & Henriksen. So this is where the formula for the surface brightness comes from in Briel et al: $$ S(R) = S_0\left(1 + \left(\frac{R}{a}\right)^2\right)^{-3\beta+0.5}. $$ Unfortunately, in their notation they use $r$ instead of $R$ for the projected radius.
The total mass inside a radius $r_b$ then follows from the spatial electron density: $$ M(r_b) = 4\pi\,m_\text{A}\int_0^{r_b}n_e(r)\,r^2\text{d}r = 4\pi\,m_\text{A}n_0\int_0^{r_b}\left(1 + \left(\frac{r}{a}\right)^2\right)^{-3\beta/2}\,r^2\text{d}r, $$ where $m_\text{A}$ is the average mass of an atom in the cluster gas. To calculate $m_\text{A}$, we can assume that the cluster gas is made up entirely of hydrogen and helium atoms. We need to be careful though: each hydrogen atom corresponds with one electron, but each helium atom corresponds with two electrons. Therefore, if $\rho$ is the total mass density, $$ \rho = n_em_\text{A}= (n_\text{H} + 2n_\text{He})m_\text{A}, $$ where $n_\text{H}$ and $n_\text{He}$ are the number densities of hydrogen and helium atoms, respectively. Now, let's call $X$ the mass fraction of hydrogen atoms. Then $$ X\rho = n_\text{H}m_\text{H},\qquad (1-X)\rho = n_\text{He}m_\text{He} \approx 4n_\text{He}m_\text{H}, $$ where we used in the last line the fact that the mass of a helium atom is about 4 times the mass of a hydrogen atom. Putting it all together, we find $$ m_\text{A} = \frac{\rho\, m_\text{H}}{(n_\text{H} + 2n_\text{He})m_\text{H}}\approx \frac{2\rho\, m_\text{H}}{2X\rho + (1-X)\rho} = \frac{2m_\text{H}}{1+X}. $$ I don't know what value of $X$ is used by Briel et al, but a common value is $X = 0.768$ (see e.g. eqs. (16) & (17) in Wu et al, 1999). Since $$ m_\text{H} = 1.67\times 10^{-27}\,\text{kg} = 8.42\times 10^{-58}\,\text{M}_\odot, $$ we obtain $$ m_\text{A} = 9.52\times 10^{-58}\,\text{M}_\odot. $$ The other values are listed in Briel et al: $$ \begin{align} n_0 &= 2.89\times 10^{-3}\,h_{50}^{1/2}\,\text{cm}^{-3} = 8.49\times 10^{70}\,h_{50}^{1/2}\,\text{Mpc}^{-3},\\ r_b &= 5\,h_{50}^{-1}\,\text{Mpc},\\ a &= 0.42\,h_{50}^{-1}\,\text{Mpc},\\ \beta &= 0.75, \end{align} $$ which indeed gives $$ M(r_b) = 5.1\times 10^{14}\,h_{50}^{-5/2}\,\text{M}_\odot. $$
Update: the relation between angles and intrinsic size in Coma:
At the time this article was, written, its redshift was measured to be $z=0.0235$ (Sarazin et al, 1982). All distances were also given in terms of $h_{50}$, a dimensionless constant defined as $$ H_0 = 50h_{50}\,\text{km}\,\text{s}^{-1}\,\text{Mpc}^{-1}. $$ In other words, a value of $h_{50}=1$ corresponds with a Hubble constant of $H_0 = 50\,\text{km}\,\text{s}^{-1}\,\text{Mpc}^{-1}$. For a modern-day value $H_0 = 68\,\text{km}\,\text{s}^{-1}\,\text{Mpc}^{-1}$, you get $h_{50}=68/50=1.36$.
From Hubble's Law, we get the co-moving distance to Coma: $$ D_c \approx \frac{cz}{50h_{50}\,\text{km}\,\text{s}^{-1}\,\text{Mpc}^{-1}} = 141h_{50}^{-1}\,\text{Mpc}. $$ But to convert angles into intrinsic sizes, we need to use the angular diameter distance, which is a small cosmological correction: $$ D_A = \frac{D_c}{1+z} = 138h_{50}^{-1}\,\text{Mpc}. $$ Therefore, $$ a = 10.5' = 0.00305\,\text{rad} \rightarrow aD_A = 0.42h_{50}^{-1}\,\text{Mpc}, $$ which is given in the first paragraph on page L33 in Briel et al.