Part of the work my research group does involves calculating mass-loss rates for hot, massive main sequence and post-main sequence O stars ($\gtrsim20M_{\odot}$), so I'll say a little bit about our methods.
The radiation-driven winds in these stars produce shocks through a phenomenon known as the line-driving instability. The shock heating in turn leads to x-ray emission, which is also influenced by absorption within the wind itself; this absorption is the key to how we determine the mass-loss rate of the star. The resulting spectrum has a number of key lines in the $5Å<\lambda<25Å$ range, which we typically fit with several free parameters: elemental abundances (we typically let O, N and C vary), the radius at which emission begins, and a parameter called $\Sigma_*$ defined by
$$\Sigma_*=\frac{\dot{M}}{4\pi R_*v_{\infty}}$$
where $R_*$ is the stellar radius and $v_{\infty}$ is the speed of the wind as $r\to\infty$. ($\Sigma_*$ is proportional to optical depth $\tau_*=\kappa \Sigma_*$, which is what affects absorption at different points in the wind.) $R_*$ and $v_{\infty}$ are typically already known, so once $\Sigma_*$ is fit, we get an easy expression for $\dot{M}$. Cohen et al. 2014 goes into the basic methodology in more detail than I've provided here, but this is essentially how we do it (they use $\tau_*$ instead of $\Sigma_*$).
Warrick cited Vink et al. 2001 as one of the major works on theoretical mass-loss rates for these stars (if not the key work!), and it certainly is. That said, over the past decade and a half there have been a number of indications that those predictions are inconsistent with observations, well beyond uncertainties. Vink et al.'s models predict mass-loss rates for many of these stars of $\sim1$-$2\times10^{-6}M_{\odot}\text{ yr}^{-1}$, while various diagnostics (involving not just x-ray but also UV and optical measurements) shift that downwards by a factor of 2-3 (Puls et al. 2006). $\zeta$ Ori is an example I like to quote, with $\dot{M}_{\text{theory}}=1.2\times10^{-6}M_{\odot}\text{ yr}^{-1}$ and, in sharp contrast, $\dot{M}_{\text{obs}}=3.4\pm0.6\times10^{-7}M_{\odot}\text{ yr}^{-1}$. The cause of the discrepancy is still unknown.