How was the mass of the moon first calculated? How was the mass of the moon first calculated? How do we measure it now?
 A: In the history of astronomy there was no correct concept of masses before Newton.
The Greeks had found reasonably accurate volumes of the Earth and Moon, but masses were unkwnown. 
The planetary mass determination relies on Kepler's third law. Newton indirectly measured the mass of the Moon, trying to estimate the ratio between the solar and lunar masses looking at sea tides. 
I suggest this lecture to clarify any doubts (in particular the first two pages).
http://articles.adsabs.harvard.edu//full/2002Obs...122...61H/0000061.000.html
A: Although this might be more suitable for the History of Science and Mathematics page, I thought I would add a few brief details on the physical details behind these measurements.
Newton
The first to calculate (or, estimate) the mass of the moon was probably Newton himself. As mentioned by Baso, Newton used measurements of tides to find the ratio of the lunar and solar masses. This is realised through the tidal force equation:
$$ f_{S} = \frac{2G M_S M_E}{R_E^3},$$
$$f_{M} = \frac{2G M_M M_E}{R_E^3}. $$
Where we take the absolute value. Then Newton considered the case of when the moon was aligned with the sun, and when it was not, to calculate the relative effects. The details can be found in Morin (chapter 10.3). This method, however, estimated a lunar mass significantly larger than the real value ($\approx \frac{M_E}{41}$), even resulting in the Earth-moon centre of mass lying outside of Earth. As Kevin Brown mentions in his website (which I definitely recommend):

The problem is that the height of tides near various land masses is a complicated function of many different factors and resonance effects, so it can’t be used to give a simple estimate of the tidal forces.

Pre-Moon Landing
Prior to the Apollo missions, measurements had to be taken indirectly, and one of the method was to use the fact that the Earth-moon system orbits their common centre of mass, which causes a periodic "wobble" in Earth's solar orbit.
It was observed that every month there is a 6.3'' parallax angle of the sun's position, which repeats periodically. So it became obvious that this could be used to determine the lunar mass. This indeed turned out to be a good method, and was the method until the moon landings.
With the parallax angle known, the distance of the Earth to the CoM, $D_E$ is:
$$ D_E = (1.5 \times 10^{11}) \times \tan(6.3'') $$
Let $D$ be the Earth-moon distance, then:
$$ \frac{M_E}{M_M} = \frac{D}{D_E} -1 = 82.2... $$
Knowing the mass of the Earth from Kepler's third law gives us the mass of the moon.
References:

*

*David Morin - Introduction to Classical Mechanics
With Problems and Solutions (ch 10.3)

*https://www.mathpages.com/home/kmath469/kmath469.htm

*https://en.wikipedia.org/wiki/Parallax
