From the graviational law Newton had the relation $$\frac{GM}{R^2}=g \tag{1}$$ The gravitational acceleration at the surface of the earth ($g=9.8\ \text{m/s}^2$) and the radius of the earth ($R=6400\ \text{km}$) were known at Newton's time.

For the average density of the earth there was at least an imprecise estimate. It should be roughly the density of stone, $\rho=3.5\ \text{g/cm}^3$.

Then the mass of the earth is $$M=\rho\cdot \frac{4}{3}\pi R^3 \tag{2}$$

So you have two equations (1) and (2) for the two unknowns $G$ and $M$. And with a little math you can solve these for $G$ and $M$.

From the graviational law Newton had the relation $$\frac{GM}{R^2}=g \tag{1}$$ The gravitational acceleration at the surface of the earth ($g=9.8\ \text{m/s}^2$) and the radius of the earth ($R=6400\ \text{km}$) were known at Newton's time.

For the average density of the earth there was at least an imprecise estimate. It should be roughly the density of stone, $\rho=3\ \text{g/cm}^3$.

Then the mass of the earth is $$M=\rho\cdot \frac{4}{3}\pi R^3 \tag{2}$$

So you have two equations (1) and (2) for the two unknowns $G$ and $M$. And with a little math you can solve these for $G$ and $M$.