Suppose we have two neutron stars of mass $M$, orbiting a common centre at a distance $R$. (i.e. the two neutron stars are $2R$ apart from each other, and $R$ apart from the centre). I've attempted to calculate the total mechanical energy, though I'm not sure which one is the correct answer.
Attempt:
$F=\frac{GMm}{r^2}=\frac{GM^2}{(2R)^2}=\frac{GM^2}{4R^2}$ (Newton's law of universal gravitation).
Consider one neutron star,
$F = Ma = \frac{Mv^2}{R}=\frac{GM^2}{4R^2}$ (centripetal force)
Therefore, $v=\sqrt{\frac{GM}{4R}}$ for one of the neutron stars
Hence, the Kinetic energy of one neutron star is
$K=\frac{GM^2}{8R}$,
making the total kinetic energy of the system:
$$K_{tot}=2K=\frac{GM^2}{4R}$$
Now, this is the part where I'm stuck.
We know that the potential energy of an orbit is given by $U=-\frac{GMm}{r}$, but in this case does $r=2R$ (i.e. the distance between the two neutron stars), or does $r=R$, the distance between a neutron star and its centre?
So depending on which expression for $U$ is correct, we would have either $$U=-\frac{GM^2}{R}$$ or $$U=-\frac{GM^2}{2R}$$
So our total mechanical energy $E=K_{tot}+U$ will be either $$E=-\frac{GM^2}{4R}$$ or $$E=-\frac{3GM^2}{4R}$$
By the way, if it helps, $U$ is defined as the mutual potential energy between the two neutron stars.