Galilean invariance of Newton's universal gravitation law

So, I was reviewing special relativity and this question came across: How can one prove that Newton's universal law of gravitation is invariant under transformations of the galilean group?

As far as I know, one should take the law $$m\frac{\partial ^2 \vec{x}}{\partial t^2}=-G\frac{Mm}{|\vec{x}|^2}\frac{\vec x}{|\vec x|}$$ and then apply on $$\vec{x}$$ a transformation of the galilean group as in: $$\vec{x} \rightarrow \vec{x}' = t\vec{v} + R\vec{x} + \vec{d}$$ Where $$\vec{d}$$ is the displacement vector, $$\vec{v}$$ is the speed of the new reference system in regard to that in rest and R is a rotation matrix. On the other hand, time must transform as: $$t \rightarrow t'=t+s$$ Where $$s$$ is a real number.

On the lefthand side of the equation, it seems easy to apply, since the time derivative doesn't change and eliminates the parts with $$\vec{v}$$ and $$\vec{d}$$, but I can't figure out how to proceed on the righthand side of it... Any ideas on how could I prove that?

I think the misleading part is that $$\vec{x}$$ on the right hand side is the relative position of the two masses $$m$$ and $$M$$ so it actually reads

$$\vec{x}=\vec{x}_m-\vec{x}_M$$ and Newton's equation would read

$$m\partial_t^2\vec{x_m} = -G{mM \over |x|^3}\vec{x}$$ (and the same for $$x_M$$ on the LHS for the other mass)

To transform $$\vec{x}$$ you need to transform $$\vec{x}_m$$ and $$\vec{x}_M$$ separately as

$$\vec{x}_m'=t\vec{v}+\vec{d}+R\vec{x}_m$$ and $$\vec{x}_M'=t\vec{v}+\vec{d}+R\vec{x}_M$$

and then find the new relative position, so that now $$\vec{x}'=\vec{x}_m'-\vec{x}_M'=R(\vec{x}_m-\vec{x}_M)=R\vec{x}$$

because $$R$$ is the rotation matrix, you are basically just rotating the relative positions vector, so $$|x'|=|x|$$ i.e. the magnitude does not change (its just a rotation).

[The following part contains a bit of hand-waving about rotation matrices and relative positions - but the result should be correct]

On the other hand, $$\vec{x'}$$ now points in a different direction defined by $$R$$ but you also have $$R$$ being applied on the left hand side so that (using the fact that as you said the derivative does not change) and applying it in the case of $$\vec{x}_m$$ (is the same for $$M$$):

$$\partial_t^2\vec{x_m}'=\partial_t^2R\vec{x_m}=R\partial_t^2\vec{x_m}$$ as the rotation is constant - it basically just rotates the derivative too - so you get

$$mR\partial_t^2\vec{x_m} = -G{mM \over |x|^3}R\vec{x}$$ which you might rewrite as $$R \left(m\partial_t^2\vec{x_m} \right) = R \left(-G{mM \over |x|^3}\vec{x} \right)$$

which is exactly the same as before minus a rotation of the reference frame, which acts on both sides so the equation is formally the same as before.

Indeed, the way that you have stated it, the law seems like it is not invariant under Galilean transformations!

Remember that the Universal Law of Gravitation holds between two particles that are interacting with each other. For convenience, one of them (say $$M$$) is usually taken to be at the origin, and therefore the force exerted by it on another mass $$m$$ is supposed to be (in one dimension): $$m \frac{\text{d}^2\mathbf{x}}{\text{d}t^2} = - \frac{G M m}{x^2}\hat{\mathbf{x}}.$$

The reason that your formulation looks like it's not invariant is because you have forgotten the second mass in the problem! As a result, you are applying the transformation to one mass, while the other stays at the origin. This is not a Galilean boost, you're just giving one mass a velocity $$v$$!

Of course the more general form is when the two masses (say, $$m_1$$ and $$m_2$$) are at positions $$x_1$$ and $$x_2$$, respectively, in which case the force on mass "2" for example would be:

$$m_2 \frac{\text{d}^2\mathbf{x}_2}{\text{d}t^2} = - \frac{G m_1 m_2}{(x_1 - x_2)^2}\hat{\mathbf{x}}_{12},$$

where $$\hat{\mathbf{x}}_{12}$$ is the vector directed from mass 1 to mass 2.

You should now be able to show that under a Galilean transformation, $$x_1$$ and $$x_2$$ transform in such a way as to keep $$|x_1 - x_2|$$ constant.