To express a line element in different coordinates, you don't need any transformation formulas because it doesn't transform. It is an invariant, the same in all coordinate systems. So all you do is substitute for the old coordinates and their differentials in terms of the new coordinates and their differentials.
In your example, you want to express the line element
$$ds^2=dr^2+r^2\,d\phi^2$$
in terms of a new radial coordinate, $r'$, where
$$r=\frac{a^2}{r'}.$$
Simply compute
$$dr=-\frac{a^2}{r'^2}dr'$$
and find
$$ds^2=\left(-\frac{a^2}{r'^2}dr'\right)^2+\left(\frac{a^2}{r'}\right)^2d\phi^2=\left(\frac{a}{r'}\right)^4(dr'^2+r'^2\,d\phi^2).$$
There is no need to think in terms of the components of the metric tensor and how they transform. You can do it that way; it's just needlessly harder.
Your introduction of the flat coordinates $x$ and $y$ is not going to possible when you are dealing with a metric for a space with curvature.
An expression like
$$\dfrac{\partial x^\alpha}{\partial x}\dfrac{\partial x^\alpha}{\partial x}g_{\alpha\alpha}$$
where four indices are contracted is never correct. Only two indices ever get contracted together. Of course, you can contract multiple pairs of indices, but you must write them as different symbols so that you can tell what is being contracted with what.
If you really want to transform the metric tensor, then don't introduce an intermediate set of coordinates like $(x, y)$. Go directly from the original coordinates
$$x^\mu=(r,\phi)$$
to the new coordinates
$$x^{\mu'}=(r',\phi)$$
using the transformation rule for a 2-index covariant tensor:
$$g_{\mu'\nu'}=\frac{\partial x^\alpha}{\partial x^{\mu'}}\frac{\partial x^\beta}{\partial x^{\nu'}}g_{\alpha\beta}.$$
For example,
$$\begin{align}
g_{r'r'}&=\left(\frac{\partial r}{\partial r'}\right)^2g_{rr}+2\frac{\partial r}{\partial r'}\frac{\partial\phi}{\partial r'}g_{r\phi}+\left(\frac{\partial \phi}{\partial r'}\right)^2g_{\phi\phi}\\
&=\left(\frac{-a^2}{r'^2}\right)^2g_{rr}\\
&=\left(\frac{a}{r'}\right)^4
\end{align}.$$
You may be used to seeing primes on the vectors or tensors, not on their indices. That notation can end up being more confusing, because it leads to ambiguous-looking things like
${g′}_{rr}$ rather than clear things like $g_{r'r'}$.