Given a Lagrangian using the standard cartesian coordinates. $$ \mathcal{L} = \frac{1}{2}m(\dot{x}^2 + \dot{y}^2) - \frac{1}{2}k(x^2 + y^2) $$
How to move to the hyperbolic coordinates given as $$2 x y = \mu $$ $$ x^2 - y^2 = \lambda $$
One way is to invert the expression of the hyperbolic coordinates and express $x$ and $y $ as a function of $\mu$ and $\lambda$ and substitute it into the lagrangian.
Is there a more direct method which does not involve inverting the function?
What you propose is needlessly busy, as it forfeits the visible symmetries of the system, notably in its quadratic forms.
Instead, note that, in polar coordinates, $$ x=r\cos\theta, \qquad y= r\sin\theta,~~\leadsto \\ x^2+y^2= r^2, \qquad\dot{x}^2+ \dot{y}^2 =\dot{ r}^2 + r^2 \dot {\theta}^2, $$ and further, $$ \mu= r^2 \sin 2\theta , \qquad \lambda = r^2 \cos 2\theta, ~~\leadsto \\ \lambda^2 + \mu^2 = (r^2)^2, \qquad \dot{\lambda}^2+ \dot{\mu}^2= 4r^2 (\dot{ r}^2 + r^2 \dot {\theta}^2)= 4\sqrt{\lambda^2 + \mu^2 } ~~(\dot{ r}^2 + r^2 \dot {\theta}^2). $$ Behold! The trig functions are gone.
Starting with hyperbolic coordinates
$$g1:=2\,x\,y=\mu\\ g2:=x^2-y^2=\lambda$$
from here with $~x=r\cos(\phi)~,y=r\sin(\phi)~$
$$g1:=r^2\,\sin(2\phi)=\mu\\ g2:=r^2\cos(2\phi)=\lambda$$
solve those two equations for $~r~,\phi~$ you obtain
$$\phi=\frac 12\arctan\left(\frac \mu\lambda\right)\tag 1$$ $$r=\pm\,(\mu^2+\lambda^2)^{1/4} \tag 2$$
the lagrangian with polar coordinates is
$$\cal L=\frac m2 (\dot r^2+r^2\dot\phi^2)-\frac k2 r^2$$
with equations $~(1)~,(2)$
$$\cal L=\frac 18\frac{m}{\sqrt{\mu^2+\lambda^2}} \,(\dot\mu^2+\dot\lambda^2)-\frac k2\,\sqrt{\mu^2+\lambda^2}$$
$$\dot r=\frac 12\,{\frac {\mu\,\dot\mu +\lambda\,\dot\lambda }{ \left( {\mu}^{2}+{\lambda }^{2} \right) ^{3/4}}}$$
$$\dot\phi=\frac 12\,{\frac {\lambda\,\dot\mu -\mu\,\dot\lambda }{{\mu}^{2}+{\lambda}^{2}}}$$
