I have the Lagrange function:
$$L=\sqrt{\frac{\dot{x}^2+\dot{y}^2}{-y}}.\tag{1}$$
The energy is then:
$$H=\dot{x}\frac{\partial L}{\partial \dot{x}}+\dot{y}\frac{\partial L}{\partial \dot{y}}-L=0.\tag{2}$$ Can I somehow apply in this case the Hamilton or Hamilton-Jacobi formalism to find the motion?
OP's Lagrangian (1) is the square root Lagrangian for a massive relativistic point particle (or equivalently, geodesics) in a curved space with metric $${\bf g}~=~\frac{\mathrm{d}x\odot \mathrm{d}x+ \mathrm{d}y\odot \mathrm{d}y}{-y}, \qquad y~<~0 .\tag{1}$$
The square root Lagrangian (1) has worldline reparametrization invariance, i.e. gauge symmetry. As a result, the Lagrangian energy function $h(x,y,\dot{x},\dot{y})$ [and the Hamiltonian $H(x,y,p_x,p_y)$ in the corresponding Hamiltonian formalism, cf. e.g. this Phys.SE post] vanish.
The easiest way to proceed is to consider the corresponding non-square root Lagrangian, cf. e.g. this Phys.SE post. Then the Legendre transformation, Hamiltonian & Hamilton-Jacobi theory are in principle straightforward to set up.
