Variable mass systems in Lagrangian mechanics When we write the Lagrangian $\mathcal{L}=\frac{1}{2}m\dot{x}^2-U(x)$, where $U$ is the potential energy, we are assuming that the mass $m$ is constant, the only variables being the velocity $\dot{x}$ and position $x$. What can be done to determine the equation of motion of the particle in case the mass is changing?
I know that we cannot simply use the formula $$\dot{p}=m\ddot{x}+\dot{m}\dot{x},$$ with $p=\frac{\partial \mathcal{L}}{\partial \dot{x}}$, because it isn't Galilean invariant and the system is not closed, so some other procedure must be used.
Perhaps the method of Lagrange multipliers may be used? Or via a non-standard Lagrangian that somehow reproduces the equation of motion given here?
 A: If you just give the mass an explicit time-dependence,
$$L = \frac12 m(t) \dot{x}^2 - U(x)$$
then the Euler-Lagrange equation is
$$\frac{d}{dt} (m \dot{x}) = \dot{m} \dot{x} + \dot{m} \ddot{x} = - \frac{dU}{dx}.$$
It's unclear to me why you think "we simply cannot use" this result. It isn't Galilean invariant, but once you let $m(t)$ have arbitrary time-dependence, the action isn't Galilean invariant either. 
A: updated response
The clearest discussion I have found for the Lagrangian for a variable mass system is the following reference (you must pay for access):
https://aapt.scitation.org/doi/full/10.1119/10.0000304
Based on this reference, for a one dimensional system with variable mass (e.g., a rocket or a falling chain), the Lagrangian approach results in ${d \over dt}({{\partial T} \over {\partial \dot x}}) - {{\partial T} \over {\partial x}} = F + M$, where T is the kinetic energy, $T = {1 \over 2}m(t)\dot x^2$,  $F$ is the net external force, and $M$ is  the momentum flux term for variable mass systems. $M = \dot m u - {1 \over 2}{{\partial m} \over {\partial x}}v^2$.  (There is an additional term if the mass is velocity dependent, not included here.  For more details see http://www.scielo.br/scielo.php?script=sci_arttext&pid=S1678-58782006000400015) $u$ is the velocity in the inertial frame of the changing mass, and $v$ is the velocity of the mass in the inertial frame.
Using this relationship you can derive the rocket equation of motion $m\dddot x = \dot mc + F$, where $c$ is the velocity of the ejected mass relative to the rocket:$\enspace c = u - \dot x$.  See
https://arc.aiaa.org/doi/abs/10.2514/3.1626.  For the rocket case, $M = \dot m u $; the second term in $M$ is zero.
A: I  propose the following form for the Lagragian with varying mass:
$$
L(\dot{x},x;t) = \frac{1}{2} m \dot{x}^2  - V(x) - \frac{1}{2} \Delta m \left( \dot{x} - u  \right)^2=L_0  - \frac{1}{2} \Delta m \left( \dot{x} - u  \right)^2,
$$
Where $\Delta m = \frac{dm}{dt} \Delta t$ is the mass change between $t$ and $t+\Delta t$ (negative for rocket problem), and $u$ an extra parameter for the ejection speed. The second term is to make up the kinetic energy loss in the varying mass problem.
The corresponding  action integral:
$$
S = \int_{t_1}^{t_2} L_0(\dot{x}, x; t) dt - \frac{1}{2} \int_{t_1}^{t_2} \Delta m \left( \dot{x} - u  \right)^2 dt
$$
Then, we can go throught the variation for Euler-Lagrangian Equation. The first term render the same Euler-Lagrangin eqaution as usual. An addtional term from the second part of action integral  using integral by parts:
$$
-\delta  \frac{1}{2} \int_{t_1}^{t_2} \Delta m \left( \dot{x} - u  \right)^2 dt =-\int_{t_1}^{t_2} \Delta m \left( \dot{x} - u  \right) \delta \dot{x} dt 
$$
$$
 =+\int_{t_1}^{t_2} \frac{dm}{dt} \frac{\Delta t}{\Delta t} \left( \dot{x} - u  \right) \delta x dt
 =\int_{t_1}^{t_2}  \frac{dm}{dt} \left( \dot{x} - u  \right) \delta x dt
$$
Since the $\Delta t$ is already the first order infinitesmal quantity, the variation are not applying to other terms in the integrand (they will vanish in the limit $\Delta t \to 0$. Thus, the modified Euler-Lagrangian equation:
$$
  \frac{\partial L_0}{\partial x} - \frac{d}{dt} \left\{ \frac{\partial L_0}{\partial \dot{x}}\right\} + \frac{dm}{dt} \left( \dot{x} - u  \right) = 0
$$
Appy to the rocket problem with constant gravity:
$$
L_0 = \frac{1}{2} m \dot{x}^2  - mgx
$$
The Equation of motion:
$$
  \frac{\partial L_0}{\partial x} - \frac{d}{dt} \left\{ \frac{\partial L_0}{\partial \dot{x}}\right\} + \frac{dm}{dt} \left( \dot{x} - u  \right) = -mg - \frac{d}{dt} m\dot{x} - \frac{dm}{dt} \left( \dot{x} - u  \right) 
$$
$$
= -mg -  m\ddot{x} - \dot{m}\dot{v} + \dot{m} \left( \dot{x} - u  \right) = 0
$$
$$
  m \dot{v}   =  | \dot{m}| u - m g.
$$
Another example, throw $dm/dt$ mass rate vertically outside a boat, $u=0$:
$$
  m \dot{v} = 0.
$$
The masses casted outside is moving with same $v$, therefore doesn't change the momentum of the remaining boat (neglecting the vertical shift within our 1-d treatment.)
A: Im my previous naswer, I included a terms of order $dt$ in the Lagrangian. It makes the variation somewhat awkward. After some thoughts, I find it is rather nature to employ the dissipation function and incorporated as a generalized force Rayleigh Dissipation.
The dissipation function is basically the power dissipated by the friction force. In this case, the mass ejected is taking away a part of the knietic energy from the system in a similar way of dissipation force.
$$
  G(v) = \frac{1}{2} \frac{dm}{dt} (\dot{x}-u)^2
$$
The mass changing rate $\dot{m} =\frac{dm}{dt}$ is negative for rocket motion.
The generalized force
$$
  Q = \frac{\partial G}{\partial \dot{x}} = \frac{dm}{dt} (\dot{x}-u)
$$
The Euler-Lagrangian equation with generalized force:
$$
   \frac{d}{dt} \left\{ \frac{\partial L}{\partial \dot{x}}\right\} -\frac{\partial L}{\partial x} = Q 
$$
Appy to the rocket problem with constant gravity:
$$
L = \frac{1}{2} m \dot{x}^2  - mgx
$$
The Equation of motion:
$$
  \frac{d}{dt} \left\{ \frac{\partial L}{\partial \dot{x}}\right\} -\frac{\partial L}{\partial x} =  \frac{d m\dot{x}}{dt} + mg = Q
$$
$$
  m\ddot{x} + \dot{m}\dot{x}  + mg = \dot{m} (\dot{x} - u) 
$$
Render the correct rocket equation of motion.
$$
  m \dot{v}   =  -\dot{m} u - m g = |\dot{m}| u - m g.
$$
Another example, throw $dm/dt$ mass rate vertically outside a boat, $u=0$:
$$
  m \dot{v} = 0 \to v(t) = \text{constant}
$$
The masses casted outside is moving with same $v$, therefore doesn't change the velocity of the remaining boat (neglecting the vertical shift within our 1-d treatment.)
