Legendre Transformation of Lagrangian with constraints I have problems with obtaining a Hamiltonian from a Lagrangian with constraints. My overall goal is to find a Hamiltonian description of three particles independent of any Newtonian Background and with symmetric constraints for positions and momenta. For this I start with the 3-particle Lagrangian
$$L= \frac{1}{2} \sum _{i=1}^3 \dot{x}_i^2 - \frac{1}{2\cdot 3} (\sum _{i=1}^3 \dot{x}_i)^2 - V(\{x_i - x_j\})$$
which only depends on relative variables, who are however still defined with respect to an absolute reference frame. To get rid of these (unphysical) dependencies I define new variables:
$$x_1 - x_2 = q_3\\
x_2 - x_3 = q_1 \\
x_3 - x_1 = q_2\\
x_1 + x_2 + x_3 = q_{cm}.$$
The reverse transformation is not uniquely definded. We can choose
$$x_1 = \frac{1}{3} \left( q_{cm} + q_3 - q_2 \right) \\ x_2 = \frac{1}{3} \left( q_{cm} + q_1 - q_3 \right) \\ x_3 = \frac{1}{3} \left( q_{cm} + q_2 - q_1 \right)$$
along with the constraint
$$ q_1 + q_2 + q_3 = Q = 0.$$
From this I can derive
$$ \dot{q}_1 + \dot{q}_2 + \dot{q}_3 = \dot{Q} = 0.$$
I now want to rewrite the Lagrangian in the new Variables. After a little work with the sums I arrive at
$$ \tilde L(q_i, \dot{q}_i) = \dot q_1^2 + \dot q_2^2 + \dot q_3^2 - V(q_1,q_2,q_3) $$
But now i don't know: Is the new Lagrangian of the form
$$L_{tot} = \tilde L + \alpha Q$$
or
$$L_{tot} = \tilde L + \alpha Q + \beta \dot{Q}~?$$
In a next step, and this is the core of my question, I would like to obtain the Hamiltonian and the conjugate momenta from this Lagrangian, but I have no idea how to treat the constraints. Is it possible to arrive at an Hamiltonian, where the constraint $Q=0$ holds along with a constraint for the conjugate momenta? For every help I'd be extremely grateful!
Another way of doing this could be legendretransforming the original Lagrangian and then finding a canonical transformation which has the same result. But how this could be achieved is even more mystical to me.
Regarding my Background: I am writing my Master's thesis in physics about Quantum Reference Frames. I have some knowledge about singular Lagrangians and constrained Hamiltonian systems (Like treated in the first chapters of Henneaux and Teitelboim's "Quantization of gauge systems). And I do know about the very basics of differential geometry, but I am not really profound in this topic.
 A: On a mathematical level, a Lagrange multiplier in the Lagrangian is no different from a "real" coordinate whose velocity does not appear in the Lagrangian, such as $A_0$ in the context of Maxwell field theory. One can therefore subject a Lagrangian containing a Lagrange multiplier to the standard Hamilton-Dirac procedure and obtain a corresponding constrained Hamiltonian.  I'll sketch out the Hamilton-Dirac analysis for this Lagrangian and leave the details to you.
The transformed Lagrangian is
$$
L = \frac{1}{6} (\dot{q}_1^2 + \dot{q}_2^2 + \dot{q}_3^2) - V(q_1, q_2, q_3) + \alpha (q_1 + q_2 + q_3),
$$
where $\alpha$ is a Lagrange multiplier.
One can construct a Hamiltonian that generates the same equations of motion by treating all of the variables, including the Lagrange multiplier, as having conjugate momenta:
\begin{align}
p_i \equiv \frac{\partial L}{\partial \dot{q}_i} = \frac{1}{3} \dot{q}_i \, \, (i &= 1,2,3)  & p_\alpha \equiv \frac{\partial L}{\partial \dot{\alpha}} = 0
\end{align}
Since the last of these quantities vanishes identically, it is therefore a primary constraint of the model.
The base Hamiltonian of the model is then (as usual)
$$
H_0 = \sum p_i \dot{q}_i - L = \frac{3}{2} \left( p_1^2 + p_2^2 + p_3^2 \right) - V(q_1, q_2, q_3) - \alpha (q_1 + q_2 + q_3)
$$
but this Hamiltonian will not, in general, generate the correct equations of motion (i.e. the evolution will generally leave the "constraint surface" $q_1 + q_2 + q_3 = 0$.)
To obtain a Hamiltonian that generates the correct equations of motion, we first construct the augmented Hamiltonian
$$
H_A = H_0 + u p_\alpha
$$
where $u$ is an auxiliary Lagrange multiplier, left arbitrary for now.  We must now see whether the requirement that the system stays on the constraint surface places any requirements on $u$.  To do this, we take the Poisson brackets of the primary constraint $p_\alpha = 0$ with the augmented Hamiltonian $H_A$.  This will lead to a secondary constraint:
$$
0 = \dot{p}_\alpha = \{ p_\alpha, H_A \} = q_1 + q_2 + q_3.
$$
So we have to have $q_1 + q_2 + q_3 = 0$ to preserve the primary constraint.
This secondary constraint must also be preserved by the time evolution, which gives rise to another secondary constraint, which gives rise to another, and so forth.  However, in this case, eventually one arrives at an equation that can be solved for the unknown Lagrange multiplier $u$.  (I haven't gone through the algebra carefully, but it looks like you will be able to express $u$ in terms of $\alpha$ and the second derivatives of $V$ with respect to $q_i$.)
The full Hamiltonian is then equal to the augmented Hamiltonian with the auxiliary Lagrange multiplier $u$ set equal to this value.  In general, one would have to add in the so-called first-class constraints—those which commute with all other constraints—at this stage as well, along with Lagrange multipliers for them.  However, I do not believe this model has any first-class constraints.

Further reading:
The best reference I know for this is Dirac's Lectures on Quantum Mechanics (a set of lecture notes from the mid-50s, and not to be confused with his better-known Principles of Quantum Mechanics.)  An excellent summary of the procedure can also be found in Appendix B of

Isenberg & Nester, "The effect of gravitational interaction on classical fields: A Hamilton-Dirac analysis."  Annals of Physics (NY) 107, pp. 56–81 (1977).

Alternately, you could look my recent paper that discusses this technique for constrained field theories.  However, it focuses on a field-theory context and I do not go into as much detail about the procedure there.

Seifert, "Constraints and degrees of freedom in Lorentz-violating field theories", Phys. Rev. D99 045003 (2019).  arXiv:1810.09512.

