Constraints of relativistic point particle in Hamiltonian mechanics I try to understand constructing of Hamiltonian mechanics with constraints. I decided to start with the simple case: free relativistic particle. I've constructed hamiltonian with constraint:
$$S=-m\int d\tau \sqrt{\dot x_{\nu}\dot x^{\nu}}.$$
Here $\phi=p_{\mu}p^{\mu}-m^2=0$  $-$ first class constraint.
Then $$H=H_{0}+\lambda \phi=\lambda \phi.$$ 
So, I want to show that I can obtain from this Hamiltonian the same equation of motion, as obtained from Lagrangian.
But the problem is that I'm not sure what to do with $\lambda=\lambda(q,p)$. I tried the following thing:
$$\dot x_{\mu}=\{x_{\mu},\lambda \phi\}=\{x_{\mu},\lambda p^2\}-m^2\{x_{\mu},\lambda\}=\lambda\{x_{\mu},p^2\}+p^2\{x_{\mu},\lambda\}-m^2\{x_{\mu},\lambda\}$$$$=2\lambda \eta_{\mu b} p^b+p^2\{x_{\mu},\lambda\}-m^2\{x_{\mu},\lambda\}=2\lambda \eta_{\mu b} p^b+p^2\frac{\partial \lambda}{\partial p^{\mu}}-m^2\frac{\partial \lambda}{\partial p^{\mu}},$$
$$\dot \lambda=\{\lambda, \lambda \phi \}=\{\lambda,\lambda p^2\}-m^2\{\lambda,\lambda\}=\lambda\{\lambda,p^2\}+p^2\{\lambda,p^2\}=2\lambda\eta_{ak}p^{a}\frac{\partial \lambda}{\partial x^{k}},$$
$$\dot p_{\mu}=\{p_{\mu},\lambda p^{2}-m^2\lambda \}=p^{2}\{p_{\mu},\lambda\}-m^2\{p_{\mu},\lambda\}=-p^{2}\frac{\partial \lambda}{\partial x^{\mu}}+m^2\frac{\partial \lambda}{\partial x^{\mu}}.$$
If we recall that $p^2-m^2=0$, then we get from the third equation: $\dot p=0$, and from the first: $$\dot x_{\mu}=2\lambda\eta_{ak}p^{a}.$$
So we have 


*

*$\dot x_{\mu}=2\lambda\eta_{\mu b}p^{b}.$

*$\dot \lambda=2\lambda\eta_{ak}p^{a}\frac{\partial \lambda}{\partial x^{k}}.$

*$\dot p=0.$
But I don't know what to do next. Can you help me?
 A: *

*We cannot resist the temptation to generalize the background spacetime metric from the Minkowski metric $\eta_{\mu\nu}$ to a general curved spacetime metric $g_{\mu\nu}(x)$. We use the sign convention $(-,+,+,+)$.


*Let us parametrize the point particle by an arbitrary world-line parameter $\tau$ (which does not have to be the proper time).


*The Lagrange multiplier $\lambda=\lambda(\tau)$ (which OP mentions) depends on $\tau$, but it does not depend on the canonical variables $x^{\mu}$ and $p_{\mu}$. Similarly, $x^{\mu}$ and $p_{\mu}$ depend only on $\tau$.


*The Lagrange multiplier $\lambda=\frac{e}{2}$ can be identified with an einbein$^1$ field $e$. See below where we outline a simple way to understand the appearance of the mass-shell constraint
$$\begin{align}p^2+(mc)^2~\approx~&0, \cr p^2~:=~&g^{\mu\nu}(x)~ p_{\mu}p_{\nu}~<~0.\end{align}\tag{1}$$


*Start with the following square root Lagrangian for a massive relativistic point particle
$$\begin{align}L_0~:=~& -mc\sqrt{-\dot{x}^2}, \cr \dot{x}^2~:=~&g_{\mu\nu}(x)~ \dot{x}^{\mu}\dot{x}^{\nu}~<~0, \end{align} \tag{2}$$
where dot means differentiation wrt. the world-line parameter $\tau$. Here the action is $S_0=\int \! d\tau~ L_0 $. The stationary paths includes the geodesics. More precisely, the Euler-Lagrange equations are the geodesics equations.


*Introduce an einbein field $e=e(\tau)$, and Lagrangian
$$L~:=~\frac{\dot{x}^2}{2e}-\frac{e (mc)^2}{2}.\tag{3}$$
Contrary to the square root Lagrangian (2), this Lagrangian (3) also makes sense for massless point particles, cf. this Phys.SE post.


*Show that the Lagrangian momenta are
$$p_{\mu}~=~ \frac{1}{e}g_{\mu\nu}(x)~\dot{x}^{\nu}.\tag{4}$$


*Show that the Euler-Lagrange equations of the Lagrangian (3) are
$$\begin{align} \dot{p}_{\lambda}~\approx~&\frac{1}{2e}\partial_{\lambda}g_{\mu\nu}(x)~ \dot{x}^{\mu}\dot{x}^{\nu}, \cr \dot{x}^2+(emc)^2~\approx~&0.\end{align}\tag{5}$$


*Show that the Lagrangian (3) reduces to the square root Lagrangian (2) when integrating out the einbein field
$$ e~>~0.\tag{6}$$
The inequality (6) is imposed to remove an unphysical negative branch, cf. my Phys.SE answer here.$^2$


*Perform a (singular) Legendre transformation$^3$ of the Lagrangian (3), and show that the corresponding Hamiltonian becomes
$$H~=~ \frac{e}{2}(p^2+(mc)^2).\tag{7}$$
This Hamiltonian (7) is precisely of the form 'Lagrange multiplier times constraint' (1).


*Show that Hamilton's equations are precisely eqs. (4) and (5).


*The arbitrariness in the choice of the world-line parameter $\tau$ leads to reparametrization symmetry$^4$
$$\begin{align}\tau^{\prime}~=~&f(\tau), \qquad 
d\tau^{\prime} ~=~ d\tau\frac{df}{d\tau},\cr 
\dot{x}^{\mu}~=~&\dot{x}^{\prime\mu}\frac{df}{d\tau},\qquad 
e~=~e^{\prime}\frac{df}{d\tau},\cr  
p_{\mu}~=~&p_{\mu}^{\prime},\qquad 
L~=~L^{\prime}\frac{df}{d\tau},\cr 
H~=~&H^{\prime}\frac{df}{d\tau}\qquad 
S~=~S^{\prime},\end{align}\tag{8}$$
where $f=f(\tau)$ is a bijective function.


*Thus one may choose various gauges, e.g. $e={\rm const.}$
References:

*

*J. Polchinski, String Theory, Vol. 1, Section 1.2.

--
Footnotes:
$^1$ An einbein is a 1D version of a vielbein.
$^2$ As a consistency check of the sign (6), if we in the static gauge $$ix^0_M~=~x^0_E~=~c\tau_E~=~ic\tau_M\tag{9}$$
Wick rotate from Minkowski to Euclidean space, then in eq. (3), the Euclidean Lagrangian $L_E=-L_M>0$ becomes positive as it should.
$^3$ Strictly speaking, in the singular Legendre transformation, one should also introduce a momentum
$$p_e~:=~\frac{\partial L}{\partial \dot{e}}~=~0\tag{10}$$
for the einbein $e$, which leads to a primary constraint (10), that immediately kills the momentum $p_e$ again. The corresponding secondary constraint is the mass-shell constraint (1). The corresponding tertiary constraint vanishes identically because of skewsymmetry of the Poisson bracket. Note that $\frac{\partial H}{\partial e}\approx 0$ becomes one of Hamilton's equations.
$^4$ Reparametrization is a passive transformation. For a related active transformation, see this Phys.SE post.
A: From your equation (1) you can get 
\begin{equation*}
\sqrt{\dot x_\mu \dot x^\mu}= 2\lambda \sqrt{p_\mu p^\mu}=2\lambda m
\end{equation*}
Combining this with your (1) you get
\begin{equation*}
\frac{\dot x_\nu}{\sqrt{\dot x_\mu \dot x^\mu}}=\frac{ p_\nu}{m}.
\end{equation*}
Finally, combining with your (2) you get
\begin{equation}
\frac{d}{d\tau}\left(\frac{{\dot x_\nu}}{\sqrt{\dot x_\mu \dot x^\mu}}\right)=0,
\end{equation}
which is exactly the equation you can find from the original Lagrangian
