Motivating the Legendre Transform Mathematically If I  begin with a functional of the form
$$J[y] = \int_a^b f(x,y,y')dx$$
and find its Euler-Lagrange equations
$$\frac{\partial f}{\partial y} - \frac{d}{dx}\frac{\partial f}{\partial y'} = 0 =  \frac{d}{dx}\frac{\partial f}{\partial y'} - \frac{\partial f}{\partial y}.$$
I end up with a second order ODE
$$\frac{d}{dx}\frac{\partial f}{\partial y'} - \frac{\partial f}{\partial y} 
=  
\left( \frac{\partial ^2 f}{\partial y' \partial y'} \right) \frac{d^2 y}{dx^2} 
+ \left(\frac{\partial ^2 f}{\partial y \partial y'}\right) \frac{dy}{dx}  
+ \frac{\partial ^2 f}{\partial x \partial y'}   
-  \frac{\partial f}{\partial y} 
= 0$$
Now every higher order ODE can be broken up into a system of first order ODEs in $y$ and the derivative $M = \frac{dy}{dx}$, giving
$$ \frac{d}{dx}\frac{\partial f}{\partial y'} - \frac{\partial f}{\partial y} 
=  \left( \frac{\partial ^2 f}{\partial y' \partial y'} \right) \frac{d M}{dx} 
+ \left(\frac{\partial ^2 f}{\partial y \partial y'}\right)M 
 + \frac{\partial ^2 f}{\partial x \partial y'}  
 -  \frac{\partial f}{\partial y} 
= 0. $$
From this perspective, Hamilton's equations
$$\left\{
\begin{array}
& \frac{dy}{dx} = \phantom{-}\frac{\partial \mathcal{H}}{\partial p}\\
  \frac{d p}{dx} = - \frac{\partial \mathcal{H}}{\partial y}
\end{array}\right.$$
are merely a system of first order equations making my above system of first order ODEs look more symmetric, after a suitable change of variables.
My question is, looking at 
$$\frac{d}{dx}\frac{\partial f}{\partial y'} 
- \frac{\partial f}{\partial y} 
=  \left( \frac{\partial ^2 f}{\partial y' \partial y'} \right) \frac{d^2 y}{dx^2} 
+ \left(\frac{\partial ^2 f}{\partial y \partial y'}\right) \frac{dy}{dx}  
+ \frac{\partial ^2 f}{\partial x \partial y'}  
-  \frac{\partial f}{\partial y} 
= 0$$
it should be possible to see why the Legendre transformation arises, first because its a transformation using derivatives to change variables but also because it should just make some terms go to zero in this second order ode so that everything looks nicer, but how do you see this explicitly?
It'd be great if you could use my notation, ie. $J[y]$ etc... as you see I snuck in $\mathcal{H}$ in above which shouldn't really be there, I'd love to see how that comes about in my notation - thanks!
 A: If I understand your question correctly, you are asking for quite a pedestrian view of the Legendre transformation, which is much more of an elegant way to transform a system of ODEs into first-order ODEs than can be visible in this way. I would recommend you have a look at V. Arnold's Mathematical methods of Classical Mechanics for a look at how the Legendre transform really works and why we use it.
You are slightly misunderstanding the transformation to the momentum variable, which is defined to be
$$p=\frac{\partial f}{\partial y'}\tag{1}$$
and not directly proportional to $y'$. The two are proportional only in the limited circumstance that $f=\frac 12 m(y')^2-V(y)$, and the relationship is in general more complicated. The only thing you need is for (1) to be a proper coordinate transformation, which means asking for the hessian $$\frac{\partial^2 f}{\partial y'\partial y'}$$ to be nonsingular and positive-definite; it's clear from that that the functional dependence of $p$ on $y'$ and $y$ can be very general indeed.
Given this, if you just want to see how this cleans up the notation, you're much better off not dismembering the original Euler-Lagrange equation,
$$\frac d{dx}\frac{\partial f}{\partial y'}-\frac{\partial f}{\partial y}=0\tag{2},$$
which turns into Hamilton's equations
$$
\left\{\begin{array}
& \frac{\partial f}{\partial y'}=p,\\
\frac{d p}{dx}=\frac{\partial f}{\partial y},
\end{array}\right.
$$
simply by substituting in the correct definition (1) and either inverting the first equation to get $\frac{dy}{dx}$ in terms of $p$ and $y$, or simply realizing that it is part of a system of two ODEs of first order in $y$ and $p$.
A: First o minor comment on Emilio's helpful answer. In his equation (2) it's clear that "f" is the Lagrangian, usually called "L". The interesting thing is that in Emilio's finding of "Hamilton's equations" he finds them in terms of "f", so
a) that means that L satisfies Hamilton's equations, if I'm not mistaken.
b) This was done without using the full Legendre Transformation. Specifically, the Legendre term
"p * x" was not applied.
Now to the original question, how and why does the Legendre Transformation arise (in going from Lagrangian mechanics to Hamiltonian Mechanics.
i) Qmechanic stated one reason, i.e. so a"higher order ODE can be broken up into a system of first order ODEs". It seems hopeful that the first order equations might be more easily solved.
ii) For me the second the second most motivating reason is not found in Lagrange's equations, but in the Legendre Transform itself. In my experience transforms, e.g. Fourier or Laplace, are used to get to a different form where they can be solved, but then that solution "want's" to be transformed back to the original set of variables to be interpreted and used. So, we'd like the transformation from a second order equation in (x,dx/dt) to a pair of first order equations in (x,p) to be reversible. Now look at the definition of the Legendre Transformation:
H(x,p) = px - L(p,x)           (IV)
having used eq. (1) in an straight-forward way to restate L in terms of different variable names.In the same way we can rename the variables in H and rearrange (IV) to get:
L(x, dx/dt) = px - H(x,dx/dt)    (V)
Voila! We have the inverse Legendre Transform.
If you are no yet sufficiently motivated, one you get to the Hamiltonian form there are other symmetries and benefits. See https://www2.ph.ed.ac.uk/~mevans/sp/LT070902.pdf
