About the general concept of 'motivation of the Lagrangian formalism'.
We have that textbook authors have converged on presenting Hamilton's stationary action as the motivation of the Lagrangian formalism, so in this answer I will discuss Hamilton's stationary action.
That said: the approach to classical mechanics introduced by Joseph Louis Lagrange predates Hamilton's stationary action. If Hamilton's stationary action would never have been introduced the development of Lagrangian mechanics would not have been impacted; Lagrangian mechanics has the same capabilities either way. What Lagrange introduced was systematic use of generalized coordinates, in combination with expressing the physics taking place in terms of energy. In that sense Lagrangian mechanics is independent of Hamilton's stationary action.
The following exposition consists of demonstrations for the purpose of gaining intuition.
To make this a general answer I will start with a short exposition of calculus of variations.
Differential calculus and calculus of varations both have a unit of operation. In differential calculus the unit of operation is a pair of points, the line through those two points is tangent to the curve you are looking to solve for. In calculus of variations the unit of operation is a triplet of points.
Diagram 1 is an animated GIF that represents that unit of operation.
(The animated GIF is composited from screenshots of an interactive diagram.)
Diagram 1 - Unit of operation
In diagram 1 the dashed line is a parabola. This parabola represents height as a function of time of an object that has been thrown upwards. A uniform force is acting in downward direction. To obtain the simplest possible parabola the value of the (uniform) acceleration is set at 2 $m/s^2$
The outer points ($t_1$ and $t_3$) are treated as fixed; variation is executed by varying the position coordinate of the middle point.
In the diagram two adjacent time intervals are evaluated; $t_{1,2}$ and $t_{2,3}$
The sweet spot is the point where the change of velocity is such that the difference in kinetic energy matches the difference in potential energy.
(In this diagram a uniform acceleration is used, which means that in this diagram the potential increases linear with height, hence $\Delta E_p$ is in this diagram constant. In the general case the force changes as a function of the position coordinate. Then the potential is not linear, and the value of $\Delta E_p$ changes as a function of the position coordinate.)
Diagram 2 - valid at any scale
Diagram 2 illustrates that the logic of this unit of operation is valid at any scale, down to the limit of infinitisimal time intervals.
Diagram 3 - Concatenation
Diagram 3 illustrates concatenation of the unit of operation over the entire domain of the function you are looking to solve for.
The analogy with the unit of operation of differential calculus: you concatenate the unit of operation over the domain of the function you are looking to solve for, and then you take the limit of infinitisimally small intervals.
As an example of a source where this idea of concatenation of the unit of operation is discussed: the book 'Calculus of variations' by Gelfand and Fomin. Gelfand and Fomin demonstrate that the Euler-Lagrange equation can be derived using the functional derivative concept.
In your question you describe that Lancaster and Blundell present expressions for 'average kinetic energy' and 'average potential energy', but the diagrams 1, 2, and 3 illustrate that the logic can be stated such that it applies at infinitisimal scale. From down at infinitisimal scale the logic propagates out to the evaluation of the entire trajectory.
The next stage is to state explicitly where we obtain the criterion that must be satisfied.
To that end we derive the Work-Energy theorem.
The starting point is $F=ma$, and evaluating the integral with respect to position, from starting point $s_0$ to end point $s$
$$ \int_{s_0}^s F \ ds = \int_{s_0}^s ma \ ds \tag{1.1} $$
The right hand side $\int_{s_0}^s ma \ ds$ can be worked out, since acceleration is the second derivative of position.
In the course of the derivation the following relations will be used:
$$ ds = v \ dt \tag{1.2} $$
$$ dv = a \ dt \tag{1.3} $$
For the time being I will not write the factor
$m$, it is a multiplicative factor that is just carried from step to step. In the final expression I will include the factor
$m$ again.
The integral for acceleration from a starting point
$s_0$ to a final point
$s$
$$ \int_{s_0}^s a \ ds \tag{1.4} $$
Use (1.2) to change the differential from $ds$ to $dt$. Since the differential is changed the limits change accordingly.
$$ \int_{t_0}^t a \ v \ dt \tag{1.5} $$
Change the order:
$$ \int_{t_0}^t v \ a \ dt \tag{1.6} $$
Change of differential according to (1.3), with corresponding change of limits.
$$ \int_{v_0}^v v \ dv \tag{1.7} $$
So we have:
$$ \int_{s_0}^s a \ ds = \tfrac{1}{2}v^2 - \tfrac{1}{2}v_0^2 \tag{1.8} $$
Here (1.8) is worked out for the entities position, velocity, and acceleration. Note that this result generalizes; it is valid in all cases that feature the cascade of first and second derivative.
Using the result (1.8) for (1.1) gives the the Work-Energy theorem:
$$ \int_{s_0}^s F \ ds = \tfrac{1}{2}mv^2 - \tfrac{1}{2}mv_0^2 \tag{1.9} $$
We have that potential energy is defined as the negative of the integral of force over distance.
$$ E_p = -\int_{s_0}^s F \ ds \tag{1.10} $$
Hence:
$$ \Delta E_k = -\Delta E_p \tag{1.11} $$
This gives us our criterion explicitly. Over any interval the amount of change of kinetic energy must match the amount of change of potential energy.
Below, between the horizontal lines, is a group of three statements. I present these three statements as a unit to emphasize the tight interconnection; while the statements are different mathematically, the physics content of these three is the same.
$$ \begin{array}{rcl}
F & = & ma \\
\int_{s_0}^s F \ ds & = & \tfrac{1}{2}mv^2 - \tfrac{1}{2}mv_0^2 \\
-\Delta E_p & = & \Delta E_k
\end{array} $$
The validity of (1.11) extends down to infinitisimal change:
$$ d(E_k) = d(-E_p) \tag{1.12} $$
The energy is a quantity that we assign as an object is moving through some potential. We now have two options for taking a derivative: derivative with respect to the time coordinate and derivative with respect to the position coordinate.
$$ \frac{d(E_k)}{dt} = \frac{d(-E_p)}{dt} \tag{1.3} $$
$$ \frac{d(E_k)}{ds} = \frac{d(-E_p)}{ds} \tag{1.14} $$
Now: since kinetic energy and potential energy are constructed by way of integration with respect to the position coordinate we know that taking the derivative with respect to the position coordinate will recover $F=ma$. (Taking the derivative of the energy with respect to the time coordinate is unpractical. I'm not aware of that derivative ever being used.)
Diagram 4 - Evaluation of area
Diagram 4 demonstrates an area evaluation property that we get for free. We get it for free in the sense that it doesn't require any additional assumption; it follows logically.
The two time intervals $t_{1,2}$ and $t_{2,3}$ are displayed in three of the four subpanels: upper-left, upper-right, and lower-left. The time intervals are set up to be of equal duration.
In the upper-right sub-panel: the height of each bar represents the value of the energy corresponding to that time interval. The number displayed in red is the summed area of the two red bars; the number displayed in green is the summed area of the two green bars.
Reminder: the energies that are displayed in this diagram are hypothetical energies. The variation sweep is hypothetical: at each position of the movable point the diagram shows what the energies would be if the object would move along that particular trial trajectory.
Stating the two different kinds of change explicity:
- Rate of change of actual energy as an object is moving along the true trajectory.
- Rate of change of hypothetical energy (as a function of variation sweep)
In motion along the true trajectory one form of energy is transformed into the other; the energies are counter-changing; $\Delta E_k = -\Delta E_p$
As illustrated in diagram 4: in sweeping out variation the (hypothetical) energies are co-changing.
Rate of change
In the case of Hamilton's stationary action the variation sweep is variation of position coordinate. This means that the derivative of Hamilton's action with respect to variation boils down to taking the derivative with respect to position.
For emphasis: taking the derivative (of Hamilton's action) with respect to variation results in an expression for derivative with respect to position.
At this point I repeat (1.14)
$$ \frac{d(E_k)}{ds} = \frac{d(-E_p)}{ds} \tag{1.14} $$
The true energy satisfies (1.14) Therefore when the variation sweep is at the sweet spot the trial trajectory coincides with the true trajectory
Subtraction
In the hypothetical variation sweep the energies are co-changing, so to identify the point where they change at the same rate one energy is subtracted from the other. (By convention it is the potential energy that gets the minus sign.)
Given that we are subtracting the potential energy from the kinetic energy: in the lower-left subpanel the green area is displayed as area below the coordinate's zero line. We can treat the green area as signed area; area is below the zero point of the coordinate system is counted as negative.
The lower-right sub-panel represents in blue the result of the subtraction.
The motion of the blue dot over the diagram represents how the value 'area($E_k-E_p$)' is responding to variation sweep.
The two time intervals $t_{1,2}$ and $t_{2,3}$ are set to be equal in duration. Because of that the value of the area is proportional to the value of the energy:
At the point where the variation hits the sweet spot the value of 'area($E_k-E_p$)' is stationary. That means: at the point where the variation hits the sweet spot the rate of change of summed red area matches the rate of change of summed green area.
Summerizing:
There is a point in the variation sweep where the rate of change of (hypothetical) kinetic energy matches the rate of change of (hypothetical) potential energy. If the time intervals are set to be of equal duration then it follows mathematically that the rate of change of summed red area matches the rate of change of summed green area.
Integral
The logic that is demonstrated in diagram 4 is valid at any scale, down to infinitisimal.
When concatenating units of operation along the length of the overall time the bars end up exactly adjacent; no overlap.
This explains why the integral of the quantity ($E_k - E_p$) has the property that it is stationary when the variation hits the sweet spot. The value of ($E_k - E_p$) already has that property at the infinitisimal scale, and from there it propagates to the value of the integral of ($E_k - E_p$), as a function of variation sweep.
The crucial point: the above reasoning does not require making any additional physical assumption. Hence: in any situation where the Work-Energy theorem holds good it follows that Hamilton's stationary action will hold good.
For the sake of completeness I give an outline for how to derive the Euler-Lagrange equation using the unit of operation demonstrated in diagram 1
I learned this approach to the derivation from an article by Preetum Nakkiran, titled 'Geometric derivation of Euler-Lagrange equation'.
In the following the notation conventions for mechanics are used:
t for 'time'
s for 'position'
v for 'velocity'
a for 'acceleration'
$S$ for Hamilton's action
$S_k$ for the kinetic energy component of Hamilton's action
$S_p$ for the potential energy component of Hamilton's action
The small variation of position coordinate will be represented with a factor $\delta s$. The potential energy and the kinetic energy each respond to that variation, responding differently.
Recapitulating:
-as an object is in motion along the true trajectory the kinetic energy and potential energy are counter-changing.
-When you are sweeping out variation: the kinetic energy and potential energy are co-changing
The following derivation is for the hypothetical variation, therefore the potential energy must be entered into the derivation as minus potential energy.
The response of the potential energy component of Hamilton's action:
$$ \delta S_p = -\frac{dE_p}{ds} \delta s \tag{2.1} $$
The response of the kinetic energy component of Hamilton's action:
On both sides of $t_2$ the velocity changes; one side increases, the other side decreases.
Before the change:
$$ v_{1,2} = \frac{\Delta s_{1,2}}{\Delta t} \qquad v_{2,3} = \frac{\Delta s_{2,3}}{\Delta t} \tag{2.2} $$
After adding $\delta s$:
$$ v_{1,2} = \frac{\Delta s_{1,2 } + \delta s}{\Delta t} \qquad v_{2,3} = \frac{\Delta s_{2,3} - \delta s}{\Delta t} \tag{2.3} $$
The changes of the velocities:
$$ \Delta v_{1,2} = \frac{\delta s}{\Delta t} \qquad \Delta v_{2,3} = \frac{- \delta s}{\Delta t} \tag{2.4} $$
The change of the kinetic energy component of Hamilton's action:
$$ \delta S_k = \frac{dE_{k(1,2)}}{dv_{1,2}}\Delta v_{1,2} + \frac{dE_{k(2,3)}}{dv_{2,3}}\Delta v_{2,3}
\tag{2.5} $$
$$ \delta S_k = - \left(\frac{dE_{k(2,3)}}{dv_{2,3}} - \frac{dE_{k(1,2)}}{dv_{1,2}} \right) \frac{\delta s}{\Delta t} \tag{2.6} $$
The total time is divided in equal length sub-intervals, so the move from interval (1,2) to the adjacent (2,3) can be rearranged to the form of taking a time derivative:
$$ \frac{dE_{k(2,3)}}{dv_{2,3}} - \frac{dE_{k(1,2)}}{dv_{1,2}} = \frac{d}{dt}\frac{dE_k}{dv}\Delta t \tag{2.7} $$
Combining (2.6) and (2.7) we obtain an expression for the change of the kinetic energy component of Hamilton's action as a function of δs:
$$ \delta S_k = -\frac{d}{dt} \frac{dE_k}{dv} \delta s \tag{2.8} $$
Combining the two responses:
$$ \delta S = \delta S_p + \delta S_k = \left( -\frac{d E_p}{ds} - \frac{d}{dt} \frac{dE_k}{dv} \right) \delta s \tag{2.9} $$
The variation δs, having served its purpose, can now be eliminated. The stationary-in-response-to-variation condition is satisfied when the following is satisfied:
$$ -\frac{d E_p}{ds} - \frac{d}{dt} \frac{dE_k}{dv} = 0 \tag{2.10} $$
Flipping the two minus signs does not change the equation:
$$ \frac{d E_p}{ds} + \frac{d}{dt} \frac{dE_k}{dv} = 0 \tag{2.11} $$
The Euler-Lagrange equation with the Lagrangian denoted with the symbol '$L$':
$$ \frac{\partial L}{\partial s} - \frac{d}{dt} \frac{\partial L}{\partial v} = 0 \tag{2.12} $$
Substituting the Lagrangian ($E_k - E_p$) into that:
$$ \frac{\partial (E_k - E_p)}{\partial s} - \frac{d}{dt} \frac{\partial (E_k - E_p)}{\partial v} = 0 \tag{2.13} $$
(2.13) and (2.11) are the same equation.
Narrowing down to what is necessary
The Euler-Lagrange equation is very compact. The Euler-Lagrange equation is an equation that has been narrowed down to what is necessary. You don't need more; the Euler-Lagrange equation is sufficient.
In other words: in retrospect we can see that all elements that were removed during the process of deriving the Euler-Lagrange equation were unnecessary elements.
The standing convention is to state the functional as an integral, but at the end the integral is no longer there; the Euler-Lagrange equation is a differential equation.
The unit of operation of diagram 1 explains why the Euler-Lagrange equation is a differential equation: the unit of operation is valid at infinitisimal scale. The logic operates at the infinitisimal scale, and from there it propagates out to the trajectory as a whole.
The integral that is present at the start is redundant, and is accordingly removed in the process of deriving the Euler-Lagrange equation
The relation between variational calculus and differential calculus
To make the comparison I take a simple case that is a natural case for application of variational calculus: the catenary.
As we know, mathematically the curve of a hanging chain is described by the hyperbolic cosine; the hyporbolic cosine gives the height as a function of the horizontal coordinate.
We have with the catenary that the force that is acting is perpendicular to the horizontal. And if the length of the catenary is changed then the change in the curve is change perpendicular to the horizontal.
Variational calculus pivots the direction of the differentiation. Variational calculus is a form of differential calculus where instead of differentiating with respect to $x$ (the horizontal coordinate), the differentiation is with repect to $y$, in such a way that you do end up with a function that gives the height of the catenary as a function of the horizontal coordinate.
The meaning of the 'stationary' criterion
I have noticed that authors of physics textbooks tend to have a cavalier attitude towards the question: "What is the meaning of the 'stationary' criterion?" The authors will acknowledge that the name 'stationary action' is better, but will then express that to actually use that name would be pedantic.
The purpose of the following discussion is to show that to understand the meaning of the 'stationary' criterion is key.
Diagram 5 - The blue curve has a point where it is stationary
In Diagram 5 the red curve and the green curve are both ascending functions.
Red: $f(x) = \frac{1}{2} x^2$
Green: $ g(x) = ln(x) $
Problem: at what x-coordinate do the functions $f(x)$ and $g(x)$ have the same slope?
Phrased differently: at what x-coordinate is the rate of change of $f(x)$ equal to the rate of change of $g(x)$?
The direct way to answer that question is to start with taking the derivative of each function, and then solve for the point where the two derivatives have the same value:
$$ \frac{f(x)}{dx} = \frac{g(x)}{dx} \tag{3.1} $$
$$ \frac{\tfrac{1}{2}x^2}{dx} = \frac{ln(x)}{dx} \tag{3.2} $$
$$ x = \frac{1}{x} \tag{3.3} $$
There is also a more convoluted way to address the problem:
I will refer to this more convoluted way as constructing a 'mathematical action':
Create a third function by subtracting the second function from the first function.
$$ h(x) = f(x) - g(x) \tag{3.4} $$
$$ h(x) = \tfrac{1}{2} x^2 - ln(x) \tag{3.5} $$
In diagram 5 the blue curve represents $h(x)$, the 'mathematical action'.
In diagram 4 the motion of the blue dot in response to variation sweep traces a curve, and that curve has a stationary point. That stationary point is the point where the rate of change of kinetic energy matches the rate of change of potential energy.
