1) Off-shell vs. on-shell action. What may cause some confusion is that Noether's theorem in its original formulation only refers to the off-shell action functional
$$\tag{1} I[q;t_i,t_f]~:=~ \int_{t_i}^{t_f}\! {\rm d}t \ L(q(t),\dot{q}(t),t), $$
while Feynman's proof (see approximately 50 minutes into this video) mostly is referring to the Dirichlet on-shell action function
$$\tag{2} S(q_f,t_f;q_i,t_i)~:=~I[q_{\rm cl};t_i,t_f], $$
where $q_{\rm cl}:[t_i,t_f] \to \mathbb{R}$ is the extremal/classical path, which satisfies the equation of motion (e.o.m.)
$$\tag{3}\frac{\delta I}{\delta q}
~:=~\frac{\partial L}{\partial q}
- \frac{ d}{dt} \frac{\partial L}{\partial \dot{q}}~\approx~ 0,$$
with the Dirichlet boundary conditions
$$\tag{4} q(t_i)~=~q_i \qquad \text{and}\qquad q(t_f)~=~q_f.$$
See also e.g. this Phys.SE answer. [Here the $\approx$ symbol means equality modulo e.o.m. The words on-shell and off-shell refer to whether e.o.m. are satisfied or not.]
2) Noether's theorem. Recall the setting of Noether's theorem. Let the off-shell action be invariant
$$\tag{5} I[q^{\prime};t^{\prime}_i,t^{\prime}_f]~=~ I[q;t_i,t_f] $$
under an infinitesimal global variation
$$\tag{6} t^{\prime}-t~=~\delta t~=~\varepsilon X(t) \qquad \text{and}\qquad q^{\prime}(t^{\prime})- q(t)~=~ \delta q(t) ~=~ \varepsilon Y(t).$$
Here $X$ is a horizontal$^1$ generator, $Y$ is a generator, and $\varepsilon$ is an infinitesimal parameter that is independent of $t$.
Noether's theorem. The off-shell symmetry (5) implies that the Noether charge
$$\tag{7} Q~:=~p Y - h X $$
is conserved in time
$$\tag{8} \frac{dQ}{dt}~\approx~0$$
on-shell.
Here
$$ \tag{9} p~:=~\frac{\partial L}{\partial \dot{q}} \qquad \text{and}\qquad
h~:=~p\dot{q}-L $$
is the momentum and energy function, respectively.
3) Assumptions. Let us assume$^2$:
that the Lagrangian $L(q,v,t)$ is a smooth function of its arguments $q$, $v$, and $t$.
that there exists a unique classical path $q_{\rm cl}:[t_i,t_f] \to \mathbb{R}$ for each set $(q_f,t_f;q_i,t_i)$ of boundary values.
that the classical path $q_{\rm cl}$ depends smoothly on the boundary values $(q_f,t_f;q_i,t_i)$.
4) Differential ${\rm d}S$.
Lemma. The Dirichlet on-shell action function $S(q_f,t_f;q_i,t_i)$ is a smooth function of its arguments $(q_f,t_f;q_i,t_i)$. The differential is
$$ \tag{10} {\rm d}S(q_f,t_f;q_i,t_i) ~=~ (p_f {\rm d}q_f - h_f {\rm d}t_f) -(p_i {\rm d}q_i - h_i {\rm d}t_i), $$
or equivalently,
$$ \tag{11} p_f~=~\frac{\partial S}{\partial q_f}, \qquad p_i~=~-\frac{\partial S}{\partial q_i}, $$
and
$$ \tag{12} h_f~=~-\frac{\partial S}{\partial t_f}, \qquad h_i~=~\frac{\partial S}{\partial t_i}. $$
Proof of eq. (11): Consider a vertical infinitesimal variation $\delta q_{\rm cl}$ between two neighboring classical paths $q_{\rm cl}$ and $q_{\rm cl}+\delta q_{\rm cl}$. The change in the Lagrangian is
$$ \tag{13} \delta L
~=~ \frac{\partial L}{\partial q} \delta q + \frac{\partial L}{\partial \dot{q}} \delta \dot{q}
~\stackrel{(3)+(9)}{=}~ \frac{\delta I}{\delta q} \delta q +\frac{ d}{dt}(p~\delta q)~\stackrel{(3)}{\approx}~\frac{ d}{dt}(p~\delta q),$$
so that
$$ \tag{14} \delta S ~\stackrel{(2)}{\approx}~\delta I ~\stackrel{(1)}{=}~ \int_{t_i}^{t_f}\! {\rm d}t \ \delta L~\stackrel{(13)}{\approx}~[p~\delta q]_{t_i}^{t_f}~=~p_f~\delta q_f- p_i~\delta q_i. $$
This proves eq. (11).
Proof of eq. (12):
^ q
|
q^*_f|------------------/
| /|
| / |
| / |
q_f|--------------/ |
| /| |
| / | |
| / | |
q_i|----------/ | |
| /| | |
| / | | |
| / | | |
q^*_i|------/ | | |
| | | | |
|------|---|---|---|-----> t
t*_i t_i t_f t*_f
Fig. 1.
Imagine that we infinitesimally extend the time interval $[t_i,t_f]$ to $[t^{*}_i,t^{*}_f]$, where
$$\tag{15}\delta t_i~:=~t^{*}_i - t_i \qquad \text{and}\qquad \delta t_f~:=~t^{*}_f - t_f$$
both are infinitesimally small. This induces a change of the boundary positions (4) of the fixed classical path $q_{\rm cl}$ as follows
$$\tag{16} \delta q_i~:=~ q^{*}_i - q_i~=~\dot{q}_i ~\delta t_i
\qquad \text{and}\qquad
\delta q_f~:=~ q^{*}_f - q_f~=~\dot{q}_f ~\delta t_f,$$
which are dictated by the end point velocities. We would now like to calculate the variation
$$ S(q^{*}_f,t^{*}_f;q^{*}_i,t^{*}_i) - S(q_f,t_f;q_i,t_i)~=~\delta S~\stackrel{(11)}{=}~p_f \delta q_f +\frac{\partial S}{\partial t_f} \delta t_f -p_i \delta q_i + \frac{\partial S}{\partial t_i}\delta t_i $$
$$\tag{17} ~\stackrel{(16)}{=}~(p_f \dot{q}_f +\frac{\partial S}{\partial t_f}) \delta t_f -(p_i \dot{q}_i - \frac{\partial S}{\partial t_i})\delta t_i $$
Since the new classical path is just an infinitesimal extension of the same old classical path, we may also estimate the variation as
$$ \tag{18} \delta S~=~S(q^{*}_f,t^{*}_f;q_f,t_f)+S(q_i,t_i;q^{*}_i,t^{*}_i)~=~ L_f \delta t_f - L_i \delta t_i.$$
Comparing eqs. (9), (17) and (18) yields eq. (12).
5) Feynman's four-point argument. We are finally ready to discuss Feynman's four-point argument.
^ q
|
| A' B'
| ___________________________
| | virtual/off-shell |
| | |
| | |
| |___________________________|
| A classical/on-shell B
|
|
|------------------------------------------------> t
Fig. 2. (Note that the two horizontal and the two vertical straight ASCII lines are in general an oversimplification of the actual paths.)
We start with the on-shell action
$$\tag{19} S(A\to B)~=~I(A\to B)$$
for some classical path $q_{\rm cl}$ between two spacetime events $A$ and $B$. We then apply the infinitesimal transformation (6) to produce a virtual path $q^{\prime}$ between two infinitesimally shifted spacetime events $A^{\prime}$ and $B^{\prime}$. In turn, the virtual path $q^{\prime}$ has an off-shell action
$$\tag{20} I(A^{\prime}\to B^{\prime})~=~I(A\to B)$$
equal to the original action due to the off-shell symmetry (5).
Next we would like to consider the shifted path $A\to A^{\prime}\to B^{\prime}\to B$. Unfortunately, the two infinitesimal pieces $A\to A^{\prime}$ and $B^{\prime}\to B$ (which we will choose to be classical paths) may correspond to constant time. In such cases we replace Feynman's four points with six points, i.e. we extend infinitesimally the original classical path $A\to B$ to a classical path $A^{*}\to B^{*}$, in such way that the two new infinitesimal paths $A^{*}\to A^{\prime}$ and $B^{\prime}\to B^{*}$ (which we also will choose to be classical paths) do both not correspond to constant time.
^ q
|
| A' B'
| ____________________________
| /| virtual/off-shell |\
| / | | \
| / | | \
| A* /___|___________________________|___\ B*
| A classical/on-shell B
|
|
|------------------------------------------------> t
Fig. 3.
Since the virtual path $A^{*}\to A^{\prime}\to B^{\prime}\to B^{*}$ is an infinitesimal variation of the classical path $A^{*}\to A\to B\to B^{*}$, we conclude that the difference
$$S(A^{*}\to A^{\prime})+I(A^{\prime}\to B^{\prime})+S(B^{\prime}\to B^{*})$$
$$-S(A^{*}\to A)-S(A\to B)-S(B\to B^{*})$$
$$\tag{21} ~=~I(A^*\to A^{\prime}\to B^{\prime}\to B^*)
-S(A^*\to A\to B\to B^*)~=~{\cal O}(\varepsilon^2)$$
cannot contain contributions linear in $\varepsilon$.
We next apply the Lemma from Section 4. The six infinitesimal classical paths mentioned so far are all described by the differential (10), which is linear and hence obeys a (co-)vector addition rule. Therefore
$$\tag{22} S(A^{*}\to A^{\prime})-S(A^{*}\to A) +{\cal O}(\varepsilon^2)
~=~S(A\to A^{\prime})
~\stackrel{(6)+(7)+(10)}{=}~\varepsilon Q_i+{\cal O}(\varepsilon^2),\qquad $$
$$\tag{23} S(B\to B^{*}) - S(B^{\prime}\to B^{*})+{\cal O}(\varepsilon^2)
~=~S(B\to B^{\prime})
~\stackrel{(6)+(7)+(10)}{=}~\varepsilon Q_f+{\cal O}(\varepsilon^2).\qquad $$
Comparing eqs. (19)-(23), we arrive at the main statement of Noether's theorem,
namely that the Noether charge is conserved,
$$\tag{24} Q_f~=~Q_i.$$
--
$^1$ Feynman and OP use the opposite convention for horizontal and vertical than this answer.
$^2$ Noether's theorem works with less assumptions, but to avoid mathematical technicalities, we impose assumption 1, 2 and 3. Note that it is easy to find examples that satisfies assumption 1 and 2, but where the classical path $q_{\rm cl}$ may jump discontinuously for varying boundary values $(q_f,t_f;q_i,t_i)$, so that assumption 3 is not satisfied.