But I'm wondering if this derivation is correct, because the second law of Newton is formulated for point masses.
So my question is: is this formula, that I frequently encounter in syllabi and books, correct? Where lies my mistake.
Newton's second law is not meant only for point masses, but for any massive body which does not lose or gain massive parts.
The formula
$$
\mathbf F_{ext} = \frac{d\mathbf p}{dt},~~~\tag{1}
$$
expressing 2nd law mathematically, thus $is$ correct, provided $\mathbf F_{ext}$ is external force on the system and $\mathbf p$ is momentum of the system, where system is a body which does not lose or gain massive parts. Thus mass of the body is assumed constant. But then we have $d\mathbf p/dt = m\mathbf a$, and we can rewrite the equation as
$$
\mathbf F_{ext} = m\mathbf a.~~~\tag{2}
$$
It turns out, surprisingly, that the former equation is not easily adaptable to the case of variable mass systems, but the latter equation can be generalized to the case of variable mass easily, by adding new force term on the left-hand side.
But there is a persistent idea in literature and many minds, that (1) is actually the more general equation, and that it applies to variable mass systems.
An example of such a variable mass system is e.g. all matter considered to be part of a rocket (thus including the fuel and oxidizer inside, but without the expelled gases far away from the engine nozzle). Obviously, such system has variable mass. More generally, we can consider all matter inside a well-delimited, possibly moving control volume, as the subject system for which we seek the equation of motion.
The above idea that (1) is more general and is applicable to variable mass systems keeps around in part because of a) Newton writing 2nd law this way; b) some mechanics textbooks suggesting so and also c) (probably) because people get the impression that since $\mathbf F_{ext} = d\mathbf p/dt$ is the more accurate equation in special relativity, where "mass increases with speed", while the other equation $\mathbf F_{ext} = m\mathbf a$ is less correct, this "ability to account for variable mass" should be true also in Newtonian mechanics.
But a) Newton meant bodies that do not lose/gain mass; b) those textbooks are either misleading/confusing, or even wrong on this; c) notice that even in special relativity, (1) is valid only for systems of constant rest mass. If rest mass changes, like for a rocket that loses rest mass, (1) is not correct in special relativity. Similarly, in non-relativistic mechanics, (1) is not correct when the system loses or gains massive parts.
In the interesting case where the system of interest (inside the control volume) does lose or gain massive parts (like a rocket), the equation (1) is no longer valid. An easy way to see this is that this equation is not Galilei-invariant, and thus cannot be true in all inertial frames; when we write it in different frame nevertheless, the external force does not change, but the right-hand side does, so the equation cannot hold in all frames, and thus cannot be general equation of motion for such system.
The second equation (2) is not correct in this case either, as the system with variable mass can have non-zero acceleration even in absence of external forces (let's not count the force of the exhaust gases on the rocket nozzle as part of the external force for now, as it is due to mass which is very close to the system (touching it), and which may be considered part of the system, when we put the boundary of the control volume far enough from the surface of the rocket nozzle).
However, a different and correct equation for $\mathbf p$ (a variable-mass system's momentum) may be derived from Newton's 2nd law when applied to each particle in the control volume1. This can be written as
$$
\mathbf F_{ext} + \mathbf F_{parts~outside} = \frac{d\mathbf p}{dt} + \frac{d\mathbf p_{lost}}{dt}~~~(4).
$$
1)
This can be done because each particle $i$ obeys the equation $\mathbf F = m_i\mathbf a_i$, as it does not lose or gain parts. One way to derive the equation above goes like this.
Let us use a convention where $\mathbf F_{a,-b}$ means force due to particle (or set of particles) $a$ acting on the particle (or set of particles) $b$.
Let $V(t)$ be the region of space that is our control volume, which defines "the system". It is easy to see that at time $t$
$$
\sum_{i\in V(t)} \mathbf{F}_{ext,-i} + \sum_{i \in V(t)} \mathbf F_{parts~outside,-i} = \sum_{i\in V(t)} m_i \mathbf a_i.~~~(a)
$$
We would like to express this without summing over index $i$, using only quantities referring to the system inside and outside as a whole.
Let $\mathbf p$ be momentum inside the control volume $V$. This changes in time for two reasons:
- the particles that stay inside change their momentum in time;
- some particles leave or come in to the control volume.
So we can write
$$
\frac{d\mathbf p}{dt} = \sum_{i\in V} m_i \mathbf a_i - \frac{d\mathbf p_{lost}}{dt}~~~(b)
$$
where $d\mathbf p_{lost}$ is momentum lost from the control volume due to particles leaving the control volume, per time $dt$.
Comparing $(a)$ and $(b)$, we can see that summation of terms $m_i\mathbf a_i$ over $i$ can be replaced by an expression involving $d\mathbf p/dt$ and $d\mathbf p_{lost}/dt$.
Introducing $\mathbf F_{ext}=\sum_{i\in V} \mathbf F_{ext,-i}$ and $\mathbf F_{parts~outside}=\sum_{i\in V} \mathbf F_{parts~outside,-i}$, we can express the above equation as
$$
\mathbf F_{ext} + \mathbf F_{parts~outside} = \frac{d\mathbf p}{dt} + \frac{d\mathbf p_{lost}}{dt},
$$
which is the sought equation (4).
Thus we can see that net force on the particles in the system (those inside the control volume) does not equal rate of change of momentum of the system; but there is an additional term, rate of change of momentum lost from the control volume by particles crossing the boundary. This is why the idea that (1) applies to variable mass systems is untenable. Howsoever we re-interpret the term $\mathbf F_{ext}$ to include the forces due to outside particles acting back on the system, we won't get from (1) the correct equation involving $d\mathbf p/dt$. We have to use (4) instead, to take into account the change of momentum not due to forces, but due to gain/loss of particles.
When we apply this to a rocket (with control volume that follows the rocket shell, completely containing it), we can see that momentum of the rocket $\mathbf p$ changes due to 1) external force, 2) force of the exhaust gases acting back on the system in the control volume (rocket+some exhaust gas inside the control volume), and 3) some momentum going out of the control volume (due to exhaust gas leaving the system).
This expression of the equation of motion in terms of momenta is somewhat foreign to the usual treatment of a rocket in mechanics, because for the purpose of rocket mechanics in gravity field, we are interested in its velocity, rather than momentum.
In the simplest case where the lost particles all leave in direction same or opposite to the body velocity $\mathbf v$ (idealized rocket), the equation of motion can be further simplified, as is common in textbooks, in terms of velocities. Let the boundary of the comoving control volume be far from the rocket, so that force $\mathbf{F}_{parts~outside}$ acting back on the system in the control volume is negligible (the exhaust gas is rarified and has low pressure on the control volume boundary). Velocities of gas particles crossing the boundary of the control volume are close to average value $\mathbf v + \mathbf{c}$, where $\mathbf c$ is the average final velocity of the particles when very far away from the nozzle, in the frame of the rocket. Then the momentum that gets lost per unit time is
$$
\frac{d\mathbf{p}_{lost}}{dt} = - \frac{dm}{dt}(\mathbf c+\mathbf v).~~~(5)
$$
Using these findings, the equation of motion (4) simplifies into
$$
\mathbf{F}_{ext} + \frac{dm}{dt} \mathbf c = m\frac{d\mathbf{v}}{dt}.~~~(6)
$$
This equation has the same form as (2), only with a different net force, where we have to add the term $\frac{dm}{dt} \mathbf c$ to the external force. It is tempting to interpret this term as value of the force of the gas acting on the nozzle; however, such interpretation is hard to justify based just on the above derivation. Notice that this term comes from the term giving lost momentum per unit time, not from the term giving the force of the outside gas on the system in the control volume. We would have to do another derivation with a different control volume whose boundary would copy the rocket surface, including the nozzle surface and the pipes surface right to the place where the fuel is at rest wrt to the rocket shell. Then, comparing to the above, we would find out that force acting on the nozzle and pipes above it is indeed $\frac{dm}{dt} \mathbf c$, where $\mathbf c$ is the final velocity of the exhaust gas, far away from the nozzle.
It is also interesting to note that $\mathbf v$ in this equation is not velocity of the center of mass of the rocket, but velocity of its solid shell. It satisfies the relation $\mathbf p = m \mathbf v$ where $\mathbf p$ and $m$ are momentum and mass inside the (latter, close to the rocket surface) control volume. But because most of the mass gets lost, and center of mass moves closer to the tip of the rocket, this center moves somewhat faster than the solid rocket shell.