So my question is: is this formula, that I frequently encounter in syllabi and books, correct? Where lies my mistake.
The equation and the subsequent expression of the derivative
$$\mathbf F_{ext} = \frac{d\mathbf p}{dt} = \frac{d}{dt}(m\mathbf v) = \frac{dm}{dt}\mathbf v + m\mathbf a,~~~(1)$$
are based on the erroneous idea that the equation
$$
\mathbf F_{ext} = \frac{d\mathbf p}{dt}~~~(2)
$$
is valid for systems that lose or gain mass (where $\mathbf F_{ext}$ is external force on the system and $\mathbf p$ is momentum of the system.
The system in this context is often a rocket without the expelled gases far away. Obviously such system has variable mass. More generally, we can consider any mass system in a specified well-delimited control volume as the subject for which we seek the equation of motion.
The above idea keeps around (probably) in part because of some special relativity texts saying $\mathbf F_{ext} = d\mathbf p/dt$ is more general than $\mathbf F_{ext} = m\mathbf a$ because the former is valid equation for relativistic particles.
But in non-relativistic mechanics this equation is valid only when the system does not lose or gain parts.
In the subtler case where the system of interest (inside the control volume) does acquire or lose material parts (like a rocket), the equation (2) is no longer valid. An easy way to see this is that this equation is not Galilei-invariant: when written in different frame, the external force does not change, but the right-hand side does.
However, a different and correct equation for $\mathbf p$ (a variable-mass system's momentum) may be derived from Newton's laws when applied to each particle the system is made of 1:
$$
\mathbf F_{ext} + \mathbf F_{parts~outside} = \frac{d\mathbf p}{dt} + \frac{d\mathbf p_{lost}}{dt}~~~(4).
$$
1)
This can be done because each particle obeys Newton's law $\mathbf F= m\mathbf a$, as it does not lose or gain parts. One way to derive this equation goes like this.
Let us use convention where $\mathbf F_{a}$ means force due to body $a$ on something, $\mathbf F_{-b}$ means force acting on body $b$ due to something, and $\mathbf F_{a,-b}$ means force due to body $a$ acting on the body $b$.
It is easy to see that at time $t$
$$
\sum_{i\in V(t)} \mathbf{F}_{ext,-i} + \sum_{i \in V(t)} \mathbf F_{parts~outside,-i} = \sum_{i\in V(t)} m_i \mathbf a_i.~~~(a)
$$
We would like to express this without summing over index $i$, using only quantities referring to the system inside and outside as a whole.
Let $\mathbf p$ be momentum inside the control volume $V$. This changes in time for two reasons:
1. particles that stay inside change their momentum
2. some particle leave or come in the control volume
So we can write
$$
\frac{d\mathbf p}{dt} = \sum_{i\in V} m_i \mathbf a_i - \frac{d\mathbf p_{lost}}{dt}~~~(b)
$$
where $d\mathbf p_{lost}$ is momentum lost from the control volume due to particles leaving, per time $dt$).
Comparing $(a),(b)$ we see that the sum equation can be expressed more over $i$ can be removed resulting equation of motion for the system (in the control volume) can be written more concisely as
$$
\mathbf F_{ext} + \mathbf F_{parts~outside} = \frac{d\mathbf p}{dt} + \frac{d\mathbf p_{lost}}{dt}
$$
which is the equation (4).
In case of mass leaving the system (a rocket), we can write this in an easier-to-remember way
$$
\mathbf F_{ext} + \mathbf F_{parts~outside} - \frac{d\mathbf p_{lost}}{dt} = \frac{d\mathbf p}{dt}~~~(5).
$$
When we apply this to a rocket, we can see that momentum of the rocket changes due to 1) external force, 2) force of the exhaust gases acting back on the rocket, but decreased by momentum lost from the rocket per unit time (due to exhaust gas leaving the system).
Although more general, this is somewhat foreign to the engineering viewpoint on rockets, because for the purpose of travel, rather than in momentum of the rocket, we are interested in its velocity.
In the simplest case where the lost particles all leave in direction same or opposite to the body velocity $\mathbf v$ (idealized rocket), this can be further simplified, as is common in textbooks. Let the boundary of the control volume be far from the rocket, so that velocity of particles crossing the boundary (relatively to the rocket) is constant $\mathbf{c}$, and $\mathbf{F}_{parts}$ acting back on the system in the control volume is negligible (the exhaust gas is rarified). Then the lost momentum per unit time is
$$
\frac{d\mathbf{p}_{lost}}{dt} = - \frac{dm}{dt}(\mathbf v+\mathbf c)~~~(6)
$$
and the equation of motion simplifies into
$$
\mathbf{F}_{ext} + \frac{dm}{dt} \mathbf c = m\frac{d\mathbf{v}}{dt}.~~~(7)
$$
It is important to realize also that $\mathbf v$ is not velocity of center of mass of the system, but is defined as $\mathbf p/m$ where $\mathbf p$ and $m$ are momentum and mass inside the control volume. The necessity of this distinction is best seen from this example: let the body have constant velocity $\mathbf v$, but let the control volume shrink so that less and less of the body is inside. Center of mass of the control volume has different velocity from $\mathbf v$, in fact it accelerates due to moving boundary of the control volume. However, velocity of the material particles does not change at all.
