Center of mass motion and variation of mass Here are the proofs regarding the center of mass motion as reported on my book.
$$\vec{r_{cm}}=\frac{\sum\vec{r_i} m_i}{\sum m_i}$$
$$\vec{v_{cm}}=\frac{d{\vec{r_{cm}}}}{dt}=\frac{1}{M}\sum \frac{d}{dt} m_i \vec{r_i}=\frac{1}{M} \sum m_i \vec{v_i}=\frac{1}{M} \vec{P} \tag{1}$$
$$\vec{a_{cm}}=\frac{d{\vec{v_{cm}}}}{dt}=\frac{1}{M}\sum \frac{d}{dt} m_i \vec{v_i}=\frac{1}{M} \sum m_i \vec{a_i}=\frac{1}{M} \vec{F^{(EXT)}}=\frac{1}{M} \frac{d\vec{P}}{dt} \tag{2}$$

Both in $(1)$ and $(2)$ derivatives this assumption was made:  the mass of the system $M$ and the mass of each point $m_i$ are constant. Otherwise the derivatives would have been much more complicated.
But I also read that $$\vec{F^{(EXT)}}= \frac{d\vec{P}}{dt}\tag{3}$$
Holds true also if the mass is not constant.
Nevertheless to prove $(2)$ (and so $(3)$) the assumption of constant mass was used, so how can $(3)$ be true without that assumption?
And if $(3)$ holds which mass can vary? The mass of the system $M$ or the mass of the single points $m_i$?
 A: For Newton's Laws to hold, mass must not vary.  Wherever you read otherwise was mistaken.
Take a look at this answer.  It contains a description of why mass must be constant in Newton's Laws in the context of the rocket equation ... but the analysis applies generally.  Newton's Laws are not valid for variable mass systems.
A: One could not defend a book you are most probably misquoting. I strongly suspect the book says "the mass distribution is not constant", that is M is constant but the distribution and number of constituent $m_i$s may vary, i.e. they may split or agregate, a common feature in astrophysics. You are confusing yourself with symbols and definitions and proofs. 
The problem is trivial if you start with one particle of mass M which splits into two equal particles of mass m each, by a small explosion, and you find the cm position, velocity, and acceleration before and after the split, so you convince yourself (1), (2) and (3) are preserved. Then, and only then, bother with generic splits, constituent $m_i$s and positions. 
A: Just a note, you can rewrite (3) as
$$\vec{F} = \frac{d\vec{P}}{dt} = \frac{dm}{dt}\vec{v} + m\frac{d\vec{v}}{dt}$$
If you have a system where the total mass is not changing, then $\frac{dm}{dt}=0$, which brings us back to the form we're more commonly used to. 
