Why does low magnitude of force imply that all components of a system have same acceleration? 

The surface between block M and ground (green) is frictionless, the surface between the both block M and m have coefficient of static friction $\mu$. An external force $\vec{F}$ is applied on the right block of mass $M$

It turns out that for small magnitude of $F$, the acceleration $a$ of all blocks are the same.. but why? I got the set up from this video by Prof. Walter lewin , He mentions this point at 5:46 but I don't understand what the idea behind it is ( click here for time stamped moment)
Ideally looking for a mathematical proof
 A: The large block on the right is accelerated directly by the force $F$, but the block on top of it it accelerated by the frictional force between the two blocks. This frictional force has an upper limit on the magnitude of force it can impart, dictated by the coefficient of friction and normal force. If the large block is accelerated too quickly, friction won't be able to compensate, and the small block on top will slip relative to the large block below it. When $F$ is small enough, friction is able to accelerate the small block at the same rate that $F$ accelerates the large block, meaning there is no relative movement and everything accelerates at the same rate. You have the same situation on the left in reverse, where the small block must accelerate the large one through friction alone.
This is basically how the trick of pulling a tablecloth out from a fully set table works. If you pull lightly, friction is sufficient to make everything on the table move with the tablecloth, causing everything to crash to the floor. If you pull sharply with a lot of force, however, friction can't compensate, and there is relative movement between the tablecloth and table setting, allowing you to remove the tablecloth while moving the table setting very little. In this case, a light tug will cause all the blocks to move together since there is no slipping, but a sudden yank will pull out the rightmost block and leave everything else in almost the same place.
Ultimately, we see that as long as the force is low enough, all blocks must have the same acceleration. The large block on the right accelerates at $a$ from the force directly. If the magnitude of acceleration is low enough, the block on top doesn't slip, and also accelerates at $a$. The rope between the small blocks is idealized (massless and cannot stretch), so the block on the top left must also accelerate at $a$ (or else the rope would stretch/snap). Finally, as long as we don't have slipping, the large block on the bottom left must also accelerate at $a$ (or else there would be slipping). When the force is low enough, there is no relative motion between any of the blocks, so they therefore must have the same velocity and acceleration at all times.
A: 
I read through the comments, here's my take. Consider the simplified setup above.
Imagine two stacked blocks, A on top of B, friction between A and B but not between B and the ground. Apply a small force to the bottom block B.
To prevent block B from sliding with respect to block A, static friction between blocks A and B kicks in. We know that block B will definitely move, since there is an unbalanced external force acting on it (ground exerts no friction on B). However, the following is implied: if block B moves, block A must also move so there is no sliding.
Static friction will always oppose sliding between 2 surfaces, and the only way that sliding can be prevented while block B accelerates, is if block A moves the same way.
Now, why does this only work with a small force? Apply a force to block B; provided you also don't lift block B, the normal force between blocks A and B won't change.
Our friction model states that the maximum friction is $f_{\rm max} = \mu N$. We just constrained the normal force $N$ to a constant, which means that the maximum friction force depends only on the coefficient of friction.
Therefore, since friction is the only force accelerating block A, there is a maximum acceleration value that corresponds to block A before friction becomes too weak to increase the acceleration any more.
Mathematically, the maximum acceleration of block A is given by the friction force $f=m_Aa=\mu m_Ag\implies a=\mu g$. Since, $m_Ba = F - f$, you can solve for the maximum force $F$.

Now, for your case with 2 sets of blocks attached to each other: the way I would think about it is by splitting the system into 2. The first system is the top and bottom block which you're pulling. The only difference with my simplified example is that you also have a tension force.
Then, the same principle is applied to the second set of blocks. The tension force is applied to the top block, but the bottom block does not want to slide, so it exerts a static friction force on the top block to prevent the top block from moving, and by Newton's third law, the bottom block must accelerate.
TLDR: The key idea here is that static friction will prevent sliding by applying a force such that there is no sliding. The only constraint is the magnitude of that force, which depends only on the coefficient of friction (provided normal force does not change).
Hope this helps.
A: If you apply a large enough $F$, you will be able to pull out the right block of mass $M$ from under the right block of mass $m$. The assumption of "small magnitude of F" is to prevent that from happening (and avoid pulling off the left block of mass $m$ from the top of left block of mass $M$, and also prevent the string in between from breaking up) so all blocks of the system will move together as they have been. This is what that assumption is for.
