Newton's second law: system with three blocks and a pulley Here's a common Newtonian mechanics problem.
What horizontal force must be applied to a large block of mass $M$ shown in the figure so that the blocks of mass $m_1$ and $m_2$ remain stationary relative to the large block? Assume all surfaces, string and pulley are inextensible(nonstretchable), massless and frictionless.

To remain stationary, acceleration of $m_1$ should be equal to that of $M$, and let it $a$.
Applying Newton's second law to the horizontal motion of $m_1$ yields $T=m_1 a$, where $T$ is tension of the string. For vertical of $m_2$, $T-m_2 g=0$. Therefore, $a=\frac{m_2}{m_1}g$.
For whole system, I came up with two solutions.


*

*Since "all surfaces, string and pulley are inextensible(nonstretchable), massless and frictionless", force $F$ cannot exert force on $m_1$, so force $F$ is exerted on system consisting of $M$ and $m_2$. Therefore, $a=\frac{F}{M+m_2}$ and $F=\frac{(M+m_2)m_2 g}{m_1}$.

*Force $F$ is exerted on system consisting of $M$, $m_1$ and $m_2$. Therefore, $a=\frac{F}{M+m_1+m_2}$ and $F=\frac{(M+m_1+m_2)m_2 g}{m_1}$.


Which is right? If latter one is right, how force $F$ is exerted on $m_1$?
 A: The latter is right. 
If you take the forces acting on $M$ alone, you have - the normal force from $m2$. This is just $m2*a$. You also have the horizontal component of the force from the string on the pulley. This is (think about it) equal exactly to $m1*a$
A: The latter is right. Note, that "F  is exerted on $m_1$" is no physical requirement. What you do require though, is that the force $F$ is applied on the whole system consisting of all three masses. Whatever happens internally; inside the system, pulleys, strings, mass blocks etc is nothing bother about as far as acceleration of the whole system is concerned. That is just total force applied divided by total mass.
As for your question, "which force accelerates $m_1$ horizontally with acceleration $a$?", it's the string (through tension), which is in turn pulled by the pulley. How? Note that the pulley applies a force on the string in the $\frac{1}{\sqrt{2}}(\hat{x}+\hat{y})$ direction (the direction normal to the surface of the pulley-string contact); The horizontal component of which causes the tension.
A: The latter solution ie 2nd one is correct...it can be explained by the concept of psuedo force....The larger block exerts a psuedo force on the mass m1 in the backward direction which balances the tension in the string which turn balances the weight of the block (=m2xg). This psuedo force is exerted on the mass m1 due to the acceleration of the larger block.
Now to understand the concept of psuedo force in regard to the question...which says that the two blocks are at rest relative to the larger block.....all u have to do is to imagine urslef on the larger block and write down the Newtonian eqns of motion of the blocks m1 and m2....now since the block m1 is at rest in ur new reference frame....u have to mention a psuedo force in ur eqns...since ur new frame is non inertial....solve the new eqns. and u would get the required result which is the same as in ur 2nd solution.
A: Gravity pulls down on M and m1, but the ground holds them up. These vertical forces are irrelevant in the absense of friction to convert them to horizontal forces.
Gravity also pulls down on m2. For it not to move, the tension in the string must be just enough to hold it up.
T = m2 g
Force F pushes horizontally on cart M, which pushes on block m2. They therefore accelerate at
a = F / (M + m2).
In the absence of friction, F cannot impart any horizontal force to m1. The tension in the string is the only horizontal force on m1, so it determines m1's acceleration.
a = T / m1
These accelerations need to be equal for the system to move together.
m2 g / m1 = F / (M + m2)
A: 1st one is correct.The equation describes the acting of the external force,as theres no external horizontal force acting on m1,its not in the equation,however it is accelerated by the tension of the string. horizontally on cart M, which pushes on block m2. They therefore accelerate at
a = F / (M + m2).
In the absence of friction, F cannot impart any horizontal force to m1. The tension in the string is the only horizontal force on m1, so it determines m1's acceleration.
a = T / m1
These accelerations need to be equal for the system to move together. m2 g / m1 = F / (M + m2)
