Moving wedge and pulley system A wedge of mass $M$ rests on a rough horizontal surface with coefficient of static friction $\mu$. The face of the wedge is a smooth plane inclined at an angle $ \alpha$ to the horizontal. A mass  $m_1$ hangs from a light string which passes over a smooth peg at the upper end of the wedge and attaches to a mass $m_2$ which slides without friction on the face of the wedge.
A) Find the accelerations of $ m_1, m_2 $ and the tension in the string when $ \mu $ is very large.
B)Find the smallest coefficient of friction such that the wedge will remain at rest.
Attempts:
No specification on whether or not any of the masses $m_i$ are bigger than the others, so let us assume that $m_1$ goes down and $m_2$ goes up the slope. Then A) is fine. (a minus error in the final answer for A) is introduced if I assume the other orientation)
As for B) if we consider the wedge+masses as a system, then to conserve linear momentum if the masses move as above, then the wedge must move rightwards.  So the frictional force acts leftwards until the wedge slips.  The only other force acting on the wedge is via the normal contact force from $m_2$.  A component of this force acts to accelerate the wedge horizontally.  Let's write this relative to some fixed inertial frame of reference with $x$ rightwards. Then the above recast into symbols gives $$-\mu (M+m_1+m_2)g - m_2 g \cos \alpha \sin \alpha = 0 $$ at the point of slip of the wedge.  However, solving this for $\mu$ gives a negative result.  Where did I go wrong?

 A: Part A
$\mu$ large implies that the wedge does not move. Therefore the two forces on the small masses along their direction of movement are:
$$F_1 = m_1 g, \quad F_2 = m_2 g \sin(\alpha).$$
They pull in different directions on the pulley, so the net acceleration (with respect to $m_1$ going down) is:
$$F = F_1 - F_2 = g [m_1 - m_2 \sin(\alpha)].$$
Then the acceleration of both masses is:
$$ a = \frac Fm = \frac{g}{m_1 + m_2}[m_1 - m_2 \sin(\alpha)].$$
You should not be able to introduce a minus error just by choosing a convention. I assume that you just did not flip all the minus signs.
Part B
The wedge has to be at rest, so the force of friction has to compensate the tangential forces exactly. This means
$$ \mu F_\text{down} \overset != F_\text{tangential}. $$
I now construct the forces that act on the masses:

The tensions will change with the acceleration:
$$ T_1 = m_1[g-a], \quad T_2 = m_2[g\sin(\alpha) + a].$$
To digest this, set $a =0$, the static case. Then the tensions are just the forces that gravity exerts on the masses. The extreme case, where $m_1$ is free falling is $a = g$. Then $T_1$ has to be zero, since the object is free falling.
The mass $m_2$ exerts a normal force onto the wedge that will push it to the left. The tension $T_2$ will drag on the pulley which will then push on the wedge to the right. The force to the left is given by
$$ m_2 g \cos(\alpha)\sin(\alpha)$$
whereas the force to the right (just the $x$-component) is given by
$$ m_2[g \sin(\alpha) + a] \cos(\alpha).$$
In the static case $a = 0$, those two forces are equal and do not make the wedge move. In the dynamic case, they try to move the wedge. This is what you meant by conservation of linear momentum, just written down with the forces.
The net force to the right (assuming $a > 0$ then is
$$ m_2 a \cos(\alpha) = F_\text{tangential}.$$
Now the forces that push the wedge downward and give it a frictional force. The mass $M$ will drag downward. Then $T_1$ will push the pulley down, so does the $y$-component of $T_2$. The last thing is the $y$-component of the normal force. The last two mostly add up to just $m_2 g$, there is still a $m_2 a \sin(\alpha)$ left from the dynamics in $T_2$:
$$\underbrace{m_1[g-a]}_{T_1} + \underbrace{m_2[g\sin(\alpha) + a]\sin(\alpha)}_{{T_2}_y} + m_2 g \cos(\alpha)^2$$
Together, this is:
$$F_\text{down} = Mg + m_1[g-a] + m_2[g + a\sin(\alpha)].$$
Now set them equal, plug in $a$ and solve for $\mu$. You might want to omit all the terms with $a$ here, since that makes it really complicated. If $M$ is large, you can do that without a large error.
