Net force and CM acceleration pointing in different directions This is problem 15 from Chapter 9 of Halliday & Resnick, 10th Ed. (I am only considering part a.)

The answer turns out to be that the acceleration of the CM is (2.35 m/s/s, -1.57 m/s/s). This makes sense, as just from intuition I can see the CM must move down and to the right.
However, if I think of this in terms of the Newton's 2nd Law equation for a composite system,
F_net = M*a (where a is CM acceleration and these are vectors)
it no longer makes sense to me, because the net force on the system is the weight of the hanging block acting only downward, whereas the CM accelerates down and to the right. So as a vector equation it seems like the two sides can't be equal.
What am I missing here? How can the CM acceleration have a horizontal component when the net force is only vertical?
 A: You're missing Newton's third law. The system of masses pushes down on the rail and the pulley. The rail and the pulley push up on the system of masses with an equal and opposite force and push down on the planet with the original force. The planet pushes up on the rail and the pulley with an equal and opposite force.
If you took away the planet but kept the gravity (i.e. you dug a big hole and dropped the system of masses into it), obviously the system of masses would plummet straight down at $g$.
Regarding the horizontal acceleration of the center of mass: the planet is part of the system and is accelerated ever so slightly to the left. The center of mass of the planet-blocks-rail-pulley system is indeed stationary (that is: comoving with the the original trajectory of the system before the blocks were released).
A: The external forces on the cart-cord-pulley-block system are:

*

*gravity on the cart, $-m_1 g \ {\hat {\bf y}}$

*a normal force on the cart from the track, $m_1 g \ {\hat {\bf y}}$ (equal and opposite to the gravity force on the cart)

*gravity on the block, $-m_2 g \ {\hat {\bf y}}$

*a normal force on the pulley from the axle, ${\bf N} = N_x {\hat {\bf x}} + N_y {\hat {\bf y}}$
If you draw a free body diagram of the pulley and a portion of the cord (with tension $T$), you can see that the sum of the forces is ${\bf N} -T \ {\hat {\bf x}} - T \ {\hat {\bf y}} = 0$, which implies $N_x = N_y = T$.
Newton's second law for the cart-cord-pulley-block system with mass $M = m_1 + m_2$ is then
$$
T \ {\hat {\bf x}} + \left(T - m_2 g\right) {\hat {\bf y}} = M {\bf a}
$$
If you consider the cart and block separately, you should find that
$$
T = \frac{m_1 m_2 g}{M}
$$
