How does the mass loss in a binary system affect the semi-major axis of the orbit? It is common that during the stellar evolution one of the stars in a binary system would transfer mass into the the other, resulting in the increase of mass of one star and decrease of mass of the other. Or, if one of the stars happens to evolve through the red giant phase while the other is still on the main sequence, the loss of the mass should be really significant and consequently the orbit should also be changed dramatically. 
I have a hard time visualize how the semi-major axis is changed as a result of mass loss. According to the Kepler's third law the semi-major axis a is proportional to $(M_1+M_2)^{1/3}$ but I am not sure whether the period can be fixed or not.
 A: The behaviour depends on whether the mass is gained by the companion (conservative mass loss) or lost from the system altogether (unconservative mass loss).
For conservative mass loss, one conserves angular momentum and the total energy and requires that $M_1+ M_2 =M$, which is constant. In this case we have
$$ M_1 M_2 a^{1/2} = {\rm constant},$$
so
$$\frac{a_f}{a_i} =\left(\frac{M_{1,i}M_{2,i}}{M_{1,f}M_{2,f}}\right)^2$$
and what happens to $a$ depends on whether the more massive or less massive star loses mass, since the product $M_1 M_2$ is maximised when the two are equal. Thus if the mass ratio becomes more similar (more massive star loses mass) then $a$ decreases, and vice versa.
Unconservative mass loss is more complicated and depends on how the mass leaves the system. The most simple case is spherically symmetric mass loss that escapes without interacting with the other star and leaving with the specific angular momentum of the mass-losing star. In this case
$$\frac{a_f}{a_i} = \frac{M_i}{M_f},$$
where $M$ is the total system mass. Since $M_f < M_i$ then the separation always widens.
