The angular momentum of a star in orbit is $m r^2 \omega$. Thus the total angular momentum of a binary system is constant (if no mass is lost from the system) and given by $$ J = (m_1 a_1^2 + m_2 a_2^2) \omega,$$
where $a = a_1 + a_2$ is the separation, $m_1 a_1 = m_2 a_2$, the total mass $m = m_1+m_2$ and (from Kepler's 3rd law) $a^3 \omega^2$ is constant (call it $\alpha$), if the mass lost by one star is accreted by the other.
Thus
$$ J = m_1 a_1 a \omega = m_1 a_1 a^{-1/2} \sqrt{\alpha}$$
and
$$ ma_1 = m_1 a_1 + m_2 a_1 = m_2 a$$
thus
$$J m\alpha^{-1/2} = m_1 m_2 \sqrt{a}$$
The LHS of the above expression is a constant, so the ratio of final to initial separation after any conservative mass loss is
$$\frac{a_f}{a_i} =\left(\frac{m_{1,i} m_{2,i}}{m_{1,f} m_{2,f}}\right)^2$$
The product $m_1 m_2$ is maximised (if the total mass is constant), when $m_1 = m_2$. Thus if mass is lost by the less massive star, then the final $m_1 m_2$ becomes smaller and the binary widens. This is what happens in an Algol-type system -- a less massive red giant loses mass to a more massive main sequence companion and the orbit widens.