I'm considering a simple binary system made of two stars, of mass $m_1$ and $m_2 < m_1$, on circular orbits around their center of mass. Using Newton's theory of gravitation, it is easy to prove the following formulae for the total mechanical energy and angular momentum of the whole binary ($a \equiv r_1 + r_2$ is the separation of the two masses) : \begin{align} E \equiv K + U &= -\, \frac{G m_1 \, m_2}{2 a}, \tag{1} \\[12pt] L \equiv L_1 + L_2 &= \frac{m_1 \, m_2}{M} \, \sqrt{G M a}. \tag{2} \end{align} Suppose that one star is transfering matter to the other star, without any loss. Conservation of matter implies $M = m_1 + m_2 = \textit{cste}$, so $\dot{m}_2 = -\, \dot{m}_1$. Now, many papers/lectures I've found state that angular momentum is still conserved : $\dot{L} = 0$. This gives an equation for the rate of change of the distance $a$ (the orbits are slowly evolving, while remaining approximately circular) : \begin{equation}\tag{3} \frac{\dot{a}}{a} = -\, 2 \Big( \frac{\dot{m}_1}{m_1} + \frac{\dot{m}_2}{m_2} \Big). \end{equation} See for example these references :
http://www.ast.cam.ac.uk/~pettini/STARS/Lecture18.pdf (see page 8) http://jila.colorado.edu/~pja/astr3730/lecture32.pdf (see page 12)
Equation (3) implies the following : \begin{equation}\tag{4} a(t) = a(0) \Big( \frac{m_1(0) m_2(0)}{m_1(t) m_2(t)} \Big)^2. \end{equation}
My trouble comes from conservation of energy, if the system is really assumed to be isolated (no loss/gain of wathever). Then the derivative of equation (1) above gives this (from $\dot{E} = 0$) : \begin{equation}\tag{5} \frac{\dot{a}}{a} = \frac{\dot{m}_1}{m_1} + \frac{\dot{m}_2}{m_2}, \end{equation} which is incompatible with equation (3) (unless of course $\dot{m}_1 = -\, \dot{m}_2 = 0$, but then there is no mass transfer).
So what is going on here ? Why angular momentum should be conserved while energy is not ? Why not the reverse, i.e. that energy is conserved but not angular momentum ? The authors of the lectures cited above says nothing about the binary's mechanical energy.
EDIT : About the total angular momentum, if we add the contributions from the size and rotation spin of the stars, we get the true conserved total angular momentum. I'm assuming tidal locked stars, with parallel rotation and revolution axes : $\omega_{\text{rot} \, 1} = \omega_{\text{rot} \, 2} = \omega_{\text{orbital}} \equiv \omega$ : \begin{align} L_{\text{tot}} &= L_{\text{orbital}} + S_1 + S_2 \nonumber \\[12pt] &= \frac{m_1 \, m_2}{M} \, \sqrt{G M a} + (I_1 + I_2) \, \omega, \tag{6} \end{align} where $\omega$ is given by Kepler's third law : \begin{equation}\tag{7} \omega = \sqrt{\frac{G M}{a^3}}. \end{equation} I think that the spin contributions and their time variations are negligible in front of the orbital angular momentum, since $I_1 \propto m_1 \, R_1^2$, and $R_1 \ll a$ (same for star 2).
Any clue on this ?