I was reading this notes about chemical potential. I am confused about some things in section 2: Potential energy and the chemical potential
Let us look at two systems A and B a the same temperature that may exchange particles, but the two systems are not yet in diffusive. We assume that we start from $\mu_B>\mu_A$ so that particles will flow from B to A. The difference in chemical potential is $\Delta \mu=\mu_B-\mu_A$.
...
Let us now reanalyze the system thermodynamically. We have changed the potential energy of all particles in system A, but we have not made any changes in system B.
The author claims that while $U_A$ and $F_A$ increase, $U_B$ and $F_B$ do not change. This means the chemical potential of $\mu_A$ increases until it is equal to $\mu_B$ but $\mu_B$ stays constant for the whole process.
I don't understand why it is true. I believed that chemical potential is dependent on particle count, so as $N_A$ increases, $\mu_A$ also increases. If $N_A$ is increasing, then it follows that $N_B$ is decreasing since the total number of particles stays constant. When $N_B$ decreases, then $\mu_B$ should decrease as well. This process continues until $\mu_A=\mu_B$.
Am I missing something?