Can't understand derivation of relativistic mass in Kleppner In Kleppner and Kolenkow chapter 13, they derive the expression of relativistic mass by considering a symmetric glancing elastic collision.

It was analyzed from two reference frames. One in which the velocity of A in the x direction was zero and another one in which the velocity of B in the x direction was zero.

Here is how the derivation goes in the book:

Our task is to find a conserved quantity analogous to classical momentum. We suppose that the momentum of a particle moving with velocity $\mathbf{w}$ is  $$\mathbf{p} = m(w) \mathbf{w}$$ where $m(w)$ is a scalar quantity yet to be determined, analogous to Newtonian mass but which could depend on the speed $w$.
The x momentum in A’s frame is due entirely to particle B. Before the collision B’s speed is
$w = \sqrt{V^2 + u_0^2/\gamma^2}$ and after the collision it is $w' = \sqrt{V^2 + u'^2/\gamma^2}$. Imposing conservation of momentum in the x direction yields $$m(w)V = m(w')V$$ It follows that $w=w'$, so that  $$u' = u_0$$ In other words, y motion is reversed in the A frame.
Next we write the statement of the conservation of momentum in the y direction as evaluated in A’s frame. Equating the y momentum before and after the collision gives
$$-m_0 u_0 + m(w) \frac{u_0}{\gamma} = m_0 u_0 - m(w) \frac{u_0}{\gamma}$$
which gives  $$m(w) = \gamma m_0$$ In the limit $u_0 \rightarrow 0$, $m(u_0) \rightarrow m(0)$, which we take to be the Newtonian mass, or "rest mass" $m_0$, of the particle. In this limit, $w = V$. Hence $$m(V) = \gamma m_0 = \frac{m_0}{\sqrt{1 - V^2/c^2}}$$
Consequently, momentum is preserved in the collision provided we define the momentum of a particle moving with velocity $\mathbf{v}$ to be  $$\mathbf{p} = m \mathbf{v}$$ where
$$m = \frac{m_0}{\sqrt{1 - v^2/c^2}} = \gamma m_0$$

Now I have a few problems with this derivation. They are:

*

*They assumed that both A and B has the same mass. Unless I'm mistaken, the momentum equation in the $x$ direction should remain unchanged because during the collision the impulse is in the $y$ direction. So suppose the masses were different, namely $m_A$ and $m_B(w)$. Then since
$$m_B(w)V = m_B(w')V$$
it follows that $$u' = u_0$$ But then the y equation becomes
$$-m_A u_0 + m_B(w) \frac{u_0}{\gamma} = m_0 u_0 - m_B(w) \frac{u_0}{\gamma}$$
or
$$m_B(w) = \gamma m_A$$
which is weird because B's mass shouldn't depend on A's. Personally I think that their argument for $u' = u_0$ is flawed. Because it doesn't matter whether A and B's masses are different but intuitively I think it should. I don't see how the collision could be elastic and symmetrical without the two particles having the same mass. Because I could always take extreme cases when one is much more massive than the other and following their argument we would still have $u' = u_0$. Or maybe I misunderstood their argument and the masses really do matter. This is all very confusing to me.

*It seems that when writing the momentum equation in the $y$ direction, the author represented $m(u_0)$ as $m_0$ while in the final equation they meant $m_0$ to be the rest mass which makes sense because A was also moving in the y direction in A's frame so it's mass can't be just the rest mass $m_0$. However before taking the limit $u_0 \rightarrow 0$, $m(u_0) \rightarrow m(0)$, the equation for $m(w)$ was
$$m(w) = \frac{m(u_0)}{\sqrt{1 - V^2/c^2}}$$
and after taking the limit it became
$$m(V) = \frac{m_0}{\sqrt{1 - V^2/c^2}}$$
However both equations are supposed to be true and using the final result we should have
$$m(w) = \frac{m_0}{\sqrt{1 - w^2/c^2}}$$
and similarly for A in A's frame,
$$m(u_0) = \frac{m_0}{\sqrt{1 - u_0^2/c^2}}$$
Substituting this in the first equation
$$m(w) = \frac{m(u_0)}{\sqrt{1 - V^2/c^2}}$$
$$ = \frac{m_0}{\sqrt{(1 - V^2/c^2)(1 - u_0^2/c^2)}}$$
$$ = \frac{m_0}{\sqrt{1 - (u_0^2/c^2 + V^2/c^2) + (Vu_0)^2/c^4}}$$
$$ = \frac{m_0}{\sqrt{1 - w^2/c^2 + (Vu_0)^2/c^4}}$$
$$ \neq \frac{m_0}{\sqrt{1 - w^2/c^2}} $$
I'm probably missing something but I can't figure out what.

 A: 
I don't see how the collision could be elastic and symmetrical without the two particles having the same mass.

You're right, but this isn't a mistake in the argument. The authors are assuming a specific situation and using that to derive general constraints. If you change the assumptions, you would get a different, more complicated setup, which would not be useful.
What you are saying is analogous to this:
Klepper: Let $x$ be the number of cows. Since you can't have a negative number of cows, $x \geq 0$.
You: But what if $x$ isn't the number of cows? Then it could be negative, so your argument is flawed.
In response to comments: there's indeed another step that Kleppner has let go implicit. Kleppner has assumed that it is possible for such a collision to happen. And as you point out, it wouldn't be possible if the masses weren't equal, either in relativistic or nonrelativistic physics.
So here's an argument for why it's possible when the masses are equal. The initial state in the lab frame has equal mass particles traveling with opposite velocities. As long as "momentum" flips sign when the sign of the velocity is flipped, the initial momentum must be zero. By the same logic, the final momentum is also zero. So the setup is consistent with momentum conservation.
Then you might ask, how do you know that momentum flips sign when the velocity flips sign? But this is similar to asking "how do you know that $x$ means the number of cows?" We are searching for conserved quantities in a new context, and a conserved quantity would only deserve the name "momentum" if it satisfied that basic requirement.

It seems that when writing the momentum equation in the $y$ direction, the author represented $m(u_0)$ as $m_0$

The argument is correct, but their notation is very confusing, because it's not explicit enough. By making all mass-dependence in the $m$'s and $\gamma$'s explicit, their $y$-momentum equation rearranges to
$$m(u_0) u_0 = m(w) u_0 / \gamma(V).$$
Upon cancelling a $u_0$ and taking $u_0 \to 0$ on both sides we have
$$m(0) = m(V) / \gamma(V)$$
which is precisely the desired conclusion. Now, your question is whether this is self-consistent, if we substitute it back into the original equation. If we do that, we get
$$\gamma(u_0) m(0) u_0 = \gamma(w) m(0) u_0 / \gamma(V)$$
and cancelling factors gives
$$\gamma(u_0) \gamma(V) = \gamma(w).$$
Taking the inverse square of both sides gives
$$(1 - u_0^2)(1-V^2) = (1-w^2)$$
where I set $c = 1$. Simplifying a bit gives
$$u_0^2 + V^2 - u_0^2 V^2 = w^2.$$
Since the velocity $w$ has components $V$ and $u_0 / \gamma(V)$, we have
$$w^2 = V^2 + (u_0 / \gamma(V))^2 = V^2 + u_0^2 (1 - V^2)$$
which precisely matches the desired left-hand side. So it is self-consistent.
