How can the Lorentz transformation cause two objects' masses to both increase? Suppose there are two people A and B. A is standing on the earth and B is on a rocket in space. Now B passes by the earth at a speed of 0.9c and A sees him. According to A, B's mass would have increased. But since the lorentz factor considers only relative velocity, we can say that even A is moving at a speed of 0.9c relative to B. So will A's mass also increase according to B after applying the the lorentz transformation formula?
If yes then my doubt is how can the mass of both increase?
 A: To add to Jinawee's comment:

The relativistic mass of the other will increase for each observer because velocities are relative

and Salvador2395's answer: if you don't like the concept of relativistic mass, presumably you are just as worried by any quantity scaled by $\gamma$ under a boost and so your question is wholly analogous to the pondering of the Twin Paradox. Read the Wiki page of this name and I'm sure you'll be some wiser. Although it is a bit weird, there is no contradiction in both relative-to-one's own-time-lapse being shorter because both observers are in different inertial frames, and so are not comparing the same quantities, so one doesn't get to say a quantity is strictly less than itself! Three points you might find enlightening:


*

*If you impart the full Lorentz transformation, i.e. a matrix transformation involving all the spacetime co-ordinates to take you from A to B and then impart the Lorentz transformation back, you find that they are indeed inverses of one another and their composition is indeed the identity matrix, even though the time dilation factors $\gamma$ are the same (not reciprocals of one another) in both directions! To understand this in detail you probably have to derive the transformations and their composition in detail: see the section "From Group Postulates" on the "Lorentz Transformaton" Wikipedia page to help you. 

*If the two observers are brought back to the same inertial frame so that they can truly compare notes and times and distances measured by one another, then at least one of the observers must accelerate, in other words, they are not in an inertial frame at all times, and so the simple time dilation calculation as you are thinking of it can only be done from the inertial frame twin. Therefore the twin who runs off, decelerates, then comes back truly does age less than the always-in-inertial-frame-twin. You need to imagine the thought experiment done with both observers taking a system of accelerometers with them so that they can be aware of whenever their frame is not inertial. Acceleration defined in this way in relativity, general or special, is absolute (it defines the geometry, and its deviation from Euclid, so it is a "shape of a fabric"), so you may need to ponder that carefully: Einstein nowadays seems to be held up in Arts and Letters Faculties as a pseudo-justification that everything is relative, so this erroneous idea has become part of our culture. You can also think of the problem from the standpoint of the twin who decelerates but this is much more complicated and the calculation is also sketched for you on the Twin Paradox Wiki page. The decelerating twin's reckoning of the inertial twin's time is dilated as they coast away and back, but there is a "catchup" wrought in their frame by the acceleration phase, so that the nett effect is the same as calculated from the inertial frame.

*The twin paradox is "resolved" (i.e. the point in 2. vindicated) everyday in particle accelerators, where we accelerate short lived particles and "bring them back to our (almost) inertial frame" by detecting their decay product in our laboratory. We witness experimentally that they do live longer by exactly the time dilation factor $\gamma^{-1}$.
Edit: After OP's question:

So what do you believe. Will it increase for both observers or not? 

I answer thus: If you want to stick with relativistic mass, then, yes, for both observers the other's relativistic mass will increase, because each perceives the other to have more kinetic energy than they do. This is one of the reasons why relativistic mass is thought of as a bad teaching conception, but it is no more contradictory than the idea that each observer seems to have more energy than the other (each thinks, "I'm still, he's going by swiftly"). However, if we concentrate on the total energy rather than construing it as showing up as increased relativistic mass, then we "resolve" (note my quotes, there is simply weirdness that needs to be grasped rather than a contradiction to be untangled) the "paradox" of each observer seeming to have more energy than the other exactly as we do in my point 1. above. Namely, energy is part of a four-vector with an invariant length: it belongs, as the time component, to the momentum four-vector:
$$\mathbf{P} = \left(\frac{E}{c},\,p_x,\,p_y,\,p_z\right)$$
and the invariant length is $m_0\,c$, where here $m_0$ is now the invariant rest mass, not the relativistic mass. $E$ is the total energy and $p_j$ the spatial components of the vector momentum. So if we write energy instead of time and vector momentum instead of position in my point 1. above, the full Lorentz transformation imparted to $\mathbf{P}$ to take you from frame A to frame B and then impart the Lorentz transformation back, you find that they two Lorentz transformations are inverses of each other, even though they both have the same dilation factors $\gamma$. You can hopefully see that it is a little awkward to talk about the same idea in terms of relativistic mass, because the relativistic mass does not belong as a component to a four vector. This is the reason that relativistic mass has fallen out of favour as a concept.
Edit 2: In answer to OP's comment:

Thanks though I did not understand the vector part because I am still a grade 9 student.

I'm sorry, your question was well written at an early university level. Think of kinetic energy, and, if you like, forget about special relativity even: the following statement holds in "everyday" Galilean relativity as well: eergy is a relative concept - an inertial observer's reckoning of an object's energy depends on the frame of reference. Kinetic energy is the same as increased relativistic mass. So your proposed scenario is no less paradoxical than the following "everyday" case: if you're driving a bus in deep space and you see the same kind of bus coming towards you: the question of who has THE kinetic energy is meaningless: each of you drivers could equally well construe yourself to be stationary and the other to have all the kinetic energy. Or even that each of you has the same kinetic energy. Your scenario with both observers measuring the other to have increase relativistic mass is exactly the same as this (increased mass standing for kinetic energy) and the conclusion is actually exactly the same in my bus scenario and independent of whether you use special or "everyday" relativity (although the numerical values will be slightly different in both relativities).
Now let me try to reword the vectors bit: energy to momentum is what time is to the position vector. Both sets of quantities transform between different inertial frames of reference in exactly the same way. So what I'm saying is: think of energy as being relative - you can describe this by the Lorentz transformation. Relativistic mass, on the other hand is an "orphan" - it doesn't mix with other quantities like time/distance or energy/momentum. So nowadays we simply think of the total energy rather than mass: energy has inertia - that's always a given - so there is no need to describe it by saying the mass increases. We simply talk about the rest mass as the relevant property: then we write $E^2−p^2\,c^2=m_0^2\,c^4$: this last equation expresses the contant "length" of the energy/ momentum four-vector and generalises the equation $E=m\,c^2$, which is actually not always true: it only holds in a frame of reference that is stationary relative to the object in question, otherwise you need to use the more general equation $E^2−p^2\,c^2=m_0^2\,c^4$.
A: There are some authors who do not consider the concept of 'relativistic mass' to be useful. In particular, the concept does not hold up for $F=ma$ since relativistic force is not $ F = \gamma ma $. For more please read: David Morin: "Introduction to Classical Mechanics". Regarding the issue, perhaps you may consider it the same as if it were time or length, in each case, each observer sees the corresponding relativistic effect (time dilation and length contraction respectively) because it is relative to his particular reference system. 
A: Mass is the rest mass. It is a constant and does not vary with speed.
