I know that relativistic mass is calculated as:
But when mass m0 gets bigger, also mass of the accelerating body mass m gets bigger so m0 gets bigger and so on.
So if body accelerates with acceleration a in time t and original mass m, how big will be the m0?
If the formula will be just the same, but with different v, why doesn't it change, why do we calculate only the original mass? And what really happens to mass when it accelerates?
The mass $m_0$ is the rest mass, also known as the invariant mass. It is a constant and is the same for all observers regardless of how they are moving. So you are overcomplicating your calculation - just hold $m_0$ constant and calculate $m$ by putting in the velocity $v$.
However, as you will find if you study physics further, the relativistic mass is not a useful concept as the mass does not change. The idea of relativistic mass was introduced (I think) because at relativistic velocities the momentum and kinetic energy of a moving body do not follow the simple Newtonian equations. Specifically, the momentum is given by:
$$ p = \frac{m_0v}{\sqrt{1 - v^2/c^2}} \tag{1} $$
and the total energy by:
$$ E^2 = p^2c^2 + m_0^2c^4 \tag{2} $$
If we write $m = m_0/\sqrt{1 - v^2/c^2}$ then it simplifies the first equation to $p = mv$, which looks like the Newtonian equation. However this doesn't help simplify equation (2). No physicist today uses the concept of relativistic mass, and if you go on to study physics you will not be taught the concept. It is entirely unnecessary and creates the sort of confusion that you ran into.
