Introduction:
In Special Relativity, when we say a quantity is an invariant, we mean that it is a Lorentz invariant. What this means is that when you apply the Lorentz transforms to any frame of reference, the magnitude of the Lorentz invariant does not change. The Lorentz invariant is a scalar, which means that it can only have a magnitude, and can carry no other information. Instead, if we take something like a vector (something which has magnitude but also contains information about direction), then the magnitude would still be the same, but the information about the direction changes, so it could not be an invariant.
Also, in Special Relativity we say that events occur in 'spacetime', which is just a 4-dimensional 'stage' for events, including 3 dimensions of space and 1 dimension of time. In ordinary Newtonian mechanics, we conversely use space and time separately, and things like momentum are described only in 3 dimensions.
Because of this fact, physical concepts, like momentum and velocity, are described not by 3-dimensional vectors (for eg. $p = (p_x + p_y + p_z)$), but rather by four dimensional vectors, called four-vectors. Four-vectors have four components: the first one of time and the other three of space.
One more thing: in any transformation, the dot product of a vector with itself will always be an invariant, whether you are talking about $3D$ space or $4D$ spacetime. 
The dot product $\vec{V} . \vec{V}$, remains the same when you change your coordinate axes (dotted lines).
Before we start: Conventions:- We assume natural units ($c = 1$), and the $(+---)$ metric. We label the time component as $0$, while the space components are labelled $1, 2, 3$.
1. How to take dot products of four vectors:
Let's say we have a four vector $A^{\mu}$. Now, how do you take the dot product of this vector with itself? If instead you had a $2D$ vector (say, $\vec{V}$), the dot product would have been $\vec{V} . \vec{V} = V_x^2 + V_y^2$. Now for four vectors, the dot product would be: $$A_{\mu} A^{\mu}$$
Notice that we have $A_{\mu}$, which is not the vector we started with. How do we get this? Like this: $$A_{\mu} = g_{\mu \nu} A^{\nu}$$
where $g_{\mu \nu}$ is the metric tensor. In flat spacetime, we use the Minkowski metric $\eta_{\mu \nu}$, given by:
$$ g_{\mu \nu} =
\begin{pmatrix}
1 & 0 & 0 & 0 \\
0 & -1 & 0 & 0 \\
0 & 0 & -1 & 0 \\
0 & 0 & 0 & -1 \\
\end{pmatrix}
$$
This is what I meant by the (+---) in the convention.
So, $$A_{\mu}A^{\mu} = (A^{0})^{2} - (A^{1})^{2} - (A^{2})^{2} - (A^{3})^{2} = (A^{0})^{2} - (A)^{2}$$
where $A$ is the vector describe in the three dimensions of space only. We subtract the space components, because of the negatives (+---) convention (the + means that time components remain positive, while the three - means that space components get negative).
2. Why mass is an invariant.
In SR, momentum (like other quantities) is also described by a four-vector. Mathematically, $$P^{\mu} = (E, p_x, p_y, p_z)$$
where $E$ is the energy, and the other components are the $3D$ momentum. Now, there is another relation: $$E^2 = \vec{p}^2 + m^2$$ where $m$ is the mass and the $p$ is the momentum (not the four-vector though!). Now, taking the dot product of $P^{\mu}$ with itself, $$P_{\mu}P^{\mu} = (P^0)^2 - \vec{p} = E^2 - \vec{p}^2$$ But, $E^2 = \vec{p}^2 + m^2$, therefore $$E^2 - \vec{p}^2 = m^2$$ So we get that the mass squared is the dot product of the momentum four-vector with itself. Therefore, $m^2$ is invariant. Therefore, $m$ is also an invariant.
3. So what's the deal about relativistic mass?
When an object is at rest, it has no momentum. Therefore we get $$E = m$$ When you apply the Lorentz transformation on the energy you get $$E^{'} = \gamma E - \beta \gamma P^X$$ but the momentum is $0$, so you get (because $E = m$) $$E^{'} = \gamma m$$ This is just $$E = \gamma mc^2$$ where I have taken $c = 1$, remember? But, this doesn't look so elegant, so some people replaced $\gamma m = M$ and got the world's most famous equation: $$E = Mc^2$$ But, relativistic mass is just a mathematical trick. We showed that mass is an invariant. When you multiply that by $\gamma$, you get $m_{rel}$, but it's not real. Your mass does not increase or decrease depending on your velocity. Rather, your inertia increases as your speed increases, that is why it takes infinite energy to accelerate an object to the speed of light.
For further info:
About how we see mass today: Why is there a controversy on whether mass increases with speed?
When relativistic mass can be useful: http://math.ucr.edu/home/baez/physics/Relativity/SR/mass.html
More about four-vectors in relativity: http://hyperphysics.phy-astr.gsu.edu/hbase/Relativ/vec4.html
Some resources on vectors and tensors (generally, the field of linear algebra):
https://www.mathsisfun.com/algebra/vectors.html
https://www.khanacademy.org/math/algebra-home/alg-vectors
https://www.youtube.com/watch?v=f5liqUk0ZTw
And a brilliant introductory series on the topic by 3Blue1Brown:
https://www.youtube.com/watch?v=fNk_zzaMoSs&list=PLZHQObOWTQDPD3MizzM2xVFitgF8hE_ab
And of course; the Wikipedia page: https://en.wikipedia.org/wiki/Mass_in_special_relativity
