I tried to check the Lorentz invariance of the standard special relativity action for free particle directly: ($c=1$)

$$S=\int L dt=-m\int\sqrt{1-v^{2}}dt$$

Lorentz boost:

$$dt=\frac{dt^{'}+udx^{'}}{\sqrt{1-u^{2}}}$$, $$dx=\frac{udt^{'}+dx^{'}}{\sqrt{1-u^{2}}}$$, $$v=\frac{dx}{dt}=\frac{u+v^{'}}{1+uv^{'}}$$

Substitute these expressions into the action:

$$S^{'}=-m\int \sqrt{1-\left(\frac{u+v^{'}}{1+uv^{'}} \right)^{2}} dt=-\int \sqrt{1-\left(\frac{u+v^{'}}{1+uv^{'}} \right)^{2}} \left(\frac{dt^{'}+udx^{'}}{\sqrt{1-u^{2}}}\right)$$

$$S^{'}=-m\int \left(\frac{\sqrt{1-\left(\frac{u+v^{'}}{1+uv^{'}} \right)^{2}}}{\sqrt{1-u^{2}}} \right)dt^{'}-m\int \left(\frac{u\sqrt{1-\left(\frac{u+v^{'}}{1+uv^{'}} \right)^{2}}}{\sqrt{1-u^{2}}}\right)dx^{'}$$

But I know that action of this type is invariant under lorentz transformations, so we want to get that $$S^{'}=-m\int \sqrt{(1-v'^{2})} dt^{'}$$

But it doesnt seem to me that these expressions for action are equal. So I'm confused. Maybe there is a mistake in my calculations?

joshphysics's answer was excellent, and I'd like to draw attention to another aspect of the question.

$S=-m\int\sqrt{1-u^{2}}dt$

Can be written:

$S=-m\int\sqrt{(dt)^{2}-(udt)^{2}}$

$=-m\int\sqrt{(dt)^{2}-(dx)^{2}} = -m\int ds$

And the transformed action $S' = -m \int ds'$.

So, for the action to be Lorentz invariant, it suffices to show that the interval, $s$, is Lorentz invariant, i.e. $s=s'$:

Say $s^2 = t^2 - x^2$; $x' = \gamma (x-vt)$; $t' = \gamma (t-vx)$; and finally $\gamma = (1-v^2)^{-1/2}$. Then,

$(s')^2 = (t')^2 - (x')^2$

$=\gamma^2((t-vx)^2 - (x-vt)^2) = \gamma^2(t^2 +v^2x^2 - x^2+ -v^2t^2)$

(Cross terms cancel)

$= (1-v^2)^{-1}(t^2(1-v^2) -x^2(1-v^2)) = t^2-x^2$

So $ds'=ds$, and consequently your action is Lorentz invariant.

I too will set $c=1$ and I'll ignore the mass and minus sign outside of the integral for simplicity since they don't affect Lorentz-invariance.

Let $x^\mu(\lambda) = (t(\lambda), x(\lambda)$) denote a parameterized path in two-dimensional Minkowski space. We can mathematically define the action of Lorentz transformations on such parameterized paths by \begin{align} t'(\lambda) &= \gamma_u(t(\lambda)-ux(\lambda)) \\ x'(\lambda) &= \gamma_u(x(\lambda) - ut(\lambda)) \end{align} Physically, the transformed spacetime path defined in this way corresponds to what an observer in another inertial frame (with the inertial frame moving at velocity $u$) would measure. The claim of Lorentz invariance of the free particle is simply that for a given parameterized path, if we define the following: action functional $$S[t, x] = \int d\lambda \sqrt{\frac{dt}{d\lambda}^2 - \frac{dx}{d\lambda}^2}$$ then $$S[t', x'] = S[t, x]$$ to see this, we simply compute \begin{align} S[t', x'] &= \int d\lambda \sqrt{\frac{dt'}{d\lambda}^2 - \frac{dx'}{d\lambda}^2} \\ &= \int d\lambda \sqrt{\gamma_u^2\left(\frac{dt}{d\lambda} - u\frac{dx}{d\lambda}\right)^2 - \gamma_u^2\left(\frac{dx}{d\lambda} - u\frac{dt}{d\lambda}\right)^2} \\ &= \int d\lambda \sqrt{\gamma_u^2(1-v^2)\left(\frac{dt}{d\lambda}^2 -\frac{dx}{d\lambda}^2\right)} \\ &= \int d\lambda \sqrt{\frac{dt}{d\lambda}^2 -\frac{dx}{d\lambda}^2} \\ &= S[t,x] \end{align}

Note that if $t(\lambda) = \lambda$, then $\lambda$ corresponds to the time as measured in the unprimed inertial frame. In this case we could denote $v = \frac{dx}{dt}= \frac{dx}{d\lambda}$ and note that $\frac{dt}{d\lambda}= 1$ so that $$S = \int dt\sqrt{1-v(t)^2}$$ In this case, we would have additionally that \begin{align} t'(t) &= \gamma_u(t- u x(t))\\ x'(t) &= \gamma_u(x(t) - u t) \end{align} and you can go through the same analysis with the action.