Four-velocity of an accelerating particle In the rest frame of a particle, the acceleration four-vector is
$$ \mathbf{a} = (0, g) $$
(see https://users.physics.ox.ac.uk/~smithb/website/coursenotes/rel_A.pdf , Eq.3.51).
By definition
$$ \mathbf{a} = \frac{d\mathbf{u}}{d\tau} $$
where $\tau$ is the proper time. It follows from the above that
$$ \frac{du^t}{d\tau} = 0 $$
$$ \frac{du^x}{d\tau} = g $$
So the velocity four-vector is:
$$ \mathbf{u} = (u^t, g\tau) $$
where $u^t$ can be found from the condition $\mathbf{u} \cdot \mathbf{u}=-1$, which gives $u^t = \sqrt{1 + g^2\tau^2}$.
Anyway, what confuses me here is that we started in the rest frame of the particle, and did not do any transformations to other frames using Lorenz boosts, yet the spatial component of the velocity is not zero, so obviously the particle is moving. Shouldn't the velocity of the particle in the rest frame of this same particle be zero? What is wrong with this reasoning?
 A: Your source should've worded that more carefully. This is not the acceleration of the particle in the rest frame; instead, it's the acceleration in a momentarily comoving inertial frame, that is, an inertial frame in which the particle is momentarily at rest. In the actual rest frame, which is a non-inertial frame, the velocity and acceleration of the particle are by definition identical to zero. 
Let's go over this in more detail; I will mostly use the notation in your source. In an inertial frame, the 4-displacement, 4-velocity, and 4-acceleration of a particle are
$$
\begin{align}
X &= (ct,\, \boldsymbol{r}),\\
U &= \frac{\text{d}X}{\text{d}\tau} = (U^t,\, \boldsymbol{U}) = (\gamma_u c,\, \gamma_u\boldsymbol{u}),\\
A &= \frac{\text{d}U}{\text{d}\tau} = (A^t,\,\boldsymbol{A}) = \left(\gamma_u^4\frac{\boldsymbol{u}\cdot\boldsymbol{a}}{c},\, \gamma_u^4\frac{\boldsymbol{u}\cdot\boldsymbol{a}}{c^2}\boldsymbol{u} + \gamma_u^2\boldsymbol{a}\right),
\end{align}
$$
where 
$$
\boldsymbol{u} = \frac{\text{d}\boldsymbol{r}}{\text{d}t},\qquad
\boldsymbol{a} = \frac{\text{d}\boldsymbol{u}}{\text{d}t},\qquad
\gamma_u = \frac{\text{d}t}{\text{d}\tau} = \frac{1}{\sqrt{1 - \frac{u^2}{c^2}}}.
$$
Now let's express these quantities in a different inertial frame, moving with a constant velocity $\boldsymbol{v}$ with respect to the original frame. The corresponding Lorenz transformation of any 4-vector $S = (S^t, \boldsymbol{S})$ is given by (see also wiki)
$$
\begin{align}
{S'}^{t} &= \gamma_v\left(S^{t} - \frac{\boldsymbol{v}}{c}\cdot \boldsymbol{S}_\parallel\right),\\
\boldsymbol{S}'_\parallel &= \gamma_v\left(\boldsymbol{S}_\parallel - \frac{\boldsymbol{v}}{c}S^{t}\right),\\
\boldsymbol{S}'_\perp &= \boldsymbol{S}_\perp,
\end{align}
$$
where $\boldsymbol{S} = \boldsymbol{S}_\parallel + \boldsymbol{S}_\perp$ is split into a vector parallel to and a vector perpendicular to $\boldsymbol{v}$:
$$
\boldsymbol{S}_\parallel = \frac{\boldsymbol{S}\cdot\boldsymbol{v}}{v^2}\boldsymbol{v},\qquad \boldsymbol{S}_\perp = \boldsymbol{S} - \boldsymbol{S}_\parallel.
$$
If we apply this to the particle, we find
$$
\begin{align}
X' &=  \left(\gamma_vct-  \gamma_v\frac{\boldsymbol{r}\cdot\boldsymbol{v}}{c},\, \boldsymbol{r}_\perp + \gamma_v\boldsymbol{r}_\parallel - \gamma_v\boldsymbol{v}t\right),\\
U' &= \frac{\text{d}X'}{\text{d}\tau} = \left(\gamma_u\gamma_v c - \gamma_u\gamma_v\frac{\boldsymbol{u}\cdot\boldsymbol{v}}{c},\, \gamma_u\boldsymbol{u}_\perp +  \gamma_u\gamma_v(\boldsymbol{u}_\parallel - \boldsymbol{v})\right),\\
A' &= \gamma_u^2\left(\gamma_v\gamma_u^2\frac{\boldsymbol{u}\cdot\boldsymbol{a}}{c}\left(1 - \frac{\boldsymbol{u}\cdot\boldsymbol{v}}{c^2}\right)
-\gamma_v\frac{\boldsymbol{a}\cdot\boldsymbol{v}}{c}
,\, \gamma_u^2\frac{\boldsymbol{u}\cdot\boldsymbol{a}}{c^2}\left(\boldsymbol{u}_\perp + \gamma_v(\boldsymbol{u}_\parallel - \boldsymbol{v})\right) 
 + (\boldsymbol{a}_\perp + \gamma_v\boldsymbol{a}_\parallel)\right)
\end{align}
$$
If you have a few hours to spare, then you can verify that these 4-vectors can be written as
$$
\begin{align}
X' &= (ct',\, \boldsymbol{r}'),\\
U' &= (\gamma_{u'} c,\, \gamma_{u'}\boldsymbol{u}'),\\
A' &= \left(\gamma_{u'}^4\frac{\boldsymbol{u}'\cdot\boldsymbol{a}'}{c},\, \gamma_{u'}^4\frac{\boldsymbol{u}'\cdot\boldsymbol{a}'}{c^2}\boldsymbol{u}' + \gamma_{u'}^2\boldsymbol{a}'\right),\tag{1}
\end{align}
$$
with
$$
\boldsymbol{u}' = \frac{\text{d}\boldsymbol{r}'}{\text{d}t'},\qquad
\boldsymbol{a}' = \frac{\text{d}\boldsymbol{u}'}{\text{d}t'},\qquad
\gamma_{u'} = \frac{\text{d}t'}{\text{d}\tau} = \frac{1}{\sqrt{1 - \frac{u'^2}{c^2}}}
= \gamma_u\gamma_v \left(1 - \frac{\boldsymbol{u}\cdot\boldsymbol{v}}{c^2}\right),
$$
so the 4-vectors maintain the same form, as we should expect. Now, if the particle is moving with velocity $\boldsymbol{u}$ at some time $t$ in the original frame, and we set $\boldsymbol{v} = \boldsymbol{u}$, we end up in an inertial frame where the particle is momentarily at rest. Indeed, we have, at the corresponding time $t'$,
$$
\begin{align}
U' &= (c,\, \boldsymbol{0}),\\
A' &= \left(0,\, \boldsymbol{a}'\right) = \left(0,\, \boldsymbol{g}\right)  =\left(0,\, \gamma_u^3\boldsymbol{a}_\parallel + \gamma_u^2\boldsymbol{a}_\perp\right).\tag{2}
\end{align}
$$
So, the spatial component of the 4-acceleration, with I will call $\boldsymbol{g}$ (in your source it's called $\boldsymbol{a}_0$), is not zero because we're still in a non-accelerating frame. Therefore, the particle doesn't stay at rest and $U'$ and $A'$ will have the general form $(1)$ again at any later time.
The norm of $\boldsymbol{g}$ is a Lorentz-invariant, called the proper acceleration:
$$
\begin{align}
g &= |\boldsymbol{g}| = \sqrt{-(A^t)^2 + (\boldsymbol{A})^2}
 = \gamma_u^2\sqrt{\gamma_u^2\boldsymbol{a}_\parallel^2 + \boldsymbol{a}_\perp^2} \\
&= \gamma_u^2\sqrt{\gamma_u^2a^2 + (1 - \gamma_u^2)a^2\frac{u_\perp^2}{u^2}}
= \gamma_u^2\sqrt{\gamma_u^2a^2 - \gamma_u^2a^2\frac{u_\perp^2}{c^2}}
= \frac{\gamma_u^3}{\gamma_{u_\perp}}a,
\end{align}
$$
where $\boldsymbol{u}_\perp$ is the part of $\boldsymbol{u}$ perendicular to $\boldsymbol{a}$. But while different inertial observers agree on the norm of $\boldsymbol{g}$, they in general do not agree on its direction. Indeed, note from $(2)$ that $\boldsymbol{a}$ is in general not parallel to $\boldsymbol{g}$. In other words, the direction of the acceleration of the particle is not the same in different inertial frames.
Finally, let's take a look at motion with constant $\boldsymbol{g}$, starting from rest. For simplicity, consider motion in the $x'$-direction: $\boldsymbol{g} = (g,0,0)$. Let's write the velocity in the form
$$
\boldsymbol{u}' = (c\tanh\eta, 0, 0),
$$
such that $\gamma_{u'} = \cosh\eta$, and 
$$
\begin{align}
U' &= (c\cosh\eta,\, c\sinh\eta, 0, 0),\\ \tag{3}
A' &= \left(c\sinh\eta\frac{\text{d}\eta}{\text{d}\tau},\, c\cosh\eta\frac{\text{d}\eta}{\text{d}\tau}, 0, 0 \right) = (g\sinh\eta,\, g\cosh\eta, 0, 0),
\end{align}
$$
with
$$
g = \sqrt{c^2\cosh^2\eta\left(\frac{\text{d}\eta}{\text{d}\tau}\right)^2 - c^2\sinh^2\eta\left(\frac{\text{d}\eta}{\text{d}\tau}\right)^2} = c\frac{\text{d}\eta}{\text{d}\tau}.
$$
Even if $g$ is constant, $\boldsymbol{A'}$ is not, so you cannot write $\boldsymbol{U'} = \boldsymbol{g}\tau$. Instead, $\boldsymbol{U'} = (c\sinh\eta,0,0)$, where $\eta = g\tau/c$. For more details, see this post.
