The proper acceleration in the formula you're trying to derive refers to what's measured in the inertial frame instantaneously co-moving with the particle, and not in the accelerated co-moving frame of the particle where the measured acceleration is zero.
This is how I would do the derivation using either a 3-vector or 4-vector approach:
3-vector approach
Let frames $S$ and $S'$ move at a constant velocity, and $p$ be an accelerated point. The strategy is to find how the acceleration of $p$ transforms between $S$ and $S'$, and then to make the velocity of $S'$ equal to the instantaneous velocity of $p$ which then gives us the relationship between the proper acceleration and lab acceleration of $p$. Without loss of generality and to simplify things, $p$ accelerates along $x$, and $S'$ moves along the $x-\text{axis}$ at constant velocity $v$ as usual.
In $S'$, the coordinates of $p$ are $(x',y',z',t')$ which can be expressed in terms of the $S$ coordinates via the Lorentz transformations. Using these we can also find $\frac{dt'}{dt}$:
$$ x' =\gamma(x-vt),\quad t' = \gamma(t - vx/c^2),\quad \frac{dt'}{dt}= \gamma(1-\frac{vu_x}{c^2})$$
$p$ is accelerated along the x'-axis:
$$\begin{align*}
\frac{d^2x'}{dt'^2}
&=\frac{d}{dt'}\frac{dt}{dt'}\frac{d}{dt}\gamma(x - vt)
=\frac{dt}{dt'}\frac{d}{dt}\frac{u_x - v}{1-\frac{vu_x}{c^2}}
=\frac{1}{\gamma(1-\frac{vu_x)}{c^2}}
\cdot
\frac{1}{\gamma^2(1-\frac{vu_x}{c^2})^2}
\frac{d^2x}{dt^2}\\\
&=\frac{1}{\gamma^3(1-\frac{vu_x}{c^2})^3}\frac{d^2x}{dt^2}
\end{align*}$$
We now make $S'$ the co-moving and therefore proper frame of $p$ with $u_x$ the velocity of $p$ in $S$ which we now set equal to $v$, finally giving
$$\frac{d^2x'}{dt'^2} = \gamma^3\frac{d^2x}{dt^2}$$
4-vector approach
The four acceleration is given by
$$\left(
{\gamma_u}^4\frac{\mathbf{a\cdot u}}{c},
{\gamma_u}^2\mathbf{a}+
{\gamma_u}^4\frac{(\mathbf{a\cdot u)}}{c^2}\mathbf{u}
\right)$$
In the proper frame where $\mathbf{u=0}$, this immediately simplifies to $(0,\mathbf{a_p})$. We just Lorentz tranform the time part to establish the relationship between the lab and proper accelerations
$$
\begin{align*}
\gamma^4\frac{\mathbf{a_lv}}{c^2}&=
\gamma\left(0 + \frac{\mathbf{va_p}}{c^2}\right)\\
\mathbf{a_p}&=\gamma^3\mathbf{a_l}
\end{align*}
$$
method in question
It's not clear to me what your strategy is, but it could be made to work based upon the 3-vector approach above:
$$\frac{dx}{d \tau} = \frac{dx}{dt} \frac{dt}{d \tau} = \gamma \frac{dx}{dt}$$
where we used $\frac{dt}{d \tau} = \gamma $
Correct
Now,
$$\frac{d^2 x}{d \tau^2} = \frac{ d}{d \tau}( \gamma \frac{dx}{dt}) = \frac{d \gamma}{d \tau} \frac{dx}{dt} + \gamma^2 \frac{ d^2 x}{dt^2} $$
This won't work because $\gamma$ is a function of the relative velocity $v$ between the two frames where the acceleration is being measured, and we keep this velocity constant:
$$\frac{d^2 x}{d \tau^2} = \frac{ d}{d \tau}( \gamma \frac{dx}{dt}) = \gamma^2 \frac{ d^2 x}{dt^2} $$
The term on the RHS isn't an acceleration for any frame because it's mixing the space coordinate from the proper frame with the time coordinate from the lab frame. You need to transform $x$ to $x'$ using the Lorentz transformation $x=\gamma(x' + vt)$ so that finally, keeping $\gamma$ and $v$ constant:
$$\frac{d^2 x}{d \tau^2} = \gamma^2 \frac {d^2x}{dt^2} = \gamma^2\frac {d^2}{dt^2}\gamma (x' + vt)= \gamma^3\frac{d^2x'}{dt^2}$$
