The way you formulated your question confused me a bit. I think that you trying to consider the case where there a constant external force and not a constant acceleration. If that's the case here is my solution.
As you suggested if the value of $h$ is rather large then we must make use of the full formula describing gravitational force,
\begin{equation}
F_g=G\frac{m M}{(R+h)^2},
\end{equation}
where $R$ is the radius of the earth and $M$ is the mass of the earth. On top of that we also assume that there is a constant external force $F$ pushing the rocket upward. Then Newton's equation is written as
\begin{equation}
m \frac{d^2 h}{dt^2} = F - G\frac{m M}{(R+h)^2}.
\end{equation}
Now you can try to solve the equation directly but to me it seems more advantageous to take a secondary route and use conservation of energy. Recall that the variation of total energy is equal to the work done by the external forces (in this case $F$), and that the gravitational force admits a potential energy given by $U=- G(m M)/(R+h)$. Then we may write
\begin{equation}
\begin{aligned}
\Delta E=& W_{est} = F \Delta h \\
\Delta U +& \Delta K = F \Delta h \\
- G\frac{m M}{R+h} + G\frac{m M}{R+h_0} &+ \frac{1}{2} m v^2 - \frac{1}{2} m v_0^2= F (h-h_0).
\end{aligned}
\end{equation}
Now we fix the initial conditions as $v_0=0, h_0=0$ and the equation simplifies to ($v= dh/dt$)
\begin{equation}
\frac{1}{2} m \biggl(\frac{dh}{dt}\biggl)^2= Fh+ G\frac{m M}{R+h}- G\frac{m M}{R} \rightarrow \\
\rightarrow \frac{dh}{dt}=\sqrt{\frac{2}{m} \biggl(Fh+ G\frac{m M}{R+h}-G\frac{mM}{R}\biggl)}.
\end{equation}
In the last equation we can use separation of variables and perform an integral in $h$ (probably using Mathematica) to get the correct answer.
