Jerk is the change in acceleration, and in your scenario acceleration is constant and therefore jerk is zero.
But no, because now we are considering the change in gravity due to height. So at any point, the acceleration is a function of height $h$ only.
$$ a= \frac{F}{m} - \frac{GM}{(R+h)^2} \tag{1}$$
Considering the velocity as $v = \tfrac{\rm d}{{\rm d}t} h$, jerk is defined by the chain rule
$$ j = \tfrac{\rm d}{{\rm d}t} a = \tfrac{{\rm d} a}{{\rm d}h} \tfrac{{\rm d}h}{{\rm d}t} = v \left( \frac{2 G M}{(R+h)^3} \right) \tag{2}$$
But since velocity $v$ changes with height, this is still not an answer. But we can integrate (1) to get the velocity at each height $h$.
$$ \tfrac{1}{2} v^2 - \tfrac{1}{2} v_0^2 = \int_0^h a\,{\rm d}h $$
$$ v = \sqrt{ v_0^2 + \frac{2 F h}{m} - \frac{2 G M h}{R(R+h)} } \tag{3} $$
So now we can find jerk as a function of height
$$ j = \sqrt{ v_0^2 + \frac{2 F h}{m} - \frac{2 G M h}{R(R+h)} } \left( \frac{2 G M}{(R+h)^3} \right) \tag{4} $$
If initially, the rocket is at rest $v_0=0$ and considering different forces which are more than the weight at the surface, then the shape of jerk vs. height looks something like this
Now the peak of the curve (maximum jerk) occurs at the following height (with $\gamma$ the ratio of Force/Weight)
$$ h_\text{max jerk} = R \frac{\sqrt{9 \gamma^2 -17 \gamma + 9} -2 \gamma + 3}{5 \gamma} \tag{5}$$
In the limiting case where $\gamma=1$, the force equals the weight, then $ h_\text{max jerk} = \tfrac{2}{5} R$, which is a rather large distance, considering $R$ is the radius of the earth.
The higher the force, the sooner peak jerk occurs. The closest it can be though is $\tfrac{1}{5}R$ when $F \rightarrow \infty$