The term you are asking about does come from the $\frac{1}{4}\dot{x}^\mu\dot{x}_\mu$ term of the Lagrangian. To evaluate it, insert the first of Eqs.(2.23) back into Eq.(2.22) to obtain the expression for $\dot{x}(\lambda)$: $$ \dot{x}(\lambda) = e^{2e{\bf F}\lambda}\dot{x}(0) = e^{2e{\bf F}\lambda}\frac{1}{e^{2e{\bf F}s} - 1} 2e{\bf F}\; (x' - x") = \frac{2e{\bf F}\; e^{2e{\bf F \lambda}} }{e^{2e{\bf F}s} - 1} \; (x' - x") $$ The rearrangement in the last form is possible because all factors are functions of ${\bf F}$ only. Taking this in the corresponding action integral term yields, up to a factor of $\frac{1}{4}$, $$ \int_0^s{d\lambda \;\dot{x}_\mu(\lambda)\dot{x}^\mu(\lambda) } = \int_0^s{d\lambda \;(x' - x")_\alpha \left[ \frac{2e{\bf F}\; e^{2e{\bf F \lambda}} }{e^{2e{\bf F}s} - 1} \right]^\alpha_{\;\;\beta} \left[ \frac{2e{\bf F}\; e^{2e{\bf F \lambda}} }{e^{2e{\bf F}s} - 1} \right]^\beta_{\;\;\gamma} (x' - x")^\gamma } \\ = (x' - x")_\alpha \int_0^s{d\lambda \;\left[ \frac{(2e{\bf F})^2\; e^{4e{\bf F \lambda}} }{\left(e^{2e{\bf F}s} - 1\right)^2} \right]^\alpha_{\;\;\beta} } (x' - x")^\beta\\ = (x' - x")_\alpha \left[ \frac{e{\bf F}}{\left(e^{2e{\bf F}s} - 1\right)^2}\int_0^s{d\lambda \; 4e{\bf F}\;e^{4e{\bf F} \lambda} }\right]^\alpha_{\;\;\beta} (x' - x")^\beta = \\ = (x' - x")_\alpha \left[ \frac{e{\bf F}}{\left(e^{2e{\bf F}s} - 1\right)^2}\int_0^s{d\lambda \; \frac{\partial}{\partial \lambda}e^{4e{\bf F}\lambda}} \right]^\alpha_{\;\;\beta} (x' - x")^\beta $$ Now the integration is trivial and results in $$ \int_0^s{d\lambda \;\dot{x}_\mu(\lambda)\dot{x}^\mu(\lambda) } = (x' - x")_\alpha \left[ e{\bf F} \frac{e^{4e{\bf F}s} - 1}{\left(e^{2e{\bf F}s} - 1\right)^2}\right]^\alpha_{\;\;\beta} (x' - x")^\beta $$ The fraction in the square brackets is then rearranged as $$ \frac{e^{4e{\bf F}s} - 1}{\left(e^{2e{\bf F}s} - 1\right)^2} = \frac{\left(e^{2e{\bf F}s} - 1\right)\left(e^{2e{\bf F}s} + 1\right)}{\left(e^{2e{\bf F}s} - 1\right)^2} = \\ = \frac{\left(e^{2e{\bf F}s} + 1\right)}{\left(e^{2e{\bf F}s} - 1\right)} = \frac{\left(e^{e{\bf F}s} + e^{-e{\bf F}s}\right)}{\left(e^{e{\bf F}s} - e^{-e{\bf F}s}\right)} = \coth(e{\bf F}s) $$ Taking it back in the square bracket and expanding the summation gives the expression after Eq.(2.23): $$ \int_0^s{d\lambda \;\dot{x}_\mu(\lambda)\dot{x}^\mu(\lambda) } = (x' - x")_\alpha \;eF^\alpha_{\;\;\beta} \left[ \coth(e{\bf F}s) \right]^\beta_{\;\;\gamma} (x' - x")^\gamma $$ Well, up to rewriting the contraction.

The term you are asking about does come from the $\frac{1}{4}\dot{x}^\mu\dot{x}_\mu$ term of the Lagrangian.

Start by inserting the first of Eqs.(2.23) back into Eq.(2.22) to obtain the expression for $\dot{x}(\lambda)$: $$ \dot{x}(\lambda) = e^{2e{\bf F}\lambda}\dot{x}(0) = e^{2e{\bf F}\lambda}\frac{1}{e^{2e{\bf F}s} - 1} 2e{\bf F}\; (x' - x") = \frac{2e{\bf F}\; e^{2e{\bf F \lambda}} }{e^{2e{\bf F}s} - 1} \; (x' - x") $$ Now notice that the equation of motion $$ \ddot{x}^\mu(\lambda) = 2e {\bf F}^{\mu\nu} \dot{x}_\nu(\lambda) $$ implies $$ \ddot{x}^\mu(\lambda)\dot{x}_\mu(\lambda) = 2e\; \dot{x}_\mu(\lambda) {\bf F}^{\mu\nu} \dot{x}_\nu(\lambda) = 0 $$ and $$ \frac{d}{d\lambda}\left[ \dot{x}_\mu(\lambda)\dot{x}^\mu(\lambda) \right] = 2\; \ddot{x}^\mu(\lambda)\dot{x}_\mu(\lambda) = 0 $$ Hence we have
$$ \int_0^s{d\lambda \;\dot{x}_\mu(\lambda)\dot{x}^\mu(\lambda) } = s\; \dot{x}_\mu(0)\dot{x}^\mu(0) $$ But from the expression for $\dot{x}(\lambda)$ above we have $$ \dot{x}(0) = \frac{2e{\bf F}}{e^{2e{\bf F}s} - 1} \; (x' - x") = (x' - x") \frac{2e{\bf F}e^{2e{\bf F}s}}{e^{2e{\bf F}s} - 1} $$ where the last form on the right hand side follows from $F^{\mu\nu} (x'-x")_\nu = - (x'-x")_\nu F^{\nu\mu}$. With this the action integral term becomes $$ \int_0^s{d\lambda \;\dot{x}_\mu(\lambda)\dot{x}^\mu(\lambda) } = s\;(x' - x") \frac{2e{\bf F}e^{2e{\bf F}s}}{e^{2e{\bf F}s} - 1}\frac{2e{\bf F}}{e^{2e{\bf F}s} - 1} \; (x' - x") =\\ = (x' - x") e{\bf F}s \frac{4e{\bf F}e^{2e{\bf F}s}}{\left(e^{2e{\bf F}s} - 1\right)^2}\; (x' - x") = (x' - x") e{\bf F}s \frac{e{\bf F}}{\sinh^2(e{\bf F}s)}\; (x' - x") = \\ = - (x' - x")\; (e{\bf F}s) \frac{d}{d s}\coth(e{\bf F}s)\; (x' - x") $$ Assuming $x'$ and $x"$ are fixed, the last expression can be rearranged to obtain $$ \int_0^s{d\lambda \;\dot{x}_\mu(\lambda)\dot{x}^\mu(\lambda) } = (x' - x")\; e{\bf F} \coth(e{\bf F}s)\; (x' - x") - \frac{d}{d s} \left[(x' - x")\; (e{\bf F}s) \coth(e{\bf F}s)\; (x' - x") \right] $$ The first term is the one we are looking for. The second one is not only a total time derivative, but can also be rewritten as the integral of a total time derivative, since $$ \int_0^s{d\lambda \frac{d^2}{d \lambda^2} \left[(x' - x")\; (e{\bf F}\lambda) \coth(e{\bf F}\lambda)\; (x' - x") \right] } = \\ = \frac{d}{d s} \left[(x' - x")\; (e{\bf F}s) \coth(e{\bf F}s)\; (x' - x") \right] - \lim_{\lambda \rightarrow 0} \frac{d}{d \lambda} \left[(x' - x")\; (e{\bf F}\lambda) \coth(e{\bf F}\lambda)\; (x' - x") \right] $$ and $$ \lim_{\lambda \rightarrow 0} \frac{d}{d \lambda} \left[(x' - x")\; (e{\bf F}\lambda) \coth(e{\bf F}\lambda)\; (x' - x") \right] = 0 $$ But a term of the form $\int_0^s{d\lambda \frac{d^2}{d \lambda^2} \left[(x' - x")\; (e{\bf F}\lambda) \coth(e{\bf F}\lambda)\; (x' - x") \right] }$ would only add a total time derivative to the Lagrangian, so it can be safely discarded and the final result is $$ \int_0^s{d\lambda \;\dot{x}_\mu(\lambda)\dot{x}^\mu(\lambda) } = (x' - x")\; e{\bf F} \coth(e{\bf F}s)\; (x' - x") $$