The factorization in equation $(3)$ is not justified in general.
In fact, there are boundary conditions for which the Green function cannot have the form given in Jackson. This expansion is valid in certain cases such as when the Green function is required to vanish on the entire boundary. These are the kinds of cases that Jackson discusses in this section, so he probably only had that in mind.
Suppose we have a boundary value problem where we require the potential to equal $V$ on the sphere of radius $a \gt 0$, where $V$ is given by $V(\theta, \varphi) = \sum\limits_{l=0}^\infty\sum\limits_{m=-l}^l 2^{-l}Y_{lm}(\theta, \varphi)$, and we are interested in the potential in the interior of the ball of radius $a$.
Using the Green function expansion assumed in Jackson and substituting $r = a$, we have
$$V(\theta, \varphi) = G(a, \theta, \varphi, r', \theta', \varphi') = \sum\limits_{l=0}^\infty \sum\limits_{m=-l}^l A_{lm}(\theta',\varphi')g_l(a, r')Y_{lm}(\theta, \varphi)$$
Then using orthogonality we have
$$A_{lm}(\theta', \varphi')g_l(a, r') = \int_0^{2\pi}\int_0^\pi V(\theta,\varphi)Y_{lm}^*(\theta, \varphi)\sin\theta\,d\theta\,d\varphi = 2^{-l}$$
This implies $g_l(a, r') \neq 0$ so dividing by it we get
$$A_{lm}(\theta', \varphi') = \frac{1}{2^l g_l(a, r')}$$
The LHS is purely a function of $\theta'$ and $\varphi'$ while the RHS is purely a function of $r'$, so both sides must be constant.
Therefore, we see that $A_{lm}(\theta', \varphi') = c_l$ is a nonzero constant depending only on $l$.
Substituting back into the Green function expansion, we get
$$G(r, \theta, \varphi, r', \theta', \varphi') = \sum\limits_{l=0}^\infty c_l g_l(r, r')\sum\limits_{m=-l}^l Y_{lm}(\theta, \varphi)$$
But this is absurd since it does not even depend on $\theta', \varphi'$ and contradicts the requirement that $A_{lm}(\theta', \varphi') = Y_{lm}^*(\theta', \varphi')$ which was deduced from the delta function expansion. It will not be able to satisfy the equation $\nabla^2_{\mathbf{x}} G = -4\pi\delta(\mathbf{x}-\mathbf{x}')$.
So you are correct that this expansion is missing terms, and does not work for all such boundary value problems.
But if we force $G(\mathbf{x}, \mathbf{x}') = 0$ on the entire boundary (which is what Jackson assumes in the sentence after equation (3.116)) then the expansion is valid. We further assume that our boundaries are always spherical, and in the general case consist of a sphere of radius $a$ and a sphere of radius $b$ with $0 \lt a \lt b$.
In that case, starting from your expansions (2) and (5), which are completely general, and substituting into the Poisson equation with the spherical harmonic expansion of $\delta(\mathbf{x} - \mathbf{x}')$, we get
$$A_{lm}(r | r', \theta', \varphi') = \sum\limits_{l'=0}^\infty\sum\limits_{m'=-l'}^{l'} H_{lml'm'}(r, r') Y_{l'm'}^*(\theta',\varphi')$$
$$\frac{1}{r}\frac{\partial^2}{\partial r^2}\left(r H_{lml'm'}(r, r')\right) - \frac{l(l+1)}{r^2}H_{lml'm'}(r, r') = 0\text{ if }(l', m') \neq (l, m)$$
$$\frac{1}{r}\frac{\partial^2}{\partial r^2}\left(r H_{lmlm}(r, r')\right) - \frac{l(l+1)}{r^2}H_{lmlm}(r, r') = -\frac{4\pi}{r^2}\delta(r - r')$$
The first equation has the general solution
$$H_{lml'm'}(r, r') = B_{lml'm'}(r')r^l + C_{lml'm'}(r')r^{-(l+1)}\text{ if }(l', m') \neq (l, m)$$
Now we impose the boundary condition that $G$ vanishes at $r = a$ and $r = b$.
By orthogonality of the spherical harmonics $Y_{lm}(\theta, \varphi)$ we see that $A_{lm}(a|r',\theta',\varphi') = 0$ and then by orthogonality of the spherical harmonics $Y_{l'm'}(\theta',\varphi')$ we see that $H_{lml'm'}(a, r') = 0$ when $(l', m') \neq (l, m)$. Similarly, $H_{lml'm'}(b, r') = 0$ when $(l', m') \neq (l, m)$.
Therefore we have the system
$$B_{lml'm'}(r')a^l + C_{lml'm'}(r')a^{-(l+1)} = 0$$
$$B_{lml'm'}(r')b^l + C_{lml'm'}(r')b^{-(l+1)} = 0$$
Its determinant is nonzero since $0 \lt a \lt b$ so it has the unique solution
$$B_{lml'm'}(r') = 0$$
$$C_{lml'm'}(r') = 0$$
which holds when $(l', m') \neq (l, m)$.
Therefore, $H_{lml'm'}(r, r') = 0$ when $(l', m') \neq (l, m)$.
This means that our expansion for $A_{lm}(r|r',\theta',\varphi')$ simplifies to
$$A_{lm}(r|r',\theta',\varphi') = H_{lm}(r, r')Y_{lm}^*(\theta',\varphi')$$
where $H_{lm}(r, r') = H_{lmlm}(r, r')$ for simplicity.
This is almost the factorization given in Jackson, except that we still have a dependence on $m$ in the radial factor. Showing that there is no dependence on $m$ takes longer, this is the next part.
We now analyze the second radial equation:
$$\frac{1}{r}\frac{\partial^2}{\partial r^2}\left(r H_{lm}(r, r')\right) - \frac{l(l+1)}{r^2}H_{lm}(r, r') = -\frac{4\pi}{r^2}\delta(r - r')$$
The general solution is
$$H_{lm}(r, r') =
\begin{cases}
A_{lm}(r')r^l + B_{lm}(r')r^{-(l+1)} & r \lt r' \\
A_{lm}'(r')r^l + B_{lm}'(r')r^{-(l+1)} & r \gt r' \\
\end{cases}
$$
We can write this more compactly with the Heaviside step function:
$$H_{lm}(r, r') = [1 - H(r - r')]\left[A_{lm}(r')r^l + B_{lm}(r')r^{-(l+1)}\right] + H(r - r')\left[A_{lm}'(r')r^l + B_{lm}'(r')r^{-(l+1)}\right]$$
We need to substitute this in the radial equation to find a relationship between $A_{lm}(r')$ and $B_{lm}(r')$ that makes the equation also hold at $r = r'$ in the sense of distributions.
First we compute
$$r H_{lm}(r, r') = [1 - H(r - r')]\left[A_{lm}(r')r^{l+1} + B_{lm}(r')r^{-l}\right] + H(r - r')\left[A_{lm}'(r')r^{l+1} + B_{lm}'(r')r^{-l}\right]$$
$$\frac{\partial}{\partial r}\left(r H_{lm}(r, r')\right) =$$
$$-\delta(r - r')\left[A_{lm}(r')r'^{l+1} + B_{lm}(r')r'^{-l}\right] + [1 - H(r - r')]\left[(l+1)A_{lm}(r')r^l - lB_{lm}(r')r^{-(l+1)}\right]$$
$$+\delta(r - r')\left[A_{lm}'(r')r'^{l+1} + B_{lm}'(r')r'^{-l}\right] + H(r - r')\left[(l+1)A_{lm}'(r')r^l - lB_{lm}'(r')r^{-(l+1)}\right]$$
$$\frac{\partial^2}{\partial r^2}\left(r H_{lm}(r, r')\right) =$$
$$-\delta'(r - r')\left[A_{lm}(r')r'^{l+1} + B_{lm}(r')r'^{-l}\right] - \delta(r-r')\left[(l+1)A_{lm}(r')r'^l - lB_{lm}(r')r'^{-(l+1)}\right]$$
$$+l(l+1)[1 - H(r - r')]\left[A_{lm}(r')r^{l-1} + B_{lm}(r')r^{-(l+2)}\right]$$
$$+\delta'(r - r')\left[A_{lm}'(r')r'^{l+1} + B_{lm}'(r')r'^{-l}\right] + \delta(r - r')\left[(l+1)A_{lm}'(r')r'^l - lB_{lm}'(r')r'^{-(l+1)}\right]$$
$$+l(l+1)H(r - r')\left[A_{lm}'(r')r^{l-1} + B_{lm}'(r')r^{-(l+2)}\right]$$
where I used the identity $\delta(r - r')f(r, r') = \delta(r - r')f(r', r')$.
Then we get
$$r\frac{\partial^2}{\partial r^2}\left(r H_{lm}(r, r')\right) - l(l+1)H_{lm}(r, r') = $$
$$+\delta'(r - r')\left\{\left[A_{lm}'(r') - A_{lm}(r')\right]r'^{l+2} + \left[B_{lm}'(r') - B_{lm}(r')\right]r'^{-(l-1)}\right\} + \delta(r - r')\left\{(l+1)\left[A_{lm}'(r') - A_{lm}(r')\right]r'^{l+1} - l\left[B_{lm}'(r') - B_{lm}(r')\right]r'^{-l}\right\}$$
We want this to equal $-4\pi\delta(r - r')$ so we need
$$\left[A_{lm}'(r') - A_{lm}(r')\right]r'^{l+2} + \left[B_{lm}'(r') - B_{lm}(r')\right]r'^{-(l-1)} = 0$$
$$(l+1)\left[A_{lm}'(r') - A_{lm}(r')\right]r'^{l+1} - l\left[B_{lm}'(r') - B_{lm}(r')\right]r'^{-l} = -4\pi$$
Using Cramer's rule, this system has solution
$$A_{lm}'(r') - A_{lm}(r') = -\frac{4\pi}{2l+1}r'^{-(l+1)}$$
$$B_{lm}'(r') - B_{lm}(r') = \frac{4\pi}{2l+1}r'^l$$
Therefore we can eliminate $A_{lm}'(r')$ and $B_{lm}'(r')$ to obtain
$$H_{lm}(r, r') = A_{lm}(r')r^l + B_{lm}(r')r^{-(l+1)} - \frac{4\pi}{2l+1}H(r - r')\left(\frac{r^l}{r'^{l+1}} - \frac{r'^l}{r^{l+1}}\right)$$
Finally we substitute $r = a$ and $r = b$ and use the vanishing boundary condition to obtain for $a \lt r' \lt b$:
$$A_{lm}(r')a^l + B_{lm}(r')a^{-(l+1)} = 0$$
$$A_{lm}(r')b^l + B_{lm}(r')b^{-(l+1)} = \frac{4\pi}{2l+1}\left(\frac{b^l}{r'^{l+1}} - \frac{r'^l}{b^{l+1}}\right)$$
Using Cramer's rule again, we get
$$A_{lm}(r') = \frac{4\pi}{2l+1}\frac{b^{-(l+1)}}{b^{2l+1} - a^{2l+1}}\left(\frac{b^l}{r'^{l+1}} - \frac{r'^l}{b^{l+1}}\right)$$
$$B_{lm}(r') = -\frac{4\pi}{2l+1}\frac{a^{2l+1}b^{l+1}}{b^{2l+1} - a^{2l+1}}\left(\frac{b^l}{r'^{l+1}} - \frac{r'^l}{b^{l+1}}\right)$$
We see that $A_{lm}(r')$ and $B_{lm}(r')$ do not depend on $m$.
So we can write $A_{lm}(r') = A_l(r')$ and $B_{lm}(r') = B_l(r')$.
Similarly we have $A_{lm}'(r') = A_l'(r')$ and $B_{lm}'(r') = B_l'(r')$.
We didn't need to solve the equations explicitly, we could have just noticed that both the boundary conditions and the constraint of the delta function at $r = r'$ involve some conditions on $A_{lm}(r'), A_{lm}'(r'), B_{lm}(r'), B_{lm}'(r')$ that do not involve $m$ explicitly.
Anyway, after all this work, we finally conclude that
$H_{lm}(r, r') = H_l(r, r')$ is independent of $m$, so our factorization is
$A_{lm}(r|r', \theta', \varphi') = H_l(r, r')Y_{lm}^*(\theta',\varphi')$
which is what is in Jackson.