@J.G. solved the problem for you. As a pedagogical lark, and because it is a wonderful thing to know (I levitated, in the mid-70s, when I learned the method in Gilmore's standard text), I'll sketch how your method could have actually come close, if only you had not mangled the CBH formula.
The degenerate form you wrote down only works (by dropping the plain wrong 1/2 in front of the commutator in the last exponent) if that commutator is central, i.e., commutes with everything else. While this is true for the Heisenberg algebra of oscillators, it is not for the SU(1,1) of bilinears you have here and recognized in your question--think of it loosely as a non-hermitian SU(2). The mutual commutators keep ever rotating each other.
The proper CBH braiding identity to use, instead, is,
$$
G\equiv e^{\lambda a^2} e^{\lambda (a^\dagger)^2} = e^{\frac{\lambda}{1-4\lambda^2} (a^\dagger)^2} e^{ -\ln (1-4\lambda^2)\cdot ~ (1/2+ a^\dagger a)} e^{\frac{\lambda}{1-4\lambda^2} a^2} .
$$
(Read on, below, after the separating line, if you were interested in obtaining it.)
For $\lambda <1/2$, all is fine; you can see that
$$
\langle 0| G |0\rangle = \frac{1}{\sqrt{1-4\lambda^2}}~,
$$
by inspection, following your argument on trivial action on the vacuum.
It diverges for $\lambda\to 1/2$. Also note the indeterminate $\infty \cdot 0$s in the exponents of the leftmost and rightmost exponentials you rightly ignored acting on the vacuum. Unless one were a strong geometer, beyond the call of duty here, one would not dare sensibly continue past the singularity we saw.
So, now, we can indulge in the pedagogy of deriving the CBH formula, cf
appendix of this answer.
You already noted the SU(1,1) (SU(2)), in your question, namely
$$
L_-\equiv -a^2/2, \qquad L_+ \equiv a^{\dagger 2 }/2, \qquad 2L_3\equiv 1/2 +a^\dagger a,
$$
closing into the familiar Lie algebra
$$
[L_+, L_-]=2 L_3, \qquad [L_3, L_{\pm} ]=\pm L_{\pm}~~.
$$
But you already know the simplest faithful Pauli matrix irrep of this algebra,
$$
L_+= \begin{pmatrix}
0&1\\
0&0
\end{pmatrix} , \qquad
L_-= \begin{pmatrix}
0&0\\
1&0
\end{pmatrix} , \qquad
2L_3= \begin{pmatrix}
1&0\\
0&-1
\end{pmatrix} .
$$
- The key point:A group element identity (product of exponentials of generators) holds for all representations; conversely, if it holds for a faithful irrep, such as this doublet (the Pauli matrices), it holds in general, for all reps, as the combinatorics of a putative CBH expansion would be identical, and the Lie algebra structure constants are in common.
(This is nontrivial: it requires Poincaré's exponential theorem to the effect the CBH series in the exponent is fully in the Lie algebra, and hence the representation is immaterial.)
So one need only derive the above group product braiding identity for the doublet irrep, and one is done!!
$$
e^{-2\lambda L_-}~ e^{2\lambda L_+} =
\begin{pmatrix}
1&0\\
-2\lambda&1
\end{pmatrix} \begin{pmatrix}
1&2\lambda\\
0&1
\end{pmatrix} = \begin{pmatrix}
1&2\lambda\\
-2\lambda&1-4\lambda^2
\end{pmatrix} \\ = \begin{pmatrix}
1&\frac{2\lambda}{1-4\lambda^2}\\
0&1
\end{pmatrix} \begin{pmatrix}
\frac{1}{1-4\lambda^2}&0\\
0&1-4\lambda^2
\end{pmatrix} \begin{pmatrix}
1&0\\
\frac{-2\lambda}{1-4\lambda^2} &1
\end{pmatrix}\\ = e^{\frac{2\lambda}{1-4\lambda^2} L_+} ~e^{ -\ln (1-4\lambda^2)\cdot ~ 2L_3} ~e^{\frac{-2\lambda}{1-4\lambda^2} aL_-} .
$$
Substituting the oscillator bilinear realization above yields the formula to be proven.