From $(a^\dagger)^n|0\rangle=\sqrt{n!}|n\rangle$ we have $e^{\lambda (a^\dagger)^2}|0\rangle =\sum_{n=0}^\infty\frac{\lambda^n\sqrt{(2n)!}}{n!}|2n\rangle$, which has norm $\sum_{n=0}^\infty\frac{\lambda^{2n}(2n)!}{n!^2}$. The Stirling approximation gives $\frac{(2n)!}{n!^2}\approx\frac{4^n}{\sqrt{n\pi}}$ for large $n$, so the convergence condition is $4\lambda^{2}< 1$. (The case $\lambda =\frac{1}{2}$ doesn't work because $\zeta(\frac{1}{2})$ diverges.) In fact, the sum is $(1-4\lambda^2)^{-1/2}$.

From $(a^\dagger)^n|0\rangle=\sqrt{n!}|n\rangle$ we have $e^{\lambda (a^\dagger)^2}|0\rangle =\sum_{n=0}^\infty\frac{\lambda^n\sqrt{(2n)!}}{n!}|2n\rangle$, which has norm $\sum_{n=0}^\infty\frac{\lambda^{2n}(2n)!}{n!^2}$. The Stirling approximation gives $\frac{(2n)!}{n!^2}\approx\frac{4^n}{\sqrt{n\pi}}$ for large $n$, so the convergence condition is $4\lambda^{2}< 1$. (The case $\lambda =\frac{1}{2}$ doesn't work because $\sum_n n^{-1/2}$ diverges.) In fact, the sum is $(1-4\lambda^2)^{-1/2}$.