Consider a free scalar quantum field
$$ H = \int d^3 x \left( \, \Pi(x)^2+(\nabla\phi(x))^2 \right). $$
Introducing the creation and annihilation operators we find the "vacuum catastrophe"
$$ H = \int d^3p \, \big(\omega_p a^\dagger(p)a(p) +\frac{1}{2}[a(p),a^\dagger(p)] \big),$$
a diverging energy of the vacuum ground state. No problem, this is just an unobservable constant energy shift, as is argued for example in Peskin Schröder and claimed all over the place when discussing this issue.
However, what is glossed over in this explanation is that the variance of the field at any given point is also infinite:
$$ \Delta\phi^2=\langle0|\phi^2|0\rangle=\langle0| \int d^3p \int d^3q \frac{1}{2\sqrt{\omega_q \omega_p}}(a(p)e^{ipx}+a^\dagger(p)e^{-ipx})(a(q)e^{iqx}+a(q)e^{-iqx})|0\rangle \\ =\int d^3p \frac{1}{2\omega_p}=\infty $$
Since the quantum field countains all possible modes, and every mode has groundstate fluctuations which contribute to the field at a given point, the total field fluctuations diverge.
Clearly, the variance of the field is observable by just measuring the field. We can ignore an infinite energy shift but not infinite values of the field can we?