In classical (say Hamiltonian) mechanics, adding a constant energy offset has no effect on the dynamics of the physical system. One way to understand this might be to understand that Hamilton's equations of motion which determine the time evolution of the system variables depend only on derivatives of the Hamiltonian, and the derivative of a constant is zero.
In the Heisenberg picture we get the Heisenberg equations of motion in place of the Hamilton's equations (they are the exact same save for hats on the variables). By the same argument a constant offset to the energy has no effect on the dynamics.
In the Schrodinger picture we know that an energy offset simply adds a global time-evolving phase to every state. This global phase has no observable effects.
Main Question:
How then can any argument be made that zero point energy has an physical consequences?
Note 1:
I would say that the following Hamiltonians result in the exact same physics:
\begin{align} \hat{H} =& \hbar \omega \left(\hat{a}^{\dagger}\hat{a}+ \frac{1}{2}\right)\\ \hat{H} =& \hbar \omega \hat{a}^{\dagger}\hat{a}\\ \hat{H} =& \hbar \omega \left(\hat{a}^{\dagger}\hat{a}+ 1,000,000\right) \end{align}
Note 2:
Note that I distinguish between zero point energy and zero point motion. I do believe that zero point motion is physical and has observable consequences. I also believe that the theory predicts zero point motion even in absence of including zero point energy in the Hamiltonian. See Consistent, complete, and generalized description of the quantum harmonic oscillator