In non-relativistic and non-gravitational physics (both conditions have to be satisfied simultaneously for the following proposition to hold), energy is only defined up to an arbitrary additive shift. In this restricted context, the choice of the additive shift is an unphysical, unobservable convention.
Special relativity
However, in special relativity, energy is the time component of a 4-vector and it matters a great deal whether it is zero or nonzero. In particular, the energy of the empty Minkowski space has to be exactly zero because if it were nonzero, the state wouldn't be Lorentz-invariant: Lorentz transformations would transform the nonzero energy (time component of a vector) to a nonzero momentum (spatial components).
General relativity
In general relativity, the additive shifts to energy also matter because energy is a source of spacetime curvature. A uniform shift of energy density in the Universe is known as the cosmological constant, and it will curve the vacuum. So it's important to know what it is - and it is not just a convention. Also, in general relativity, the argument from the previous paragraph may be circumvented: dark energy, regardless of its value, preserves the Lorentz (or de Sitter or anti de Sitter, which are equally large) symmetry because the stress energy tensor is proportional to the metric tensor (because $p=-\rho$). However, as long as there is gravity, the additive shift matters.
In practice, we don't measure the zero-point energy by its gravitational effects, and the value of the cosmological constant remains largely mysterious. So I surely have a different, more observationally relevant answer.
Casimir energy, comparison of situations
The additive shifts to the energy are also important when one can compare the energy in two different situations. In particular, the Casimir effect may be measured. The Casimir force arises because in between two metallic plates, the electromagnetic field has to be organized to standing waves - because of the different boundary conditions. By summing the $\hbar\omega/2$ zero-point energies of these standing waves (each wavelength produces a harmonic oscillator), and by subtracting a similar "continuous" calculation in the absence of the metallic plates, one may discover that the total zero-point energy depends on the distance of the metallic plates if they're present, and experiments have verified that the corresponding force $dE/dr$ exists and numerically agrees with the prediction.
There are many other contexts in which the zero-point energy may be de facto measured. For example, there exist metastable states that behave like the harmonic oscillator for several low-lying states. The energy of these metastable states may be compared with the energy of the free particle at infinity, and the result is $V_{\rm local\,minimum}-V_{\infty}+\hbar\omega/2$. This is somewhat analogous to calculating the energies of the bound state in a Hydrogen atom - which may be measured (think about the ionization energy).
So yes, whenever one adds either special relativity or gravity or comparisons of configurations where the structure and frequencies of the harmonic oscillators differ, the additive shift becomes physical and measurable.