I have seen explanations that canonical momentum for charged particles $p = mv + qA/c$ is not a measurable quantity/observable because it is not gauge invariant. However, there are many quantities that also depend on an "arbitrary choice", for example even the Hamiltonian (which corresponds to the energy observable) involves an arbitrary choice of where the zero point of the potential energy is.
What's the difference between the two quantities? One might further argue that "ok, it is the change in energy that is observable, not absolute energy" -> In this case, can I not look at change in canoncial momentum? The arbitrary choice in the gauge transformation $\nabla f$ will similarly "cancel out" when I look at change in canonical momentum.