In the framework of special relativity (SR), why is it that only energy differences are observable?

Is it simply because, starting our with some action $$S=\int d^{4}x\,\mathcal{L}(x)$$ one can always add some constant $M^{4}$ to the Lagrangian resulting in a "shifted" action $$S'=S+M^{4}\int d^{4}x$$ such that the equations of motion are unaffected by this new contribution, since $\delta\left(M^{4}\int d^{4}x\right)=0$ and so $$\delta S'=\delta S$$ Hence the dynamics of the theory are unaffected by this constant shift. In particular, the corresponding Hamiltonians of the "shifted" and "un-shifted" theories are equal up to an arbitrary constant, which we can set to zero.

Also, when one takes into account gravity, is the reason why it is sensitive to absolute energies because in the this case general covariance (or diffeomorphism invariance) demands that $$S'=S+M^{4}\int d^{4}x\sqrt{-g}$$ and as the metric is a dynamical quantity, this additional constant contribution $M^{4}\int d^{4}x\sqrt{-g}$ does affect the dynamics of the theory, since under a variation of the metric $$\frac{\delta\left(M^{4}\sqrt{-g}\right)}{\delta g_{\mu\nu}}=M^{4}g^{\mu\nu}$$ and hence makes a measurable contribution - since its inclusion affects the dynamics of the metric and hence the local curvature of spacetime.