Indeed, non-observable quantities are not necessarily gauge invariant.
Consider first the effective action, $S_{\textrm{eff}} = \int d^{4}x\, \mathcal{L}_{\textrm{eff}}(x)$. It's straightforward to see that this will be gauge invariant, but this in turn implies that the effective Lagrangian, $\mathcal{L}_{\textrm{eff}}(x)$, is only gauge invariant up to total derivatives. But this is ok, because the equations of motion etc. are insensitive to such differences so we can identify different Lagrangians evaluated in different gauges, $\mathcal{L}_{\textrm{eff}}(x) \cong \mathcal{L}_{\textrm{eff}}(x) + \partial_{\mu} \Delta \mathcal{L}^{\mu}(x) = \mathcal{L}_{\textrm{eff}}^{\prime}(x)$.
For the propagator, however, things are different.
We can, in fact, consider a more general case: "dressed propagation" in which the particle emits or absorbs an arbitrary number of (external) photons. This is closely related to the tree level photon amplitudes (non-linear Compton scattering, for example). Now we can consider loop corrections to this object (OP's original question deals with $N = 0$ external photons at one-loop order).
This propagator does depend upon various gauge choices:
- The gauge choice of external photons.
- The gauge choice of internal (loop) photons.
The first case is easily understood thanks to the Ward identity - a change in gauge of an external photon (in momentum space, $\epsilon \rightarrow \epsilon + \lambda k$, for a photon of momentum $k$) can only produce insertions at the ends of the propagator. When we put external legs on-shell and apply the LSZ amputation such terms are killed, so at the level of the observable cross-section gauge invariance holds.
In the second case, a change in gauge of the loop photons really does change the propagator. A systematic study of the change in the propagator induced by variation of the gauge of internal photons was started by Landau and Khalatnikov and around the same time by Fradkin:
- L. Landau, I. Khalatnikov, Sov. Phys. JETP2,69 (1956).
- E. S. Fradkin, Zh. Eksp. Teor. Fiz.29, 258261 (1955).
These "LKF" transformations were restudied with more modern techniques by Zumino:
- K. Johnson, B. Zumino, Phys. Rev. Lett.3, 351 (1959).
- B. Zumino, J. Math. Phys. 1, 1 (1960).
and have recently become quite a hot topic of research. See, for example,
- H. Sonoda, Phys. Lett. B499 (2001) 253-260
- A. Bashir, A. Raya, Phys. Rev. D66:105005, (2002)
- T. De Meerleer, D. Dudal, S. P. Sorella, P. Dall'Olio, A. Bashir, Phys. Rev. D 97, 074017 (2018)
- H. Kißler, PoS(LL2018)032
In fact, it is now known that Landau and co. discovered a special case of more general rules describing how arbitrary Green functions transform non-perturbatively under a change of gauge:
- T De Meerleer, D. Dudal, S. P. Sorella, P. Dall'Olio, A. Bashir, Phys. Rev. D 101, 085005 (2020)
- J. Nicasio, J. P. Edwards, C. Schubert, N. Ahmadiniaz, arXiv:2010.04160 [hep-th]
So short answer: Yes, your loop correction depends upon the gauge. However, this gauge dependency cannot be observed: if we think of the one-loop vertex correction ($N = 1$ external photons) and extract the electromagnetic form factors then, of course, quantities such as $g-2$ which are directly measurable turn out independent of the gauge.