The gauge invariant lagrangian is $$\mathcal{L} = -\frac{1}{4}F_{\mu\nu}F^{\mu\nu}$$ and is such that, under the transformation $A_\mu\to A_\mu^\prime = A_\mu+\partial_\mu\alpha$ $$\mathcal{L}^\prime = \mathcal{L}$$ Now when we add the gauge-fixing term $$-\frac{1}{2\xi}(\partial_\mu A^\mu)^2$$ we brake gauge invariance of the lagrangian. In fact, under a gauge transformation, the gauge fixing term becomes $$(\partial_\mu A^{\mu\prime})^2 = \left(\partial_\mu(A^\mu+\partial^\mu\alpha)\right)^2 = (\partial_\mu A^\mu+\Box\alpha)^2 = (\partial_\mu A^\mu)^2+(\Box\alpha)^2+2(\partial_\mu A^\mu)\Box\alpha$$ which is clearly different from the initial gauge fixing term, in fact $$\delta\mathcal{L} = \frac{1}{2\xi}(\Box\alpha)^2+\frac{1}{\xi}(\partial_\mu A^\mu)\Box\alpha$$ where now $$\mathcal{L} = -\frac{1}{4}F_{\mu\nu}F^{\mu\nu}-\frac{1}{2\xi}(\partial_\mu A^\mu)^2$$ If the variation of the lagrangian under a general gauge transformation is not zero, then the lagrangian is not invariant under the gauge transformation.
Whenever you go ahead and fix a gauge, so for example by using the Lorentz gauge, you get back a gauge invariant theory, in fact the Lorentz gauge poses $\partial_\mu A^\mu=0$ and the additional term in the lagrangian cancels.
The idea of gauge fixing is that we take aside gauge invariance for a moment so that our theory is better behaved (the functional integral is convergent whenever eliminating the gauge redundance). Now if the theory is not gauge invariant anymore, we'll get some additional unphysical polarizations in the photon. Whenever quantising we can get rid of this unphysical states by using a specific condition, the Gupta-Bleuer condition, which again, amounts to putting $\partial_\mu A^\mu = 0$ but in an operatorial way as $$\langle\text{phys}|\partial_\mu A^\mu|\text{phys}^\prime\rangle = 0$$ this condition exactly cancels the unphysical states.
Remember that whenever handling QFT, we need to work with expectation values.
