In every textbook I've read the only argument why gauge bosons can't have mass is "because a mass term would break gauge invariance". Even fermions can't have mass because it would break the $SU(2)_L$ electroweak invariance. But now I wonder why we'd like to have a gauge invariant Lagrangian in the first place. Why is this necesary? Gauge theory is just a great way of finding interaction terms but we don't need to actually have those symmetries.
What would be the problem if a fermion had its own mass term? Or a gluon? Or any other particle? I've read the following questions but I haven't found them useful to answer my question: Why can't gauge bosons have mass?, What goes wrong if we add a mass term for gauge bosons without the Higgs mechanism?