Can gauge fields be avoided by not specifying a coordinate system? I've been studying quantum field theory, and recently my focus has been on the complex scalar field $\phi(x_\mu)\in\mathbb{C}$ because it seems to be the simplest field that can couple to an interaction like electromagnetism. In particular, I'm interested in seeing how the 4-potential manifests as the gauge field necessary for keeping the Lagrangian invariant under a local phase shift $\phi\to e^{-i\alpha(x)}\phi$.
However, I am having difficulty understanding why we want this local gauge invariance in the first place. A global symmetry makes sense to me because a uniform rotation of every vector has no physical significance. However, rotating the field by a different amount at each point through a local transformation seems like it would produce a completely different field. I am having trouble seeing how this transformation does not change all of the physics. I have been reading through textbooks like Baez, Henneaux & Teitelboim, and Naber, so things are starting to take shape, but I wanted to ask to see whether I was on the right track.
I really like Terrance Tao's perspective of gauge transformations as changes of coordinates. He talks about how "coordinate systems identify geometric or combinatorial objects with numerical (or standard) ones". Connecting to my focus on the complex scalar field, he writes:

to convert a position on a circle to a number (modulo multiples of
2\pi), or vice versa, one needs to pick an orientation on that circle,
together with an origin on that circle.  Such a coordinate system then
equates the original circle to the standard unit circle

To me, this means that in order to assign a complex number to each point we must choose a reference direction within the plane to call "1"/the positive real line. This is an arbitrary (smooth) assignment at each point in spacetime that should have no physical meaning, so the the physics, i.e. the Lagrangian, should be invariant under an alternative choice of direction. However, all choices of directions for the positive real line are related to each other by a rotation in the complex plane, so the invariance required is really that of a local phase shift. This would make sense to me. Just thinking about putting a complex number everywhere in space seemed simple, but I had not realised the nuances of how that is not possible without implicit choices being made that bring in a the need for a gauge to compensate.
That being said, could this be avoided by not specifying a coordinate system at all? For example, rather than have $\phi\in\mathbb{C}$ could we have $\phi\in V$ for some 2-dimensional vector space $V$? Or perhaps there is another way to have coordinate invariance? The first approach seems problematic, as we'll want to do calculus with the field, so it will need to be described with some smooth manifold. However, differential geometry is meant to be coordinate-free and does not introduce gauge fields in order to do calculus.
 A: No, gauge symmetry cannot be "avoided" by not specifying coordinates. Tao's analogy is supposed to say that just as we are free to choose any coordinates we want for a manifold, we are free to choose any gauge (fixing) we want for our gauge theory. The fixing of a gauge resembles a choice of origin for a coordinate system, but it is not equivalent to it.
We can talk about a gauge-theory in entirely coordinate-free language, this doesn't change that it is a gauge theory. For instance, in a standard Yang-Mills theory on a manifold $M$ you have some gauge group $G$, some matter fields $\phi_i$ valued in representations $(V_i,\rho_i)$ of $G$, and a gauge field $A$ valued in the Lie algebra $\mathfrak{g}$ of $G$. Saying this is a gauge theory means that for any smooth function $g : M \to G$ with $g(x) = \mathrm{e}^{\mathrm{i}\alpha(x)}$ for some $\alpha : M\to G$, the transformations
\begin{align}
 \phi_i(x) & \mapsto \rho_i(g(x))\phi_i(x) \\
 A(x) & \mapsto g(x)A(x)g^{-1}(x) + \mathrm{d}\alpha(x)
\end{align}
are a symmetry of the Yang-Mills action and that two sets of $(\phi_i, A)$ related by such a transformation are physically indistinguishable. There is no choice of coordinates here.
What Tao is likely thinking of is that, in the $G$-principal bundle over $M$ associated with such a gauge theory, the gauge transformation $g$ corresponds to "choosing" a new way to identify the fibers, which are abstractly diffeomorphic to $G$ but for which no "natural" diffeomorphism exists, with the group $G$. I.e. what was the identity ("origin of coordinate system") in the fiber over $x$ before now is $g(x)$, and what was $g^{-1}(x)$ before is now the identity.
But we cannot refrain from "choosing coordinates" in this sense - I didn't have to choose an identification of the fibers with $G$ in order to write down the gauge transformations of YM theories above! The essence of a gauge theory is that our chosen dynamical variables, i.e. the $\phi_i$ and $A$, are an overcomplete description of the physical system we're trying to model, and the gauge symmetry is the manifestation of that redundancy: The underlying system has fewer degrees of freedom than we're using in our model, and the gauge symmetry identifies distinct values of the dynamical variables in our model that correspond to the same physical states of the underlying system.
Gauge symmetry is therefore not something "desirable", it is just something forced on us when describing systems with such overcomplete sets of variables. See this answer of mine and this answer of mine for related discussion of "why" we would "want" gauge theories.
