We are looking for a vector field $A_{\mu}(x)$ which has spin 1 particle excitations, and does NOT require gauge invariance to describe it. Let's figure this out systematically, although I won't go through the gory details (references are below). First of all, the vector field is in the $(\frac{1}{2},\frac{1}{2})$ representation of the Lorentz group, so translating this into what spins this field could possibly produce, it is spin $0$ and spin $1$. If we wish to kill the spin $0$ component of the field, which would be of the form $A_{\mu}(x)=\partial_{\mu}\lambda(x)$, we could
1) Require that our theory has a gauge invariance $A_{\mu}\to A_{\mu}+\partial_{\mu}\lambda$.
2) Require that the field $A_{\mu}$ satisfies the "Lorentz gauge" constraint $\partial_{\mu}A^{\mu}=0$ (although calling it a gauge in this context is misleading).
Or we could simply leave the spin $0$ excitation alone and let it propagate.
Let's now consider the effect of the particle's mass. Starting with a massless spin $1$ particle. It turns out, that on very general circumstances, it is impossible to construct a vector field with massless excitations which transforms under Lorentz transformations the following way
$$U(\Lambda)A_{\mu}U^{-1}(\Lambda)=\Lambda^{\nu}_{\mu}A_{\nu}$$
Where $U(\Lambda)$ is a unitary representation of the Lorentz group. To understand why this is, it has to do with how the vector field is represented in terms of creation and annihilation operators. In general we have
$$A_{\mu}(x)=\int d^3p(2\pi)^{-3/2}(2p^0)^{-1/2}\sum_{\sigma=\pm 1}\Big(e_{\mu}(p,\sigma)e^{ip\cdot x}a(p,\sigma)+e^*_{\mu}(p,\sigma)e^{-ip\cdot x}a^{\dagger}(p,\sigma)\Big)$$
Where $e_{\mu}(p,\sigma)$ is the polarization vector, $\sigma$ is the helicity and $a(p,\sigma)$ is an annihilation operator of the massless spin $1$ particle. The $U(\Lambda)$ acts on the $a(p,\sigma)$ and they represent the true particle content of the field excitation. Somehow, we need a polarization vector $e_{\mu}(p,\sigma)$ which transforms appropriately to go from the massless spin $1$ particle representation the $a(p,\sigma)$ transform under, to the $(\frac{1}{2},\frac{1}{2})$ representation that the field $A_{\mu}$ should transform with respect to. This simply cannot be done.
The best that can be done is the following
$$U(\Lambda)A_{\mu}U^{-1}(\Lambda)=\Lambda^{\nu}_{\mu}A_{\nu}+\partial_{\mu}\Omega(x,\Lambda)$$
This is a combination of a Lorentz transformation and a GAUGE transformation. This is highly concerning, as this implies that the theory can no longer be unitary. In order to remedy this we must require gauge invariance! Thus any massless spin 1 particle must be described by a gauge invariant vector field. Similar arguments can be made for non-abelian vector fields.
Once we add a mass, these issues don't arise, and gauge invariance is not required. But constructing a unitary and renormalizeable theory is another story. One could imagine adding a simple mass term to the Yang-Mills action $-m^2A_{\mu}A^{\mu}$ to describe a massive spin $1$ particle, however the pesky minus sign in the Minkowski inner product creates an unstable vacuum. One can attempt to remedy this by adding a term $-\lambda (A_{\mu}A^{\mu})^2$, but then we run into the issue of renormalizability, as well as a spontaneous breakdown of Lorentz symmetry.
The fact of the matter is it is much "easier" to describe a massive spin $1$ particle using gauge invariance. This is done in the standard model, where the $W$ and $Z$ bosons are massive spin $1$ particles. These are described by an $SU(2)\times U(1)$ gauge theory which is spontaneously broken, leaving some of the modes massive. This can be done in a unitary and renormalizeable way.
Most of the details here can be found in Volume I of The Quantum Theory of Fields by Steven Weinberg, Sections 5.3 and 5.9.
