Consider the Maxwell kinetic Lagrangian
$$L_{kin}=-\frac{1}{4}(F_{\mu\nu})^2$$
where the field strength is $F_{\mu\nu}=\partial_\mu A_\nu-\partial_\nu A_\mu$. A vector quantity can contain at most 4 degrees of freedom (d.o.f.). Just from the definition of $F_{\mu\nu}$ it is clear that the longitudinal degree of freedom $\partial_\mu\chi$ in the vector
$$A_\mu=A_\mu^\perp+\partial_\mu \chi$$
cancels out of $L_{kin}$. Additionally, one can use the gauge invariance $A_\mu\to A_\mu+\partial_\mu\xi$ to e.g. eliminate the spatially longitudinal component in $A_i=A_i^T+\partial_i A^L$. This shows that the massless Maxwell field only propagates 2 degrees of freedom.
However, if we add a mass term to the Lagrangian:
$$L_m=-\frac{1}{2}m^2A_\mu A^\mu$$
This reintroduces a kinetic term for the longitudinal component $\partial_\mu\chi$ above. Additionally, the gauge invariance is broken.
My question is:
Without gauge invariance, how can we show in this case that the massive field propagates 3 degrees of freedom and not 4?