The fact that $m^2A_\mu A^\mu$ is both Lorentz invariant tells that this term is an allowed term in the Lagrangian. But it does not explain why $m^2 A_\mu A^\mu$ represent the mass term.
The reason why $m^2\phi^2$ is called the mass term of the scalar field is that after quantization, each single particle state $|p\rangle$ in the Fock space is characterized by definite eigenvalue $m^2$ for the operator $P_\mu P^\mu$ i.e., $$P_\mu P^\mu|p\rangle=p_\mu p^\mu|p\rangle=m^2|p\rangle$$ which amounts to the relativistic dispersion relation $E^2-\textbf{p}^2=m^2$. Therefore, one can identify $m$ as the mass of the quanta of the theory.
The reason is same for a massive Proca field ("massive photon"). If you start with a term $m^2A_\mu A^\mu$ in the Lagrangian, after quantization, the operator $P_\mu P^\mu$ yields the eigenvalue $m^2$ on the one-particle states. It is the same $m^2$ that appear in the relativistic dispersion relation for that particle.