Why can't a massive particle decay to a lesser mass and a photon? I want to calculate a process $M \to \gamma ~ m$ where m is a lesser mass state and M is a heavier state. Going through the Feynman diagram I get a matrix element ${\cal{M}}^2=2g^2(~(M-m)^2-2mM)$, but I don't think this can be correct. If $M\approx m$ then ${\cal{M}}^2<0$ which is of course bad. I've done this calculation myself twice, my advisor and I did it, I evaluated it with CalcHep, all give the same answer.
Now maybe I've missed a minus sign, but then you get a similar problem when $m=0$.What is going on here?
This is an unusual process that you don't see in the standard model, but I don't see anything wrong with it. There are deeper questions one could ask, like "how does this off diagonal coupling between $m,M$ arise?" but I can imagine some crazy model where you get this vertex and these questions should not matter for this discussion. I should never get ${\cal{M}}^2<0$.
Usual kinematic arguments such as an electron radiating off a photon don't apply due to the mass splitting between $M$ and $m$.
Is there something inherently wrong with this diagram? Is this some sort of cousin to renormalization, where I don't worry about infinity's but I worry about negative signs? ${\cal{M}}^2>0$ demands a certain mass splitting, but why on earth should this be?
Thanks in advance and I look forward to the discussion
Edit: including my calculation...
incoming M state has momenta p, outgoing m state has momenta k.
${\cal{M}}=\bar{u(k)}(-ig\gamma^{\mu})\epsilon_{\mu}u(p)$
squaring...
${\cal{M}}^2=\bar{u(k)}(-ig\gamma^{\mu})u(p)\bar{u(p)}(ig\gamma^{\nu})u(k)\epsilon_{\mu}\epsilon^*_{\nu}$
Averaging over initial spins,summing over final spins...
=$\frac{-g^2}{2}\rm{Trace}(\gamma_{\nu}({\not p}+M))\gamma^{\nu}({\not k}+m))$
=$\frac{g^2}{2}(8p\cdot k-16Mm)$
and
$p\cdot k=\frac{m^2+M^2}{2}$
So,
${\cal{M}}^2=2g^2((M-m)^2-2mM)$
 A: Alright I have a possible solution, and I would like to hear others weigh on it. Bottom line: this process via this vertex is not allowed.
With a vector coupling and a process $M\to m \gamma$, this violates the Ward Identity. i.e. $q_{\gamma}^{\mu} \cdot {\cal{M}}_{\mu}\not=0$ where $q_{\gamma}$ is the momentum of the outgoing photon and ${\cal{M}}_{\mu}$ is the Matrix-Element with $\epsilon^{\mu}$ removed. I think the only way for this process to satisfy the Ward Identity is for $q_{\gamma}=0$, but in this case the decay width goes to zero (and there is no photon).
Now you could look at something like a top quark decaying to a bottom quark via a massive $W^{\pm}$. So we could do the same thing with our $M$ and $m$ state but have a massive vector boson instead. In this case the polarization sum has an additional factor that I am guessing would remove the nonsense that we see, while also satisfying the Ward Identity. (I suppose this must be the case otherwise I don't think the top quark could decay like this).
I think a violation of the Ward Identity indicates that gauge invariance is violated, and cannot be saved. You can find a similar comment about this in this paper in the paragraph above Eq. 3.2. This paper goes on to produce a similar vertex, although with a tensor current instead, which I think satisfies the Ward Identity.
Does this make sense to people? I haven't been able to prove whether or not the Ward Identity is satisfied for the vector current; it looks like $q_{\gamma}^{\mu} \cdot {\cal{M}}_{\mu}>0$ for $q_{\gamma}^{\mu}>0$ but does someone know how to show this explicitly so I can finally sleep at night?
Is my interpretation of the Ward Identity accurate? It's a bit weird to me that I know gauge invariance is violated simply from this identity, and with no knowledge of other terms in my Lagrangian. For all I know, there could be other terms that save gauge invariance, no?
A: The OP's answer is sound, but I will summarize the answer to his

Is there something inherently wrong with this diagram?

to refocus the question.
The answer to this is yes, the vertex supporting it is not gauge invariant, and fits into the broad rubric of SM-compatible interactions:

*

*There are no flavor-changing neutral (& photon) currents!

Call the M state μ for the sake of simplicity and the m state e. There  may be, of course, a kinematically allowed hyper-rare, undetected, decay $\mu\to e\gamma$ in the SM,  but it is a loop-based term involving Ws which mix flavors (including lepton flavors, hence neutrino such). Its effective interaction is proportional to
$$
F_{\mu\nu}~ \overline {\mu _R} \sigma^{\mu\nu}e_L +  \hbox{h.c.}
$$
which is gauge invariant, even though your pathological vertex
$$
\propto A_\mu ~\bar\mu \gamma^\mu e
$$
isn't: for the infinitesimal transform of it to vanish, you'd need, after an integration by parts, $-\Lambda \partial_\mu (\bar\mu \gamma^\mu e)$ to vanish.
This is the current whose charge $\int\!\!dx ~\bar\mu \gamma^ 0 e $ rotates  s to es and vice versa. But you already all-but posited it is not conserved, since you chose s and es to not be degenerate. You may check directly it is not conserved, violated by the mass difference. (If they were, the current would be conserved on-shell, also being the Noether current of the evident O(2) symmetry in that limit, but plain relativistic kinematics would stanch the amp.)
You can reverse-scope such a term cannot be there, since for vector currents it would have to come from the covariant completion of a kinetic term such as $\bar \mu   D\!\! / ~e$ which you can't have, as mentioned by @Andrew.
This is  because you already resolved any μ-e mixing when you unitarily diagonalized the mixing states to get eigen- kinetic and mass terms.  As an aside, this is exactly what happens in the Z neutral current sector, as the separate mass eigenstates are precisely the separate eigenstates coupling to the Z. This unitary cancellation fails for the charged currents precisely because the unitary diagonalization matrices of the upper and lower isocomponents need not be the same, so they do not cancel in unitary multiplication.
