Why is the critical exponent $\alpha$ negative at the Ising spin-glass transition?

The specific heat usually diverges at a phase transition - typically as a power-law, giving a critical exponent $\alpha > 0$. (Although in 2D, sometimes the divergence is only logarithmic, as with the Ising model, and sometimes the specific heat remains smooth but non-analytic so there is no divergence at all, as with the Kosterlitz-Thouless transition. In both cases we formally have $\alpha = 0$.) This makes sense intuitively: the heat capacity roughly counts the number of "actively fluctuating" internal degrees of freedom through which a temperature increase can raise the internal energy, and at a second-order phase transition you have huge domains that are actively fluctuating together. (Or at a first-order transition, any additional thermal energy you add to the system goes into changing chunks of the low-temperature phase into the high-temperature phase, so it doesn't raise the temperature at all.)

But this paper claims that for 3D locally-coupled spin glasses (of either Ising, XY, or Heisenberg spins), $\alpha \approx -2$ is strongly negative at the spin-glass transition, so the singular part of the specific heat goes as $\sim |\tau|^{2 + \epsilon}$ for some $|\epsilon| \ll 1$. This seems to imply that the specific heat reaches a local minimum at the spin-glass transition temperature, which seems very counterintuitive to me. What is the physical interpretation of this strange temperature dependence of the specific heat?

Actually, a negative exponent doesn't mean the specific heat has to be at a local minimum at criticality. For instance, Fischer's paper Renormalization of Critical Exponents by Hidden Variables, writes (Eq. 2.46, where $\alpha_\mathrm{x}>0$)$^1$

$$C(\tau) \approx A - B\tau^{\alpha_\mathrm{x}},$$

which means the specific heat reaches a (finite) maximum at the critical temperature.

As for why $\alpha$ is negative, it's already clear from the paper linked in the question: absence of divergence, since a positive $\alpha$ would mean $|\tau|^{-\alpha}$ diverges. Fisher argues in the paper mentioned above that that is a general characteristic of non-ideal systems, a statement that should stand at least for frustrated systems.

But it can be a minimum: in 2D, for instance, $T_c=0$ and, as Jorg et al (arxiv) put it:

at any low temperature, a fraction [...] of the system’s spins can be thermally activated

which explains why $C(\tau\to 0)$ should tend to a minimum, given the intuition that "the heat capacity roughly counts the number of "actively fluctuating" internal degrees of freedom through which a temperature increase can raise the internal energy".

$^1$: Here the positive exponent $\alpha_\mathrm{x}$ is the exponent of the real system, which can be written as a function of the ideal exponent $\alpha$ as $\alpha_\mathrm{x}=-\alpha/(1-\alpha)$.