Paramagnet: Negative specific heat? for a simple paramagnet ($N$ magnetic moments with values 
$-\mu m_i$ and $m_i = -s, ..., s$) in an external magnetic field $B$, I have computed the Gibbs partition function and thus the Gibbs free energy $G(B,T)$ and obtained 
a simple approximation for small $B$ fields, which has the basic form 
$$G(B,T) = N k T a - N b B^2/T$$
for suitable constants $a$ and $b$. From this, I would like to compute the heat capacities for both constant $B$ and constant magnetization, $M$. I can obtain the magnetization from 
$$M = -\frac{\partial G}{\partial B} = 2 N b B/T$$
and from this the total energy via
$$E = -MB$$
For the heat capacity at constant field, I would then write this in terms of $B$ and $T$ alone, i.e.
$$E = - 2 N b B^2 / T$$
so 
$$C_B = 2 N b B^2 / T^2$$
this makes sense for small $B$ and large $T$ as we have a finite-state system that achieves saturation: Eventually, there simply are no more available states to add more energy to the system.
Now for the heat capacity at constant magnetization. My idea was to just rewrite $E$ again, this time as a function of $M$ and $T$ alone, which gives 
$$E = -MB = -\frac{M^2 T}{2Nb}.$$
This time, obviously
$$\frac{\partial E}{\partial T} = -\frac{M^2}{2Nb}$$
so I have a negative heat capacity.
Now, I have a hard time visualizing this. I increase the temperature and thereby 
lower the energy? Is this because increasing the temperature increases 
spin fluctuations so they are not as aligned anymore and, in absence of a kinetic energy term, these fluctuations don't carry energy themselves?
Or have I gone wrong at some part in my derivation? 
 A: It is the constant magnetization assumption which causes this.  If you increase the temperature at constant $B$, then the magnetization would reduce, so to keep magnetization constant, you have to increase the magnetic field.  The energy, $E=-MB$, so as you increase $B$ at constant $M$ you are making energy more negative.
Does that make sense?
A: Shanth answers is, basically, what I was after. There is one additional interesting point to note about such systems, and that is that they can have infinite or negative temperature!
Recall that temperature can be defined as 
$$1/T = \frac{\partial S}{\partial E}$$
that is, temperature is related to how entropy changes with energy. I won't go through the exact calculations for my system, but here's a simple argument to see how we get infinite and negative temperature:
At constant magnetic field, the entropy is a function of internal energy. The paramagnet can have a maximum energy of $N \mu B$ if and a minimum energy of $-N\mu B$. These are obtained if either all magnetic moments are anti-parallel or parallel to the magnetic field. But for these extremal energies, the entropy is zero, because there is exactly one micro-state associated with these. 
On the other hand, for a magnetization of $0$, we have maximum entropy as this is the state where we have the most possibilities for the individual magnetic moments to arrange themselves. 
So, the curve $S(E)$ has a zero at some energy, another zero at another energy, and a maximum in between. At the maximum, $\partial S / \partial E = 0$, so $1/T = 0$ so $T = \infty$. Beyond the maximum, $\partial S / \partial E < 0$, so $1/T < 0$ and hence 
$T < 0$. 
The physical meaning of infinite and negative temperature is that if you bring such a system into contact with a system of a finite, positive temperature, it will be more than happy to give up its energy so that it can gain entropy.
