Ideal Paramagnet of spin 1/2 and canonical Ensemble In the ideal Paramagnet of spin $\frac{1}{2} $ system, I was looking into the derivation of its partition function as covered under standard undergrad course.
The Hamiltonian is given by,
$$\textbf{H}= -\mu_0H \sum_{j=1}^{N}\sigma_j $$
Then canonical partition function is defined as:
$$Z=\sum\exp{(-\beta E_i)}$$
Here,
$$Z=\sum_{(\sigma_j)}\exp{(-\beta \textbf{H})}$$
Now I can play with Helmholtz free energy, magnetic moment and all.
In dealing with this system, I didn't get the essence of canonical ensemble, except the fact that I have used $Z=\exp{(\beta \textbf{H})}$.
The whole proof goes along with basic logic and I was not able to appreciate it as a good example of the canonical ensemble. If I have given this system, why should I have to start with the canonical ensemble?
Where I would have faced the problem if I have not used Canonical ensemble (or I have not used the concept of the ensemble at all). It all worked like the magic of $Z=\exp {(\beta \textbf{H})} $.
Can't I deal with this system using other ensembles?
 A: You can handle the systems in other ensembles. For "harmless" systems the thermodynamic variables will behave the same for the different ensembles in the thermodynamic limit, so usually it does not matter what ensemble you use.
Of course you'll have to tune the thermodynamic variables to match the observables. E.g. when describing a gas in the canonical or the grand canonical ensemble, to match the observables, you have to tune your chemical potential to make
$$ \left<N\right>_{\mu,T,V}^{\text{grand}} = N^{\text{can}}. $$
But with this matching you will get matching answers for all thermodynamic observables, the equations of state, etc.
So usually you just use the ensemble that makes your calculation simple (either because of the concrete way your system is prepared or due to technical simplifications). For the paramagnet the canonical ensemble leads to the thermodynamics of the system with minimal effort, and thus is a great example to appreciate the canonical ensemble.
The small but
For small systems coupled to heat or particle baths (that is, when you are not working in the thermodynamic limit) the results will be different in the different ensembles (think, e.g. of an impurity in a metal, where you have a heat and particle bath, or a localized spin in a quantum gas, where you only have a heat bath).
You can understand the equivalence for large systems by considering a sub-system of a much larger system, the the bath plus your system act as bath for the sub-system. This way, you can, for example, consider a sub-system of a canonical system as a system in the grand-canonical ensemble. Or a sub-system of a micro-canonical system as a canonical system (where the rest of the system acts as canonical bath).
The big but
The paramagnet is not a "harmless" system (because its energy spectrum is bounded from above).
If we look at the system in the canonical ensemble in the limit $T \to \infty$ we will see that the energy of the system goes to zero. That is, not all states of the system can be reached in the canonical ensemble (since there are states up to the energy of $E_\text{max} = \mu_0 H N$).
We therefore look at the system microcanonically. This is much more work than the canonical approach, but it will allow to explore the states that cannot be reached in the canonical ensemble. From now on, w.l.o.g, we assume $H > 0$ to simplify the discussion.
For an energy $E$ the number of states is (assuming the energy is admissible for the system, otherwise we extend microcanonics to count states in an intensive energy interval and take the gauss bracket of the fraction)
$$ \Omega(E, N, H) = \binom{ N }{ \frac{E+\mu_0 H N}{2\mu_0 H} }. $$
To simplify the calculations we make a change of variables use the fraction $\epsilon$ of spins that are in the up state instead of $E$:
$$E = \mu_0HN \epsilon - \mu_0 H (1-\epsilon) N = (2\epsilon-1) \mu_0 H N. $$
This simplifies the number of states to:
$$ \Omega(\epsilon, H, N) = \binom{N}{\epsilon N}.$$
Now we can expand the extensive part of the entropy for large $N$ using the Stirling formula:
\begin{align*}
 S(E, N, H) &= \ln\big(\Omega(E, N, H)\big) = \ln \left( \frac{N!}{(\epsilon N)! \big((1 - \epsilon) N\big)! } \right) \\
            &= \ln(N!) - \ln\big((\epsilon N)!\big) - \ln\Big( \big((1 - \epsilon) N\big)! \Big) \\
            &= N \log(N) - N - \epsilon N \ln(\epsilon N) + \epsilon N - (1-\epsilon)N \ln\big( (1-\epsilon) N \big) + (1-\epsilon) N + O\big( \ln(N) \big) \\
            &= - N \epsilon \ln(\epsilon) - N (1-\epsilon) \ln(1-\epsilon) + O\big( \ln(N) \big)
\end{align*}
For large $N$ we can treat the energy as continuous.
The entropy has a maximum for $\epsilon = \frac 1 2$ and is zero at $\epsilon \in \{0, 1 \}$. The temperature
$$ \frac 1 T = \partial_E S $$
goes to infinity as $E \to 0$.
If we prepare the system in a state with $\epsilon > \frac 1 2$, the entropy will decrease as we increse the energy, if we take the formula above seriously, this means that the system has a negative temperature. (And that negative temperatures are in a way higher than $T \to \infty$).
We can also see where the mapping between the canonical and microcanonical ensembles fails for small $N$: We neglected the non-extensive part of $S$, and that part becomes relevant for small systems. The microcanonical ensemble does not even have a well defined temperature for small systems (since the energy can't be treated as a continuous variable).
(On a side note, those states can be reached by using a heat bath of negative temperature in the canonical ensemble, but we usually don't allow this, as it leads to unphysical results. E.g. when coupling a system with a spectrum that is not bounded from above to a heat bath of negative temperature, then the bath and the system won't equilibrate, but the bath will transfer an infinite amount of energy to the system. In real systems the energy spectrum is never bounded from above, but we you prepare negative temperatures in subsystems whose relaxation processes with respect to the rest of the system are strongly suppressed.)
More ensembles
Finally, for magnetic systems we have another interesting ensemble. We can do a Legendre transform of $F$ on the variable $H$ to the magnetization $\mu$ (since $\mu = -\partial_H F$, don't nail me on the sign). This results a "modified" free energy (I've seen several names for this ensemble and free energy):
$$ \bar{F}(T, N, \mu) = F(T, N, H(T, N, \mu)) + \mu H(T, N, \mu) $$
As usual $H(T, N, \mu)$ is obtained by inverting $\mu(T, N, H) = -\partial_H F$.
This ensemble is very useful for analysing certain problems (e.g. the spontaneous magnetization of ferromagnets, the minima of $\bar F$ show the spontaneous magnetization).
Again we use that in the thermodynamic limit the thermodynamic observables are the same regardless of the ensemble.
Summary
You can approach the system with different ensembles, in the thermodynamic limit (and for positive temperature situations) you will get equivalent results. But the canonical ensemble is the simplest to work with for this kind of system.
If you have special questions, you may have to do a Legendre transformation to change over to another ensemble. Using Legendre transformation from the canonical ensemble will be easier than working things out directly (just look how complicated the microcanonical calculation was, compared to the canonical one, I am not even sure how to directly calculate $\bar F$, it is probably possible but certainly more difficult).
For other systems and preparations, other ensembles may simplify the work. E.g. for non-interacting electrons in a lattice, the grand canonical ensemble will be the simplest approach (even though your experimental preparation will probably be canonical).
