What is so special about spontaneous symmetry breaking? (time reversal example) I have serious trouble understanding the concept of spontaneous symmetry breaking (in condensed matter specifically). Let's take time reversal in magnetic systems as an example. 
Ferromagnetism is said to spontaneously breaks time reversal symmetry. As I understand it, time reversal symmetry can be understood with a time reversal operator $\mathcal{T}$ that reverses the sign of all momentum and spin, so that $\mathcal{T} S_{zi} =- S_{zi} \mathcal{T}$. 
We take an hamiltonian that can generate ferromagnetism like Ising
$$ H_0 = -J \sum_{<i,j>} S_{zi} \ S_{zj} $$
and notice that $[\mathcal{T}, H_0] = 0$ because there are two spin operators. So no symmetry breaking here.
Apparently, a spontaneous symmetry breaking is manifested in the asymmetry of the ground state rather than that of the hamiltonian. There are two ground states for the hamiltonian one with all spins up 
${\left|\left. \uparrow \right>\right.}^{\otimes n}$ and one with all spins down ${\left|\left. \downarrow \right>\right.}^{\otimes n}$. Since $\mathcal{T} {\left|\left. \uparrow \right>\right.}^{\otimes n} = {\left|\left. \downarrow \right>\right.}^{\otimes n}$, time reversing keeps you in the ground state, so no symmetry breaking here.
A point that is often made is that passing below the critical temperature breaks the time reversal symmetry because you will find the system exhibits non vanishing magnetization and thus is in a particular ground state not in a superposition of both. But this is only the case because some noise in the environment (it could also be an irregularity in the system) caused the system to choose a particular direction. We could simply add that noise into the model by saying 
$$H= H_0 + \delta H$$ with $[\mathcal{T}, \delta H] \neq 0$. Then the total hamiltonian is not symmetric. 


*

*Is that the essence of the so called "spontaneous symmetry breaking"? If it is, what is so special about it?

*Couldn't we just say that below the critical temperature the system is greatly susceptible (literally since susceptibilities are discontinuous) to small perturbations? 

*Is there a rigorous definition of what a spontaneous symmetry breaking is?

 A: You've already mentioned the exact definition of spontaneous symmmetry breaking:
Spontaneous symmetry breaking for a system that is described by a hamiltonian $H$ with ground state $\left| g \right\rangle$ happens where there is a symmetry transformation of $H$ that doesn't leave the ground state invariant
$$[T,H] = 0  \text{ but } T \left| g \right\rangle \neq 0. $$
Much like a stick that's standing on its tip can be rotated around itself but will eventually fall back to a "ground state" that doesn't have this rotational symmetry anymore.
The example you are quoting is a statistical theory, so the fluctuations that drive the system into a state of non-vanishing magnetisation below the critical temperature are already built in from the start.
Spontaneous symmetry breaking is a key ingredient in the Standard Model of particle physics, it is used to explain why the particles that mediate the weak force are massive (W and Z bosons, this is remarkable because you can't achieve that by just putting a mass term in the Lagrangian!).
But it also works the other way around, in that it explains why sometimes there are massless modes in a system (see Goldstone bosons).
A: Indeed, one of the definitions of spontaneous symmetry breaking is in terms of its susceptibility:

Suppose we add a symmetry breaking perturbation $h \; \delta H$ to our
  Hamiltonian (as you do), if
  $$ \lim_{h \to 0} \lim_{N \to \infty} \langle m \rangle \neq 0 $$ 
  then we say our system has spontaneous symmetry breaking.

(Note: $N$ is the number of spins in our system. Indeed, on a mathematical level, non-analyticities can only arise in the thermodynamic limit.)
What is special is that any arbitrarily small perturbation will do. Imagine you have a million spins. If the state is originally in a symmetric state (i.e. not symmetry broken yet), then even if I just apply an arbitrarily small magnetic field on a single spin, the whole system will choose that orientation.
You suggest that the fact one in principle needs the environment to 'make the choice' that this is not really spontaneous. It is true that in that philosophical sense of the word, the direction of magnetization is not 'spontaneous'. But what can be called spontaneous in the universe? If I perfectly balance an egg, then the direction it will eventually roll when it loses its balance is spontaneous (or not spontaneous) in exactly the same sense. And note that once the egg has rolled down (and stopped), the tiny perturbations in the air which influenced its original direction are now no longer sufficient to change its position. I.e.: after the `spontaneous' process, the system is now stable.
The same thing happens in the above magnet: once it has chosen a direction of magnetization, then changing the applied magnetic field on that single spin I mentioned before will not change the total magnetization. So in that sense it is not true that it is so susceptible! One needs to apply an extensive magnetic field (i.e. a field that acts on most of the spins) to change the direction of the magnetization.
That is what is so funny about these systems: 

An arbitrarily small perturbation can create a magnetization, but it
  cannot change it!


On a more quantum-mechanical note, if one has a Hamiltonian whose ground state should display spontaneous symmetry breaking, then if one takes the ground state to be in a symmetric superposition (which one can always do), then this state has ridiculously long entanglement. These are called cat states (in reference to Schrodinger's cat). This is a natural consequence of the above: an interaction with a single spin has to influence all spins at once, which is only possible if every single spin is entangled with every other spin. An example is the state $|\uparrow \uparrow \uparrow \cdots \rangle + |\downarrow \downarrow \downarrow \cdots \rangle$. (Indeed: an interaction with a single spin will collapse this 'cat state' to a product state, and then it is clear that any subsequent single-spin interaction cannot flip the state to the other product state.) Indeed, the way symmetry breaking phases are classified in one spatial dimension is in terms of these entanglement properties [Schuch et al., 2010].
