The answer lies in a refinement of your claim that phases with topological order are robust to all local perturbations. The more precise statement is:
A (zero-temperature) phase of matter with (intrinsic) topological order is robust to all local perturbations which do not close the energy gap above the ground state.
If we close the energy gap, critical modes appear which destroy the topological order, and on the 'other side' of the critical point we can then have a 'boring' phase (such as a symmetry-breaking magnet etc).
Hence, the energy gap (defined as the smallest amount of energy necessary to create a quasiparticle excitation) is paramount. It turns out that in many cases, the gap $\Delta$ is rather small, making these phases hard to see. First, it means we have to achieve temperatures $T \ll \Delta$ such that we will effectively see the quantum regime (remember that topologically ordered phases are really only well-defined at zero temperature). Second, a small gap $\Delta$ means that these phases are typically constrained to live in a small region of parameter space, since perturbations (e.g., an external field $h\gtrapprox\Delta$) can close the gap and trivialize the phase.
Ultimately, then, we arrive at the question: why is the gap so small? In principle, the gap can be large. Many solvable models are known where the energy gap is of the same order as the interaction strength of the model. A paradigmatic example is the Kitaev toric code model. However, these models involve interactions where multiple constituents interact at once (these are call multi-body interactions); this is often necessary to give rise to the exotic correlations that characterize topological phases. But of course physical systems tend to have at most two-body interactions. It has been well-appreciated that for such two-body interactions, one can still realize an effective multi-body interaction by using perturbation theory. However, this naturally means that the energy gap will be quite small (since, after all, we are relying on some perturbative approximation). I think this is ultimately the reason why we are not being buried in examples of topologically ordered materials.
There is, however, a celebrated instance of intrinsic topological order in solid-state materials: the fractional quantum Hall states. How do these evade the aforementioned issue? Here, very large magnetic fields are used to create well-separated Landau levels. Due to the non-triviality of these levels (and due to the magic of Pauli exclusion), it turns out that the two-body Coulomb interactions are sufficient to deform (and split) these Landau levels into topological phases with exotic correlations---one does not need to rely on higher-order perturbation theory. (For further reading in this direction, you can look up the Haldane pseudo-potential.) In this case, the gap is thus rather favorable (at least relatively speaking).
Let me mention that in addition to the issue of the energy gap, there is a second subtlety that makes it hard to find experimental realizations of topological order. Even if you were lucky and found such a material, it is very non-trivial to devise a measurement that actually tells you that you have such a phase of matter. Indeed, almost by definition local probes cannot give you conclusive evidence. Hence, it might very well be that many known materials exhibit topological phases, but that they have been overlooked for this reason. Again, the quantum Hall states are an exception since in that case we can detect them through the very accessible quantized Hall conductance, which is sadly not a generic feature of other topological phases.
