What is the topological space in “topological materials/phases of matter”? I’m embarrassed to admit that after sitting in on several “topological physics” seminars, I still don’t understand the basic ideas of the area. In particular, when physicists talk about the “topology” and “topological properties” of a system, what do they mean exactly?
I know the definition of topological spaces and the fundamentals of (point-set) topology and topological manifolds. With this background, the first, most basic question to be answered (though I have never seen it addressed directly) is what topological space is being dealt with when describing a physical system. Sometimes the answer to this seems to be paths through space (say, when describing anyons), sometimes it seems to be about the surface of a physical object (say, when describing quantum Hall currents), and sometimes it seems to be about the spectrum of a system’s Hamiltonian.
I am hoping someone can give a concise summary of the form: “Topological physics deals with _____ systems. When analyzing these systems mathematically, a topological space that naturally appears is _____ . Topological properties of this space, such as ______, can correspond to physical properties, such as ______ .”
 A: Based on OP's elaboration in the comments on their background, I will try to make this answer pedagogical and self-contained, erring ont he side of simplicity in the beginning; however, since OP mentioned some familiarity with topological manifolds and vector bundles, the concluding section will become more sophisticated. Before beginning, however, I'll do my best to give the requested concise, mad-lib style summary.
“Topological physics deals with systems which possess global properties which are insensitive to local perturbations. When analyzing these systems mathematically, some topological spaces that naturally appear are the Bloch and conduction/valence bundles. Topological properties of these spaces, such as Chern number $C$ (i.e. Berry curvature integrated over the BZ), can correspond to physical properties, such as Hall conductivity $\sigma_H = \frac{e^2 C}{h}$. Since the Chern number is insensitive to local perturbations which do not induce singularities in the Berry curvature, so too is the Hall conductivity, which is manifested in the integer quantum hall effect.”

Consider a particle hopping on an infinite 1-dimensional lattice. The appropriate Hilbert space is $\ell^2(\mathbb Z)$, the space of square-summable maps $\mathbb Z \rightarrow \mathbb C$.  The simplest Hamiltonian is$$H:= v\left(\sum_{m\in \mathbb Z}|m \rangle\langle m+1| + |m+1\rangle\langle m|\right)$$
where $v\in \mathbb R$ is called the nearest-neighbor hopping amplitude.
Recalling the identities $$\sum_{m\in \mathbb Z} e^{imk}= 2\pi \delta(k)  \qquad \int_{-\pi}^\pi \mathrm dk \ e^{imk} = 2\pi \delta_{m0}$$
we make use of the semi-discrete Fourier transform:
$$|m\rangle = \frac{1}{\sqrt{2\pi}}\int_{-\pi}^\pi \mathrm dk \  e^{ikm} |k\rangle \iff |k\rangle = \frac{1}{\sqrt{2\pi}}\sum_{m\in \mathbb Z} e^{-ikm}|m\rangle$$
$$\implies H = \int_{-\pi}^\pi \mathrm dk \ 2v\cos(k)\ |k\rangle\langle k| \equiv \int_{-\pi}^\pi \mathrm dk \ \mathcal{E}(k) \ |k\rangle\langle k|$$
The spectrum of this Hamiltonian is clearly $\sigma(H)=[-2v,2v]$. The $|k\rangle$'s are generalized eigenstates, and are the discrete analog of plane waves in continuous spaces; for the time being, we ignore any technicalities related to their non-normalizability.
Critically, note that $|k+2\pi\rangle = |k\rangle$; as a result, the parameter space in which the $k$'s live is not the interval $[-\pi,\pi]\subset \mathbb R$ but rather the circle $S^1$, which you can think of as the aforementioned interval with opposite ends identified.  In $d$ spatial dimensions, this parameter space is the $d$-torus $\mathbb T^d$ (note that $\mathbb T^1 \simeq S^1$). In general, this space is called the Brillouin zone which I will denote $\Gamma$.

This model is somewhat boring, so let's imagine now that our lattice is bipartite; alternating adjacent sites are labeled $A$ and $B$. Each unit cell now consists of two sites, and the intracell hopping amplitude $v$ and intercell hopping amplitude $w$ are generally different.  A schematic of the lattice is shown below, with the $A$ sites colored gray and the $B$ sites colored white.

This is called the Su-Schrieffer-Heeger model. It is convenient to decompose the Hilbert space as $\ell^2(\mathbb Z)\otimes \mathbb C^2$, with a generic site $|m\rangle\otimes |A/B\rangle \equiv |m,A/B\rangle$ labeled by the cell $m$ and intracell site $A/B$. The Hamiltonian is now given by
$$H:= v\left(\sum_{m\in \mathbb Z}|m,A\rangle\langle m,B| + h.c.\right) + w\left(\sum_{m\in \mathbb Z} |m,B\rangle\langle m+1,A| + h.c.\right)$$
where $h.c.$ stands for "Hermitian conjugate." Making the substitution $|m\rangle = \frac{1}{\sqrt{2\pi}}\int_\Gamma \mathrm dk \ e^{imk}|k\rangle$ as before, this becomes
$$H = \int_\Gamma \mathrm dk \pmatrix{0 & v+we^{-ik} \\ v+we^{ik} & 0}|k\rangle\langle k|$$
where the matrix acts on the $\mathbb C^2$ part of the Hilbert space. This is much more interesting; we have effectively decomposed the Hamiltonian as an integral superposition of $2\times 2$ matrices $H_k$.  These can be individually diagonalized with little effort; we have $\sigma(H_k) = \big\{\pm \sqrt{ v^2+w^2+2vw\cos(k)}\big\}$, and so taking the union over all $k\in \Gamma$, the spectrum of the full Hamiltonian is $\sigma(H) = [-(v+w),-|v-w|] \cup [|v-w|,v+w]$ (where I've assumed that $v,w>0$).  As long as $v\neq w$, this spectrum consists of two continuous bands labeled (arbitrarily) with band indices $n=\pm 1$ and separated by a gap $\Delta = 2|v-w|$.


We now reframe the SSH model (assuming half-filling and $v\neq w$, so it models an insulator) and examine it through the lens of topology.  We do this by decomposing the Hilbert space $\mathscr H$ as a direct integral of Hilbert spaces $\mathscr H_k \simeq \mathbb C^2$ for $k\in \Gamma$; we then have that
$$\mathscr H = \int_\Gamma^\oplus \mathscr H_k \mathrm d\mu(k) \iff \mathscr H \ni \Psi = \int_\Gamma \mathrm dk \ \psi_k |k\rangle$$
where $\psi_k\in \mathscr H_k$.  The key thing to formalize is that while $\mathscr H_k$ and $\mathscr H_{k'}$ are isomorphic as Hilbert spaces, we should think of them as distinct copies of the same space attached to the distinct points $k,k'\in \Gamma$.
The appropriate machinery for this is that of a fiber bundle. Define the Bloch bundle $E \overset{\pi}{\longrightarrow} \Gamma$, where the total space $E=\sqcup_{k} \mathscr H_k$ is the disjoint union of all of the $\mathscr H_k$'s, and where the projection operator $\pi:(k,\psi_k)\mapsto k$ simply maps an element of $(k,\psi_k), \psi_k\in \mathscr H_k$ down to the point $k\in \Gamma$ to which it is attached. There are two important subbundles of the Bloch bundle, called the conduction and valence bundles. One first decomposes each $\mathscr H_k \simeq \mathscr H_k^+ \oplus \mathscr H_k^-$, with $\mathscr H_k^\pm$ the sub-Hilbert spaces of positive and negative energy, respectively.  Then the conduction and valence bundles are defined as $E^{\pm}\overset{\pi^{\pm}}{\longrightarrow} \Gamma$, with $E^{\pm} = \sqcup_k \mathscr H^{\pm}_k$ and  $\pi^{\pm}$ the corresponding projection maps.
In "ordinary" systems (at least time-reversal symmetric systems in $d\leq 3$ dimensions), the Bloch bundle is trivial - meaning that there exists a continuous global map $u:\Gamma \rightarrow E$ such that $\big(\pi \circ u\big)(k)=k$ (called a (global) section), and that $E$ is globally homeomorphic to $\Gamma \times \mathbb C^2$.  However, it is entirely possible that while the Bloch bundle is trivial, the conduction and valence bands are not; this lack of triviality is a defining characteristic of topological materials.
Concretely, we saw above that the SSH Hamiltonian decomposes into a family of $2\times 2$ matrices
$$H_k = \pmatrix{0&v+we^{-ik}\\v+we^{ik} & 0}$$
with eigenvectors $\phi_{+1,k}$ and $\phi_{-1,k}$.  The maps $u_{\pm}:k\mapsto \phi_{\pm 1,k}$ constitute local sections of the conduction and valence bundles, respectively.  We define the (abelian) Berry connections $A_{\pm}$ and Berry curvatures $F_{\pm}$ to be
$$A_{\pm} := -i\langle u_{\pm} , \mathrm d u_{\pm}\rangle \iff A_{\pm}(k) = -i \langle \phi_{\pm 1,k}, \frac{\partial}{\partial k} \phi_{\pm 1,k}\rangle$$
$$F_\pm := dA_{\pm} \overset{\text{in 1D}}{\longrightarrow} 0$$
where $\mathrm d$ is the exterior derivative and $F_{\pm}=0$ is a trivial consequence of working in 1D which does not extend to higher dimensions.  The principle quantity of interest in the 1D SSH model is the Zak phase
$$ \gamma_- := \int_\Gamma \mathrm dk A_-(k) = \nu \frac{\pi}{2}$$
where $\nu\in\{\pm 1\}$; this $\nu$ is an example of a topological invariant which distinguishes different topological phases. In this case, we find that $\nu = \mathrm{sign}(v-w)$, with $v>w$ called the trivial phase and $v<w$ called the topological phase; one can only pass from one to the other if at some point $v=w$, but at this point the gap closes and the system is no longer insulating. Therefore, as long as local perturbations do not close this gap (and do not destroy the symmetries of the Hamiltonian), the Zak phase cannot change. One can show that this Zak phase is related to the electrical polarization and, if we consider a finite chain for which the infinite model is a bulk approximation, the existence of zero-energy states localized to the edges.
To conclude, a few remarks for future study. Generic models include $N$ bands, where we have included only two. To each section $u_n:\mathbf k\mapsto \phi_{n\mathbf k}$ derived via Bloch's theorem, we can associate an abelian Berry connection and curvature $A_{n} := -i\langle u_n, \mathrm d u_n\rangle$ and $F_n := \mathrm dA_n$. It is also possible to generalize to the non-Abelian case by including multiple bands, so the connection and curvature become matrix-valued.
The sections $u_n$ are only defined up to a phase, so we can always define a different section $u_n'$ such that $u_n'(k) = e^{i\theta_k} u_n(k)$ for some smooth function $\theta_k$.  Implementing such a change sends $A_n \mapsto A_n' = A_n + \mathrm d\theta_k$, which we can interpret as a change of gauge; this formally associates the Bloch/conduction/valence bundles with a $\mathrm U(1)$ principal bundle.
In 2D systems, a crucial topological invariant is the integral of the Berry curvature over the (now 2-dimensional) Brillouin zone; this is called the Chern number. In 3D systems, one can compute the Chern numbers over various special 2-planes in the (now 3-dimensional) Brillouin zone, and non-triviality of the corresponding topological indices are the defining characteristics of topological insulators.
Finally, the interplay between topology and symmetry is very deep. Certain topological invariants only make sense if some corresponding symmetries are respected; breaking of these symmetries leads to the breakdown of the topological protection physical features such as metallic edge states.
References:

*

*J Cayssol and J N Fuchs, Topological and Geometrical Aspects of Band Theory


*MichelFruchart and David Carpentier, An Introduction to Topological Insulators


*For a wonderful resource which avoids the sophisticated language of bundles, see An Online Course in Topology and Condensed Matter


*Similarly, see A Short Course on Topological Insulators
A:  [Note: There are already some nice answers here, but they focus on the case of non-interacting fermions. While that is certainly an interesting case, it is quite a small fraction of the landscape of topological phases of matter, and some of its key features (in particular, the topological space being a vector bundle defined by the energy eigenstates) do not generalize to the more general case. Let me thus offer a different perspective, which I think covers all known cases of topological phases of matter (at least in the sense that this term is conventionally used).]
The defining property of topological phases of matter (be they non-interacting, or symmetry-protected, or intrinsically topologically ordered) is that their universal description only relies on topological information of the spacetime manifold on which they live (that is to say, it does not depend on the metric).
To unpack this, let me first explain what I mean by 'universal'. Technically, this refers to the properties which remain when one performs a renormalization group flow, which gets rid of short-distance physics. Practically, one can think of these as the properties which can be probed at long distances. The reason these universal properties are of interest is as the name suggests: they are independent of any specific realization of the phase. More precisely, two states of matter in the same phase will have the same universal properties. Indeed, suppose one has a path in parameter space where along each point the universal part only relies on topological information (we say that the low-energy theory is described by a topological quantum field theory (TQFT)): this implies that we did not encounter a quantum critical point along this path, since at a quantum critical point we have massless particles, implying a non-trivial stress-energy tensor which couples to the metric and is thus not topological (remember that $T^{\mu,\nu} \propto \frac{\delta S}{\delta g_{\mu,\nu}}$).
As already said, 'topological' means that these universal properties do not depend on the metric. As just discussed, this excludes the case of gapless fields. Conversely, does any gapped theory flow to a TQFT under RG? It is tempting to think so: if local excitations have some nonzero mass, then under RG their mass scale will increase, eventually flowing to infinite-mass. In this RG fixed point limit, no local degrees of freedom (d.o.f.) remain; the only remaining d.o.f. would thus have to be topological (we will discuss examples shortly). For this reason, for a long time, the study of 'topological phases of matter' was seen to be equivalent to the study of gapped (quantum) phases of matter. Now we know this is not the full story: there is something known as fracton order, which are gapped phases of matter which are not topological in the aforementioned sense. The loophole is quite subtle and not yet fully understood: it turns out that the usual notion of RG does not quite apply to these phases, and its resolution is an active field of research. (I will not discuss fractons further in this post, as they go beyond the conventional notion of topological order.)
So what answer does this suggest to your question? We see that, arguably, the 'topological space' relevant to topological phases of matter is spacetime itself. Of course in principle spacetime has a lot more structure than just a topological space, but the point of topological phases of matter---which are described by a TQFT---is that they do not care about any additional structure of spacetime. Note that this answer implies we cannot quite use the 'answer prompt' that you suggested: that prompt implicitly presumed that the topological space in question was encoding the information of the phase of matter. I am referring to your sentence: ''Topological properties of this space, such as ______, can correspond to physical properties''. This is indeed the case for free-fermion phases of matter, where it is possible to think of the vector bundle of energy eigenstates as being a topological space encoding the information of whatever phase you are in, with e.g. the Chern number of that vector bundle encoding physical properties such as Hall currents (see J. Murray's nice answer on this perspective). In a sense, this is a curiosity of free-fermion systems. More generally, we should not consider topological phases as being characterized by a topological space, but rather a topological map. To say it differently: for a topological quantum field theory, the (topological) spacetime manifold is part of the input, not the output. I.e., once you fix your TQFT, it can assign interesting invariants to whatever spacetime manifold you choose to put it on. E.g., for toric code topological order (which is a particular topological phase of matter popularized by A. Kitaev [although many people contributed to its discovery, including X.G. Wen, S. Sachdev and N. Read]), its low-energy theory is a topological map which takes as input an arbitrary closed spacetime manifold $X$ and as its output it gives a stable ground state degeneracy given by $4^g$ where $g$ is the genus of $X$ (this is just an example, ground state degeneracy is not the only quantity to look at).
This way of viewing topological phases of matter as topological maps can indeed be formalized by defining TQFTs as 'functors' (essentially: maps) from the category of cobordisms (essentially: spacetime manifolds with spatial boundaries) to the category of vector spaces (essentially: the Hilbert space of your physical states). I will not delve into this abstract notion here. Instead, I will go through the three main instances of topological phases of matter (i.e., (1) free-fermion topological insulators as explored in the previous posts here, albeit in a different jacket, (2) interacting symmetry-protected topological phases, and (3) intrinsic topological order) and give some examples of the TQFTs that describe them.
Example: topological insulators
A typical example of a topological free-fermion phase of matter is a 2D Chern insulator. Instead of giving an explicit lattice model (see e.g. these notes), I will discuss the effective TQFT that characterizes this phase of matter. To define this, we imagine coupling our system to an external gauge field $A$. Then its low-energy theory is claimed to be described by the following action:
$$ S_\textrm{CS}[A] = \frac{k}{4\pi} \int AdA = \frac{k}{4\pi} \int \varepsilon^{ijk} A_i \partial_j A_k \mathrm d x \mathrm d y \mathrm d t. $$
This action is well-defined on an arbitrary (orientable closed) manifold without having to appeal to a metric; this makes the action topological. This particular topological action is known as the Chern-Simons action. Moreover, one can argue that gauge-invariance implies that $k \in \mathbb Z$ (see e.g. Dunne's notes). This integer is a quantized topological invariant that characterizes this phase of matter.
When we say that 'this action describes our phase of matter', it means that all the salient features of the phase can be deduced from this action. E.g., the current can be obtained as $j^\mu = \frac{\partial S}{\partial A_\mu} = \frac{k}{2\pi} \varepsilon^{\mu \nu \lambda} \partial_\nu A_\lambda$. In particular, this tells us that the current in the $x$-direction is proportional to the electric field in the $y$-direction: i.e., we derive the quantum Hall effect (with the quantization of $k$ implying the quantization of the Hall conductance). Also other physical features, such as chiral edge modes, can be deduced from this action (see the aforementioned notes by Dunne for more on this). The fact that all these physical features are wrapped up in this topological action implies that these are robust features of a topological phase of matter. (In fact, one neat consequence is that it shows that these properties are stable upon adding interactions to the Chern insulator, since this effective action does not explicitly use the non-interacting nature of the underlying system.)
All good and well: a Chern insulator is apparently described by this topological action. But who ordered that? I.e., where does it come from? One practical answer is a phenomenological one, where one simply mutters 'if the shoe fits...'. By this I mean: we have found a topological action which reproduces the physical features which we know that Chern insulators have, hence it must be the correct action. However, this answer is perhaps not very satisfactory. Fortunately, there is a more systematic way to arrive at this action. Firstly, one starts from the underlying fermionic Hamiltonian $\hat H$ (e.g., a particular lattice model realizing a Chern insulator). This defines a partition function:
$$ Z = \textrm{tr} e^{-\beta \hat H}. $$
Using the usual tricks, one can rewrite the fermionic lattice operators in terms of Grassmannian field operators to essentially have something of the form
$$ Z = \int e^{-S[\psi]} \mathcal D[\psi]. $$
Using minimal coupling, one can couple this to an external gauge field. 'External' means that we do not integrate over it in the partition function:
$$ Z[A] = \int e^{-S[\psi,A]} \mathcal D[\psi] .$$
Now imagine explicitly performing the integral over the fermionic degrees of freedom. In the non-interacting case, this can be done explicitly since everything is quadratic in the fermions. One writes the result as
$$ Z[A] = e^{-S_\textrm{eff}[A]}.$$
While doing this integration is doable, it has some subtleties, as discussed extensively and explicitly in this paper by Witten and Yonekura. The end result turns out to be the Chern-Simons action!
What determines the integer $k\in \mathbb Z$ of the resulting Chern-Simons action? In fact, it turns out to be equivalent to the Chern number of the vector bundle of single-particle energy eigenstates in momentum space! This makes a link with the approach discussed in J. Murray's post. This non-trivial equivalence between the level $k$ and the Chern number is discussed pedagogically in Section 2.5 of these lecture notes by Witten.
For more on this field-theoretic perspective on topological insulators, see the seminal paper by Xiao-Liang Qi, Taylor Hughes and Shou-Cheng Zhang. Xiao-Liang Qi also wrote a review chapter on this topic, which you can partially access here on google books (but perhaps/hopefully there is a better source somewhere).
 Disclaimer: Above I was a bit sloppy in referring to the effective Chern-Simons action as giving us a TQFT. It is true that we are looking at a quantum phase of matter, and we started with a partition function where we integrate over quantum-mechanical d.o.f. (the fermionic fields $\psi$). However, at the end we just have an action for an external gauge field $A$---the latter is not integrated over and thus not quantum-mechanical. Hence, in this case, the topological Chern-Simons action actually leads to a classical topological field theory, rather than a quantum-mechanical one. This is a general property of non-interacting fermionic topological phases of matter as well more generally of symmetry-protected topological phases of matter (see next example). These phases do not exhibit intrinsic topological order, which means their effective action is a classical one rather than a quantum-mechanical one (see our last example, namely that of intrinsic topological order, to see a genuine TQFT). 
 Disclaimer of the disclaimer: The current example under discussion---the Chern insulator---is actually a bit of an odd one out in the world of non-interacting topological phases. The reason for this is that unlike other topological insulators, it actually does not require the fine-tuning of symmetries to be stable. (Compare this to e.g. the Su-Schrieffer-Heeger chain discussed in J. Murray's post, which requires sublattice symmetry to have a well-defined topological invariant.) This means that it actually does admit a genuine TQFT description (despite not having genuine topological order); I will briefly come back to this point in the last section/example of this post. For this reason, the Chern insulator lives in the twilight zone between symmetry-protected topological phases and intrinsic topological order. Hence, it is not actually the best representative of the world of topological insulators, but it's just so nice that I could not resist discussing it.

Example: symmetry-protected topological phases of matter
Since my first and last example are quite long, let me attempt to keep this middle example brief. The main point is that topological insulators have a natural extension to the case of interactions, resulting in a concept known as 'symmetry-protected topological (SPT) phases of matter'. Like all topological phases, they can be characterized by an effective topological action describing their universal features.
SPT phases require the presence of symmetries in order to be non-trivial. This means that the fundamental degrees of freedom have some well-defined charges under a symmetry, which in turn implies we (usually) have a well-defined notion of how to couple these particles to an external gauge field for that symmetry. Similar to before, by integrating out the physical degrees of freedom, this then gives an effective partition function or action of the form $Z[A] = e^{iS_\textrm{eff}[A]}$. Here I work in real time, such that action is multiplied by the imaginary unit. Non-trivial SPT phases are characterized by non-trivial phase factors in this partition function.
As a concrete example, we can consider the Haldane phase which is realized in the spin-1 Heisenberg chain and deformations thereof. This is a non-trivial SPT phase in the presence of spin rotation symmetry $SO(3)$. In fact, one does not need the full spin-rotation symmetry: it is sufficient to preserve its $\mathbb Z_2 \times \mathbb Z_2$ subgroup generated by $\pi$-rotations around the three axes. As before, one can couple the spin chain to an external gauge field for these two $\mathbb Z_2$ symmetries and then integrate out the spins (in fact, one can do this calculation exactly for a clever choice of initial Hamiltonian, but I won't include this here since this post is already bursting at the seams). The result is a topological action/partition function $Z[A]$ on a spacetime torus (i.e., we imagine space and time both being a circle). This partition function has the special property that it is negative if one of the $\mathbb Z_2$ gauge fields has a non-trivial flux along the spatial direction and the other $\mathbb Z_2$ flux along the timelike direction. This negative sign serves as a topological invariant which distinguishes it from the trivial phase!
Example: intrinsic topological order
Now for the big kahuna: intrinsic topological order. This is the richest class of the three. In the examples discussed thus far, we had to couple the system to an external gauge field to get a non-trivial topological action. This is related to the fact that topological insulators and SPT phases have no interesting bulk excitations, i.e., there are no emergent anyons with unusual statistics and no emergent gauge fields. In contrast, phases of matter with intrinsic topological order give rise to these exotic features, which also means their low-energy topological action is non-trivial even in the absence of coupling it to external fields.
In the simplest types of topological order, the topological action is again the Chern-Simons action we saw above! (Fortunately recycling is a virtue.) However, it is now a genuine quantum theory: the gauge field is no longer a fixed external classical gauge field, but an 'internal' gauge field which we integrate over in our partition function:
$$ Z = \int e^{-S_\textrm{CS}[a]} \mathcal D[a]. $$
Two natural questions might be: (1) what does this internal gauge field $a$ refer to? And (2) what is the difference/consequence of having it be a dynamical gauge field that we now integrate over?
With regard to the first question: this internal gauge field $a$ refers to an emergent concept. It is seemingly not there in the underlying lattice model but only arises as an effective degree of freedom at low energies. It represents a collective motion, similar to how water waves represent the collective motion of billions of water molecules. One typically does not directly derive this emergent property. Instead: one writes down possible effective TQFTs, one analyzes their properties, and then one matches those properties with known properties of certain lattice models/phases of matter. Once these match, we declare that we have found the effective TQFT describing this phase of matter. In case this sounds haphazard: the universality classes describing intrinsic topological order have been classified and it is known what are the minimal set of properties one needs to check in order to claim one knows all of its universal properties.
Let us now address what the physical consequence is of integrating over our gauge field. It turns out to drastically alter the physics. In particular, for $k>1$ the physics is no longer that of Chern insulators (which is essentially integer quantum Hall physics) but instead that of fractional quantum Hall physics! For instance, the topological action implies the presence of fractionalized quasiparticles. I will not go into that here. Instead, I will show how this topological action implies that there is a topological ground state degeneracy which depends on the genus of the manifold. For concreteness, I will show how to derive that the ground state on a torus is $|k|$. (Here I largely follow the lecture notes by Sachdev.)
The Lagrangian density is
$$ \mathcal L = \frac{k}{4\pi} a \mathrm d a = \frac{k}{4\pi} \left( a_t \partial_x a_y - a_t \partial_y a_x + a_x \partial_y a_t - a_x \partial_t a_y + a_y \partial_t a_x - a_y \partial_x a_t \right). $$
If we are on a closed spatial manifold (or even on a manifold with boundary where we choose the gauge $a_t=0$ at the boundary), then by partial integration we can write
$$ \mathcal L = \frac{k}{2\pi} a_t \underbrace{\left( \partial_x a_y - \partial_y a_x\right)}_{=\textrm{ magnetic field}} + \frac{k}{4\pi}\left( a_y \partial_t a_x - a_x \partial_t a_y \right). $$
Note that the only place where $a_t$ appears is in front of the magnetic field; hence, it functions as a Lagrangian multiplier such that integrating over $a_t$ effectively pins us to gauge fields where the field strength is zero (we say they are flat):
$$ Z = \int e^{-\frac{k}{2\pi} \int a_t \left( \partial_x a_y - \partial_y a_x\right) \mathrm dt \mathrm d x \mathrm dy} \mathcal D[a_t] e^{ -\frac{k}{4\pi} \int \left( a_y \dot a_x - a_x \dot a_y \right) \mathrm d t \mathrm dx \mathrm d y} \mathcal D[a_x,a_y] = \int e^{ -\frac{k}{4\pi} \int\left( a_y \dot a_x - a_x \dot a_y \right) \mathrm d t \mathrm dx \mathrm d y} \mathcal D[a_x,a_y]_\textrm{flat}.$$
We should thus only integrate over gauge fields with zero field strength. Fortunately, we can do this by fixing our gauge such that we only have gauge fields which are constant in space. Indeed, to see this, let $a_x(x,y)$ and $a_y(x,y)$ be a general position-dependent gauge field. Now define the function $\lambda(x,y) = \int_{(0,0)}^{(x,y)} \vec a \cdot \mathrm d \vec r$. Note that the flatness condition of the gauge field implies that this function is well-defined, i.e., it is path-independent. It is then easy to confirm that the gauge transformation $a_i \to a_i - \partial_i \lambda$ (for $i=x,y$) makes the gauge field constant (i.e., in this gauge we have that $\partial_i a_j = 0$). Hence, instead of a path integral for fields defined on space, our partition function is now a simpler object where we have two scalars which depend on time
$$ Z = \int e^{-\frac{k}{4\pi} \int \left( a_y \dot a_x - a_x \dot a_y \right) \mathrm dt \;\times \iint \mathrm d x \mathrm d y} D[a_x(t)] D[a_y(t)] = \int e^{-\frac{k}{4\pi} \int \left( b_y \dot b_x - b_x \dot b_y \right) \mathrm dt } D[b_x(t)] D[b_y(t)],$$
where in the last equation we introduced the rescaled fields $b_i = a_i \times L_i$ for convenience (where $L_{x,y}$ is the length in the two directions of the torus).
Hence, the resulting Lagrangian (after partial integration on a closed manifold) is
$$ L= \frac{k}{2\pi} b_y \dot b_x .$$
Note that the momentum conjugate to $b_x$ is $\Pi_x := \frac{\delta L}{\delta \dot b_x} = \frac{k}{2\pi} b_y$. Hence, the gauge fields $b_x$ and $b_y$ are conjugate with the commutation relation $[\hat b_x,\hat \Pi_x] = i$, i.e.:
$$ [\hat b_x,\hat b_y] = \frac{2\pi i}{k} .$$
Also, observe that the Hamiltonian is trivial: $H = \Pi_x b_x - L = 0$. This is typical of TQFTs, since the theory has no local content (and energy is local). This means that any operator that is well-defined in our theory is a zero-energy operator, in the sense that it maps zero-energy states to zero-energy states. Remarkably, our theory has non-trivial operators: defining the Wilson loop operators that wrap around the torus, i.e. $\hat W_x = e^{i\int_0^{L_x} \hat a_x \mathrm d x} = e^{i \hat b_x}$ and $\hat W_y = e^{i \hat b_y}$, the aforementioned commutation relations imply
$$ \hat W_x \hat W_y = e^{2\pi i/k} \hat W_y \hat W_x . $$
Since these are zero-energy operators that do not commute, it follows that the ground state manifold is degenerate. Indeed, suppose $|\psi_0\rangle$ is a ground state which is an eigenstate of $\hat W_x$, then $|\psi_n \rangle := \hat W_y^n |\psi_0\rangle$ are also ground states for any choice of $n\in \mathbb Z$, but due to the aforementioned commutation relations, the eigenvalues of these states under $\hat W_x$ depend on $n \mod k$ and we thus have $k$ linearly independent ground states on the torus! QED
The easiest way to go beyond this simple Chern-Simons action is by including multiple emergent gauge fields. E.g., $\mathbb Z_2$ topological order (which I mentioned toward the beginning of this post) is described by a mutual Chern-Simons action: $S[a,b] = \frac{1}{2\pi} \left( a \mathrm db + b\mathrm d a\right)$, where we integrate over both $a$ and $b$ in the partition function. More complicated topological orders will require more complicated topological actions to describe them. For those interested in further reading, the early seminal paper by X.G. Wen has an illuminating discussion on these actions.
 Extra tidbit: We see that the case $k=1$ above does not lead to a degeneracy. Indeed, this special case does not have intrinsic topological order. In fact, it is a different description of the Chern insulator we described in our first example! To see this, we first need to include a coupling to an external gauge field: $S[a,A] = \frac{1}{4\pi} a \mathrm d a + \frac{1}{2\pi} a \mathrm d A$. The partition function is then:
$$ Z[A] = \int e^{-S[a,A]} \mathcal D[a]  = \int e^{-\frac{1}{4\pi} \int (a+A) \mathrm d (a+A) + \frac{1}{4\pi} \int A \mathrm d A} \mathcal D[a] = e^{ \frac{1}{4\pi} A \mathrm d A} \times \int e^{-\frac{1}{4\pi} \int b\mathrm d b} \mathcal D[b] $$
where $b = a+A$ is a change of variables. We indeed recover the topological action for the external gauge field $A$, agreeing with what we saw for the Chern insulator with Chern number $k=1$. To get a higher Chern number, we just need to include multiple (decoupled) internal gauge fields, each described by the Chern-Simons TQFT at level $k=1$.
A: The misunderstanding many have is that topology just the study of topological spaces.  It is really also about continuous functions between two topological spaces.
If one has an infinite lattice model, where the Hilbert spaces is more-or-less $\ell^2(\mathbb{Z}^d)$, and if the Hamiltonian is periodic (a huge limitation) then one has momentum space that is naturally a torus $\mathbb{Z}^d$.  This is for non-interacting Fermions.
If the Hamiltonian is gapped at $E_0$ then you get another topological space.  For simplicity, assume $H$ has spectrum only at $0$ and $1$.  Then over in momentum space the Hamiltonian becomes a continuous function from the torus  to the set of all $k$-by-$k$ matrices that are Hermitian and with spectrum only in $0$ and $1$.  (I take $k$ finite, limiting this even more.)  This is a Grassmann manifold.
Thus we naturally find our Hamiltonian leads to a continuous function from a torus to a Grassmann manifold.  In fact, this will be differentiable, and so what is often performed is often calculations using geometry.  Also, many prefer to talk about a vector bundle, which is an equivalent view point.
That is the historical picture.  Many other topological spaces arise in the study of a topological insulator, and often what is needed is more that a normal topological space.  Finally, many more modern calculations involve operator theory or operator algebras, so there is only noncommutative topology where there are not really any topological spaces at all.  These sorts of things are needed to get beyond infinite, perfect models.
A: Edit: this was written before J. Murray's excellent answer

Saying solid state physics isn't my expertise is an understatement, so I'm not sure my answer is correct, but I'll do my best:
According to my understanding, the topological space involved is the electron band structure (see for example this gif from Wikipedia which shows the structure for some materials). A lot of physical processes perturb the electrons such that they don't "hop" between bands, so the topology of these graphs (for example, intersection points between bands) can be a relevant character of the material. Moreover, performing some actions on the material (maybe for example applying a voltage, I'm not so sure..) apparently don't change the shapes of each band but not the topology, which is also useful to acknowledge.
I'd be glad to learn more if anyone can add to my answer.
