Hold on there now: don't be too hasteful in declaring TQFT's some artificial construct only benificial for abstract mathematics. This is simply not true, as there are examples of physical systems exhibiting behavior which is encaptured by the phyics of a TQFT!
First things firsts: it's true that the model which physicists use for elementary particle physics does not use any topological field theories. This is the standard model, and it is based on a large construct of 4D QF Theories, such as QED, Yang-Mills theory, the Higgs boson, Dirac Field theory and so on. You can obtain properties of single particles, or the scattering amplitidues of a bunch of particles, etc.
However, in condensed matter physics, such as metals, insulators, cold atom gases, lattice systems, you deal with not one or a few particles, but with many, many (>10^10 --- 10^26) particles. Due to the impossibility of even attempting to write something that looks like a solution of the theory, starting from first principles, is simply impossible and highly ineffecient. Instead, one tries to come up with an effective model which encompasses the essence of the many-body problem. This is standard practice in condensed matter physics and frequently leads to an effective quantum field theory, which has little resemblence with the basic interactions of the original constituents (try to apply QED to an ionic lattice with electrons flowing around. Can you derive the electronic transport properties of such a system? The answer is: no, it's far too complex).
What does this elaboration have anything to do with topological quantum field theories? The reason is simple: some systems are effectively described by a TQFT! Namely, the fractional quantum Hall effect. This effect arises as follows: two types of material are grown on top of each other. An interface between the two layers arises, and this interface "confines" the motion electrons. The electrons can only move in two out of three spatial dimensions. They are effectively living in a two-dimensional world.
Without explaning what the quantum Hall effect actually is, I will instead tell you that the effective field theory associated with this effect is a Chern-Simons theory -- an example of a TQFT.
Other examples exist as well. Topological insulators are likely to be connected to TQFT's as well (BF theories). I'm sure other examples follow.
The various links with CFT, quantum groups, knot theory, etc all have left footprints in this field. I'm sure there is plenty of "translation" of mathematical results to the physical implications left undone. The "invariants" you are talking about pop up as very physical observables. For instance, it's known since the 80's that the so-called conductivity of the fractional quantum Hall effect is nothing more but a certain Chern number.