When Atiyah wrote down his axioms for a TQFT, he was inspired by similiar axioms that Segal came up with to describe 2 dimensional CFTs. A good explanation of the physical motivation from the axiomatic viewpoint is given in Segal's lectures (he is talking about axioms for QFTs but you will recognize parts of the axioms for TQFTs), but you can also take a look at Atiyah's original paper. Another nice reference is Baez's Prehistory of n-categorical physics or Witten's ICM address.
Topological Quantum Field theories are indeed examples of Quantum Field Theories. Their common characteristic is roughly, that the "time evolution" does only depend on changes in topology. That corresponds to the axiom $Z(M \times [0,1]) = id_{Z(M)}$. A physicist would phrase this as "the Hamiltonian vanishes".
The reason the functor is usually called $Z$ is because it should remind you of "Zustandssumme" the german term for partition function. When a physicist wants to study a problem in statistical physics or quantum field theory (they are related) he often starts by writing down a partition function (also called functional/Feynman/Pathintegral in this context)
$$Z_M[J] = \int_{C(M)} D\phi\; \exp(-S[\phi] + J\phi)$$
where $C(M)$ is some space of "fields" on a fixed manifold $M$. You can think of the axioms as properties that a reasonable partition function should have. The common language of CFT/QFT/TQFT is the language of those functional integrals.
To understand this from a physical perspective you should at least understand some quantum mechanics. I am not sure what good books there are for mathematicians, but I think there has been a question on mathoverflow about that. Then there is of course the two volume set "Quantum Fields and Strings: A course for mathematicians". The notes from which the books were made can still be found online at the ias website.