TQFT associates a category to a manifold 
Any 3d TQFT (topological-quantum-field-theory) associates a number to a closed oriented 3-manifold, a vector space to a Riemann surface, a category to a circle, and a 2-category to a point.

This is properly well-known in category theory. I learned it from here.
I wish to have someone sharing their physical explanation and their best understanding about this statement.
I suppose the higher dim generalization is what Urs Schreiber describes using codimension surface. I have read the Phys.SE post: about-the-atiyah-segal-axioms-on-TQFT, but would not mind someone starts from the basic. 
 A: There are maybe three different stages to be distinguished and to be understood here:
first: maybe part of the question is why an $n$-dimensional QFT should assign numbers to closed $n$-dimensional manifolds, and vector spaces to closed $(n-1)$-dimensional manifolds. That is what I had replied to in that other discussion linked to above: the vector spaces assigned are just the spaces of quantum states assigned to a spatial hyperslice of spacetime, the numbers assigned to closed $n$-dimensional pieces of spacetimes are the partition functions, and generally the linear maps assigned to $n$-dimensional pieces of spacetime with boundary are the quantum propagators (the correlators, the S-matrix) which propagate the incoming states to the outgoing states.
second: the question is why one would want to refine this assignment ("Atiyah-Segal-type QFT") to something that also assigns data to $(n-k)$-dimensional pieces of spacetime, for all $0 \leq k \leq n$. The answer to this is that this solves what in physics is known as the "problem of covariant quantization". Namely assigning vector spaces of states to spatial hyperslices a priori means breaking the diffeomorphism invariance of the field theory, after all it involves choosing these spatial hyperslices and assigning data to them in a way that is not a prior build up covariantly.
The whole point of "extended TQFT" is to solve this "problem of covariant quantization of field theory" by enforcing that the spaces of quantum states which are assigned to codimeninson-1 spatial hyperslices arise from gluing of local data. It's the locality principle of quantum field theory, by which every global assignment must be reconstructible from gluing of local assignments.
Mathematically this is where higher categories come in: where the ordinary category of vector spaces knows about vector spaces and linear maps between them, hence about the data of spaces of quantum states and of propagators between them, an n-categorical refinement of this would also know how to build spaces of quantum states (which are then promoted from objects to $(n-1)$-morphisms) from local data (namely by composing $(n-1)$-morphisms along $(n-2)$-morphisms).
So in summary: the reason for passing from Atiyah-Segal style QFT which formalizes the assignment of spaces of quantum states to spatial hypersurfaces and of linear quantum propagator maps between them to pieces of spacetime to higher categorical extended QFT is to fully implement the locality principle  of quantum field theory in the axioms. 
The high-point of this axiomaticts is the cobordism theorem which fully clasifies all fully local ("extended") TQFTs in a rigorous fashion.
third: the question then finally is: if an $n$-dimensional fully local (topological) quantum field theory hence is an n-functor $Bord_n \to \mathcal{C}$ from the n-category of cobordisms to some n-category $\mathcal{C}$ which in its two top dimension degrees looks like vector spaces with linear maps between them, then: what should $\mathcal{C}$ be like in lower degrees?
This is actually a question of ongoing investigation. The cobordism theorem itself allows any n-category wth all duals, but many of these will not "look very physical" in fact.
In any case, the point to notice here is that $\mathcal{C}$ is a choice. It may -- but need not -- look like suggested above in the question. This is what it tends to look like for 3d TQFT of Chern-Simons theory type. The strongest theorem to that effect is now probably Douglas & Schommer-Pries & Snyder 13. See there for more.
A: I cannot comment on the category theory part, but the ideas regarding 'numbers' to a 3-manifold and vector spaces to Riemann surfaces comes about naturally in Gromov-Witten theory (see here: http://www.math.harvard.edu/~jbland/ma273x_notes.pdf for a nice introduction).
The heuristic recipe (physical explanation) is as follows: Take a closed symplectic (as one does in TQFT) manifold $\Omega$. Look at maps from Riemann surfaces of genus $g$: $R_{g}$, to a smooth space built from $\Omega$ (Like the Grassmannian $\mathbf{G}$). You can now look at the moduli stack of all such maps (i.e, the collection of pseudo-holomorphic curves $\psi$ from $R_{g}$ to $\mathbf{G}$ satisfying some conditions, call this $\mathcal{M}$). This moduli stack $\mathcal{M}$ admits a class field theory, i.e, has some equivalence relations on it $[Z,\tilde~]$ that tells you when two curves are equivalent. This is essential since Riemann surfaces have some strange behaviours, if you have ever studied branch-cuts and things you will know what I mean. Meaning that, given various $g$ you may have degeneracies between them.
One then studies intersection theory on this moduli stack and counts the number of pseudo-holomorphic curves that obey some relationships modulo the equivalence relation $\tilde~$. The intersection theory itself produces numerical invariants, which are useful to describe the topological nature of the manifold $\Omega$. These ideas are very reminiscent of the ideas of de-Rham cohomology, which study differential forms on the manifold $\Omega$ to better understand its topology. This is a very important concept in TQFT, since one would like to know precisely how 'unique' or otherwise their manifold structure is. Computation of these invariants using integrals over $\mathbf{G}$ can be done in principle.
As an example, in string theory, one can imagine that the strings of different kinds can join together to form various topological structures, which on a larger scale represent the different types of particles we observe. To exactly describe all possible configurations of how these strings join it is necessary to analyse these Gromov-Witten invariants to maintain internal consistency (i.e, you cannot have a particle which has two distinct ground state energy levels).
Not sure if that is helpful at all, but I think the study of Gromov-Witten invariants (or Donaldson-Thomas, etc) is what you are after here (see here: http://ncatlab.org/nlab/show/Gromov-Witten+invariants as well).
A: I expect this is somewhat of an axiomatic statement, since the TQFT axioms include the association of a Riemann surface $\Sigma$ to a vector space (or a module) $Z(\Sigma)$, and an element $Z(M)\in Z(\partial M)$ to a manifold $M$. They do not include direct reference to categories. It sounds like the categories are natural extensions to the lower dimensions ($d=0$ and $d=1$).
