This is explained in Section 3 of Witten's "Quantum Field Theory and the Jones Polynomial." The idea is to locally parametrize a three-manifold by $M\times \mathbf{R}$, where $M$ is some two-dimensional manifold and $\mathbf{R}$ is the time direction that we are quantizing along. Once we do this, we can fix temporal gauge, where the time component $A_0$ of the gauge field vanishes. In this gauge, the Gauss's law constraint implies that the spatial components of the field strength vanish, which in turn says that the gauge connection is flat and the only degrees of freedom are topological.

My general feeling on Chern-Simons theory, from the limited amount that I know about it, is that most confusions that one might have are addressed in Witten's paper (as long as it's not about the relatively new field of Chern-Simons-matter.)