Why are topological properties described by surface terms? An example are the anomalies in abelian and non-abelian gauge quantum field theories. 
For example, the abelian anomaly is $\tilde {F}_{\mu\nu}F^{\mu\nu}$ and the integral over this quantity is a topological invariant which measures a topological characteristic of the gauge field $A_\mu$.
All such quantities can be rewritten as total derivatives and then, using Gauss' law transformed into a surface integral. 
What is the intuitive reason that quantities which describe topological properties can always be written as surface integrals?
 A: I am giving you an answer from the path integral quantization point of view.  
When we quantize a gauge theory, we need to sum over all the configurations of connections of a gauge group on some manifold modulo gauge transformation. 
In contrast to the affine space of all  connections (without dividing by gauge transformation), the latter space (of connections modulo gauge transformation) can disconnected corresponding to different sectors of bundles which cannot be deformed into each other by any combination of diffeomorphism of the base manifold or continuous deformations of the transition functions of the fiber. 
In quantum theory, we absolutely need to sum over all topological sectors in the path integral. For example if we do not do that in the problem of a particle moving on a circle, we do not get the correct answer given by Schrödinger's equation.
Chern and Weil (please see the following expositionby Fecko) discovered a profound theorem that there exist topological invariants differentiating between the principal or vector bundles, with the same structure group, which can be expressed by means of certain polynomials of the curvature of the connection. These topological invariants do not depend on the connection (they are gauge invariants) nor on the field strengths but only on the bundle. Moreover, these topological invariants -named characteristic classes- can be expressed by means of cohomology classes of the base manifold (this is why they are closed). 
Thus, we can use these invariants to weigh different bundles in the path integral because they do not depend on the connections nor the fields but only on the bundles.
There are other topological invariants which cannot be written in terms of differential forms, such as the Stiefel–Whitney classes, on which the existence of fermions on the manifold depends. These invariants also affect the path integral, however, more advanced techniques are needed to take them into account. 
It is worthwhile to mention that not every closed form on the base manifold is an image of a characteristic class (or can be written as a combination of characteristic class). Thus, as topological terms, we can consider only special types of closed forms.
A: Pretty much every time a physicist says "topological invariant", they mean a topological invariant of a vector bundle (the vector bundle the fields take values in, usually), the most common of which are the Chern classes. 
The Chern classes can be expressed as integrals of polynomials in the curvature (it does not matter which curvature, so even if the physical theory doesn't contain a gauge field, you can simply choose/construct one ad hoc) $\int F\wedge F$, and it so happens that these polynomials in the curvature are locally total derivatives of their associated Chern-Simons forms. So, at the level of rigor of physics where one now usually disregards the "locally", the integrals of the polynomials in the curvature which yield the Chern classes are integrals of total derivatives, hence "boundary terms". The deeper mathematical story of why the curvature polynomials that represent integral cohomology classes of a bundle in DeRham cohomology should be such total derivatives is the story of secondary characteristic classes and differential cohomology.
It should be stressed, however, that once we try to be bit more rigorous than the average physicist, the Chern classes are not "surface terms". In fact, that terminology doesn't make any sense because they're invariants of vector bundles over ordinary manifolds, and ordinary manifolds do not have a boundary - any "surface terms" simply vanish on a compact manifold, and are potentially ill-defined on non-compact ones. What's really happening is that, once again, the physicist hides the global property of a bundle on something like compactified spacetime $S^4$ by just looking at it in one of the coordinate patches, pushing out all the structure of the second necessary patch "to infinity", i.e. "to the surface", precisely as in this answer of mine to your question about large gauge transformations.
Additionally, "surface term" usually carries some sort of connotation with it that the choice of function on the surface still matters - but it does not, the Chern class is independent of the choice of connection, as a true topological invariant should be, it is solely a function of the topological homeomorphism class of the bundle. Using some sort of gauge field/curvature to compute the topological invariant is merely a crutch because it's often easier than "purer" topological computations, especially for physicists who don't know such topology.
A: This is not a full answer but I'll give it in order to contemplate some aspects not covered in the previous (good) answers and explicitly mentioned in the OP's comment:

I'm asking, why a topological quantity, such as the ""knottedness of
  vortex lines in the flow" or the "winding of the gauge functions" is
  completely determined by a surface integral, i.e. completely encoded
  in the boundary of the system

In this case, the topological invariants you are referring to are related to homotopy classes. Specifically, in the case of gauge theories, we seek for finite energy (or tension in the case of a vortex line) solutions and this imposes some constraints on the asymptotic fields. Each (non negative) term in the hamiltonian density $\mathcal H$ has to vanish sufficiently fast as the fields approach the spatial infinity. In particular, the potential $V$ has to vanish asymptotically. This means that the scalar field belongs to the vacuum manifold when $r\rightarrow\infty$. This asymptotic field provides a map from the spatial infinity (which depends on the spatial dimension of the model) to the vacuum manifold and different configurations (different maps) are classified into equivalent classes according to homotopy groups. Maps belonging to different classes cannot be continuously deformed into each other and that is why non trivial maps (non trivial winding number for example) are said to be topologically stable or protected. As you can see, the topological invariants (namely the homotopy classes) in these models are encoded in the asymptotic fields or in the boundary of the system. 
