What are Connections in physics? This question arises from a personal misunderstanding about a conversation with a friend of mine. He asked me a question about the "truly nature" of spinors, i.e., he asked a question to me about what is an spinor object. After a few lines of dialogue, he asked something quite alien to me: 

"So, spinors are Levi-Civita connections?"

The relationship between a mathematical object which models physical entities in field theory (a Dirac spinor for example) and a purely mathematical entity like a Levi-Civita connection, still intrigues me. 
Now, today I encountered this question here:
Under what representation do the Christoffel symbols transform?
and in the second answer, the user made another relationship between field theory and connections: 

"The "Christoffel symbols" are now just the components of a principal connection on that bundle, where a "connection form" is better known to physicists as a gauge field"

I'm asking this question because, from the point of view of elementary general relativity, we are taught that we need a pseudo-riemanninan manifold and a (Levi-Civita) connection to, roughly speaking, make a well-defined notion of derivative of tensor fields. From this point of view a connection is nothing more than a linear map. 
So, what are Connections in physics, INDEED?
 A: Connections in physics "are" the same as they are in mathematics but are usually interpreted as field potentials, with the exception of GR.
The interpretation follows naturally from the concept of a covariant derivative: local transformations of the field being studied must not change the physics involved (i.e. the Lagrangian must be invariant) so one introduces another "gauge" field which has dynamics of its own to cancel changes from the matter field's transformation.
Take the case of quantum electrodynamics: The Lagrangian (density) is
$$\mathcal{L}=\overline{\psi}\left(i\gamma^\mu\partial_\mu + m\right)\psi$$
You can check that it is invariant under the transformation $\psi\to e^{i\lambda}\psi$ when $\lambda$ is a constant. To make the transformation local we "promote" $\lambda$ to a function, but now we have an offending term $\overline{\psi}\gamma^\mu\partial_\mu\lambda e^{i\lambda}\psi$! All is well if we introduce the covariant derivative 
$\mathcal{D}_\mu=\partial_\mu-ie_0A_\mu$ such that $A_\mu\to A_\mu +\partial_\mu \lambda$ is the corresponding transformation.
The full Lagrangian is then 
$$\mathcal{L}=\overline{\psi}\left(i\gamma^\mu\partial_\mu + m\right)\psi + e_0\overline{\psi}\gamma^\mu A_\mu \psi + \frac{1}{4}F^{\mu\nu}F_{\mu\nu}$$
where $F_{\mu\nu}$ is the electromagnetic field strength tensor introduced to account for the dynamics of the potential (photon) field $A_\mu$
Connections in the context of relativity are instead the gravitational field strength since in our currently accepted theory gravity is not a "gauge field" like the photon field. The identification of gravitation with spacetime curvature makes particles travel according to the geodesic equation which can recover the usual Gauss's Law for gravity in the Newtonian limit.
A: Here is a physical way I think of gauge transformations. We are very interested in moving objects around in space-time.  Translations in space-time (along with rotations, and boosts) clearly obey the axioms of a Lie group (eg: Poincare group). The operator $p^{\mu}$ is the Lie group generator of space-time translations, and $[p^i,p^j]=0$ for the Poincare group. This predicts that if we translate (only translate ... no rotation, boosts, or strains) an object around a loop, the resulting object will be identical to the one we started with.  Unfortunately, that's not what happens with real physical objects.
When in the presence of other charge particles, we translate a charge around a loop, a U(1) transformation Q is done, and the quantum mechanical phase of the resulting object changes.
When in the presence of other weakly interacting particles, we translate an object around a loop, transformations $SU(2)_{Weak}$   are done, and the object is rotated into other weak isospin states.
When in the presence of other strongly interacting particle, we translate an object around a loop, transformations $SU(3)_{Color}$ are done, and the object is rotated into other color states.
When in the presence of other mass which "curves space-time", we translate an  object around a loop, transformations GL(4) are done, and the resulting object is rotated,  boosted, or strained with respect to the original object (even though we carefully did no rotations, boosts, or strains along the way!).
So, we introduce gauge transformations to "patch up" the translation generators $p^{\mu}$, and a small translation now becomes:
$$ (1+dx^{\mu}p_{\mu}) \rightarrow (1+dx^{\mu} P_{\mu} )$$
$$ P_{\mu}=p_{\mu} + A_{\mu}Q + B_{\mu k}I^k + C_{\mu n}\lambda^n + \Gamma_{\mu \beta}^{\alpha}J^{\beta}_{\alpha} $$
Where  $A, B, C, \Gamma$ are the electromagnetic, weak, strong, and affine connection (=Christoffel symbol) “gauge fields”(or also can be called connection coefficients), and $Q,I,\lambda,J$ are the U(1), $SU(2)_{Weak}$, $SU(3)_{Color}$, and GL(4) gauge group generators.   I left out a coupling constant in each of the above terms so as not to confuse things. Thus it is said the Standard Model of particle physics is U(1) X $SU(2)_{Weak}$ X $SU(3)_{Color}$ … X GL(4),  where I have tacked GL(4) on the end for the people who think of General Relativity as a gauge theory.
Usually this story is presented as patching up the partial derivative $\partial^{\mu}$ to become the covariant derivative $D^{\mu}$.  These derivatives are just specific representations of the operator $p^{\mu}$ and $P^{\mu}$ when they are operating on continuous functions.
A: I think of "connections in physics" as referring to a cluster of concepts - parallel transport, covariant derivative, connection one-forms, geodesics, Christoffel symbols, gauge fields - which all exist to solve the same basic problem, which can be stated as follows.
In physics, our models come in the form of differential equations applied to fields on manifolds. So we want to express how a field changes from point to point in the manifold. The problem is that the spaces in which the objects live are independent from point to point.
Take the example of tangent vectors. We want to know how much a tangent vector field changes as we move from x to y. This means we must somehow compare these two vectors. But the tangent vector at x lives in a totally different vector space than the tangent vector at y. How does one compare vectors that don't even live in the same vector space? There just is no built-in way of comparing two objects that don't live in the same space.
So to compare the vector at y to the vector at x (to see how much it changed for our equation of motion), we need some way of specifying which vector at y counts as being 'the same as' the vector at x. A way of 'connecting' the two spaces, so to speak. Then we can compare this 'same' vector to the vector at y and say that the difference between them is the change we were looking for.
This is the concept underlying parallel transport. Intuitively, it seems the way we can compare a vector at x to a vector at y is to just slide the vector at x to y, keeping it parallel to itself the whole time. That way we know that we are really comparing the vector at y to the vector as it was at x.
However, our intuitions about parallelism are deceptive. (Think about how it would work with tangent vectors to a sphere.) So we end up defining parallel transport as transport in which the covariant derivative of the vector along the transport path is zero. But the covariant derivative itself requires a connection - i.e. a specification of which vector at y counts as 'the same as' which vector at x - otherwise how could you take the derivative to find that it is zero and thus that you parallel transported the vector?
So how to think about the covariant derivative? Again, start with the fact that the vector spaces (this applies to more general objects, I'm just using vectors for definiteness) at different points of the manifold (e.g. spacetime) are independent of each other. In particular, they can be coordinatized differently, i.e. they can be described using different basis sets. (This is called 'moving frames' in some contexts.)
Again, this poses a problem when we want to know whether a vector field has changed as we move from point x to point y. If we are to have freedom to choose a different basis for each point, that means we now have two reasons why we might detect a change in the components of a vector from point x to point y:
a) the vector may have actually changed as we moved from x to y, whether the basis changed or not
b) the basis we're using to describe the vector numerically may have changed from x to y, whether the vector changed or not
Obviously, we're only interested in the actual changes to the vector itself. The other changes are spurious artifacts of a coordinate choice that is arbitrary for purposes of doing physics. Therefore, we need to build into our differential equations a way of 'subtracting out' any change in the basis, so any change in the components that appears in our equation is assured to arise from genuine change in the vector itself.
What does the job of this 'subtracting out' is a correction term that is appended to the derivative operator. It is this correction term that we refer to in physics as 'the gauge field'. In GR, these are the Christoffel symbols. The EM potential field is also a gauge field in this sense.
To put a somewhat finer point on it, the Christoffel symbols, for example, are the components of the changes in the basis vectors as we move from one point to another. They have 3 indices: one index tracks which component that symbol represents, one tracks which basis vector you're measuring the change in, and one index tracks which basis vector's direction you're moving along. So the ijk-th Christoffel symbol means "the ith component of the change in the jth basis vector as you move in the direction of the kth basis vector."
Those are the basic ideas, though of course there's more to say about the curvature of the connection and the field strength, etc. But the rest is really just elaborating on this essentially simple idea in terms of differential geometry of Lie groups and fiber bundles.
For future hikers in these woods, here are some references I've found helpful in grasping connections and gauge fields:
Moriyasu, Elementary Primer on Gauge Theory
Fecko, Differential Geometry and Lie Groups for Physicists
Baez, Gauge Fields, Knots, and Gravity
Gron, Einstein's Theory for the Mathematically Untrained (the chapter on Christoffel symbols)
Schuller, Geometric Anatomy of Theoretical Physics Videos, Lecture Notes
Lam, Kai S. Fundamental Principles of Classical Mechanics, Non-Relativistic Quantum Theory, Topics in Contemporary Mathematical Physics
Schwichtenberg, Physics From Finance
Cheng, Einstein's Physics, Relativity, Gravitation, and Cosmology
Healey, Gauging What's Real
Viallet, The Geometric Setting of Gauge Fields of the Yang-Mills Type
Fre, Gravity: A Geometrical Course, Vol 1
