I think of "connections in physics" as referring to a cluster of concepts - parallel transport, covariant derivative, connection one-forms, geodesics, Christoffel symbols, gauge fields - which all exist to solve the same basic problem, which can be stated as follows.
In physics, our models come in the form of differential equations applied to fields on manifolds. So we want to express how a field changes from point to point in the manifold. The problem is that the spaces in which the objects live are independent from point to point.
Take the example of tangent vectors. We want to know how much a tangent vector field changes as we move from x to y. This means we must somehow compare these two vectors. But the tangent vector at x lives in a totally different vector space than the tangent vector at y. How does one compare vectors that don't even live in the same vector space? There just is no built-in way of comparing two objects that don't live in the same space.
So to compare the vector at y to the vector at x (to see how much it changed for our equation of motion), we need some way of specifying which vector at y counts as being 'the same as' the vector at x. A way of 'connecting' the two spaces, so to speak. Then we can compare this 'same' vector to the vector at y and say that the difference between them is the change we were looking for.
This is the concept underlying parallel transport. Intuitively, it seems the way we can compare a vector at x to a vector at y is to just slide the vector at x to y, keeping it parallel to itself the whole time. That way we know that we are really comparing the vector at y to the vector as it was at x.
However, our intuitions about parallelism are deceptive. (Think about how it would work with tangent vectors to a sphere.) So we end up defining parallel transport as transport in which the covariant derivative of the vector along the transport path is zero. But the covariant derivative itself requires a connection - i.e. a specification of which vector at y counts as 'the same as' which vector at x - otherwise how could you take the derivative to find that it is zero and thus that you parallel transported the vector?
So how to think about the covariant derivative? Again, start with the fact that the vector spaces (this applies to more general objects, I'm just using vectors for definiteness) at different points of the manifold (e.g. spacetime) are independent of each other. In particular, they can be coordinatized differently, i.e. they can be described using different basis sets. (This is called 'moving frames' in some contexts.)
Again, this poses a problem when we want to know whether a vector field has changed as we move from point x to point y. If we are to have freedom to choose a different basis for each point, that means we now have two reasons why we might detect a change in the components of a vector from point x to point y:
a) the vector may have actually changed as we moved from x to y, whether the basis changed or not
b) the basis we're using to describe the vector numerically may have changed from x to y, whether the vector changed or not
Obviously, we're only interested in the actual changes to the vector itself. The other changes are spurious artifacts of a coordinate choice that is arbitrary for purposes of doing physics. Therefore, we need to build into our differential equations a way of 'subtracting out' any change in the basis, so any change in the components that appears in our equation is assured to arise from genuine change in the vector itself.
What does the job of this 'subtracting out' is a correction term that is appended to the derivative operator. It is this correction term that we refer to in physics as 'the gauge field'. In GR, these are the Christoffel symbols. The EM potential field is also a gauge field in this sense.
To put a somewhat finer point on it, the Christoffel symbols, for example, are the components of the changes in the basis vectors as we move from one point to another. They have 3 indices: one index tracks which component that symbol represents, one tracks which basis vector you're measuring the change in, and one index tracks which basis vector's direction you're moving along. So the ijk-th Christoffel symbol means "the ith component of the change in the jth basis vector as you move in the direction of the kth basis vector."
Those are the basic ideas, though of course there's more to say about the curvature of the connection and the field strength, etc. But the rest is really just elaborating on this essentially simple idea in terms of differential geometry of Lie groups and fiber bundles.
For future hikers in these woods, here are some references I've found helpful in grasping connections and gauge fields:
Moriyasu, Elementary Primer on Gauge Theory
Fecko, Differential Geometry and Lie Groups for Physicists
Baez, Gauge Fields, Knots, and Gravity
Gron, Einstein's Theory for the Mathematically Untrained (the chapter on Christoffel symbols)
Schuller, Geometric Anatomy of Theoretical Physics Videos, Lecture Notes
Lam, Kai S. Fundamental Principles of Classical Mechanics, Non-Relativistic Quantum Theory, Topics in Contemporary Mathematical Physics
Schwichtenberg, Physics From Finance
Cheng, Einstein's Physics, Relativity, Gravitation, and Cosmology
Healey, Gauging What's Real
Viallet, The Geometric Setting of Gauge Fields of the Yang-Mills Type
Fre, Gravity: A Geometrical Course, Vol 1