For gravity, we have General Relativity, which is a geometric theory for gravitation.
Is there a similar analog for Electromagnetism?
1. Kaluza-Klein theory. This is similar to General Relativity, but instead of three space dimensions plus time, there are four space dimensions plus time. The fourth dimension is cyclic, and satisfies some symmetry conditions. The electromagnetic potential appears as the components of the metric in the fourth space dimension. It is usually rejected on the grounds that we can't see the fourth space dimension, or that it is made too small to be seen. In fact, the symmetry conditions along this dimension make it indistinguishable, and moving along it is equivalent to a gauge transformation. So, this is the only evidence predicted by the theory, no matter how large we make the cyclic dimension. Which leads us to
2. Gauge theory. As mentioned by DImension10 Abhimanyu PS, electromagnetism can be described by a gauge theory whose gauge group is $U(1)$; the electromagnetic potential becomes a connection, and the electromagnetic field the curvature associated to the connection. It is in fact the symmetry group of the fourth dimension in Kaluza-Klein theory. For mathematicians, a gauge theory is described in terms of principal bundles, which, if the gauge group is $U(1)$, are in fact 4+1 dimensional spaces, satisfying symmetry conditions like in the Kaluza-Klein theory. So, mathematically, they are equivalent, although there are variations of the Kaluza-Klein theory which cannot be described by a standard gauge theory.
3. Rainich-Misner-Wheeler theory. There is a way to obtain electromagnetism from geometry, in the 4d spacetime of General Relativity. Rainich was able to give in 1925 necessary and suficient conditions that spacetime is curved in a way which corresponds to the electromagnetic field. By Einstein's equation, the spacetime curvature is related to the field. So, Rainich decided to see if one can obtain the electromagnetic field from the curvature, using Einstein's equation. He found some necessary and sufficient conditions for the Ricci tensor, which are of algebraic and differential nature. This works for source free electromagnetism. There is an ambiguity, given by the Hodge duality between the electric and the magnetic fields, for the source free Maxwell equations. So, basically, the field is recovered up to a phase factor called complexion. The idea was rediscovered by Misner and Wheeler three decades later, who combined it with the wormholes of Einstein and Rosen. They interpreted the ends of the wormholes as pairs of electrically charged particles-antiparticles. The electromagnetic field, in this view, doesn't need a source, since the field lines go through the wormhole. While this idea may seem bizarre, it allowed to obtain "charge without charge", and to fix the undetermined phase factor. This model of particles had some issues, for instance it couldn't explain the spin, and Misner and Wheeler abandoned it.
The geometric interpretation comes from the similarity of the covariant derivatives between general relativity and quantum field theory -- $\nabla_\mu -\partial_\mu$ in general relativity gives you the Christoffel symbol (although it contracts in an actual operation), which is the gravitational field strength. Similarly in QED, $\nabla_\mu -\partial_\mu = ig_sA_\mu$, which measures the electromagnetic field strength.
This gives rise to the geometric interpretation of electromagnetism in which it is the curvature on the $U(1)$ bundle (and in general for a gauge theory, the gauge force is a curvature on its corresponding gauge group's bundle).