One very popular view (as espoused by Max Tegmark) is that (quoting count_to_10) :

> math works because the universe is based on math 

http://www.scientificamerican.com/article/is-the-universe-made-of-math-excerpt/  
https://en.wikipedia.org/wiki/Mathematical_universe_hypothesis
 		
Such a view was common from the time of Pythagoras, through to Kepler and Newton, with attempts to find mystical mathematical patterns in nature, and the description of God as a Geometer.  Galileo wrote in 1623 : "The book of nature is written in the language of mathematics."

*************************

An alternative view which is more "down to Earth" is that mathematics developed from the attempt to describe the world using numbers - not simply counting but also measuring (distance, angle, area, volume, weight, etc).  This is obvious in the case of Geometry (literally, 'land measurement').  Trigonometry also developed for use in surveying, navigation and astronomy (in the latter case for predicting floods or auspicious astrological events). Probability was developed to answer questions about gambling.  Calculus developed from trying to account for the shape of celestial orbits.

More recently, the mathematics of chaos arose from weather prediction, and fractal geometry from the practical question of measuring the length of a coastline.

Throughout most of its history mathematics developed as a tool of science and technology, even in the era of Euler, Lagrange, Gauss and Legendre.  So it should not be surprising that it "works" in physics.  It was not until about 1850 that Pure Mathematics became established as a separate subject.

As Paul T points out, the issue was addressed by Eugene Wigner in a famous essay, "The Unreasonable Effectiveness of Mathematics in the Natural Sciences".
http://www.maths.ed.ac.uk/~aar/papers/wigner.pdf

However, I think this description of "unreasonable effectiveness" clashes with the reality of mathematical physics.  

Take a look inside Landau & Lifschitz or any other graduate text in mathematical physics.  Seeing the horrendous mathematics required to solve many differential equations (Fourier Transforms, Bessel Functions, etc), most of which have no analytical solution anyway, you might then question whether the description of "unreasonable effectiveness" is really appropriate.  Even more so when you realise that this complex mathematics is still **only an approximation** to reality since the differential equations have themselves arisen only after making several simplifying assumptions.

In Quantum Mechanics only the most simple problems can be solved analytically.  Some are resolvable only into transcendental equations (eg finite potential barrier).  Others are tractable only as "perturbations" of known solutions, or in QED require the summing of infinite series of terms.  In some fields special tricks like Renormalization and Regularization are needed to deal with the infinities in the mathematics.

The fact that linear algebra applies quite well in numerous macroscopic situations of interest is due to the facts that (1) many phenomena are approximately linear over the narrow region of interest, and (2) they are only weakly coupled to other processes. Then empirical laws like Hooke's Law and Ohm's Law give sufficiently accurate results without making the calculations too difficult. 

The Law of Large Numbers, which is the basis of statistical mechanics, is also a great help in getting round the difficulties of solving non-linear equations at the molecular level.

Most notably in the case of turbulence, although we can write the Navier-Stokes Equation - which again rests on simplifying assumptions - nobody has yet worked out how to solve it.  But even with a system as simple as the Double Pendulum, we can write an equation of motion but we cannot predict its behaviour.

As dmckee says :   

>Think for a moment about what happens to proposed descriptions of reality whose math doesn't work for describing the system they pertain to. Kirchhoff's laws didn't end up in the texts because the man's name is fun to say. 

When mathematics doesn't provide a solution to a physics problem, it is left out of the textbooks.  Or we simplify until the problem is solvable.  We concentrate on the problems we can solve, and avoid those we can't.  That leaves the impression that mathematics can solve every physics problem.

*********************

So in summary my answer is that :

1. mathematics works in physics because it was developed (in part) for the purpose of describing the world, and
2. it doesn't actually work anywhere near as well as some people make out.