One very popular view (as espoused by Max Tegmark) is that (quoting count_to_10) :
math works because the universe is based on math
http://www.scientificamerican.com/article/is-the-universe-made-of-math-excerpt/
https://en.wikipedia.org/wiki/Mathematical_universe_hypothesis
Such a view was common from the time of Pythagoras, through to Kepler and Newton, with attempts to find mystical mathematical patterns in nature, and the description of God as a Geometer.
An alternative view which is more "down to Earth" is that mathematics developed from the attempt to describe the world using numbers - not simply counting but also measuring (distance, angle, area, volume, weight, etc). This is obvious in the case of Geometry (literally, 'land measurement'). Trigonometry also developed for use in surveying, navigation and astronomy (in the latter case for predicting floods or auspicious astrological events). Probability was developed to answer questions about gambling. Calculus developed from trying to account for the shape of celestial orbits.
More recently, the mathematics of chaos arose from weather prediction, and fractal geometry from the practical question of measuring the length of a coastline.
Mathematics developed alongside and in part to serve science and technology, so it should not be surprising that it "works" in physics.
As Paul T points out, the issue was addressed by Eugene Wigner in a famous essay, "The Unreasonable Effectiveness of Mathematics in the Natural Sciences". http://www.maths.ed.ac.uk/~aar/papers/wigner.pdf
However, I think this description of "unreasonable effectiveness" clashes with the reality of mathematical physics.
Take a look inside Landau & Lifschitz or any other graduate text in mathematical physics. Seeing the horrendous mathematics required to solve many differential equations (Fourier Transforms, Bessel Functions, etc), most of which have no analytical solution anyway, you might then question whether the description of "unreasonable effectiveness" is really appropriate. Even more so when you realise that this complex mathematics is still only an approximation to reality since the differential equations have themselves arisen only after making several simplifying assumptions.
In Quantum Mechanics only the most trivial problems can be solved analytically. Other problems are tractable only as "perturbations" of known solutions, or require summing infinite series of terms. In some fields special tricks like Renormalization and Regularization are needed to avoid the infinities in the mathematics.
The fact that linear algebra applies quite well in many macroscopic situations is due to the fact that many phenomena are approximately linear over the range of interest. Then empirical laws like Hooke's Law and Ohm's Law give sufficiently accurate results without making the calculations too difficult. The Law of Large Numbers, which is the basis of statistical mechanics, is also a great help in getting round the unpredictability at a molecular level.
In some cases, notably turbulence, although we can write the Navier-Stokes Equation - which again rests on simplifying assumptions - nobody has yet worked out how to solve it.
As dmckee says :
Think for a moment about what happens to proposed descriptions of reality whose math doesn't work for describing the system they pertain to. Kirchhoff's laws didn't end up in the texts because the man's name is fun to say.
When mathematics doesn't provide a solution to a physics problem, it is left out of the textbooks. Or we simplify until the problem is solvable. We concentrate on problems we can solve, and avoid those we can't. That leaves the impression that mathematics can solve every physics problem.
So in summary my answer is that :
- mathematics works in physics because it was developed for the purpose of describing the world, and
- it doesn't actually work anywhere near as well as some people make out.