One very popular view (as espoused by Max Tegmark) is that (quoting count_to_10) :
math works because the universe is based on math
http://www.scientificamerican.com/article/is-the-universe-made-of-math-excerpt/
An alternative view which is more "down to Earth" is that mathematics developed from the attempt to describe the world using numbers. This is obvious in the case of Geometry (literally, 'land measurement'). Trigonometry also developed for use in surveying, navigation and astronomy (in the latter case for astrological reasons or eg for predicting floods). Probability was developed to answer questions about gambling. Calculus developed from trying to describe celestial orbits.
As Paul T points out, the issue was addressed by Eugene Wigner in a famous essay, "The Unreasonable Effectiveness of Mathematics in the Natural Sciences". http://www.maths.ed.ac.uk/~aar/papers/wigner.pdf
However, the description "unreasonable effectiveness" clashes with the reality of mathematical physics. Take a look inside Landau and Lifschitz or any other graduate text in mathematical physics. Seeing the horrendous mathematics required to solve many differential equations (Fourier Transforms, Bessel Functions, etc), most of which have no analytical solution anyway, you might then question the description "unreasonable effectiveness" is appropriate. (I do!) Even more so when you realise that this complex mathematics is still only an approximation of reality since the differential equations have themselves arisen only after making several simplifying assumptions.
The fact that linear algebra applies quite well in many topics is due to the fact that many phenomena are approximately linear over the range of interest. Then empirical laws like Hooke's Law and Ohm's Law apply quite well and don't make the calculations too difficult.
So in summary my answer is that :
- mathematics works in physics because it was developed for the purpose of describing the world, and
- it doesn't actually work anywhere near as well as some people make out.