Can all of physics be described by simple math? Recently I was browsing through A Dynamical Theory of electromagnetic field by Maxwell and wondered because the paper did not seem to include any vector calculus or any vectors. I thought of the famous paper by Einstein On the electrodynamics of moving bodies and couldn't find any vector calculus either. Instead he writes out every component separately. Although the papers didn't include more (sophisticated) math they were able to describe the physical phenomena very well.
This led me to the question if not more physical problems could be reduced to simpler math (for example just calculus) but still be able to describe the physics in the same way. 
So would one be able to describe whole physical fields such as Quantum Mechanics without Linear Algebra or Relativity without Tensor algebra?
Or maybe you know any examples where the further mathematics is essential for the correct description of physics?
 A: I was thinking about answering or not this one. I am not sure if I should... But well, I'll give a try.
Well... The whole math is "backtracable" to more primitive structures, because all the more advanced structures is defined based on the simpler one. 
For instance: you can write an derivative using differential calculus. But you can also write a derivative using limits, because the definition of a derivative is a limit. You can also write a limit as a norm, because definition of a limit is based on norm (Note that we are writing a derivative not even using differential calculus). But it won't stop... what is a norm? Its definition is based on a function... What is a function? Its a ordered triple of two sets and a map. Ordered triple is a ordered pair of a number and a ordered pair. Ordered pair is defined as a set. Now one can write a derivative using maps (set operation) and sets (Very primitive isn't it?). Here, behold the derivative of a function $f'(x_0) = L$ using norms:
$$
\forall\epsilon>0, \quad\exists\delta>0, \quad\mbox{st}\quad 
0<\left|x - x_0\right|<\delta\quad\Longrightarrow\quad
\left|\frac{f(x) - f(x_0)}{x-x_0} - L\right|<\epsilon
$$
So.. this is a derivative.. not even using differential calculus. But then, what is simpler? $\frac{df}{dx}$? or that? Which one is more readable? Well.. I tend to think differential calculus is simpler, instead of this norm stuff: Try to calculate the derivative of $f(x) = x$ using calculus, and using this norm stuff. =).
Same way, since the whole math is based on set theory (not sure about that, I am not a mathematician), then probably you can reduce everything from math and eventually from physics into more primitive stuffs, and keep doing until arrive in sets and set operations. But it wouldn't be readable. And would be big. That's where comes modern notation. Its not to overcomplicate, its to simplify.
A: Writing equations in vector rather than component form makes thinking and writing easier, but as you surmise doesn't lead to new physics.  However, some "more complicated math" really does explain new phenomena.
For example, are calculus and Schrodinger's differential equation sufficient for explaining the energy levels of the hydrogen atom?  Separating the equation into a radial and angular part, we find the energies are specified by a radial integer n=1,2,... , and an angular momentum integer $l=0,1,2,...n-1$ (and its z component $l_z=-l,-l+1,...,0,...l$).  But the hydrogen atom actually has more energy levels which the half-integer spins of the electron and proton are needed to explain.  You need a new mathematical construct, the Lie Group SU(2), to explain the existence of half-integer spin angular momenta... calculus by itself was not sufficient.  The Dirac equation needs the Pauli spin matrices satisfying the algebra of SU(2).
While perhaps it is true that a mathematician may formally build up to Lie Groups from set theory and numbers, new ideas were introduced along the way, and some have been useful to physics.
