Can you derive new theories in physics by only looking at the math? Can you develop new theories in physics by only looking at the maths from past physics concepts? Or does observation of physical reality or converging of physical concepts ALWAYS come first before the math, and that the math’s just there to further explain the theory?
 A: This is about theoretical physics and experimental physics working together.
The purpose of physics is to create a set of laws that allow us to predict the behavior of physical systems. 


*

*New, untested theories (i.e. maths) are hypothesized based on previous experiments.

*These theories are used to predict behaviors in untested regimes (e.g. higher energies, larger velocities, smaller distances). 

*Experiments validate or invalidate those theories. 

*New theories are hypothesized from these new observations.

*Rinse, repeat


A great example of the necessity of experiments is the Michelson-Morley experiment which disproved the existence of an aether and eventually led to Einstein's theory of special relativity. Before that experiment, no one knew that light traveled at the same speed in all frames.
A great example of the necessity of theory is the Dirac equation which predicted the existence of antimatter from a simple equation. Before this, antimatter was never considered, but it was later experimentally confirmed. 
A: It depends on what you mean by a new theory. If you mean a complete, self-contained and logical set of axioms/postulates and their theorems/predictions, then it is not possible. To build a physical theory one has to start with a set of laws. How would you enunciate physical laws without physical intuition? That is impossible.
If you mean a further knowledge and explanation of some phenomena based in a pre existent or larger theory, then it is possible that the math plays a sole role. 
I recall three examples now: 


*

*The Dirac's antiparticles prediction based on mathematical rigour.

*The Pontecorvo's neutrino oscillation which grew up as a didactic exercise on linear algebra and quantum mechanics and provided an answer to a problem that did not exist at that time, the missing neutrinos problem.

*The General Relativity's idea that the gravitational interaction comes from a curvature of spacetime.


Note however that it is not true that the General Relativity Theory is a completely mathematically based theory. It is indeed based on principles of symmetry however those principles come from physical observation or intuitions which are then postulated to be valid (such as space homogeneity, isotropy, equivalence principle and so on).
A: You can. The Laser is a prime example. Short cersion: The effect was predicted 1917. Then 1951 the Maser was predicted from theory and the  theoretical work from 1917. Then experimentalist tried to build such a device and succeded in 1954.
Another example would be Maxwell completing the Maxwell equation from purely theoretical grounds. It took Hertz a few decades to show this to be true.
