Sometime ago, I had same doubt, but finally, I chosen Physics for the following reasons:
Pure math is very abstract, you may go along very complex structures that will bring you to nowhere, in a sense of their practical usability.
Modern math is very far of it's old philosophy "intuitionism" , and they try always to prove things that seems very obvious , this decreasing productivity very much, and dragging you back of understanding the "big picture", that of course not a wrong thing, but they are really exaggerating in that in my opinion.
Besides other reasons I will not mention to keep it short.
My advice to you:
Try to read a math book for mathematicians and another one about the same subject specially for physicists, such that both of same subject. You will understand immediately what fits you. even from the first 50 pages, that how I understood what I want, and as a good example of such subject is "differential geometry".
May be you want to consider an "in the middle" solution: since physics is more intuitive-oriented, you will see a lot of hand waving in books of physics, mathematically ill defined objects, or even totally unacceptable manipulation ways (from math point of view, like Dirac delta function), or non rigorously derived things. If this makes you uncomfortable, you can go to "Mathematical Physics", there guys tries to make physics more "math-friendly" and rigorously defined, in the same time avoiding the usual "maze" of math abstraction.