As someone who intended to study physics, but ended up studying maths; I found maths, dry & discursive and far removed from what my physical intuition found useful. It was impossible to study :-)
When I returned to Physics, I found the arguments sometimes impossible to follow, as I was always looking for the logical motivation. In other words, my physical intuition had evaporated :-(.
Mathematics books, although there exposition may be clearer, to the mathematician; has differing standards, and is trying to accomplish different things.
I suggest you stick to physics books with the appropriate mathematical technology, so that the primary physical intuition you need to develop isn't displaced. But also dip into maths texts to see what else is going on, or get a mathematician to explain, to see what you're missing out on. Expositary papers are useful.
Historically the links between the two subjects are complex, and fascinating; and I can only expect this to continue, despite the occasional falling out (Gruppenpest & Abstract Nonsense).