Skip to the last paragraph for the bottom line, otherwise read on:
First you need to take more of the core math sequence to learn vector calculus, and at every chance you get you should try to deepen your intuition for it (what does it mean physically when you evaluate the divergence of a vector field, etc). Then you'll probably want to take classes (often required) in linear algebra, methods of solutions to both ordinary and partial differential equations, and the applications of fourier analysis to just about anything you can find (including its application to the solutions of differential equations). No matter what type of physics you do, knowing those topics cold will be crucial. It's also handy to gain some familiarity with special sets of orthonormal functions that come up often in solutions of differential equations (Legendre, Bessel, Hermite, etc).
In general I recommend you master the math mentioned so far in the context of physics applications, before diving in to the more abstract realm if it interests you - at least, this is how I learn best, by first seeing the concrete applications before abstracting. Most books with titles along the lines of "mathematical methods in physics" are good for this purpose, just follow some recommendations and then choose your favorite based on your own experience. I'll second the recommendation of Boas in the Alexander's comment.
At this point you'll probably have enough of a foundation in mathematical methods to work in many areas of physics, but it would still be a wise investment to learn some abstract algebra and group theory in particular, ideally with an emphasis on Lie groups. This is crucial for gaining a deeper understanding of quantum mechanics, and will be useful regardless of whether you go on to do experiment or theory in particle physics, condensed matter / solid state, or atomic physics / optics. Again, I'd recommend you seek out classes along the lines of "group theory for physicists" first, although if you really like it you could probably benefit from an additional class from the pure math perspective, too. Elementary ideas in what's known as representation theory come up frequently here and it's good to be aware of the connections.
Next, if you're thinking of working on either particle physics or gravity, you might consider a class in differential geometry (or better yet just take General Relativity, since the first few weeks of a GR class are typically a crash course in differential geometry). You'll probably have learned pieces of this subject already by this point (i.e. what is a manifold, from your vector calculus or if you've learned about Lie groups), but a course in diff. geometry can help tie things together and teach you new conecpts.
By that point you'll have a better idea of what it is you want to research, and can plan future math classes (or the lack thereof) accordingly. Popular choices are more specialized topics in representation theory, topology and algebraic topology. And I'm sure other people might chime in with their favorite math electives.
Bottom line: Beyond the core calculus classes and linear algebra, I'd recommend taking math methods classes from the physics department first before allocating too much time to self-study on your own, and generally your knowledge of how the math "really works" will just grow with experience. If your department doesn't offer a math methods class, then make sure you have access to a math methods textbook to consult as new things come up. To learn more advanced topics (group theory, representation theory, other parts of abstract algebra, differential geometry, topology and algebraic topology), again look for classes before going it alone, but now some of them might be back in the math department. One last thing to keep in the back of your mind: you'll probably want to develop computer programming skills, too.
