In astronomy per se, little advanced mathematics are used, except statistics, error propagation, and in designing optical instruments. As soon as you start shading over from pure astronomy into astrophysics, however, you will have occasion to use the full range of, as a grad student acquaintance of mine once very memorably put it, Jedi math. Cosmology and gravitation are especially heavy. (Whoops, no pun intended.) The cutting edge of astrophysics and its close relative particle physics have historically been major drivers in pure mathematical research, in fact.
It should be stated that astronomy and astrophysics are so intertwined that there is virtually no such thing as a truly pure astronomer, nor any purely astronomical research endeavor, so the distinction is pretty artificial.
In short, pseudo-Riemannian geometry is used in General Relativity, analysis of differential equations is used widely for theories about objects like stars, and observers use quite advanced statistical methods for their large data sets. I've tried to provide some accessible examples of where the maths gets used. I don't think there's as much use of abstract mathematics as in theoretical physics. But then again, I'm also not sure how much of that is because it's genuinely useful...
(Pseudo-)Riemannian geometry is important because that's the context in which General Relativity is formulated. This isn't all that theoretical anymore, because observing gravitational waves (or trying to...) is a fairly hot topic at the moment. Non-Euclidean geometry is probably the heaviest frequently used section of mathematics.
That aside, I'd say a lot of structure theories (stars, planets, accretion discs, etc) rely on coupled non-linear differential equations. So all the associated analytical techniques for dealing with non-linear DEs are relevant. For example, linear stability analysis (finding eigensystems of Jacobians and whatnot) could come up. I've seen it done for the Lane-Emden equation and for accretion flows onto black holes.
There's quite a lot of numerical analysis that's now necessary because of all the computational work that is done. Most theorists should be well-versed in the numerical methods used in their particular field. You could read, for example, Price's introduction to smoothed particle hydrodynamics if that were your thing. Or you could read about relaxation methods in any textbook on stellar structure and evolution e.g. Section 4.7c of Collin's The Fundamentals of Stellar Astrophysics. I don't think these are usually included in junior undergraduate teaching.
Finally, observers employ a lot of statistics. Something I hadn't heard of during undergrad was the Kolmogorov–Smirnov test for determining whether two populations are drawn from the same distribution. (At least, I think that's what it's for...) I'm not well-versed in statistics, so I'm not sure what qualifies as more or less advanced in that topic.
In my opinion, the two other posts that are currently on here don't summarize a simple answer.
Short Answer: It completely depends upon your specialty in astronomy/astrophysics/planetary science. It's not like, say, accounting where anyone who does accounting needs to have a set of math they must know. Rather, it's more like asking, "What biology do I need to know if I want to study medicine?" The answer is wholly dependent upon what you want to do with it.
Longer Answer: To re-iterate what Andrew said, first we must establish the difference between "astronomy" and "astrophysics." Most consider the former to be what people did, generally, prior to around 1900. "Astronomy" is considered most of the basic observational stuff. "Astrophysics" is more adding the theory to the observations, and much more detailed and rigorous observations.
If you're an observer, you need to know a lot about statistics, error analysis, noise models, signal processing, and optics.
If you do spectroscopy, you'll need to know a lot about Fourier theory, diffraction, and a heck of a lot about optics.
If you study cosmology, you'll need a lot of calculus, the Riemannian geometry, general relativity and related math, etc.
If you study dynamics, you'll need to study a heck of a lot of coding methods and ways to approximate gravity in millions-of-particles simulations (K-tree stuff), searching and sorting, and lots of dynamical theory.
Solar and you get into what most consider the hardest physics - magnetohydrodynamics. So magnetic fields, electricity, fluid mechanics, turbulence, nuclear theory, and some quantum mechanics.
Planetary and you need to know about turbulence, atmospheres, statistics, geology, convection, etc., but even there you have so many specialties that each rely on their own thing.
I realize after writing these that I've specified more the fields of physics more than the actual fields of math that you need; while this doesn't answer your question quite as directly, I think it may more informative in general. Each of these generally require statistics and error analysis, basic algebra, and at least a working knowledge of calculus. Once you get into your specialty, you'll need to, well, specialize, and you'd really need to ask about that field and find someone who knows it more than just asking the general question you did.