The first thing that must be said is that the question is not really specific enough: Applications to what exactly are you looking for? To me, a book on algebraic geometry and mirror symmetry, and how it relates to mirror symmetry as physicists know it, is very relevant and interesting. However, I have the feeling that this is not exactly what you're looking for.
Therefore, I will mostly stick to "basic stuff", that seems relevant to anyone really interested in e.g. high energy physics (on the theory side of things, obviously), without assuming the reader is really interested in advanced stuff. However, I'll include a section of "specialized" books, where some more esoteric and/or difficult topics are included, as well as books that completely focus on one specific branch of physics (e.g. general relativity), rather than developing the general mathematical tools.
Finally, note that I'm omitting standard introductory texts for both topology & geometry if I feel the text is not really aimed specifically at physical applications, since there are so many that even with this stringent criterion there is enough left. Here we go:
Baez & Muniain - Gauge Theories, Knots and Gravity
Interesting book that develops the mathematics along with the relevant physical theories: The first chapter is on E&M and the relevant mathematical notions such as forms, the second chapter is about gauge theory, both from the physical and mathematical perspective, and the last chapter is about general relativity and Lorentzian geometry.
Bleecker - Gauge Theory and Variational Principles
Starts with a very brief treatment of tensor calculus, fiber bundles etc, quickly moving on to physical topics such as Dirac fields, unification (of gauge fields) and spontaneous symmetry breaking.
Bredon - Topology and Geometry
After seeing the added message by QuanticMan, requesting answers not to shy away from going beyond the intentions of the OP and to recommend books with geometrical intuition, this book immediately sprang to mind. In the preface, Bredon states:
While the major portion of this book is devoted to algebraic topology, I attempt to give the reader some glimpses into the beautiful and important realm of smooth manifolds along the way, and to instill the tenet that the algebraic tools are primarily intended for the understanding of the geometric world.
This seems to perfectly fit the bill, and I can personally recommend it. Note, however, that it purely a book about mathematics: No applications to physics are presented, though the tools are of course relevant in physics, too.
Burke - Applied Differential Geometry
Starts with about 200 pages of mathematical tools (from tensors to forms) and then delves into applications: From the heat equation to gauge fields and gravity.
Cahill - Physical Mathematics
This is a really basic book, that does much more than just topology and geometry: It starts off with linear algebra, spends a lot of time on differential equations and eventually gets to e.g. differential forms.
Fecko - Differential Geometry and Lie Groups for Physicists
Develops the basic theory of manifolds (the focus is not on topology), and eventually treats a bunch of topics including classical mechanics (symplectic geometry), gauge theory and spinors. There is also a (much shorter) set of lecture notes by Fecko on the same topic.
Frankel - The Geometry of Physics: An Introduction
This is a big book that covers a lot of group mathematically, but does not really focus on physical applications. The topics include differential forms, Riemannian geometry, bundles, spinors, gauge theory and homotopy groups.
Gilmore - Lie groups, physics and geometry
Subtitled "An Introduction for Physicists, Engineers and Chemists", this book could be a good starting point for someone who is really only interested in simpler, down-to-Earth topics. Includes a chapter on "hydrogenic atoms", which sounds interesting.
Isham - Modern differential geometry for physicists
A "standard introductory book" on differential geometry, translated to the language of physicists. Isham is careful to point out where mathematical notions that he introduces are used in physics, which is nice for those who prefer not to lose track of the physical relevance of it all. Covers all the basics up to fiber bundles in about 300 pages.
Jost - Geometry and Physics
Quickly gets to more advanced topics including moduli spaces, spinors and supermanifolds (all within the first 100 pages) in the first part, dedicated to mathematics. The second part is dedicated to physics and includes e.g. sigma models and conformal field theory.
Mishchenko & Fomenko - A course of differential geometry and topology
Though this is pretty much a "general introduction" book of the type I said I wouldn't include, I've decided to violate that rule. This book is Russian, and the style of Russian textbooks is very physical and interesting for physics students, in my opinion. Furthermore, the book does not focus on either differential geometry or topology, but covers both (briefly), which is also good for physics students.
Naber - Topology, Geometry and Gauge Fields (two volumes)
The first volume has a cute motivational chapter which introduces advanced notions (sadly, those are usually the ones that turn out to be relevant in physics), and first discusses homology & homotopy (in reverse order?!) before moving on to manifolds and bundles (also in reversed order?!), ending with (physical) gauge theory. The second volume covers more advanced topics such as Chern classes.
Nakahara - Geometry, Topology and Physics
The go-to book for mathematical prerequisites for e.g. gauge theory, string theory etc. if you ask 90% of physicists. I personally think it's terrible because it doesn't explain anything properly, but I guess it's good to learn buzzwords.
Nash & Sen - Geometry and Topology for Physicists
This book is not very physical, but seems very nice if you're really trying to get a good grip on the math. It takes its time to (properly, I hope) develop all the theory in order: fundamental group, homology, cohomology and higher homotopy groups are all introduced, before fiber bundles and then Morse theory and (topological) defects are treated (!!). The final chapter is on Yang-Mills theories, discussing instantons and monopoles.
Von Westenholz - Differential Forms in Mathematical Physics
After about 400 pages of preparatory mathematics (including, besides the standard topics, Frobenius theory and foliations, which is nice!), the book treats classical mechanics and relativistic physics (including fluid mechanics), each in about 50 pages.
Specialized and/or Advanced Texts
Booss & Bleecker - Topology and Analysis: The Atiyah-Singer Index Formula and Gauge-Theoretic Physics
Advanced topic---very analytic, with lots of information on elliptic differential operators.
Cartan - Theory of Spinors
Couldn't resist putting this in: The original classic on spinors, by the discoverer himself. Kind of outdated (in e.g. notation) and therefore probably not very useful to modern students.
Deligne et al. - Quantum Field and Strings: A Course for Mathematicians (two volumes)
The two volumes cover about 1500 pages, with contributions from famous mathematicians and physicists alike (Deligne, Witten...). Covers lots of advanced topics in physics from a mathematical perspective, and includes exercises.
Dunajski - Solitons, Instantons and Twistors
Because the OP mentions solitons I thought this might be interesting to mention: Basic topology and geometry is assumed to be known, and a lot of physically interesting topics are covered (such as monopoles, kinks, spinors on manifolds etc)
Levi-Civita - The Absolute Differential Calculus
Another classic, and one of the first books on tensor analysis.
Nash - Differential topology and quantum field theory
This book seems fascinating for those who are really trying to get into the more difficult parts of gauge theory. Topics covered include topological field theories (knots invariants, Floer homology etc), anomalies and conformal field theory.
O'Neill - Semi-Riemannian Geometry With Applications To Relativity
A well-known textbook on the mathematical underpinnings of general relativity.
Sachs & Wu - General Relativity for Mathematicians
Same as the previous book
Ward & Wells - Twistor geometry and field theory
This book is completely dedicated to the theory of twistors: The last part is about applications to gauge theory.