I am a physics undergrad and thinking of exploring quantum information theory. I had a look at some books in my college library. What area in QIT, is the most mathematically challenging and rigorous? From what I saw in the books, most topics was just simple linear algebra. I am looking for an area which is mathematically more rich, and uses maybe more concepts from theoretical computer science, number theory, discrete maths, algebra etc. Classical cryptography is an area on the interface of maths and TCS which uses many areas of maths such as number theory, algebra, ellptical curves. Is the quantum cryptography also rich in mathematics? What are the prerequisites? If not, please could you suggest some areas that I are mathematically rich in QIT?
I believe that the geometric point of view is superior to the algebraic one in quantum theory. Many of the achievements in understanding quantum theory emerged from the geometrical point of view, for example, Wigner's classification of relativistic particles (as irreducible representations of the Poincare group). Also, many of Witten's achievements stemmed from his deep geometrical understanding. In fact, in his seminal works he applied geometric quantization beyond the limits that were known to mathematicians at the time.
Of course, the mathematical areas relevant to this direction of research include: Analysis on manifolds, Lie groups, Fibre bundles, Symplectic geometry, Geometric quantization Etc.
In the special case of QIT, it is true that the main stream follows the algebraic point of view, but let me refer you to works adopting the geometric point of view. The basic reference is Bengtsson and Zyczkowski's book: Geometry of quantum states: An introduction to quantum entanglement. Let me also refer you to important more recent works in this direction:
Geometry of entangled states by Marek Kus and Karol Zyczkowski.
Symplectic geometry of entanglement by: Adam Sawicki, Alan Huckleberry, Marek Kus, and
Segre maps and entanglement for multipartite systems of indistinguishable particles by: Janusz Grabowski, Marek Kus, Giuseppe Marmo
These articles include many other references on the subject, also, many of the authors have additional works.
I would make the case that of the myriad of choices, a key area of mathematics that will become more and more critical to general physicists in the future (and it's already key in certain leading edge physics) is category theory. Culturally this is called abstract nonsense which should already indicate its importance to the mathematical community (you would have to see a mathematician's eyes glow when they talk about it to understand). A good paper making the case can be found here. It is also critical to understanding advances in QIT and other fields.
The other area is computational complexity theory, where I would refer you to the complexity zoo to get a good introduction. Physics and computational theory are beginning to intersect in a very real way, and an understanding of computational complexity will allow for a bridge to computer science, which will probably be the dominating field in general science in the coming years (if not already).
Note: Just a quick comment, although the linked paper implied a connection to loop quantum gravity in the context of quantum gravity, category theory also links to string theory in a more physical way. I would refer you to the nLab website for more insight on category theory and physics. See also.