What would be the requirement to learn matrix mechanics?
More specifically, what math do I need?
Can anyone recommend me a book that covers all maths needed for matrix mechanics?
Matrix mechanics is quantum physics without the Schroedinger equation; the dynamics is instead defined by the Heisenberg equation on the observables.
Matrix mechanics is not very popular nowadays, but you find everything you need (exept for background in linear algebra and calculus, which are assumed) in my book Classical and Quantum Mechanics via Lie algebras.