Hilbert space and group theory: relationship between these two approaches to quantum mechanics, and references for a beginner? I have read basic books on Quantum Mechanics like R. Shankar's "Introduction to Quantum Mechanics, Griffiths "Quantum Mechanics" and partly I followed Bransden "Atoms and Molecules".
But none of the above books treat QM in a mathematical way and I am really interested to look into the mathematical side of QM. In doing so, I found a book by Brian C. Hall "Quantum Mechanics for Mathematicians," which treats the subject from Hilbert Space approach and another book I found was by Hermann Weyl "Theory of Groups and Quantum Mechanics" which treats the subject from group theoretic approach.
What is the relation between those two approaches mentioned above and is there any other book which would deem fit for me who has limited mathematical background?
 A: I would suggest you have a look at Peter Woit's book, 

Quantum Theory, Groups and Representations: An Introduction (Springer, 2018).

It's a mathematics book for physicists rather than the other way around. It avoids going too far into esoteric maths while keeping that which is relevant for the typical models of theoretical physics. 
A: The best book I can think of for this subject at the moment is The Quantum Theory of Fields, Volume 1: Foundations. 
I am not recommending you to read quantum field theory. This book contains the best introduction of quantum mechanics in chapter 1 and chapter 2, using rigorous group representation theory in Hilbert space, but is very readable for physicists. It also includes the introduction of Wigner's theorem, which plays the key role in quantum mechanics. I learned a great deal of mathematics that is important for understanding quantum physics from this book, such as the mathematical origin of central charge, the mathematical definition of mass in Galilean quantum mechanics. The subject is rarely introduced in other maths and physics textbooks. As for me, the book opened a door to a whole new world of the interactive intertwining of representation theory and quantum physics, such as Mackey's induced representation theory, which led me to the great book Theory of Group Representations and Applications by A.O. Barut.
A: I highly recommend 

Anthony Sudbery, Quantum mechanics and the particles of nature (An outline for mathematicians), (Cambridge UP)

As the name implies it is pitched for mathematicians and written by someone with an excellent reputation as mathematical physicist.  It is NOT mathematically formal but does not suffer from the handwaviness of some texts written by physicists.  The target audience is defined by the author as follows:

"In practice I have imagined this reader as a mathematics student
  taking a third-year undergraduate course in quantum mechanics such as
  is commonly offered as a part of the mathematics degree course in
  British universities".

It remains accessible to a serious reader with interest in a more formal presentation than what is often done.  It does not start with Lie groups (or groups in general) as a topic but connections to group theory are pointed out when needed and it's a good stepping stones to more group-oriented texts.
