There is no good book aimed at physicists. Robert Hermann, Lie Groups for Physicists is worth reading, but you didn't want something only about Lie Groups. Gelfand, Graev, and Vilenkin, Les Distributions, vol. 5 or, in English, Generalized Functions, vol. 5 is good for Fourier analysis on a group closely related to the Lorentz group, but not aimed at physicists, but is eminently readable and has some mistakes which don't really matter. Representations of finite groups are covered in Boerner, Representations of Groups: With Special Consideration for the Needs of Modern Physics an old classic written for physicists. None of these books are good, but they are the best I can think of. Strichartz has written about harmonic analysis on the actual Lorentz group, perhaps it is worthwhile, perhaps I will look at it some day...

A famous mathematician once told me no one had ever understood Weyl, The Classical Groups. I think much of it is covered by Boerner.