Is there any need to prove the existence of an operator $U$ which represents the action of symmetry transformation on rays in Hilbert space? Or is it enough just to prove that it is unitary and linear or antiunitary and antilinear?
I do not see how we should go about finding this operator. I got the answer that this operator is given, but I don't see how. The only thing that is given is the symmetry transformation which takes the whole ray $R_1$ into, let's say, $R_2$. But what then is the action of the operator on this ray? Which vector in $R_1$ will be taken into which in $R_2$?
My guess is that we can show that coefficients of the old and a new transformed vector are the same, so we can just say that in this way the transformation is translated into an operator. Every orthonormal vector of the old basis has the same coefficient as the one in the new basis.