Can spin of a particle or a group of particles become infinity? Explain plz. Is there any representation for spins like dot(for $S=0$) and arrow (for $S=1$)? If so what for $S= \infty$?
Compact group $SU(2)$ really has only finite-dimensional unitary irreducible representations, but formally it is not enough to close the question, because there are unitary irreducible infinite-dimensional representations of spin group $SL(2,C)$ of four-dimensional relativistic Lorentz group and they were used in some models, below is a cite from: N.N. Bogolubov, A. A. Logunov, A.I. Oksak, I. Todorov, General principles of quantum field theory, Springer, 1989. Appendix I for chapter 9
The concept of an infinite-component field (ICF for short) is the result of abandoning the "technical" requirement that the representations of the Lorentz group according to which the fields transform (say, in the Wightman formalism) be finite-dimensional. This idea turned up at the earliest stages of quantum field theory: in 1932, Majorana gave an example of an infinite-dimensional wave equation $(i \Gamma^\mu \partial_\mu – M) \psi(x) = 0$ without negative-energy solutions of non-negative square mass, that is, without "antiparticles".
Running ahead (see §1.3), it should be noted, however, that the description of composite systems by means of ICF's has met with difficulties which, it would seem, require a weakening of the postulate of (strict) locality.
I am not aware of any theory involving a notion of infinite spin. Generally, spin is a quantum number that takes (typically) small integer or half integer values. In principle, you can have a system with as high a spin as you would like, but that's not infinite. So I'd say the answer is no.
The three-dimensional spin group is the compact group $SU(2)$, which has only finite-dimensional representations. Hence, the spin is always a non-negative integer or half-integer.