Precise definitions for higher spin operators I am trying to understand the matrices and vectors presented in this section
https://en.wikipedia.org/wiki/Spin_(physics)#Spin_projection_quantum_number_and_multiplicity
I am looking for a reference where these objects are defined in a sufficiently precise manner that I can derive the matrices and vectors from their definition. All I could find were long and quite vague descriptions of physical backgrounds which ask me to understand things intuitively but lack necessary precision. For example, the Wikipedia article talks of operators but does not provide their domain of definition. Also the $|\alpha, \beta >$ notation is used without ever defining these notations and the vector spaces behind them in a precise manner.
I also find abstract mathematics as in "are elements of a unitary representation of $SU(2)$". I know what a unitary representation is but this does not provide me with a sufficiently clear definition from which I can derive the matrices in the article.
Ideally, I am looking for a starting point which defines these objects in a mathematically precise way and from which I can dash off calculating these matrices.
Added for clarification:
I obviously get the task of interpreting this wrong but I do not know where exactly.
In order that a commutator relation $[\sigma_x, \sigma_y] = 2i\sigma_z$ makes sense,
I need to know the space where these operators live. So let's pick a space.
In the spin 1/2 case I can get 2 different experimental results.
Thus, I am using projective space ${\mathbb P}({\mathbb C}^2)$
for the particle state.
I consider Hermitean operators of the type
${\mathbb C}^2 \to {\mathbb C}^2$. The four Hermitean
operators $\sigma_x, \sigma_y, \sigma_z, \sigma_0$ form a (real) basis of the
4-dimensional space of Hermitean operators. If I only consider trace zero operators this breaks
down to $\sigma_x, \sigma_y, \sigma_z$. I can use a normed real vector $\vec{a} = (a_x, a_y, a_z)^t$
to define an observable $a_x\sigma_x + a_y\sigma_y + a_z \sigma_z$. Measuring the
observable provides two possible results, spin-up or spin-down, which I interpret as
measuring the spin in direction $\vec{a}$.
In the spin 1 case I can get 3 different experimental results. Thus, I am using
projective space ${\mathbb P} ({\mathbb C}^3)$ for the particle state.
I consider Hermitean operators
${\mathbb C}^3 \to {\mathbb C}^3$. The space of Hermitean operators of this
signature has dimension 9, reducing them to trace zero leaves 8 dimensions.
I thus expect a basis consisting of 8 Hermitian operators.
How should I now arrive at only three operators $\sigma_x, \sigma_y, \sigma_z$
since I need 8?
How should they obtain an interpretation in 3 dimensional real space? If
the characteristic thing I need is the commutator relation, I can satisfy that
with the following embeddings as well:
$
\sigma_x =
\begin{pmatrix}
0 &1 &0 \\
1 &0 &0 \\
0 &0 &0 \\
\end{pmatrix}$,
$\sigma_y =
\begin{pmatrix}
0 &-i &0 \\
i &0 &0 \\
0 &0 &0 \\
\end{pmatrix}$ and
$
\sigma_z =
\begin{pmatrix}
1 &0 &0 \\
0 &-1 &0 \\
0 &0 &0 \\
\end{pmatrix}
$
Further clarification:
What I am looking for as well is a complete definition of what spin is. In my mind this needs a definition of the domains as in $A\colon H \to H$ (and what is $H$). If a starting point is "spin obeys commutation relations such-and-such" then I expect to find an existence and uniqueness theorem somewhere. Currently I fail in these attempts...
Amendment:
The article mentions operators $S_x$, $S_y$, $S_z$ and calls them
spin operators.
They seem to make
sense on every $n$-dimensional complex vector space. The article lists them for $n=2, 3, 4, 6$.
In the case $n=2$, three operators $S_x, S_y, S_z$ are given on ${\mathbb C}^2$.
Probably they are interesting because we want to study an observable
$a_x \cdot S_x + a_y \cdot S_y + a_z \cdot S_z$ which we can
compose as real linear combinations of these operators.
That we chose these operators as Pauli matrices is a matter of convention and convenience.
In the cases $n=3, 4, 6$ the space of observables is much larger.
However, we still only consider three operators $S_x, S_y, S_z$.
Probably we again are interested in studying the observables of the form
$a_x \cdot S_x + a_y \cdot S_y + a_z \cdot S_z$.
The particular choice of the $S_x, S_y, S_z$ probably again is a matter of convention
and convenience, and they simply are obtained if one follows a particular Kronecker
or tensor product type of construction. Fine.
However there must be a particular condition by which we single-out these operators.
We are not considering all traceless Hermitean operators (as we did in the $n=2$ case)
but only a very specific subspace of traceless Hermitean operators.
One part of the conditions, so it seems, is that the operators must have full rank,
but this is not enough. A further condition might be connected with the
commutator relation. However: The structure constants of a Lie-algebra are basis dependent
and the specific choice of the $S_x, S_y, S_z$ seems a bit arbitrary - they just generate
that class of operators. So I do not expect this to translate 1:1. Moreover, I am interested in a base independent condition.
It is this condition and its physical significance which I am looking for.
Comment added only to the suggestion to use representations of SU(2) as operators: It has been suggested to define the operators as values of group elements under a representation of $SU(2)$ in a suitable $GL ({\mathbb C}^n)$. I see several problems here. 1) This definition would be dependent on the specific choice of a representation. When $\omega\colon SU(2) \to GL(V)$ is a representation also $A\cdot \omega \cdot A^{-1}$ is a representation and so we get way too many operators again (certainly more than the 3 degrees of freedom in the SU(2)). So we would again need some way of connecting this to the observable. 2) The values of a representation are unitary operators, how do we ensure Hermeticity? 3) The suggestion to start with generators of the Lie algebra then would depend on the specific choice of the generating elements, which probably also is not unique.
 A: Giving a completely precise definition of everything would take a large amount of time, and would likely not be helpful. So instead here I will spell out the general picture, and any terms which are unfamiliar should be independently researched.
Quantum field theory has as its Hilbert space the physical states of the quantum field, see also a Fock Space. Suppose we have a Lagrangian scalar which specifies the equations of motion for the evolution of the quantum field. Demanding Lorentz invariance in a Lagrangian then leads to the natural question of what types of fields can exist that satisfy a "Lorentz transformation law," or that leave the Lagrangian scalar unchanged under an arbitrary Lorentz transformation. In other words, we are interested in (finite) $n-$dimensional representations of the Lorentz group.
The Lorentz group is a continuous Lie group, so we can study the representations by looking at the Lie algebra. It turns out that $L \cong SU(2) \times SU(2)$, and any representation of the Lorentz group can be built from irreducible representations of $SU(2).$ The Casimir Operators of a Lie Algebra are something important to understand here as well.
In ordinary quantum mechanics, the spin is just its own Hilbert space that is direct-producted on to the original Hilbert space of the position/momentum. The spin Hilbert space is an $n-$ dimensional representation of $SU(2),$ which in ordinary quantum mechanics meshes well with rotations because of the correspondence between $SO(3)$ and $SU(2).$ The reason that the stuff towards the beginning of this post isn’t spelled out in introductory texts to QM (like the wikipedia article you linked) is because to actually start doing physical calculations in non relativistic QM you don’t need to know the precise details.
A: In response to your followup questions (in a different order than you asked):
(2) A general unitary matrix satisfies the condition $U^\dagger U = \mathbb{1}.$ Let $A$ be anti-Hermitian. Then
$$(\mathbb{1}+\epsilon A)^\dagger (\mathbb{1}+\epsilon A) = \mathbb{1} + \epsilon A - \epsilon A - \epsilon^2 A^2 \approx \mathbb{1}.$$
This implies that to within order $\epsilon^2$, $(\mathbb{1} + \epsilon A)$ is unitary, and so products of it are as well. For physicists, we normally write this instead as $(1+i \epsilon H)$ where $iH=A$ and $H$ is Hermitian. Therefore,
$$\lim_{\epsilon \to 0} (1+i \epsilon H)^{n/\epsilon} = \exp{iH}$$
Will be unitary. This answers your question as to how we guarantee Hermicity. The spin operators are Hermitian matrices which when exponentiated “generate” (the Lie Group of) rotations (in ordinary quantum mechanics). When we say generator, we are referring to elements of the Lie Algebra that get exponentiated, not elements in the Lie Group itself.
(1) As you pointed out, there are many different unitary-equivalent parameterizations of a Lie Group. Equivalently, the vector space of the Lie Algebra has an infinite number of possible bases choices. We can choose any representation we like because it is only the relative algebraic structure which makes physical predictions. In other words, the commutation relations alone spell out the physics, although in practice there are conventional and convenient choices for bases. This is similar to the fact that an arbitrary $(x,y,z)$ coordinate system can be chosen, and it is only the relevant dot product between basis vectors which determines physics.
(3) This question is just the same issue as (1). Ponder the fact that an infinite number of choices also exist to select between the $X, Y, $ and $Z$ position operators, but this doesn’t pose you any conceptual difficulty.
Finally, the “extra condition” which you wonder about is in fact that we are only interested in irreducible representations.
A: Thanks to all the contributions I think I finally solved the problem.
It is similar to problems structuralist mathematicians often have when
talking to physics. Stackexchange is full of questions which boil
down to similar troubles. I shall first explain the trouble in a banal setting
and will move on to my quantum physics problem and what I have learned from it hoping this might be helpful for others.
Banal Setting: Geometry
We are in the real vector space ${\mathbb R}^2$ and look at two tuples of vectors.
${\mathbf a} = \left( \begin{pmatrix}1 \\ 0\end{pmatrix}, \begin{pmatrix} 0 \\ 1\end{pmatrix} \right)$
and
${\mathbf b} = \left( \begin{pmatrix}1 \\ 7\end{pmatrix}, \begin{pmatrix} 2 \\ 3\end{pmatrix} \right)$.
Do we have an orthonormal basis somewhere?
Physicist: Yes. ${\mathbf a}$ is an orthonormal basis and ${\mathbf b}$ is not.
Structuralist: What do you mean with orthonormal basis? We are in a real vector space. This
is an algebraic structure. Both, ${\mathbf a}$ and ${\mathbf b}$ are indeed bases, which is a notion
defined in any vector space. However,
I do not have the faintest idea what you are talking about when your are asking for an
orthonormal basis.
Theorist: Well. Depends. On ${\mathbb R}^2$ it is possible to
introduce not only the structure of a real vector space but also the structure of
an inner product space. Often, implicitly, without mentioning this, we use the
inner product
$\begin{pmatrix}x_1 \\ y_1\end{pmatrix} \cdot \begin{pmatrix}x_2 \\ y_2\end{pmatrix} =
x_1 \cdot y_1 + x_2 \cdot y_2$. So if we are in this particular inner product space
then ${\mathbf a}$ is an orthonormal basis.
Problem: Initially we said: "We are in the real vector space ${\mathbb R}^2$". On this
real vector space we can define an inner product in such a way that ${\mathbf b}$ is an
orthonormal basis. We can also define an inner product in such a way that none of the two
is an orthonormal basis. We must explicitly introduce an additional structural element
in order to talk about orthonormal bases. In a real vector space we simply cannot talk about
orthonormality, about distances or angles, simply because the structure we need is not there.
Of course, we can introduce all kinds of additional structures on ${\mathbb R}^2$,
for example we could introduce holomorphic, topologic, differentiable and many more structures,
but this amounts to an additional, arbitrary choice which I must make and announce.
My Quantum Physical Problem
My problem was that I believed in the mantra "Quantum physics is done in a Hilbert space".
Since a spin 3/2 measurement can have 4 different values I was
looking at Hilbert space ${\mathbb C}^4$.
"Quantum physics is done in a Hilbert space" is correct only for $n$-level systems.
When I want to look at spin I need an additional structure identifying spin in this
Hilbert space. This is similar to above question of orthonormality, which cannot be dealt
with in a vector space but needs an inner product space as additional structure.
When I study spin then I study observables which
have the structure of the 3-dimensional Lie algebra $su(2)$.
Therefore I am interested in "finding" this structure in the observables of a Hilbert space.
Similarly as above, in the case of the inner product, this additional structure can be
found in an algebra of observables in different ways, some of which might be equivalent.
However, I must introduce this additional structure and define
what it means that they are equivalent.
A representation of $su(2)$ in the space of observables or of $SU(2)$ in the unitary
group of the vector space provides the structure; equivalency means isomorphism of
the representation.
So, spin quantum physics is done in a Hilbert space together with a representation.
When studying these representations one finds, that $SU(2)$ has many representations.
They can be classified and their classification then leads to structurally
different variants of spin, as in spin 0, spin 1/2, spin 1 etc.
Irreducible representations have no special magic, they are simply "easier"
in the following sense: I can do 2-dimensional geometry
in a ${\mathbb R}^100$ with standard inner product. The additional 98 dimensions
are not really necessary and I can simply cut them away.
When I am using a reducible representation I also can
cut away additional dimensions of the Hilbert space and restrict myself to the
invariant subspace of the representation.
Important are the isomorphism types of the representations.
The Wikipedia article provides
matrices for $S_x$ for higher spins. The article makes
implicit assumptions on the representation of $SU(2)$. These assumptions are encoded implicitly
(1) in the representation of $SU(2)$ in ${\mathbb C}^2$, which is generic, and (2) in
the manner of generating new representations in higher dimensions
${\mathbb C}^{> 2}$ using tensor products of representations (or Kronecker
products of the associated matrices).
They correspond to a choice of a specific representation of the isomorphism type required
for the necessary spin type 1, 3/2 or whatever.
So, of course, completely different $S_x$ are possible, but they then also correspond
to a different choice of the representation.
Conclusion
My problem with the article was of the same nature as the problem of the structuralist
who was calling for an inner product as additional structure.
It is the representation which "glues" the physical meaning to the elements of the
Hilbert space.
In this context also "a spinor is an element of a representation" completely makes sense.
Of course, every element of the range of a possible representation could play that role,
and so the statement looked empty to me (as in: Every vector space basis can play the role
of an orthonormal base). However, if we start with a given representation than the choice of
an element is a clear structural statement (just as in a specific inner product space the
choice of an orthonormal base is not arbitray - provided an inner product has been fixed first).
As pleasant side effect I can now also appreciate the meaning of the Stone-von Neumann
theorem and its commutator relations.
