Measuring spin at an angle. Why is the probability $p = \cos^2(\theta/2)$? In this video about quantum entanglement, at 1:38, it is said that "the probability [of spin being in the same direction with the measurement] depends on the square of the cosine of half the angle". That is, $p = \cos^2(\theta/2)$, where $\theta$ is the angle between the direction of spin and direction of measurement. Here is a video about how the spin is measured.
To give an example, for $\theta = \pi/3$:

$p = \cos^2((\pi/3)/2) = 3/4$

but, I would expect it to be (in a straightforward manner):

$p = 1 - (\pi/3)/\pi = 2/3$ ; where $\pi$ is the angle between opposite spins

So, why is the probability equal to the square of the cosine of half the angle? What is the physical interpretation and what is the mathematical derivation? Thank you.
 A: 
So, why is the probability equal to the square of the cosine of half the angle?

This is a related to a fundamental property of spin 1/2 particles.
The following discussion will show how the 1/2 in the cosine argument originates mathematically from the definition of the angular momentum operators for spin 1/2.
For example:
$$
S_z = \frac{1}{2}\   
\left( \begin{matrix} 
   1 & 0 \\
   0 & -1 \\
   \end{matrix}
\right)\;,
$$
where I am working in natural units such that $\hbar = 1$. And I am working in a basis of spin-z eigenvectors.
(Spoiler alert, as we will see, the factor of 1/2 multiplying $\theta$ in the cosine argument is going to arise from the factor of 1/2 in the angular momentum matrix definition... And unfortunately, you have to accept this definition as arising from a potentially counterintuitive fact.)
In this same basis, the spin-x operator is:
$$
S_x = \frac{1}{2}\   
\left( \begin{matrix} 
   0 & 1 \\
   1 & 0 \\
   \end{matrix}
\right)\;,
$$
and so the generator of rotations about the x-axis by angle $\theta$ is:
$$
e^{-iS_x \theta}\;.
$$
Therefore, if you rotate a spin-z eigenstate about the x axis by $\theta$ the resulting state is:
$$
e^{-iS_x \theta}|1/2\rangle \;,
$$
where $|1/2\rangle$ is the spin-z eigen state.
Use the Taylor series expansion to show that $e^{-iS_x \theta}$ is the same as
$$
-i \left( \begin{matrix} 
   0 & 1 \\
   1 & 0 \\
   \end{matrix}
\right) \sin(\theta/2) + \left( \begin{matrix} 
   1 & 0 \\
   0 & 1 \\
   \end{matrix}
\right)\cos(\theta/2)\;,
$$
For example, in the cosine part of the expansion, you will use the fact that:
$$
(S_x \theta)^2 = (\theta/2)^2 \left( \begin{matrix} 
   1 & 0 \\
   0 & 1 \\
   \end{matrix}
\right)\;,
$$
where above you see how the 1/2 that was part of the $S_x$ operator is now multiplying the $\theta$ and the matrix part got squared and became the identity matrix. The higher order terms in the cosine part of the expansion follow straightforwardly from the above expression and the fact that $\left( \begin{matrix} 
   1 & 0 \\
   0 & 1 \\
   \end{matrix}
\right)^2 = \left( \begin{matrix} 
   1 & 0 \\
   0 & 1 \\
   \end{matrix}
\right)$.
So, if you now measure if the spin in the z direction, the resulting probability amplitude is:
$$
\langle 1/2|e^{-iS_x \theta}|1/2\rangle = \cos(\theta/2)\;.
$$
The probability is the square of the probability amplitude, so the probability is:
$$
\cos^2(\theta/2)
$$
A: There are a number of ways of seeing it. A full mathematical derivation will involve representation theory of the group of rotations $SO(3)$ and the formal structure of quantum theory.
But here is a kind of "physics first" kind of derivation. We will take empirical data and ask for some basic consistency, and we will get pretty close.
First, there is the all-important fact that for spin-1/2 particles (which is what we are talking about), when you measure the spin along the axis $\hat n$, there are only two results: "parallel" or "antiparallel". Let's denote these $+\hat n$ and $-\hat n$ respectively. (Note that I am using hats to denote unit-length spatial vectors, not QM operators).
Then it makes sense that the the probability of obtaining a given result should depend only on the angle $\theta$ between the original state and the axis of measurement. So that if the spin is initially in $+\hat z$ then we require the probabilities to be given by
\begin{align}
P(+\hat n|+\hat z) &= f(\theta) \\
 P(-\hat n|+\hat z) &= g(\theta)
\end{align}
for some functions $f$ and $g$. Since there are only two options and these are probabilities, they must add up to one: $$f(\theta) + g(\theta) = 1.$$
Notice that if $\theta$ is the angle between $+\hat n$ and $\hat z$ then the angle between $-\hat n$ and $\hat z$ is $\pi - \theta$. Since we arbitrarily chose which direction of the axis to call $+$ or $-$. Choosing the other opposite convention results in switching $\theta$ with $\pi-\theta$. Since these are all just labels, and we want the probabilities to be about physical configurations, $f$ and $g$ should satisfy $$g(\theta) = f(\theta-\pi)$$ and thus $$f(\theta) + f(\pi-\theta)=1.$$
Finally, a last crucial bit of experimental info. If you measure the spin along the same axis twice, then you get the same result twice. That is
\begin{align}
1 &= P(+\hat z|+\hat z) = f(0) \\
0 &= P(-\hat z|+\hat z) = g(0) = f(\pi).
\end{align}
Putting everything together, we are seeking for a smooth (everything in physics is smooth) function
\begin{align}
f:[0,\pi]&\longrightarrow[0,1]
\end{align}
such that
\begin{align}
f(0)&=1\\
f(\pi)&=0\\
f(\theta) + f(\pi-\theta) &= 1 
\end{align}
Now, there is a pretty natural candidate solution, namely $f(\theta) = \cos^2(\theta/2)$.
This does not prove to you that this the only function that does this. Indeed, there is an infinite amount of functions that fit the above description. One might add more requirements from experiments, such as $f(\pi/2) = 1/2$ and that $f$ decreases monotonically on $[0,\pi/2]$ and so on and $cos^2(\theta/2)$ is a simple analytical function that fits these requirements.
