How is $J^{PC}$ value experimentally determined for new types of particles?
For example, this paper says
... Angular correlations in B+→X(3872)K+ decays, with X(3872)→ρ0J/ψ, ρ0→π+π− and J/ψ→μ+μ−, > are used to measure orbital angular momentum contributions and to determine the JPC value > of the X(3872) meson.
What's a description of how it works suitable to advanced undergraduate students?
Here is a nice late-undergrad or early-grad-school lab report on the determination of spin states in nuclear decays. The references to that paper (from 1940 and 1950) are relatively accessible, too.
As your pullquote says, you get $J^{PC}$ from measuring angular correlations. If a particle at rest decays into two daughters, the angular correlation is trivial: the two daughters must go out back-to-back to conserve momentum. However if you have a more complicated decay, you can have correlations where some of the decay products tend to be emitted more parallel or more antiparallel than the others. Add in the fact that the odd-$L$ spherical harmonics have negative parity and you start being able to eliminate possibilities. For example, that's the original argument for thinking the proton and neutron each obey the exclusion principle.
We can talk through the logic in your paper:
$X\to \gamma J/\psi$ implies $\hat CX = +X$.
Remember that charge conjugation, $\hat C$, is the operator which changes a particle to its antiparticle; only neutral particles may be eigenstates of $\hat C$, and the eigenvalues $C$ must have $C^2 = 1$. You can look up that the photon and the $J/\psi$ both change sign under $\hat C$, so a state which contains only those two particles does not change sign under $\hat C$.
In decays like $X\to\pi^+\pi^-J/\psi$, the angular momentum of the two-pion subsystem must be odd.
By the same logic as before, the di-pion $\pi^+\pi^-$ must have negative $C$. For this composite system, charge conjugation is the same operation as having the two particles swap places, so the pion part of the wavefunction must have negative parity. Only orbital angular momentum wavefunctions with odd $L$ have negative parity.
Dipion spins $J_{\pi\pi} \geq 3$ are ruled out because all the high-spin mesons are too heavy, so $J_{\pi\pi} = 1$.
Now we are into data analysis: you have to look at how the $J/\psi$ and the pions divvy up the energy available in decays of the $X$, and frown for a while at the Particle Data Group's list of mesons until you don't find anything and give up. They go on to further state that the channels with two pions seem to be consistent with $X\to\rho^0\,J/\psi$.
That's as far as you can really get using only the types of particles emitted and the energies involved. The next step is to actually look at decay correlation angles. When the $X$ decays to a $\rho^0$ and $J/\psi$, the spins of those two pseudovector particles are correlated with each other, and the momenta are back-to-back in the rest frame of the $X$. When those particles decay into their pairs, the decay momenta are back-to-back in their rest frames; but because of the correlated spins the two pairs of second-generation decay products aren't in independent planes. An earlier paper considered more possible quantum numbers for the $X$, and included this better figure describing the decay angles:
That paper argues against any values for $J^{PC}$ other than $1^{++}$ or $2^{-+}$, using the same sorts of arguments but less data. Your paper's final figure, below,
shows the number of events with a given $\theta_X$ and a smallish $\theta_\rho$. The data, black points with error bars, are the repeated identically in each subplot. The solid colored lines show the expected distributions of events for various internal values of $J^{PC}$. These predictions are probably based on elaborate Monte Carlo simulations, rather than first-principles calculations like the electromagnetic correlations functions in my first couple of links, but the idea is the same. You can see, for example, that the $J=0$ predictions have approximate spherical symmetry (uniform in $\cos\theta_X$), while the large-$J$ predictions are strongly peaked with the $J/\psi$ being emitted either parallel or antiparallel to the momentum of the $X$. One of those fits is not like the others, and so the spin is assigned.
