What is this nested bracket notation?

The following is an excerpt from K. Varga's paper, Precise solution of few-body problems with stochastic variational method on correlated Gaussian basis:

...The function $θ_{LM_L}(\mathbf{x})$ in Eq. (2), which represents the angular part of the wave function, is a generalization of $\mathcal{Y}$ and can be chosen as a vector-coupled product of solid spherical harmonics of the Jacobi coordinates $$θ_{LM_L}(\mathbf{x}) = [[[\mathcal{Y}_{l_1}(\mathbf{x}_1) \mathcal{Y}_{l_2}(\mathbf{x}_2)]_{L_{12}} \mathcal{Y}_{l_3}(\mathbf{x}_3)]_{L_{123}}, \ldots]_{LM_L}.\tag{5}$$ Each relative motion has a definite angular momentum...

What is the RHS of the above equation? I've never seen this nested-bracket notation before, and as far as I can tell, it isn't defined in the paper. My first guess would be something like a commutator $$[A, B] = AB - BA$$ but that doesn't explain the subscripts on the closing brackets.

This is pretty niche notation, and it is indeed not defined in the paper, but the name "vector-coupled product" does seem to be used by a few people beyond Varga and Suzuki. In essence, $$[\mathcal Y_{l_1}(\mathbf x_1)\mathcal Y_{l_2}(\mathbf x_2)]_{LM}$$ is a coupled wavefunction with total angular momentum $L$ that's made up of the single-particle wavefunctions $\mathcal Y_{l_1}(\mathbf x_1)$ and $\mathcal Y_{l_2}(\mathbf x_2)$, which have angular momentum $l_1$ and $l_2$ respectively. This means that you need to couple them via Clebsch-Gordan coefficients as usual.
Thus, the product above is given by $$[\mathcal Y_{l_1}(\mathbf x_1)\mathcal Y_{l_2}(\mathbf x_2)]_{LM} = \sum_{m_1,m_2} ⟨l_1m_1,l_2m_2|LM⟩ \mathcal Y_{l_1m_1}(\mathbf x_1)\mathcal Y_{l_2m_2}(\mathbf x_2)$$ where the sum is over all permissible magnetic quantum numbers for the individual particles.