You're halfway there. The quickest way that I can see is via explicit Cartesian components and coordinates. You start by assuming that $\mathbf B$ is uniform, so that you can pull it out of the integral (which you've already used implicitly for dealing with $Y$. That means that you're interested in
\begin{align}
\frac{\boldsymbol \tau}{I}
=\oint_L (\mathbf{r} \cdot \mathbf{B})\mathrm{d}\mathbf{r}
=\sum_{j,k}\mathbf e_k\oint_Lx_j B_j\text d x_k
=\sum_{j,k}\mathbf e_k B_j\oint_Lx_j\text d x_k.
\end{align}
On the other hand, you're trying to prove that this equals the 'goal' torque
\begin{align}
\frac{\boldsymbol \tau_G}{I}
&=\frac1I \mathbf m\times\mathbf B
=\frac12 \oint_L(\mathbf r\times\text d\mathbf r)\times\mathbf B
=\frac12\left[
\oint_L (\mathbf{r} \cdot \mathbf{B})\mathrm{d}\mathbf{r}
-
\oint_L \mathbf{r}\,(\mathbf{B}\cdot\mathrm{d}\mathbf{r})
\right]
\\ &
=\frac12\left[
\sum_{j,k}\mathbf e_k B_j\oint_Lx_j\text d x_k
-
\sum_{j,k}\mathbf e_k B_j\oint_Lx_k\text d x_j
\right].
\end{align}
These look very similar, and the thing that glues them together is the identity
$$
\oint_Lx_j\text dx_k+\oint_Lx_k\text dx_j=0.\tag{$\ast\!\ast$}
$$
This can be proved quite easily using integration by parts on either, as the boundary term vanishes because $\partial L$ is empty.
To make both torques match, you can either switch the $j$ and $k$ in $\boldsymbol \tau_G$ and collapse both terms, or split the integral in $\boldsymbol \tau$ into $\tfrac12\times$ two integrals with switched symbols and opposite signs.
Now, I'm afraid all this mathematical manipulation doesn't really go a long way towards answering the core of your question, which is why the magnetic moment comes up as the best way to summarize the ugly integral. This is typically a complicated process, and it is usually hard or impossible to really succeed without understanding the key underlying structures in the object you're attacking. In the case of dipole moments, this underlying structure is the fact that the subleading term in the Taylor expansion of the Coulomb kernel,
$$
\frac{1}{|\mathbf r-\mathbf r'|}=\frac{1}{|\mathbf r|}+\frac{\mathbf r\cdot\mathbf r'}{|\mathbf r|^3}=
$$
is linear in the 'small' vector. This means that you will be forced to sum up the essentials of your current distribution using a tensor which is linear in the coordinates. Because you have additional symmetry restrictions, the class of available tensors is already rather small.
In fact, once you make the (crucial!) step of assuming the field is uniform, you already have such a tensorial action: the process that gets you from the field $\mathbf B$ into the torque
$$
\boldsymbol \tau
=\sum_j \mathbf e_j\cdot\mathbf B\,I\oint_L x_j\mathrm{d}\mathbf{r}\,
$$
that it generates is already a linear transformation acting on the field. You could even write it up in its matrix form:
$$\tau_k=\sum_j M_{kj}B_j,$$
where
$$M_{kj}=I\oint_L x_j\mathrm{d}x_k.$$
In general, this is more or less as far as you can go. This form separates the torque into two separate ingredients, of which one describes the loop and the describes the external action on it.
In this case, however, you can go a bit further than that because of the symmetry of the underlying objects. In particular, there is a strict requirement that this linear transformation be antisymmetric. This is exactly the antisymmetry expressed in ($\ast\!\ast$), but it has a deeper and physical significance. As a mathematical requirement on the matrix $M$, this requires that for arbitrary vectors $\mathbf v$ and $\mathbf u$
$$
\mathbf v\cdot M\mathbf u=-\mathbf u\cdot M\mathbf v
$$
(and reduces to the antisymmetry in the indices by taking $\mathbf v$ and $\mathbf u$ to be coordinate vectors), but this is equivalent to requiring that
$$
\mathbf v\cdot M\mathbf v=0
$$
for any $\mathbf v$. Physically, this requires that the torque axis be orthogonal to the uniform applied magnetic field - and this is trivial, as the torque on each component is orthogonal to it.
Thus, the matrix $M$ must be antisymmetric. The integration by parts is just a nice check that we have defined it correctly, but this is in a certain sense superfluous.
Finally, it is exactly this asymmetry that allows us to condense the information in the matrix $M$ into a single vector. In three dimensions, an antisymmetric matrix has three independent components, and these are in one-to-one correspondence with the components of a (pseudo)vector:
$$
M=\begin{pmatrix}0& M_{12} & -M_{31}\\ -M_{12} & 0 & M_{23} \\ M_{31} & -M_{23} & 0
\end{pmatrix}
=:\begin{pmatrix}0& -m_3 & m_2\\ m_3 & 0 & -m_1 \\ -m_2 & m_1 & 0
\end{pmatrix}.$$
The action of this matrix is exactly the same as that of the cross product with $\mathbf m$:
$$
M\mathbf B
=\begin{pmatrix}0& -m_3 & m_2\\ m_3 & 0 & -m_1 \\ -m_2 & m_1 & 0
\end{pmatrix}
\begin{pmatrix}B_1\\B_2\\B_3\end{pmatrix}
=\begin{pmatrix}m_2B_3-m_3B_2\\m_3B_1-m_1B_3\\m_1B_2-m_2B_1\end{pmatrix}
=\mathbf m\times\mathbf B.
$$
Finally, to get an expression for the dipole moment vector, you simply define $m_i=-M_{jk}$ where $(i,j,k)$ is a cyclic permutation of $(1,2,3)$, or more compactly
$$
m_i=-\tfrac12\epsilon_{ijk}M_{jk}
=\tfrac12\epsilon_{ijk}M_{kj}
=\tfrac12\epsilon_{ijk}I\oint_Lx_j\text dx_k
.$$
You can now do away with the coordinate notation, because this is simply
$$
\mathbf m=\tfrac12I\oint \mathbf r\times\text d\mathbf r,
$$
and you're done.
