This question can be addressed using classical electrodynamics, with "electromagnetic radiation" in place of "photons." The question has two parts:
Part 1: Does a rotating produce EM radiation?
Part 2: We know that the magnet must lose rotational kinetic energy to balance the energy that is carried away as radiation, because the total energy is conserved. But what mechanism slows the rotation?
Radiation from a rotating magnet
The answer to Part 1 is yes. This is well-known, so I'll just quote a result that quantifies the rate at which energy is radiated away.
Treat the magnet as a dipole of negligible size but with a finite magnetic dipole moment $\mathbf{m}$. Suppose that the dipole is rotating about an axis perpendicular to $\mathbf{m}$, as described in the OP, and let $\omega$ denote the angular velocity of the rotation. The radiated power, integrated over all directions and averaged over a rotational period, is
$$
P = \frac{\mu_0 |\mathbf{m}|^2\omega^4}{6\pi c^3}
\tag{1}
$$
where $c$ is the speed of light and $\mu_0$ is the vacuum magnetic permeability. The condition that the magnet have "negligible size" means that its size is small compared to the wavelength $c/\omega$.
This is essentially equation (9.53) in
- Griffiths (1989), Introduction to Electrodynamics (Second Edition),
except that (1) is larger by a factor of $2$. That's because Griffiths considers a dipole that oscillates along the direction of $\mathbf{m}$ instead of a rotating dipole. To deduce (1) from Griffiths' result, a rotating dipole may be regarded as a pair of orthogonal oscillating dipoles, out of phase by $\pi/2$ so that the magnitude of the total magnetic moment remains constant.
The source of the torque: intuition
We know that the magnet must lose rotational kinetic energy to balance the energy that is carried away as radiation, because the total energy is conserved. But what mechanism slows the rotation?
I didn't find a detailed treatment in the literature, so I'm posting a relatively detailed answer here. I'll start with an intuitive preview, and then I'll show that the intuition has a solid theoretical foundation using an explicit model that includes Maxwell's equations.
We can think of the magnet as being made of lots of little magnetic dipoles distributed throughout the volume of the magnet. Consider one of these dipoles, say the one at $\mathbf{x}$.
These magnetic dipoles "want" to be aligned with each other. This can be verified empirically, using a pair of bar magnets. Therefore, any misalignment between the dipole at $\mathbf{x}$ and the magnetic field produced by the other dipoles results in a torque on the dipole at $\mathbf{x}$ that tends to restore its alignment with the others.
The dipoles interact with each other via the magnetic field. In the limit of a continuum of infinitesimal dipoles, it becomes correct to think of the magnetic field at $\mathbf{x}$ as being the superposition of the magnetic fields from all of the other dipoles. The model described below uses this idealization.
Here's the key: If the magnet is rotating, then the magnetic field at $\mathbf{x}$ is the superposition of delayed versions of the magnetic fields from the other dipoles, because any change in the field propagates at a finite speed (the speed of light). Thus the torque on the dipole at $\mathbf{x}$ tends to restore its alignment with where the rest of the magnet was at some time slightly in the past, a time that increases with the distance from $\mathbf{x}$. In other words, the torque tends to counteract the magnet's rotation, gradually slowing it down.
Intuitively, we expect the effect to be very small because the delay is very small, but does it have precisely the right magnitude to balance the energy that is carried away as EM radiation? Below, I'll show that it does. I'll do this by deriving the torque from a model that explicitly includes Maxwell's equations and that explicitly conserves the total energy.
Does the torque have the correct magnitude?
To derive the torque mathematically and show that it has the correct magnitude, I'll use a model defined by a lagrangian. The lagrangian describes a rigid magnet that can rotate about a given axis, and it describes the dynamcis of the EM field, in such a way that the total energy is conserved. In this model, the rotating magnet produces EM radiation via Maxwell's quations, so it must gradually lose rotational kinetic energy. The model will tell us exactly how this happens.
The model used here is non-relativistic (not Lorentz symmetric). In particular, it treats the magnet as a perfectly rigid object. This makes the math easier. Also, the non-relativistic approximation is consistent with the approximation that was used to derive (1): the rotation is assumed to be slow enough so that the wavelength is very long compared to the size of the magnet.
In units with $\epsilon_0=\mu_0=1$, the model is defined by the lagrangian
\begin{align}
L =\frac{1}{2}I\dot\theta^2
& + \int d^3x\ \frac{\mathbf{E}^2(t,\mathbf{x})
-\mathbf{B}^2(t,\mathbf{x})}{2}
\\
&
+\int d^3x\ \mathbf{M}\big(\theta(t),\mathbf{x}\big)
\cdot\mathbf{B}(t,\mathbf{x})
\tag{2}
\end{align}
with this notation:
$I$ is the magnet's moment of inertia. The rotation axis is such that $I$ is independent of $\theta$.
$\theta$ describes the magnet's time-dependent orientation, with time-derivative $\dot\theta$.
$\mathbf{M}(\theta,\mathbf{x})$ is (proportional to) the magnetic moment density at $\mathbf{x}$ when the magnet's orientation is $\theta$.
$\mathbf{B}=\nabla\times \mathbf{A}$ is the magnetic field in terms of the vector potential $\mathbf{A}(t,\mathbf{x})$.
$\mathbf{E}=-\mathbf{\dot A}$ is the electric field in the temporal gauge.
This lagrangian describes a perfectly rigid magnet that can move only by rotating about a given axis. The dynamic variables are the magnet's time-dependent orientation $\theta(t)$ and the electromagnetic vector potential $\mathbf{A}(t,\mathbf{x})$. The model doesn't have any prescribed time-dependent coefficients, so Noether's theorem ensures that it has a conserved total energy, which is given by
\begin{equation}
U = \frac{1}{2}I\dot\theta^2
+
\int d^3x\ \frac{\mathbf{E}^2+\mathbf{B}^2}{2}
-
\int d^3x\ \mathbf{M}\cdot\mathbf{B}.
\tag{3}
\end{equation}
The equations of motion derived from the lagrangian (2) are:
One equation that describes how the EM field reacts to
the magnet:
\begin{equation}
\mathbf{\dot E}=\nabla\times (\mathbf{B}-\mathbf{M})
\tag{4a}
\end{equation}
This is one of Maxwell's equations, with current density $\mathbf{J}\propto \nabla\times\mathbf{M}$. To derive (4), I assumed that $\mathbf{M}$ falls smoothly to zero at the "boundary" of the magnet, so that the boundary is not abrupt. This simplifies the equations without changing the insight. The definitions of $\mathbf{E}$ and $\mathbf{B}$ in terms of $\mathbf{A}$ imply another of Maxwell's equations:
\begin{equation}
\mathbf{\dot B}=-\nabla\times\mathbf{E}.
\tag{4b}
\end{equation}
Equations (4a)-(4b) together imply that EM waves propagate at a finite speed, which is equal to $1$ in the units I'm using here.
One equation that describes how the magnet's orientation
reacts to the EM field:
\begin{equation}
I\ddot\theta
=
\int d^3x\
\frac{\partial\mathbf{M}}{\partial\theta}\big(\theta(t),\mathbf{x}\big)
\cdot \mathbf{B}(t,\mathbf{x}).
\tag{5}
\end{equation}
The right-hand side of equation (5) is the source of the torque. The total energy (3) is independent of time if equations (4) and (5) are satisfied. The purpose of deriving these equations from the lagrangian (2) is to ensure that this conservation law holds, so that the rate at which the torque (5) drains the rotational kinetic energy precisely balances the rate at which energy is carried away as EM radiation through Maxwell's equations (4).
The next section explains how to relate these equations to the intuition that I described earlier.
Recovering the intuition from the math
We can think of the magnetic moment density $\mathbf{M}(\theta,\mathbf{x})$ as a continuous distribution of infinitesimal magnetic dipoles. As before, consider one of these dipoles, say the one at $\mathbf{x}$.
The right-hand side of equation (5) says that any misalignment between the dipole at $\mathbf{x}$ and the magnetic field at $\mathbf{x}$ contributes to a torque that tends to restore their alignment. To see this, recall that the magnitude of $\mathbf{M}$ at each point is fixed because the magnet is rigid. This implies that $\partial \mathbf{M}/\partial\theta$ is orthogonal to $\mathbf{M}$. Therefore, if $\theta$ is such that $\mathbf{M}$ is already aligned with $\mathbf{B}$, then the right-hand side of (5) is zero (no torque). On the other hand, if $\theta$ is slightly larger than this equilibrium value, then the right-hand side of (5) is negative, because increasing $\theta$ even further would only makes the alignment worse. (Think about the definition of the derivative.) We can use that picture to confirm that the sign of the right-hand side of (5) is what we need it to be: the torque tends to restore the alignment between $\mathbf{M}$ and $\mathbf{B}$.
Equation (4a) implies that the magnetic field at $\mathbf{x}$ is the superposition (integral) of the magnetic fields from all of the other dipoles. In a situation with no motion, this means that the torque in equation (5) tends to align the dipole at $\mathbf{x}$ with the other dipoles. This is consistent with our experience with macroscopic bar magnets. The different internal torques at different points $\mathbf{x}$ cancel each other, so that the net torque on the whole magnet is zero.
Equations (4) imply that any change in the field due to a change in the magnet's orientation propagates at a constant finite speed. Therefore, if the magnet is rotating, the magnetic field at $\mathbf{x}$ is the superposition of delayed versions of the magnetic fields from the other dipoles, with each delay proportional to the distance to $\mathbf{x}$. This completes the justification of the intuition that I described before, and it shows that the effect has precisely the right magnitude to balance the energy that is carried away by EM radiation. This conclusion follows because we derived the effect from equations (4)-(5), which manifestly conserve the total energy (3).
