There is a torque, but it points sideways, perpendicular to the orientation of the hammer at any time. Because of this, as the hammer rotates, the direction of the torque also rotates around with it. The torque only acts to rotate the system horizontally around in space, not to change the direction of its angular velocity.
Let's see this with a calculation. Suppose I model the hammer as a rod of length $L$ and mass $m_r$ with a point mass $m_p$ on the end.
The moment of inertia of the hammer at this moment can be computed by taking the moment of inertia of a similar configuration aligned along the $x$ axis and rotating it by an angle $-\theta$ in the $y$ direction:
$$\begin{align}I &= R_y^{-1}\begin{pmatrix}0 & 0 & 0 \\ 0 & \frac{mL^2}{3} + m_pL^2 & 0 \\ 0 & 0 & \frac{mL^2}{3} + m_pL^2\end{pmatrix}R_y \\
&= \begin{pmatrix}ML^2\sin^2\theta & 0 & -ML^2\sin\theta\cos\theta \\ 0 & ML^2 & 0 \\ -ML^2\sin\theta\cos\theta & 0 & ML^2\cos^2\theta\end{pmatrix}\end{align}$$
where $R_y$ is the rotation matrix around the $y$ axis and $M = \frac{m_r}{3} + m_p$. Computing the angular momentum using $\vec{L} = I\vec\omega$, where $\vec\omega = \omega\hat{z}$, I get
$$\vec{L} = ML^2\cos\theta(\hat{z}\cos\theta - \hat{x}\sin\theta)$$
The torque, on the other hand, is
$$\vec\tau = \vec{r}\times\vec{F} = (\hat{x}\cos\theta - \hat{z}\sin\theta)\times (-mg\hat{z}) = mg\hat{y}\cos\theta$$
So the torque actually pushes the angular momentum in a direction perpendicular to both its direction and the orientation of the hammer. This means there will be no change in the amount of angular momentum. It also means that the hammer rotates along with everything else, so that the angular momentum and the momenta of inertia preserve their relative orientation and thus there will be no change in its angular velocity.
