What happens is that the pendulum's effective length, $\ell_e$, increases for a time because the center of mass is moving further out as the mercury flows out of the pendulum bob. Since the period is proportional to the square root of the length, $T\propto \sqrt{\ell_e}$, then the larger effective length means a larger period. Eventually, the bob would be drained of the fluid and the effective length returns back to where it was at the beginning (when the bob was filled), so the period returns to what it used to be (decreases).
Modeling this using Newtonian mechanics might be a little difficult as the length of the pendulum and the mass of the bob are changing with time. An alternative would be to look at the Lagrangian approach in which you consider the energies (kinetic and potential): $$ L=T-V=\frac{1}{2}m(t)\left(\dot{\ell}^2+\ell(t)^2\dot{\theta}^2\right)+mg\ell(t)\cos\theta $$ This leads to the equations of motion, \begin{align} \frac{\mathrm d}{\mathrm dt}\left[m(t)\dot{\ell}\right]-m(t)g\cos\theta&=0 \\ \frac{\mathrm d}{\mathrm dt}\left[m(t)\ell(t)^2\dot\theta\right]+m(t)g\ell(t)\sin\theta&=0 \end{align} Note that since we have $m(t)$ being differentiated with respect to time, we cannot simply "divide it out" the mass as suggested by other answers without making a further assumption:
- Since you're told that $\mathrm dm/\mathrm dt=\alpha$, then $m(t)\sim m_0+\alpha t$ where $m_0$ is the initial mass of the bob (fluid + shell). If you assume that $m_0\gg\alpha t$ (i.e., mass loss is negligible), then you can factor out the $m$.
Under this assumption, and using the small angle approximation, we have a singlethe differential equationequations: $$ \frac{\mathrm{d}}{\mathrm{d}t}\left[\ell(t)^2\frac{\mathrm{d}\theta}{\mathrm{d}t}\right]+g\ell(t)\theta=0,\tag{1} $$\begin{align} \frac{\mathrm d}{\mathrm dt}\left[\frac{\mathrm d\ell}{\mathrm dt}\right]-g\left(1-\frac{1}{2}\theta^2\right)&=0\tag{1}\\ \frac{\mathrm{d}}{\mathrm{d}t}\left[\ell(t)^2\frac{\mathrm{d}\theta}{\mathrm{d}t}\right]+g\ell(t)\theta&=0,\tag{2} \end{align} whichthe latter of the two is a Sturm-Liouville equation. The solution of this depends on the actual function for the effective length, $\ell(t)$. One option would be to assume the center of mass moves linearly with time, $\ell(t)=\ell_0+\beta t$ (at least until $\beta t\geq r$, then it is just $\ell_0$ because this means the mass has drained fully from the bob). You could also integrate the volume to find the center of mass of fluid, $$ r_\text{CoM}=\frac{\int r\rho\mathop{}\!\mathrm{d}V}{\int \rho\mathop{}\!\mathrm{d}V}, $$$$ r_\text{CoM}=\frac{\int z\rho\mathop{}\!\mathrm{d}V}{\int \rho\mathop{}\!\mathrm{d}V}, $$ and add this to the initial length of the rod, $\ell_0$. Of course, assuming that $\ell_0\gg r$ would bring us back to the "uninteresting" case of the standard pendulum since the length would be roughly independent of time.
I have not tried proceeding further, but I imagine that, given appropriate boundary conditions, Eq. (1) can be solved; however, it could be the case that numerical methods are required.1
1. It may be the case that the Hamiltonian formalism provides a nicer pairing of functions, $\mathrm dp/\mathrm dt=f(t,\,\theta)$ and $\mathrm d\theta/\mathrm dt=g(t,\,p)$, for numerically integrating.