Ultimately, the Navier-Stokes equations explain this :)
OK, that's not a useful answer: here's how they explain the phenomenon in some cases. Under steady state conditions for a fluid (inviscid, incompressible) that doesn't differ too much from a cup of tea, the Vorticity Transport Equation shows that the vorticity $\omega = \nabla \times \vec{v}$ (the curl of the velocity $\vec{v}$) tends to nought in the steady state, as discussed in more detail in this answer here.
So now we have a potential flow; since $\nabla\times\vec{v}\approx \vec{0}$ we can set $v = -\nabla \phi$ where $\phi$ is a potential function and, since the continuity equation for an incompressible, steady state flow behests that $\nabla\cdot\vec{v}=0$, we must have $\nabla^2\phi=0$. So we now search for an axisymmetric solution to Laplace's equation with concentric circles for flow lines. The complex potential for a 2D vortex is (See the Wikipedia page for "Potential Flow" ):
$$\Omega(z) = \frac{\Gamma}{2\,\pi\,i}\,\log z$$
where $z = x+i\,y$ is the 2D position in the flow and $\Gamma\in\mathbb{R}$ the circulation. The implied velocity field (as a complex number field) is:
$$V(z) = (\mathrm{d}_z \Omega)* = -\frac{\Gamma}{2\,\pi\,i\,z^*} = \frac{\Gamma}{2\,\pi\,r} i\,e^{i\,\theta}$$
where we now write the position vector in polar co-ordinates $(r,\theta)$ i.e. the velocity (in the direction of $i\,e^{i\,\theta}$ is at right angles to the position vector $r\,e^{i\,\theta}$, and the streamlines are concentric circles centred at the origin.
This of course goes a bit haywire at $r=0$ and of course the description breaks down for small values of $r$, but you can see it describes pretty well the phenomenon you see in your cup of tea.
So let's summarise the physics that is expressed in the above:
The fluid evolves into a state where fluid particles have zero spin angular momentum and $\nabla\times\vec{v}=0$; they can of course have orbital angular momentum, as in a whirlpool;
Given incompressibility, conservation of mass is expressed at steady state by $\nabla.\vec{v}=0$;
These two conditions force a a potential flow: $\vec{v}=-\nabla\phi$ where $\phi$ is harmonic;
Lastly, the problem's circular symmetry defines a unique harmonic function.