The work of the friction force on the ground is zero because the application point has zero velocity even if it continuously changes. So, it is not the work of the ground on the block up to its sign!
(This is the reason why a perfect non-sliding wheel due to the presence of frictional force caused by a perfectly flat ground would run forever: The contact point (just one point because the wheel and the ground are perfect) of the wheel on the ground has zero speed and continuously changes. This way there is no work of the ground on the wheel and it preserves its kinetic energy in spite of the presence of a friction force.)
To handle these generalized cases one may use the general form of work-kinetic energy theorem of continuous mechanics
$$\frac{d}{dt} \int_{V_t} \frac{1}{2} \mu(t,{\bf x}) {\bf V}(t,{\bf x})^2 d^3x = \int_{V_t} \mu(t,{\bf x}) {\bf f}(t,{\bf x})\cdot {\bf V}(t,{\bf x}) \: d^3x+ \int_{\partial V_t} {\bf s}(t,{\bf x})\cdot {\bf V}(t,{\bf x})\: dS\:.\tag{1}$$
The left-hand side is the time derivative of the kinetic energy of a portion of continuous body $V_t$ evolving in time, preserving the number of particles along its story, with mass density $\mu(t,{\bf x})$ and velocity field ${\bf V}(t,{\bf x})$.
The two integrals in the right-hand side are the instantaneous power of the volume density of force ${\bf f}(t,{\bf x})$ at time $t$ and applied on ${\bf x}\in V_t$ and of the surface density forces ${\bf s}(t,{\bf x})$ at time $t$ and applied on ${\bf x}\in \partial V_t$ respectively.
You see that (1) is valid at time $t$ and it does not matter if application points of forces "change".
In the case of the block moving along the rigid ground at rest in a reference frame, $V_t$ is that piece of ground with ${\bf V}(t,{\bf x})=0$ and the friction force of the block on the ground can be represented as a surface density of force ${\bf s}(t,{\bf x})$ which, at fixed $t$, is different from $0$ only in a surface region given by the surface of contact of the block and the ground. This region changes in time and this change is accommodated by the time dependence of ${\bf s}(t,{\bf x})$. However, the last integral in (1) vanishes because ${\bf V}(t,{\bf x})=0$ by hypothesis.
The total surface works due to the friction force of the block on the ground is
$$W_s = \int_{t_1}^{t_2} dt\int_{\partial V_t} {\bf s}(t,{\bf x})\cdot
{\bf V}(t,{\bf x})\: dS =0\:.$$