What if the point of application of force doesn't move but the force itself moves, does it count as work? For example, If I rub my finger on a table (which doesn't move due to friction from ground), does friction between my finger and table does work (positive or negative) on it?
Or, a even simpler example - A block is moving, surely the friction is doing (negative) work on the block. But what about the ground? Although it doesn't move but the friction force itself is moving, does it count as work?
 A: Corrected answer as a result of the comments made by @ValterMoretti
I have found it easier to understand what is happening by applying the constant kinetic frictional forces on the “sides” of the surface and the block rather than on top or underneath.
I think that this way of presentation helps to understand the displacements which are involved.
When discussing or answering such question the first thing to do is to define the system under consideration.  

Consider a block as the system and it is sliding along a rough horizontal surface with an external  kinetic frictional force $-f \hat i$ acting on the block due to the surface.
If the displacement of the block is $+y\hat i$ then the work done on the block by the frictional force is $-fy$ and if you were using the work-energy theorem this would produce a change (reduction) in the kinetic energy of the block.  
Consider a surface (and the Earth) as the system.
There is an external kinetic frictional force $+f \hat i$ acting on the surface due to the block.
However the displacement of that frictional force is now $x \hat i$ and so the work done on the surface by that frictional force is $+fx$.
This produces an increase in the kinetic energy of the surface but because the displacement of the surface is less than that of the block, the gain in kinetic energy of the surface is less than the decrease in kinetic energy of the block.
The difference is the heat produced which enters both bodies.
The difference in the displacements can be shown as follows.
Assume that the mass of the surface (and the Earth) is greater than the mass of the block.
The magnitude of the impulse $f \Delta t$ on the surface and the block must be the same so the magnitude of the change in momentum of the surface and the block must be the same.
Since the mass of the surface is more massive than the block the magnitude of the acceleration of the surface must be less than that of the block.
As the interaction times $\Delta t$ are the same for both the surface and the block  the velocity vs time graphs for the two objects show the displacement of the surface (area under graph) to be less than that of the block.  
 
In practice because the Earth is so massive the displacement $x$ is taken to be zero.  

A good example of the point of contact not moving and work done by a static frictional force is you sitting on a seat with no back rest in a car which is accelerating.
Now in this case the displacement $x$ of the two static frictional forces on you and on the car seat is the same.
You and the car seat accelerate but whilst doing so you do not move relative to the car seat because there is a static frictional force between you and the car seat.  
If you are the system then you have an external static frictional force acting on you $+f\hat i$ due to the car seat and after a displacement of $x \hat i$ that static frictional force does $fx$ amount of work on you.
Your kinetic energy increases.  
At the same time there is $-fx$ amount of work done on the car seat by the static frictional force that you exert on the car seat.  
A: The work of the friction force on the ground is zero because the application point has zero velocity even if it continuously changes. So, it is not the work of the ground on the block up to its sign!
(This is the reason why a perfect non-sliding wheel due to the presence of frictional force caused by a perfectly flat ground would run forever: The contact point (just one point because the wheel and the ground are perfect) of the wheel on the ground has zero speed and continuously changes. This way there is no work of the ground on the wheel and it preserves its kinetic energy in spite of the presence of a friction force.)
To handle these generalized cases one may use the general form of work-kinetic energy theorem of continuous mechanics 
$$\frac{d}{dt} \int_{V_t} \frac{1}{2} \mu(t,{\bf x}) {\bf V}(t,{\bf x})^2 d^3x = \int_{V_t} \mu(t,{\bf x}) {\bf f}(t,{\bf x})\cdot  {\bf V}(t,{\bf x}) \: d^3x+ \int_{\partial V_t}  {\bf s}(t,{\bf x})\cdot  {\bf V}(t,{\bf x})\: dS\:.\tag{1}$$
The left-hand side is the time derivative of the kinetic energy of a portion  of continuous body $V_t$ evolving in time, preserving the number of particles along its story, with mass density $\mu(t,{\bf x})$ and velocity field ${\bf V}(t,{\bf x})$.
The two integrals in the right-hand side are the instantaneous power of the volume density of force ${\bf f}(t,{\bf x})$ at time $t$ and applied on ${\bf x}\in V_t$ and of the surface density forces ${\bf s}(t,{\bf x})$ at time $t$ and applied on ${\bf x}\in \partial V_t$ respectively. 
You see that (1) is valid at time $t$ and it does not matter if application points of forces  "change". 
In the case of the block moving along the rigid ground at rest in a reference frame, $V_t$ is that piece of ground with ${\bf V}(t,{\bf x})=0$ and the friction force of the block on the ground can be represented as a surface density of force ${\bf s}(t,{\bf x})$ which, at fixed $t$, is different from $0$ only in a surface region  given by the surface of contact of the block and the ground. This region changes in time and this change is accommodated by the time dependence of ${\bf s}(t,{\bf x})$. However, the last integral in (1) vanishes because ${\bf V}(t,{\bf x})=0$ by hypothesis. 
The total surface works due to the friction force of the block on the ground is 
$$W_s = \int_{t_1}^{t_2}  dt\int_{\partial V_t}  {\bf s}(t,{\bf x})\cdot  
{\bf V}(t,{\bf x})\: dS =0\:.$$
A: You can look at the work of each half separately. You finger, since it's moving and the force is in the opposite direction is doing work. The table, since it's not moving doesn't receive any work.
When a process produces heat, and the work in doesn't match the work out, it's often characterized by an efficiency. So, in this case, your finger is doing work on the table with a 0% efficiency.
