How can friction do no work in case of pure rolling? I have read various answers, on PSE and elsewhere, and most of them explain that the point of contact of the rolling object undergoes no instantaneous displacement in the direction of friction, I agree, but then there is a feeling that the friction does provide a torque, so how can it do no work> (The pure rolling here is under a constant external force on the object so friction does act ).
 A: Yep you're right. 
Consider the case of a cylinder rolling down a hill, starting from rest. 
Friction does provide a torque and definitely does work on the cylinder. 
However, in the case of pure rolling motion, i.e no slipping, the work done by friction = gain in rotational KE. So we can say that the change in gpe = change in total KE. 
It therefore appears based on that conservation statement that the frictional force does no work. 
 
Edit: Note delta KE means the change in linear KE, i.e $mv^2/2$
A: In a scenario of pure rolling of a rigid wheel on a flat plane, you don't need any friction. Once the wheel is rolling, it will continue to do so, even if the friction coefficient becomes zero. If this were not the case, you'd be violating conservation of angular momentum. There is no force, no torque and therefore no work done.
The more interesting case is that where the object is rolling under an external force, say down an inclined plane. See the diagram below

Now, you can analyze it in two ways. One is similar to Farcher's answer where the point of contact moves perpendicular to the frictional force and hence, there is no work done. But you were interested in it from the point of view of torques (where we consider the whole wheel, not just the point of contact) so let's do that.
Friction does two things to the wheel as a whole. It does negative work when you look at the linear motion of the wheel. Indeed,
$$W_1 = -f.dS,$$
where $f$ is the force of friction and the wheel has moved a linear distance $dS$. Next, the friction provides a torque about the center of the wheel and the wheel has angular displacement. Hence, it does positive rotational work i.e.
$$W_2 = \tau. d\alpha,$$
where $\tau$ is the torque and $d\alpha$ is the angular displacement of the wheel. But note that $\tau = fR$ and $R d\alpha = dS$. Hence $W_2 = f.dS$ and you get
$$W_{tot} = W_1 + W_2 = 0$$
A: 
The pure rolling here is under a constant external force on the object so friction does act.

As an example consider an object rolling, without slipping, down an inclined plane.
There will be a frictional force $F$ acting, at the point of contact between the object and the inclined plane, upwards and parallel to the slope as an object is rolling down an inclined plane as well as the weight of the object $mg$ and the normal reaction $N$ both of which act through the centre of mass of the rolling object.    
If you consider the motion of a point on the rolling object its path is a cycloid.
The image below is taken from the Wikipedia article on the cycloid and I need you to imagine that the black line at the bottom is inclined to the horizontal and thus representing the inclined plane. 

At the point of contact the cycloid is perpendicular to the inclined plane ie along the line $XC$ where $X$ is the point of contact and $C$ the centre of mass of the rolling object.  

This means that the direction of motion of the point of contact is perpendicular to the frictional force and so the frictional force does no work on the object.  
It terms of the torque about the centre of mass $C$ because the motion of the point of contact is towards the centre of mass of the object there is no rotation of the torque about the centre of mass and so no work is done by the frictional torque.
A: If you have a solid wheel rolling on a perfectly smooth plane and the material of the wheel and the surface have zero elasticity, no work is done . 
A: If you have a perfectly spherical cow of uniform density and zero elasticity rolling on a smooth, flat, perfectly rigid plane in a vacuum, and no external force, then no work is done.
In your question, adding an external force changes the equation, such that the linear acceleration applied by that external force is partially (but negligibly) countered by the change in rotational speed, due to friction applying torque.
As with most spherical cow analogies, the calculations are good enough in most circumstances.  A baseball's flight can be predicted fairly accurately by taking just a point-object ballistic trajectory through a vacuum most of the time. Throwing a crumpled ball of paper, though, you'll need to take drag into account.
The same with rolling objects. For most calculations, you can ignore friction, because it will be negligible compared to other forces, such as slope.  If you reduce the rigidity of either the object or the plane it's rolling in, or add other external forces though, then friction starts to matter.
