Intuitively, why is a reversible process one in which the system is always at equilibrium? A process is reversible if and only if it's always at equilibrium during the process.  Why?
I have heard several specific example of this, such as adding weight gradually to a piston to compress the air inside reversibly, by why should it be true in general?
EDIT:
Here is something that would firmly convince me of this:
Suppose I have a reversible process that is not always in equilibrium.  Describe a mechanism for exploiting this process to create a perpetual motion machine.
 A: I guess the simplest answer is just to carefully read you own words again. A reversible process is the one that can be made flow backwards. It is intuitive to think that it can be made flow backwards at any time we wish. But if the system were in a non-equilibrium state, one would need to wait a bit until it goes to equilibrium before trying to drive it back. So, it does not satisfy our desire to have the system under the control at any time.
A: Because a reversible process must not increase the entropy in the system. Any change due to lack of equilibrium will lead to an increase in entropy, and thus irreversibility. 
You can visualize this by thinking of two bottles, one full of gas and one with vacuum. If the bottles were to be connected, atoms would randomly migrate between the two bottles leading to a system with the gas divided between the containers with equal pressure. 
Thus, the disorder of the system has increased, because the atoms that were all sorted in one bottle initially are now subdivided in two partitions of a much larger volume. In other words, the lack of equilibrium in pressure leads to an uniformization of the same, and to an irreversible increase in entropy.
A: A process is reversible if and only if there is not production of entropy. If you perform a process quasi-statically, you are minimizing the production of entropy (e.g. by adding infinitesimal weight to a piston to compress the air) and the whole process can be considered, at one good approximation, reversible
A: There are two ways to see this.  A reversible process is the idealised limit of irreversible processes which go slower and slower.  In the limit, a reversible process goes "infinitely slowly" (this is really a phrase used in some thermodynamics texts).  I.e., it does not move at all.  So the points have to be at equilibrium, or else they would be moving.  When we call a process "reversible", we mean that it could go either way (but only if one disturbs the external conditions infintesimally, something which in fact alters the conditions under which the process is defined to be reversible, making it irreversible).  But how could the system decide which way? It can't, of course, so in fact it does not move at all.  It would only move if one altered the outside conditions a little bit, favouring one direction or the other, and thus making the process irreversible.  
The other way to see this is by definition of an equilibrium point: an equilibrium point means that every real process which connects to that point must be leading in: none of them can be leading out.  If a real, i.e., irreversible process started at a point and led away, the point would not be in equilibrium since that process would start going by Carnot's theorem.  
A: As I am new to Stack Exchange, I happened to see the question only now and mine is a belated answer. If Mark is still interested in an answer, then: 
It is a good question gaining added strength with the EDIT.
For thermodynamic analysys, a process must connect two equilibrium states A, B of a system. Take a state C of the system on the way from A to B. We can now consider the process A to C or C to B. To qualify for such a consideration, C must satisfy the condition that it is an equilibrium state of the system. C being chosen arbitrarily, it follows that every state of the sysyem along the path A to B must be an equilibrium state. 
Coming to the EDIT:
To simplify the discussion, let us assume the syatem to be an adiabatic system. Suppose A is an equilibrium state and B is not. Then, let the system go from A to B reversibly (if possible), then the entropy of the system (and of universe in this case) decreases. Therefore, the process B to A becomes a spontaneous process. We can then employ the spontaneous process B to A to do work for us. Therefore, we let the process go reversibly from A to B, then let it go from B to A spontaneously, doing work for us. We can repeat this process endlessly and perpetually extracting work (energy) from nothing! We thus achieve perpetual motion if an adiabatic system goes from an equilibrium state A to a non equilibrium state B reversibly.
Radhakrishnamurty Padyala
