So you need to do work to change the external pressure at every point as well which would be equal to the work done by the gas in expansion.

It is in this statement where you are going wrong.

Let's say you have the gas in a vertically oriented frictionless cylinder and piston. On top of the piston you have a number of weights which, in addition to the external air pressure, provides the external pressure on the system. You now slide a little weight horizontally off of the top of the piston. That reduces the external pressure allowing the gas to expand and perform work in raising the remaining weight.

Sliding the weight horizontally off the piston would theoretically require very little effort (work) compared to the work done by the gas in raising the remaining weight. In order to carry this process out extremely slowly (reversibly) we can imagine the weight as a pile of sand. We remove the sand one grain at a time causing an infinitely small reduction in external pressure, infinitely small transfer of heat into the gas, and an infinitely small increase in volume of the gas making the process quasi-static and reversible.

Hope this helps.

So you need to do work to change the external pressure at every point as well which would be equal to the work done by the gas in expansion.

It is in this statement where you are going wrong.

Consider the following thought experiment. (See diagrams below). Let's say you have the gas in a vertically oriented frictionless cylinder and piston. On top of the piston you have a number of weights which, in addition to the external air pressure, provides the external pressure on the system. You now slide a little weight horizontally off of a platform connected to the shaft of the piston and onto another platform along side the cylinder. That reduces the external pressure allowing the gas to expand and perform work in raising the remaining weight.

Sliding the weight horizontally off the piston would theoretically require very little effort (work) compared to the work done by the gas in raising the remaining weight. In order to carry this process out extremely slowly (reversibly) we can imagine the weight as a pile of sand. We remove the sand one grain at a time causing an infinitely small reduction in external pressure. That results in an infinitely small expansion of the gas, $Adh$. The expansion causes an infinitely small decrease in the gas temperature $dT$ and infinitely small transfer of heat $dQ$ into the gas to bring its temperature and pressure back into equilibrium with surroundings, awaiting the next removal of a weight.

Finally, for the process to be reversible, we must return both the system and the surroundings exactly to their original state. Note that after the last weight is removed (top of figure to the right), the platform is above the last weight that was removed. The requires us to take an additional weight from somewhere in the surroundings and place it on our platform to begin the reverse process. The obvious choice in order to return the system to its original state (pressure and volume) is to take the first weight that was removed and raise it to the top of the platform. At the completion of the reversed process the system (gas) has been returned to its original state but the surroundings has been altered as it had to do work to raise the first weight. This demonstrates that in order for the process to be reversible the weights must be infinitesimally small.

Hope this helps.