Consider a deformable bar, fixed at one end and acted upon by a load P (gradually increasing), as shown, through a rigid plate attached to its end. At the end of loading, potential energy is stored in the internal resistive force field (which was developed during loading). If at the end of loading I take any element as shown, it will be acted upon by equal and opposite internal resistive forces.
Now let's say we gradually decrease the applied load, P. The internal forces on the particle considered will also keep on reducing and the particle will move towards the left. During its 'left' journey some work will be done by the forces $F_1$ and $F_2$ and these works will be equal and opposite in sign so that the net work done on the particle is zero during its entire 'left' journey. The force $F_1$ does a positive work thereby trying to increase the particle's kinetic energy, but at the same time, $F_2$ does a negative work so that the energy (that otherwise would've been the kinetic energy of the particle) is sucked out.
The potential energy stored in the force field is not going to the particle (since net work done on it is zero) so where does it go?