Feynman meant that conservation of energy always holds, so that if you have a static situation, the force field on a particle is conservative. For magnetic forces, you have moving (and changing) currents in the solenoid, so its not static, and if you extract energy from the field, you just weaken the current and extract energy from the system producing the field, doing work on it.
The fact that magnetically induced EMF is non-conservative is the basis of countless claims of perpetual motion machines, so it is good to say early that you can't do this.
Magnetic fields that are changing give rise to non-conservative forces, the integral around a loop is the change in flux inside, but the process of extracting energy from the EMF reduces the magnetic field, and the amount of energy stored in it.
Feynman discusses transformers and the EMF around a loop. He also discusses something else even more counterintuitive and not at all discussed by other people. He shows two moving charges, A and B, so that A is moving perpendicular to the line joining A and B and B is moving along the line joining A and B.
In this case, the force from A on B is not equal and opposite to the force from B on A! This shows you that the (nonradiative) field is carrying momentum, and is transferring momentum to the two charges as the E and B fields rearrange. The recognition that you need to include fields in the conservation laws was long in coming, and this example is just as useful as the transformer for explaining this. Feynman also discusses a case where the field is carrying angular momentum, a collection of charged balls with a current, and when you switch off the current, the balls start to rotated around.