Note: This answer is probably cheating from a logical perspective, since I think that Feynman is trying to use his example to justify the principle I am using. However, I think it can be useful pedagogically to have the argument laid out in energy terms, and then go back to Feynman's exposition later to see exactly how he reaches the same conclusion without needing to directly use the concept of energy.
As a refresher, the work needed to raise a mass $m$ by height $h$ is $mgh$.
Let's say the "unit mass" has mass $m$, and "unit distance" is $h$. Then let's say that Machine A raises a mass $3m$ by distance $X$, and Machine B raises a mass $3m$ by distance $Y$.
We will use the following basic principle. The net change in internal energy of the system, is equal to the total work done on the system.
\begin{equation}
\Delta U = W
\end{equation}
In fact, for this reversible machine, we will always have that the net work done on the system is $0$. (I can go into more detail about this if needed).
Let's consider Machine A. Initially, Machine A consists of a mass $m$ at height $h$, and a mass $3m$ at height zero. Therefore the initial energy is $U_i=mgh$. We then lower the little mass, and raise the larger mass to a height $X$. The net work that is done on the system is zero. The final energy of the system is $U_f=3mgX$. (Of course this implies that $X=h/3$, but let's pretend we don't know this since it would lead us to conclude that $Y=X$ "too early" to follow Feynman's argument.)
Similarly, if we consider Machine B, then if we use the little mass to raise the 3 masses, the initial energy is $U_i=mgh$ and $U_f=3mgY$.
Now suppose $Y>X$. Initially, the energy is $mgh$. Then we use Machine B to raise the three masses to height $Y$, so after the three masses are in the air, the energy of the system is $3mgY$. The final step is to use Machine A to lower the three masses. At the end of this process, the little mass $m$ will be raised to a height $H$. Now, you should expect that $H=h$ -- that is, the little mass $m$ should be at the same height at the end of the cycle, as when we started. What else could the answer be? But this cannot be the case. We have previously seen for Machine A that it only takes $3mgX$ worth of energy to raise $m$ to height $h$. On the other hand, we have also said that the energy in the system is $3mgY > 3mgX$. Therefore, Machine A must end up raising the mass $m$ to a height $H>h$, as we lower the three masses to the ground. However, this implies that the final energy is $mgH>mgh$. By going through a cycle we have ended up with more energy than we started with, despite doing no work on the system. This contradicts the basic principle $\Delta U=W$.
This fact can be exploited, since we can now extract work $W=mg(H-h)$ from the system repeatedly by performing this loop over and over. By simply raising and lowering a weight on the magic seesaw formed by joining Machines A and B, we can extract work. Say we attach a piston to the seesaw, then we can raise and lower the weights and spin a wheel, to power a motor, to turn on a light.
