Say a person applies 1 N to a box with a mass of 1 kg, displacing 1 m. This is one Joule of work (1 N for 1 m).
Now say the person applies 1 N to a box with double the mass, displacing 1 m as well. This is still one Joule of work, despite the person having to push with the same amount of force for longer.
So the person expended more energy (pushing with the same effort for longer), but does the same amount of "work". How can the joule be used to measure energy then?