The work function of gold (5.2 eV) is higher than silver (4.4 eV).
Does it mean, removing an electron from the surface of gold needs more energy and is difficult than silver?
The work function corresponds to the energy that has to receive an electron of the material during an interaction to free itself from the material.
In practice, it means in the case of the photo-electric effect that the energy of a photon hitting the surface has to be above the work function :
However, the number of photons (so the intensity of the light) does not have to be important to enable the photo-electric effect.
So, in a word, it is harder in the sense that higher work functions require light of higher frequencies but not necessarily of high power.