The answer is that a big engine is less efficient at low power than a small one, because it has to lug around a lot of extra stuff (bigger cylinders, more cylinders, bigger and more bearings and so on), all of which involve frictional and other losses, which the smaller engine does not have.
This is really an engineering question rather than a physics one however: I think a physicist's answer would be that a big engine could be as efficient at low power as a small one: for instance make a big engine by strapping two small ones together, and at low power just do not run half of the cylinders. That's fine in theory but an engineer will point out that you still pay the frictional costs (I think some modern cars more-or-less do do this though).
Edit. Actually, I think that if you consider a big engine running at low power but not doing any clever tricks (see below), then it necessarily has to run with a lower temperature differential, and thus has lower best-case thermodynamic efficiency than a smaller engine running much hotter. So I was wrong: there is a physics reason, although engineering can get around some of it.
'Clever tricks' mean things like not running some of the cylinders at all, or firing only every nth stroke, or so on. Strapping two smaller engines together as I suggested above is such a trick, of course.