I'd guess it has to do with the structure of helium-3 allowing for greater Coulomb repulsion between the protons, but I'm unsure.
The Coulomb energy goes like $Z(Z-1)$, so it's typically a very small effect for light nuclei. The Coulomb energy would be responsible for the difference in binding energy between 3He and 3H.
In general, even-even nuclei are always more bound than odd nuclei. This is due to pairing.
Apart from pairing, one can usually predict nuclear binding energies quite well at a quantitative level by considering them to be the sum of a liquid-drop energy and a shell correction (Strutinsky 1968). The shell correction for 4He is large, because it's doubly magic. (I don't actually know if the Strutinsky works well for such light nuclei, but it's definitely good enough to give a correct qualitative explanation.)
“Shells” in deformed nuclei. V. M. Strutinsky, Nucl. Phys. A 122 no. 1 (1968) pp. 1-33.
See also: Curvature Correction in the Strutinsky's Method. P. Salamon, A. T. Kruppa. J. Phys. G: Nucl. Part. Phys. 37 no. 10 (2010) 105106. arXiv:1004.0079. (A variation on the technique, describes the technique itself.)
Helium-3 has less of a binding energy because the strong force is less as there are less nucleons. Not only that the electromagnetic force sort of wins over slightly. This is because in Helium-3 there are 2 protons and one neutron. However in Helium-4 there are 2 protons and also 2 neutrons. Also all the spins for the nucleons are perfect.
-one spin up proton
-one spin down proton
-one spin up neutron
-one spin down neutron
Also more nucleons means stronger strong force especially up to Iron.