I'd guess it has to do with the structure of helium-3 allowing for greater Coulomb repulsion between the protons, but I'm unsure.
The Coulomb energy goes like $Z(Z-1)$, so it's typically a very small effect for light nuclei. The Coulomb energy would be responsible for the difference in binding energy between 3He and 3H.
In general, even-even nuclei are always more bound than odd nuclei. This is due to pairing.
Apart from pairing, one can usually predict nuclear binding energies quite well at a quantitative level by considering them to be the sum of a liquid-drop energy and a shell correction (Strutinsky 1968). The shell correction for 4He is large, because it's doubly magic. (I don't actually know if the Strutinsky works well for such light nuclei, but it's definitely good enough to give a correct qualitative explanation.)
“Shells” in deformed nuclei. V. M. Strutinsky, Nucl. Phys. A 122 no. 1 (1968) pp. 1-33.
See also: Curvature Correction in the Strutinsky's Method. P. Salamon, A. T. Kruppa. J. Phys. G: Nucl. Part. Phys. 37 no. 10 (2010) 105106. arXiv:1004.0079. (A variation on the technique, describes the technique itself.)