Here is an intuitive / qualitative answer. Maybe someone else will add some math.
I wonder if it's instructive to look at diamond cleaving. As you know, diamond is extremely hard, and conventional machining is very difficult. But if you can find the right fracture plane ((111) and its symmetrical cousins), it's possible to cleave the diamond along that plane with relatively little power. This is usually done with a well aligned hardened blade, a laser-scored line to initiate fracture, and a sharp tap with a hammer. The force may be high, but the blade moves only a small distance - and the result is that the diamond cleaves along its fracture plane with relatively little work done.
So why would it be different for aluminum? Normally when a crack propagates, something interesting happens at the tip - and exactly what happens is a strong function of the rate of strain. In the case of the diamond above, the strain rate is extremely high - and the material has no chance to respond other than by breaking. In contrast, at lower strain rates you tend to get elastic deformation at the crack tip before fracture: that is, you don't just break the surface bonds, but you do a significant amount of work moving the atoms near the fracture surface around. The crack tip tends to become blunt because of this, and that only increases the volume of material that is being worked by the cutting.
So you don't just break the surface bonds - you actually induce friction and interatomic motion on a scale that goes much deeper than the surface. Exactly how much deeper depends on the material and the strain rate - but it results in work that is many times greater than you would derive from the surface energy.
Some evidence that my hypothesis might be correct is given in this article - which includes the following text:
We find partial dislocations are nucleated at the crack tip and moved away from the crack tip under stress. These partial dislocations blunt the crack tip and render a ductile fracture in the absence of H impurities.
I think that mechanism requires much more energy than a simple cleaving along the plane.