Typically, a laser will damage an optical surface in one of two ways. The first is just what you would expect: the laser heats the material up until something bad happens. The second is also pretty simple, but less common because (AFAIK) it is really only a problem with very short pulses (on the order of femtoseconds). In this case a small but rapidly changing mechanical stress is produced by the beam, either as an effect of heating or the electrostrictive effect. The resulting shock is strong enough to damage the optical surface, usually by producing microscopic chips or causing optical coatings to de-laminate.
Now, the important issue here, which should answer your question, is that average power doesn't really mean much in the context of optical damage. It can be an issue in cases where heat dissipation over long time scales becomes a problem, but in cases where the laser is pulsed, damage mechanics are primarily determined by the peak power and the pulse width.
To make analogy: I have never been shot with a gun. If I get shot in the head tomorrow, my lifetime average bullet flux will be very low, just one bullet in 26 years. Yet I'll still be dead.