Consider a stack of wood chips: each 0.5cm thick and 2x2 cm in length and width. There are 200 of them all stacked on each other. For some reason they all instantly fall. Evwn though their centre of gravity is at the centre of the stack and they have the extra added help of friction the further down the chips you go (because the second to bottom chip is squashed against the first because of the heavy load). So why does it fall?
First, let's assume the tower is a single solid rod. The tower will fall over when the center of mass extends past the base. For a longer tower, a smaller angular displacement is needed for this. This is because we want the horizontal displacement of the center of mass to be $$x=\frac 12 H\sin\theta<\frac 12 w$$ where $H$ is the length of the tower, $w$ is the width of a block, and $\theta$ is the angle the tower makes with the vertical.
Therefore: $$\sin\theta<\frac HL$$
The larger $H$ is, the smaller $\theta$ needs to be. This means that for a taller tower, smaller disturbances can cause it to fall.
Now our actual tower is not a single solid rod, but the same idea applies. Additionally, if the tower fails this at any section it will fall over, with taller portions falling onto ones below. This is why it seems like everything falls at once, as you have said. I'm not sure the friction between the blocks really matters, since there probably isn't a lot of shearing going on here.
An additional factor that I haven't seen mentioned yet is the fact that the environment acts upon a structure such as that. Errant air currents would have a deleterious effect on such a stack the higher it went. Even a tiny breeze would topple a perfect stack if it was beyond a certain height.