It probably depends on multiple factors but one of the most important is how close the static friction force is to the maximum where you have pending or imminent slipping. At that point the book is unstable and the slightest movement, vibration, etc., will cause it to fall.
For the book the static friction force is a variable reaction force that keeps the book in static horizontal equilibrium, like at the foot of a ladder. But the static friction force has a maximum possible value of $\mu N$ where $N$ is the force normal to the surface. If the book is initially leaning so that the static friction force is at or near the maximum the book will be unstable and collapse with the slightest vibration or movement.
The reason it might take several hours Is it may require the cumulative effects of imperceptible vibrations to cause the center of gravity to move and the maximum static friction force reached.
As far as I know no ordinary environment is vibration free. From a microscopic view static friction is due to microscopic irregularities at the contacting surfaces causing them to interlock. Slight vertical oscillatory vibrations briefly separate the surface. A brief resulting net horizontal force in the direction away from the wall has the surfaces come back together to interlock microscopically further away from the wall slightly reducing the normal force and thus the static friction force. Over time the maximum static friction force is reached and the book falls
Hope this helps