If the book were to be pushed to it's max, what forces would be
enacted on the book? What forces would prevent it from
falling/swiveling about the pivot?
Ignoring friction, there would be two forces acting on the book: gravity and normal reaction of the table and the balance of these forces and torques due to these forces would prevent the book from falling over the edge of the table.
The diagrams below show how the distribution and balance of these forces are changing, as the book is being moved over the edge of the table.
A. The book is away from the edge. The normal reaction force, $N$, is equal to the force of gravity, $mg$.
B. The book is partially hanging over the edge. The normal reaction force is still equal to the force of gravity, but now we have to consider the torques trying to rotate the book over the edge. Since, $m_2g<m_1g$ and $d_2<d_1$, the book does not fall. The difference between the torques due to $m_1g$ and $m_2g$ is balanced out by the torque due to the normal reaction force:
$N \times d+m_2g \times d_2 = m_1g \times d_1$
Since the normal reaction force is greater than $m_1g$, it has to be applied closer to the edge than $m_1g$ - otherwise its torque alone would overpower the torque due to $m_1g$. Due to this shift, the normal reaction force has to be redistributed along the length of the book remaining on the table, as shown by the dashed green line: the normal reaction will be increasing toward the edge.
C. This is just a progression of the movement, with the normal reaction force moving even closer to the edge.
D. A little more than half of the book is overhanging, therefore the torque due to $m_2g$ is exceeding the torque due to $m_1g$ and the book is starting to rotate over the edge. The normal reaction force has shifted all the way to the edge and does not contribute to the torque.