The key to this situation is the fact that the finger does not apply a force only at the axis of rotation. It actually provides force over an area, specifically the area of the pad of your finger. This means that, for any rotation axis you draw, the force of friction for nearly any part of the pad isn't through the rotation axis so it can induce a torque that keeps the book up. It's a small distance from any one point under the finger to the axis of rotation, so the torques are small per unit force, but they are nonzero. With enough clamping force, they can indeed generate the required torque to keep the book up.
Were you to replace the fingers in the scenario with the tips of a carpentry nail, the situation would be harder. The tip of a nail is much smaller, so its induced torques are much smaller given the same forces. This means that you have to push much harder on the nail to generate enough friction forces to hold the book up from nailpoint. In fact, you may have to push so hard that you exceed the material strength of the book and end up driving the nail through the book before generating enough torque due to friction.
In the opposite scenario we can consider a bolted joint: two pieces of metal held together by bolts. Obviously such metal joints are very good at providing the required torque. Despite what your intuition may say, the strength of a bolted joint is not caused by sheering, where the edges of the metal push on the side of the bolt. Bolts are generally weaker in this direction. Instead, we use the bolt in its strong direction to clamp the pieces of metal together, exactly like you're doing when pressing on the book with your fingers. This is the strong direction for the bolts, and it allows all of the bolts to share the load.
As a result, not only is this approach sufficient to hold up a book from the corner, but it also holds up skyscrapers and bridges. If you are interested in more information on how these wonderful joints work, I highly recommend The Incredible Strength of Bolted Joints, an incredibly accessible video by The Efficient Engineer.