Apparently simple question which teaches a lot about constraints.
I think the simplest one-sentence summary is that 'different' initial configurations dictate the final kind of equilibrium that is reached with the passage of time.
As in, the first configuration involves your finger the cap being balanced on the tip of your finger, now unless you have wide fingers, it's gonna be very difficult to keep the force you apply right on the center. If there is some inaccuracy from the center, then the gravity will twist and turn the object to tumble onto the least energy state where it will be on the ground.
To successfully achieve a balance, you need to position your fingers in such a way that the total torque generated by each finger is zero. It's a similar problem to figuring out where we should we keep the 'legs' of a table so that the structure is sturdy.
In the second configuration, your finger is being depressed a little bit and it's creating an opposite force to counteract that deformation, this counteracts gravity. Think of putting a weight on a trampoline, once it depresses, it exerts an opposite force for the object to bounced outwards.