One of the answers to this question: How does the center of gravity work? said the follows:

Since we can see that things do not start to rotate when they fall, we must assume gravity when averaged out to be pulling in the point where all torque cancel each other out, which is the center of gravity.

Also in some answers to that question, I learned that gravity isn't applied only at the center of gravity in real life, but rather every single particle of an object. Thus, I am confused why things do not start to rotate when they fall in real life as the torque caused by gravity doesn't cancel out?