Gravity pulls in every single particle in the object.
This means that gravity causes a torque at every single particle around some other point.
- If you look at a point to the left, then all those torques sum up to a net torque which pulls clockwise.
- If you look at a point to the right, then they sum up to a net torque pulling counter-clockwise.
- If you look at a point in between, then some torques pull clockwise and some counter-clockwise.
At some special point, the clock-wise and counter-clockwise torques cancel exactly out.
Since we can see that things do not start to rotate when they fall, we must assume gravity when averaged out to be pulling in this point. If we averaged it out to pull in any other point, there would be a net torque and the object would spin - and we know/see that this does not happen.
So, let's call this point the centre of gravity, while keeping in mind that we can average out gravity to pulling in this point as a simpler model to work with.