For electrical engineering purposes, Voltage is a potential difference with respect to a reference point. Absolute electrical potential is meaningless in electrical engineering contexts because circuits involve electrical current flowing from a high-potential point to a low-potential point and not to a no-potential point. Therefore electrical engineers talk about Voltage and not electrical potential. When you read "setting its potential to zero" what is really meant is "setting its Voltage (its potential difference) to zero with respect to the Earth". The former is simply a short form that is usually understood unambiguously to mean the latter in electrical engineering contexts.
Earth is simply a commonly used reference point. However, to add to the confusion, the electrical potential of Earth is not the same everywhere. One end of your street could have an excess of electric charge with respect to the other end of your street, so if you grounded two different circuits on each end of your street and connected them together, you could end up with this charge flowing from one circuit to the other. This can pose problems such as electrical noise between equipment that must be connected together but may not be grounded at the same point. This effect is referred to as a ground loop.
Your Earth may not be equal to my Earth, but as long as you always use the same Earth, you will always be using the same reference point, and you can simply call it 0V. It's a convenience thing.