According to the fact that a phone charger consists of a transfomer and a recitifier, there should - still - be a difference in potentials, even though no phone is attached to the charger.
Because the circuit is not complete. Think of it as a potential difference, but the resistance connecting them is infinite, ie, air. When phone is connected, circuit is complete and current flows. If you touch the charger tip with your tongue, you will feel slight tingling taste of electricty, as the circuit would still be complete.