What is the electric potential in free space? Is it not zero, since there is no charge around?
Whenever you are calculating fields/potentials in a region (of interest) you are always considering the fields/potentials created inside your region. Any variation in field that you’ve calculated is a result of an effect/charge outside your region. If you want to incorporate those variations then you must extend your region to include the source of the variation.
So when we say free space, we mean that in the region we are interested, there are no charges. Thus $\vec E$ will be $0$. This means that our scalar potential $\phi$ has to be a constant since $\nabla\phi=\vec E$. But this constant can be anything as long as it gives us zero electric field. This freedom is called gauge invariance.