As an example, consider an infinite line of charge with current density $\lambda$. Using Gauss's law you can find the electric field to be
$$\vec{E} = \frac{\lambda}{2\pi\epsilon_0 \rho}\hat{\rho}.$$
To find the potential at some distance $r$, we integrate from the point at which we define the zero of the potential to r. Let's define this point to be at infinity. So we have
$$V(r) = \frac{\lambda}{2\pi\epsilon_0}\int_{r}^{\infty} \frac{d\rho}{\rho}$$ $$V(r) = \frac{\lambda}{2\pi\epsilon_0}\ln{\rho}|^{\infty}_r.$$
You can immediately see that the potential evaluated at these bounds blows up. This is because the charge distribution goes out to infinity. So it appears that you can define the zero to be at infinity but you will run into trouble with infinities.
However, since we can always redefine the potential up to some arbitrary constant, we can define the zero of the potential to be at some point $R$. From this we have
$$V(r) = \frac{\lambda}{2\pi\epsilon_0}\ln{\rho}|^R_r.$$
This is well-behaved for finite $R\ne 0$.