For completeness, I'll begin my explanation from the Maxwell equations, but I'll try my best to give more intuition and qualitative explanations about them.
To understand the electric potential, we first look at the Maxwell equation $\nabla \times \mathbf{E} = - \dot{\mathbf{B}}$, where $\mathbf{E}$ is the electric (vector) field, $\mathbf{B}$ is the magnetic (vetor) field, $\nabla \times$ is the curl operator and $\dot{}$ indicates a time derivative. This is basically a representation of Faraday's Law of electromagnetic induction. The first important thing to note is that you seem to be describing potentials in the context of "electrostatics", i.e. determining the electric fields and potentials generated by charges at rest. The key assumption in this "electrostatic" regime is that any existing magnetic field is either constant with time or varies very slowly such that $\dot{\mathbf{B}} \approx \mathbf{0}$. This means that $\nabla \times \mathbf{E} = 0$. A general solution to this equation would be $\mathbf{E} = - \nabla \phi$, where $\nabla$ is the gradient operator, and $\phi$ is a scalar field. This is just because of the mathematical identity $\nabla \times (\nabla \phi) = \mathbf{0}$. The physical interpretation of the scalar field $\phi$ is the electrostatic potential. "Scalar field" just means that it is a scalar function that depends on spatial coordinates (it could also depend on time, but not in this discussion since the charges are at rest and the fields should be static). Basically, the take-away from this paragraph is that the electric potential can be seen as a quantity that can be written in a way that satisfies one of these very important Maxwell equations under electrostatic conditions.
Now, we need to unpack what $\mathbf{E} = -\nabla \phi$ means. While we probably need an introductory vector calculus lesson to fully grasp what this means, we could just take the one-dimensional case for a simpler explanation. In one-dimension, this equation is $E = - \frac{d \phi}{dx}$, i.e. the electric field is related to the spatial derivative of the potential. We can then integrate the electric field to get the potential
$\phi(x) = -\int E(x) dx = \phi_1(x) + \phi_0$
Here, $\phi_1(x)$ is the indefinite integral or antiderivative of $-E(x)$. The key point is that there's an integration constant of $\phi_0$ that appears and we don't really have any physical constraints to fix it. That's why voltage is always defined as the potential DIFFERENCE as the constant will cancel out:
$\phi(x_1) - \phi(x_2) = \left[\phi_1(x_1) + \phi_0 \right] - \left[\phi_1(x_2) + \phi_0 \right] = \phi_1(x_1) - \phi_1(x_2)$,
which is independent of the arbitrary constant.
When we make measurements, what we're really measuring is the effects of the field, which is independent of the arbitrary constant $\phi_0$ part of the potential is it disappears upon differentiation and it only depends on the $\phi_1(x)$ part that is a function of space, i.e.
$E = -\frac{d \phi}{dx} = -\frac{d}{dx} \left ( \phi_1(x) + \phi_0 \right) = -\frac{d\phi_1}{dx}$
If you google around for how voltmeters and other measurement devices work, you'll see that they are based on measuring quantities related by electric field effects such as the current (motion due to the electric force exerted on the charges) or electrostatic repulsion effects. So you can think of the electric field as the "physical" quantity that can be measured and the potential as a mathematical construction to make life easier (at least classically, some might argue about the physicality of potentials in quantum mechanics, c.f. Aharonov Bohm effect).
Hopefully, this has cleared up some of your confusion. Now onto your specific question, you can think of the charges at the poles as generating an electric field, and therefore there is a potential associated with it that you can integrate the field over space for. But again, you get that arbitrary integration constant. You don't even need a positive and negative charge, as long as you have some arbitrary charge distribution in space, $\rho(\mathbf{r})$, there will be some electric field, $\mathbf{E}(\mathbf{r})$ and associated potential, $\phi(\mathbf{r})$, over all space, which can be found by the Maxwell equations which relate the charge distribution to the fields, i.e. Gauss's equation $\nabla \cdot \mathbf{E} = \frac{\rho(\mathbf{r})}{\epsilon_0}$, or using the potential definition $\mathbf{E} = - \nabla \phi$, we have Poisson's equation $\nabla^2 \phi = - \frac{\rho}{\epsilon_0}$. Again, you can add an arbitrary constant to the potential without changing the electric field, whose effects are the ones that we physically observe.
If you have an object that is electrically neutral, I'm assuming you mean that the effective charge distribution is zero over all space. This means that the electric field is zero over all space and when you integrate "zero", you get a constant. So you CAN arbitrarily assign any constant value to the constant potential in the system - what matter's is that you get zero after you differentiate it over space, which implies zero electric field.
Sorry, if it's a bit on the technical side, but I was hopping to give you a better idea of where it all comes from. Let me know if you have any further questions, I'll try my best to help!
