No, you can't.
Consider as one example a parallel plate capacitor. The capacitors constitutive relationship can be rearranged to get
The capacitance is, of course, given by the geometry. Ignoring fringing effects, $C=\varepsilon A/d$. So, we have
That is, if charges $+Q$ and $-Q$ are spread in sheets of area $A$ and separated by distance $d$, there must be a potential difference between the two charged regions that depends not just on the amount of charge, but also on the distance between them, the size of the areas they're spread over, and the properties of the material between them.