The whole story starts from electric forces. With each conservative force,we associate a potential energy. You can understand potential energy as a measure of instability for positive charge(for your understanding). We measure with respect to positive by default. So a region of positive charges has more electric potential energy (positive charge is unstable there due to repulsion). The repulsive forces cause instability which causes charges to flow from high concentration of similar charges to low concentration of similar charges. We relate the same using concept of electric potential energy. Electric charges is most analogous to gravitational systems(easy to relate) but difference comes only that charge may repel also. Instead of energy (which is different for different charges) we use potential which is same for charges(dependent on configuration of surroundings). Like we define gravitational potential at different heights on earth. The difference of potential (voltage) causes charge to flow. (Like difference of heights is required for body to fall). Coming to your question, we reference potential to be zero at infinity (assuming no charges at infinity),but for practical situation it has no significance. It is only difference of potential that matters. In case of battery 4V means positive charge at positive terminal is at higher energy by 4q than positive charge at negative terminal. (For negative charge , energy is negative which would mean it is at higher energy at negative terminal). Hence positive charge flow from positive to negative terminal through wire. The process is analogous to free fall of body. Hope this made things clear
