The other answers are not really hitting the core of the problem and that is - on top of conductors, regardless if there is a current or not (open circuit condition for example), there exist surface charges which get created due to fact that there is an electric field present in the source. The consequence of that electric field is that a surface charge density gets established on surface of the conductors. Why on the surface? Because that's the only place where they can get in a stable equilibrium. The end result - assuming no current flows (open circuit) is that there shouldn't be an electric field anywhere inside the conductor. A small amount of positive charge (from the battery) will enter the wire that is connected to the positive terminal of the battery and it will redistribute itself along the surface of the wire connected to the positive terminal of the battery so that the field produced by the surface charge density cancels any field that would be felt from the source inside the conductors. Same thing will happen on the negative wire (the wire connected to the negative terminal of the battery) but in that case, negative charge will leak.
The end result will be something similar to this:

If you were to measure the potential along the wire, you would see that it is now constant. Your original assumption that potential should drop because of increasing distance - is correct. In fact, potential created by the battery in air drops rapidly as you move away from the battery (if I were to guess, I'd say that it drops with square of the distance, just like a dipole). However, because the charge is free to move from the battery to the conductor, any potential drop will get exactly counteracted by the creation of surface charges.
So, what happens when current flows and resistance kicks in? There will be a component of electric field that will also be tangential to the conductor. And that is the component that will be responsible for the small measurable voltage drop beacuse of resistance. But the surface charges will pretty much remain similar to the open circuit condition and it will be their job to ensure that the bulk of potential of the conductors stays the same.