There appears to be a couple layers of misunderstanding here so I'll try to clarify. Similar to what you stated, a difference in potential between two points, can be defined according to the work that an electric field would do on a given charge, $q$, by displacing it between those two points. So we have, \begin{align} \Delta V = \frac{W}{q}. \end{align} So a higher $\Delta V$ does mean that a charge that traverses that potential will have more work done on it BUT the $\Delta V$ only requires the existence of an electric field, no charge to move through it is required. If there is a field, the potential will be there, they are different representations of the same thing and the above is a quite a general definition.
Now, I'm guessing your confusion then comes from \begin{align} V=IR. \end{align}
The thing is that this relationship is not general at all, and only applies to ideal resistors (as Alfred Centauri points out), so you can't just equate this with the first definition. In the first definition the charge was just a construct you use to relate $\Delta V$ and $W$. Here you're talking about the amount of current, $I$, resulting from putting a potential difference, $V$, on a resistor with some resistance, $R$.
As for why it happens that greater $V$ means greater $I$ you can think of turning up the pressure on a water hose. The length of hose has some "resistance"''resistance" but by turning up the pressure you can get more "current''current."