You said in your question that you knew that the potential energy per charge was the volatage. One way of writing this is that the potential energy lost per unit time (power dissipated, $P$) is the charge flowed per unit time (current, $I$) times the potential $V$. Therefore we get $P=IV$ or $V=P/I$. From this equation we see that the potential is indeed inversely proportional to the current assuming constant power.
Now in "real life" we usually don't have constant power. For example, let's assume I have light bulbs of two different resistances $R_1$ and $R_2$ (let's say with $R_1 < R_2$), and I plug them both into the wall. The way electricity from the wall works is that it supplies a constant voltage $V$. Now we expect that with constant $V$, the light bulb with a lower resistance will allow more current to flow. In fact, the current $I$ resulting from a voltage across a resistance is given by $I=V/R$. Therefore we will get $I_1=V/R_1$, and $I_2=V/R_2$. Since $R_1$ is smaller, its current $I_1$ will be bigger and so the power $P_1 = I_1 V$ will also be bigger. Since the power is not the same for the two light bulbs, we do not get a contradiction with your equation from the first paragraph.