If voltage of source is increased, [then] current through the circuit increases...
Actually you've got that exactly backward.
The problem is, you misunderstand the context in which they "increase the voltage." They aren't talking about turning up the voltage in some existing circuit that supplies, for example, some existing light bulb. They are talking about tweaking the design of a power distribution system.
Suppose you want to deliver 100W to a light bulb. If you design a system in which light bulbs expect a 10V power supply, then you will have to supply 10A to light the bulb. But if you design the system to operate at 100V, then you will only have to supply 1A. "Turning up" the design voltage gives you a system in which less current will be needed.
It is commonly stated that increasing voltage of the source reduces power loss.
There, they are specifically talking about power lost in the distribution network. The wires that distribute the power have resistance. It's small, but it's enough to matter. Suppose the wire that connects your light bulb to the generating station has 1/10th of an Ohm of resistance. If the system supplies 1A of current to your light bulb, then 1/10th of a Watt ($1^2\times{}0.1$) will be "lost" in the wire. But if your system needs to supply 10A to the light bulb, then the loss will be $10^2\times{}0.1$ or 10W.
The distribution network that uses $10\times{}$ the voltage has only 1/100th of the $I^2R$ losses.