In a superconductor, the current can keep flowing "forever" since there is no resistance. But since conductors have inductance (in fact, superconductors are used most often to create magnets like for an MRI scanner), applying a voltage would not (immediately) cause an infinite current to flow.
It is instructive to see how an MRI magnet is "ramped" (turned on). From that link, I reproduce a circuit diagram here:

By turning on a very small (mW) heater, you can locally heat the superconducting loop so it becomes resistive. Now you can apply a voltage to the circuit - and it will preferentially send current around the superconducting part of the loop. The current will not be infinite though - for a given inductance $L$, the magnet will "ramp" as the current increases according to $$V = -L\frac{dI}{dt}$$.
When the desired field strength is reached, you turn off the heater - the loop will be closed and the current will keep flowing. At this point you turn off the voltage (current) supply. And you have a (strong) magnet that will persist for a very long time - as the current keeps flowing without resistance.
As was pointed out by J... And hexafraction in the comments, there is another important concept in superconductivity: that of a critical current. This is the largest current (density) that a particular superconductor can carry without becoming resistive. This is a function both of the magnetic field, and the temperature of the wire. This has a number of implications:
1) the current will never be "infinite" - you can only get up to some finite current before superconductivity stops
2) if you have a local imperfection in your wire, it might have a lower critical current and will therefore become the source of the transition to normal resistivity
3) designers of superconducting wire spend a great deal of effort on stability; for example they surround the superconducting core with normal conductor (silver or copper) to allow small temporary fluctuations in resistivity to dissipate (current temporarily diverted to the normal conductor). This greatly improves the stability of a superconducting system
4) because critical current goes down as magnetic field goes up, there is a non linear relationship between the magnetic field achieved and the amount of superconductor needed- one reason why higher field MRI systems are disproportionately more expensive.
5) sometimes a superconducting magnet will "quench" - local heat dissipation causes a runaway effect as more wire becomes resistive. You end up with all the stored magnetic energy being dissipated as heat. This will boil off the cryogen (usually liquid helium) which needs venting safely to the outside world (very expensive event!)
It is quite a complex subject...