Why is the concept of potential used so loosely? What does potential of a conductor actually mean? As I have understood it potential is the amount of work done by an external agent to bring a unit positive charge to a point of consideration. It is also something like a scalar version of electric field. However when I see the concept of a potential being used for things like an infinitely charged metallic sheet or items in an electrical circuit, I get confused. Can anybody connect the real definition of potential to these things? Is potential not a rigorous concept and meant to be used so loosely?
 A: 
Is potential not a rigorous concept and meant to be used so loosely?

The fact that zero potential can be arbitrarily assigned shows that potential is not a rigorous concept. It’s my understanding that Griffiths, in his book "Introduction to Electromagnetism",  makes the following statement:
"Evidently potential as such carries no real physical significance, for at any given point we can adjust its value at will by a suitable relocation of 0"
What is a rigorous concept, on the other hand is the potential difference, or voltage, between two points which is defined as the work required per unit charge to move that charge between the two points. That said, regarding your other comments/questions:

As I have understood it potential is the amount of work done by an
external agent to bring a unit positive charge to a point of
consideration.

What's missing in your statement is the point from which the positive charge is brought and which has been assigned a potential of zero. The nature of potential is that the zero point is, as I said, arbitrary. But once that point is set, the value of the potential at any point of consideration is measured with respect to that zero point.

It is also something like a scalar version of electric field.

The electric field is gradient of the potential, or $\vec E=\nabla V$ (volts/meter}

However when I see the concept of a potential being used for things
like an infinitely charged metallic sheet or items in an electrical
circuit, I get confused.

For a single point charge, or localized collection of charges, it is logical to set the zero point at infinity.
But this breaks down for the case of an infinite sheet of charge of charge density $\sigma$. That's because the electric field at any distance $d$ away from the sheet is constant of magnitude $E=\sigma/2\epsilon$. Then since, for a constant field, $V=Ed$ the potential becomes infinite instead of zero at infinity. On the other hand, there is no such thing as an infinite sheet of charge.
With regard to electric circuits, the concept of potential does serve a useful purpose when applying Kirchhoff's current law in nodal analysis. A point in the circuit is assigned a potential of zero and the potential at every other point (node) is measured with respect to that zero potential point. Although the assignment of zero potential is arbitrary, a point that is common to several branches, a "grounding" connection, or the negative terminal of voltage source is often considered a logical choice.
Hope this helps.
A: The first three quotations and responses in Bob D's answer are nearly exactly what I would have written.
What I'd add is that you should remember that the full name of this concept is the electrostatic potential. But electrostatic systems (systems in which none of the charges move, ever) aren't very interesting compared to other electromagnetic systems.
So we often use the potential in a non-rigorous way to describe and predict the behavior of systems that are only approximately static. That is, when there are moving charges present, but that the motion of the charge makes only a small effect on the behavior of the system, compared to what you'd predict from assuming the system is static.
A: The other answers are great, however I will add some of my analogy. We can think of potential as some peaks of mountain at different points in a space permeated by some $\vec E$. The geography (lack of better term) of all such mountains depends on $\vec E $.
If some particle descends to smaller peak (potential), from higher peak, (potential) then particle gains some energy. If particle ascends from smalller peak to higher peak, then particle loses some energy.
Now, coming to 'loose definition', actually it is not so. It is rigorously defined as, work done to bring one coloumb/kilogram charge/mass from infinity to some point in space. Mathematically: $$\Delta V=-\int \vec E \cdot \vec {dr}$$
Or: $$\vec E=\frac{dV}{d\vec{r}}$$
In atomic physics, the term 'eV', electron volt, is quite popular unit for measurement of energy. You would find even masses of particles written in this unit of measurement.
It is defined as energy required to accelerate one electron across 1 volt potential.
Or energy released/gained by electron descending/ascending a peak. where potential differnece between peaks is 1 volt.
$$1 \text{eV}=1.6 \times 10^{-19} \text{J}$$
So potential is a rigorous term and is quite useful in physics.
