I'm asking this because it seems to me that when we talk about PD around a charge (i.e. potential difference between infinity and a point), we're talking about the work done per unit charge by an external agent. But when we're talking about the PD in a circuit, we're talking about the work done by the battery's electric field.

Say on earth, there is a positive point charge at point A, then the potential at any point around point A (e.g. lets take the point as point B) will be the work done per unit charge by an external agent in moving a positive test charge from infinity to point B (i.e. to move from a region of lower potential to a region of higher potential). This potential will be positive since work has to be done by the external agent in order to move the test charge (since positive and positive charges repel). This potential can also be thought of as the potential difference between infinity to point B. Therefore, the PD between infinity to B is positive since work has to be done by an external agent (i.e. work done against the electric field).

Alternatively, in a simple circuit with a battery labelled e.g. 6 volts. The 6V refers to the PD between the positive and negative terminals of the battery. Due to the PD, there will be an electric field in the circuit's wire. The 6V also refers to the work done per unit charge in moving a positive charge from the positive terminal to the negative terminal.

However whose work done is the 6V referring to? Is it referring to the battery's electric field in the wire? Or is it referring to an external agent? Or is it referring to the work done by the positive charge? If it is referring to the battery's electric field, then 6V would make sense. If it is referring to the external agent, then the work done should be negative since the external agent do negative work to move the positive charge from a region of higher potential to a region of lower potential, and hence the battery will have a potential difference of -6V. If it is referring to the positive charge, then the charge would have done 6 joules of work per unit charge and hence the voltage will also be 6V.