" it seems to me that when we talk about PD around a charge (i.e. potential difference between infinity and a point), we're talking about the work done per unit charge by an external agent"
You're absolutely right; most textbooks define potential and potential difference due to a static charge in this way. Such definitions are (usually!) correct, but, in my view, rather clumsy. There is no need to bring in an external agency. The definition that I prefer is that the pd between A and Bthe pd between A and B (the amount by which the potential of A is greater than that of B) is the work done by the field per unit charge on a test charge going from A to B.
The amount of work per unit charge in this definition is indeed equal to the amount of work done by the field per unit charge onthat would have to be supplied by an external agency in moving a test charge going from A to B. This gives the amount by which the potential of to A is greater than that of B.
[There may actually be no external agency, and(assuming no change in the charge may simply gain or lose KE of the test charge!), or there may bebut why bring an external agency against which workinto the definition of an electrical quantity arising from an electric field?
[It is done or which does work – but there's no need, in my opinionof course, perfectly possible for a test charge to bring thego from A to B in an electric field without any external agency intoother than the definitionfield. An example would be the acceleration of electrons in an electron gun. Electrical PE is lost and kinetic energy gained.]