" it seems to me that when we talk about PD around a charge (i.e. potential difference between infinity and a point), we're talking about the work done per unit charge by an external agent"

You're absolutely right; most textbooks define potential and potential difference due to a static charge in this way. Such definitions are (usually!) correct, but, in my view, rather clumsy. There is no need to bring in an external agency. The definition that I prefer is that *the pd between A and B (the amount by which the potential of A is greater than that of B) is the work done by the field per unit charge on a test charge going from A to B*. 

The amount of work per unit charge in this definition is indeed equal to the amount of work per unit charge that would have to be supplied by an external agency in moving a test charge from B to A (assuming no change in the KE of the test charge!), but why bring an external agency into the definition of an electrical quantity arising from an electric field? 

[It is, of course, perfectly possible for a test charge to go from A to B in an electric field without any external agency other than the field. An example would be the acceleration of electrons in an electron gun. Electrical PE is lost and kinetic energy gained.]