Suppose, for sake of argument, we have a spherical grounded conductor at the origin. Additionally, let our reference voltage be at infinity. Now, I view the potential of a point in space as being the work that I personally would have to do to move a unit charge from the reference to the point in question. Thus, I interpret this grounded conductor as points in space at which I could bring in a unit charge from infinity and end up doing no work. However, that interpretation certainly cannot be true in this case. Suppose the charge that I am bringing in is unit positive. The entire time I bring that charge towards the conductor there will be a larger and larger negative charge build-up, so I will have to exert a larger and larger force in the opposite direction the entire time I bring it in. The net work that I do in moving this charge towards the conductor will clearly be negative, not zero as the potential value would seem to suggest.
Can someone resolve these conflicting interpretations? Much appreciated.