If the infinite conducting plane in the diagram above is grounded, then $V=0$ on the plane and the image problem is easy to solve - just put the dotted $-q$ charge there and the $V=0$ equipotential between the two charges satisfies the grounded plane boundary condition.
However, what if the infinite conducting plane is not grounded and so $V$ is not necessarily $0$?
Duffin, in Electricity and Magnetism 4th Edition, simply sets the conducting plane (which, as a conductor, must be an equipotential) to be the point of 0 potential. He then points out how the equipotential between the $+q$ charge and the image $-q$ charge is valued at $V=0$, so the boundary condition is satisfied. However, he set the plane as the potential reference point first when considering the initial configuration of plane and $+q$ charge and then used infinity as the reference point when he added the mirror $-q$ charge, so I don't see how his argument holds up.
How can we argue that the above image set up, with the dotted $-q$ charge, is valid even if the the potential on the conducting surface is not $0$?