You should not imagine a virtual photon as an individual object wandering from one charged particle to another. This picture is simply inapplicable. Unfortunately, Feynman diagrams mislead people to imagine such things.
Actually, Feynman diagrams are good for calculation and bad for imagination. Feynman diagrams have been introduced to help physicists to collect the mathematical terms for the perturbation series. And here they are very helpful. They have not been introduced to help people to imagine what actually happens.
That said, one always needs to keep in mind where virtual particles enter the theory: They come in when one calculates the perturbation series. A perturbation series, however, is an approximation by definition. It is not a full theory in itself. If you see it from this side you will realize that virtual particles are actually artefacts of the mathematical theory. They probably even do not have a counterpart in reality.
In QED one normally starts with free field Hamiltonian (or Lagrangian). Its eigenstates are free photon and free electron or other particle fields. Free fields are actually an idealization: They do not exist in nature since there is always some interaction. Nevertheless, if the particles are far apart from each other free fields are considered as a good approximation.
When one now turns on the interaction, the free field states are no longer eigenstates of the Hamiltonian. The new eigenstates are complicated mixtures of the free field states (if they live in the same Hilbert space at all). That means, a new eigenstate of the interaction Hamiltonian could be imagined as a new particle which is a complicated mixture out of the free particle states and therefore out of the free photon and electron states. In that way, electron and photon are no longer separate states, but mix to form new states as long as the interaction is in effect.
In any case, the new eigenstates cannot be solved for explicitely. Instead, the scientists use the perturbation series to get an approximate solution for the scattering processes. In this perturbation series the interaction is modeled as happening at certain specific points in space and time while in between the particles are propagating as "free" particles. These in between propagating particles are referred to as virtual particles. They are clearly an artefact of the perturbation series and they are not something which is happening in reality.
Charge itself is a measure of the interaction between certain particles. It is not a direct observable. Again, you should not consider it as some distinct feature of a particle which makes this particle attract or repel other (charged) particles (though, often is is viewed that way). Otherwise you indeed have a problem to understand, why the photon is contributing since it carries no charge. Instead, charge leads to an interaction term in the Langragian and causes the fields contributing in this interaction term to mix. The fields contributing are the fields of the charged particle and the photon field.
That is also, why charge cannot be measured directly, but has to be extracted from the interaction amplitudes formed by the interacting fields. This fact is important for the renormalization of quantum electrodynamics. The pure charge can even become infinite in the renomalization process as long as the charged measured via these interaction amplitudes has a reasonable, finite value.