So people usually respond to questions like this, quite reasonably, by saying that virtual particles aren't "real" and they're just a calculational device for doing perturbation theory. But this doesn't really seem to answer the "spirit" of your question: when an electron and positron interact, clearly there is something that "lets them know" what each other's charge is: an electron brought close to a positron behaves differently to an electron moved close to an electron. Your puzzle is then: if the only thing that defines the interaction between the electrons is a virtual photon, then how do they know each other's charge?
The answer is essentially: it is not just the virtual photon that defines the interaction. The electrons also know what the Hamiltonian is, or, "there's more information in the interaction than just what the virtual photon is carrying around". However you choose to incoporate that into your visualisation of how particles interact is up to you, but it highlights the limitations of visualising electromagnetic interaction as "a photon being spat back and forth" and nothing else. Let's elucidate this claim a bit more clearly with a path integral treatment.
Consider
\begin{align}
\mathscr{Z}&=\int \mathscr{D}A\exp\left(i\int d^4x\ \frac{1}{4}F_{\mu\nu}F^{\mu\nu} + A_\mu J^\mu \right)\\
&=\int \mathscr{D}A\exp\left(i\int d^4x\ \frac{1}{2}A_\mu\left(\partial^2\eta^{\mu\nu}-\partial^\mu\partial^\nu\right)A_\nu + A_\mu J^\mu \right)
\end{align}
and not worry too much about subtleties with the $\mathscr{D} A$ to do with gauge-fixing, or the dynamics of the electrons/positrons - we'll just treat the current $J^\mu$ as a classical source, and take the photon propagator to be $ -\eta_{\mu\nu}/p^2$. Our results will be unchanged under a gauge transformation, and we'll step through later how the story is changed when let the electrons be dynamical.
We can do the path integral exactly here, since everything is quadractic in $A_\mu$. The answer is
\begin{align}
\mathscr{Z}=\exp\left(iW[J]\right)
\end{align}
where
\begin{align}
W[J]=\frac{1}{2}\int\frac{d^4k}{(2\pi)^4}J^\mu(k)^*\frac{1}{k^2+i\epsilon}J_\mu(k)
\end{align}
where $J(k)$ is the fourier transform of $J(x)$ and $J^*$ is complex conjugation. The quantity $W[J]$ encodes the "potential energy" of the configuration $J$ in a way we'll make precise at the end. The zeroth component of the current vector is the charge density $J^0(k)=\rho(k)$, and so this expression tells us that the potential energy of two lumps of equal charge density is positive, while it is negative for opposite charges - i.e. the force between two like charges is repulsive and between two opposite charges is negative$^1$! Now,
the propagator factor $1/k^2$ is "a virtual photon" being exchanged between the two currents (more justification for this later), and so in this sketched example we see that it isn't the virtual photon "telling" the two particles what their respective charges are. It tells us that there is a term in the Hamiltonian that couples charges together via a photon, and the structure of QED is such that this term is positive for like charges and negative for opposite charges$^2$. It's simply the fact that a virtual photon exchanged between two like charges mediates a repulsive force, and an attractive one when exchanged between opposite charges. This is the limitation of imagining a force as nothing more than a photon spat back and forth: what that photon does depends on what the Hamiltonian tells it to do, and the Hamiltonian tells it to attract opposite charges and repel like charges. So the situation is "photon + the instructions it gets from the Hamiltonian" rather than just "photon" on its own.
Treating the electron/positron as classical sources and seeing the energy of the interaction you get is actually a perfectly fine argument, but maybe it would be a bit more persuasive to put the electrons in explicitly, and derive the Feynman rules - which is where we're used to seeing virtual photons. If we want to calculate time-ordered correlation functions, say $\langle \text{T}\{\bar{\Psi}(x)\Psi (y)\}\rangle$ we use the path integral:
\begin{align}
\mathscr{Z}&=\int \mathscr{D}A\mathscr{D}\bar{\Psi}\mathscr{D}\Psi \ \bar{\Psi}(x)\Psi (y)\exp\left(i\int d^4x\ \frac{1}{4}F_{\mu\nu}F^{\mu\nu} + i\bar{\Psi}\partial\!\!\!\big/\Psi + e\bar{\Psi}\gamma^\mu\Psi A_\mu \right)\\
&=\int \mathscr{D}A\mathscr{D}\bar{\Psi}\mathscr{D}\Psi \ \bar{\Psi}(x)\Psi (y)\exp\left(i\int d^4x\ \frac{1}{2}A_\mu\left(\partial^2\eta^{\mu\nu}-\partial^\mu\partial^\nu\right)A_\nu +i\bar{\Psi}p\!\!\!\big/\Psi + e\bar{\Psi}\gamma^\mu\Psi A_\mu\right)
\end{align}
One way to derive the Feynman rules is to separate the noninteracting piece off, then expand the exponential $\exp(i\int d^4x \mathcal{L}_{int})$. Then we get a series of integrals that look like powers of the fields times Gaussians of those fields (the noninteracting piece up in the exponent still). Integrating these objects term by term give us the Feynman diagrams. Explicitly,
\begin{align}
\mathscr{Z}&=\int \mathscr{D}A\mathscr{D}\bar{\Psi}\mathscr{D}\Psi \ \bar{\Psi}(x)\Psi (y)\sum_{n=0}^\infty\frac{1}{n!}\left(i\int d^4ze\bar{\Psi}\gamma^\mu\Psi A_\mu\right)^n\exp\left(i\int d^4x\ \frac{1}{4}F_{\mu\nu}F^{\mu\nu} + i\bar{\Psi}\partial\!\!\!\big/\Psi \right)
\end{align}
Wick's theorem tells us how to do these integrals (Wick's theorem is basically just a fancy way to tell us how to do integrals that look like powers times Gaussians). A generic term in this series looks something like
\begin{align}
\bar{\Psi}(x)\Psi (y)\left(i\int d^4we\bar{\Psi}(w)\gamma^\mu\Psi (w)A_\mu(w)\right)\left(i\int d^4ze\bar{\Psi}(z)\gamma^\nu\Psi (z) A_\nu(z)\right)
\end{align}
times the Gaussian. Wick's theorem then says to do this integral we "contract" each $A_\mu(x)$ with other terms like $A_\nu(y)$, which gives us a factor $-i\eta_{\mu\nu}/(k^2+i\epsilon)$, and contract fermion operators similarly, picking up fermion propagators. We then integrate over the positions $w$ and $z$, and obtain the value of that term in the series - this is a Feynman diagram. "Virtual photons" refer to the objects that come from contracting photon fields $A_\mu(x)A_\nu(y)$, and are represented by wavy lines in Feynman diagrams. This is why we claimed earlier that the $\eta_{\mu\nu}/(k^2+i\epsilon)$ term earlier plays the same role that the virtual photons play in the perturbation series.
Now, the way we calculate scattering amplitudes is through the LSZ prescription, which says to calculate correlation functions then amputate the external lines i.e. take factors that you got from contracting the terms like $\bar{\Psi}(x)\Psi (y)$ with each other, and then replace them with some factor (just a constant if the field is a scalar, some spinors if the field is an electron, polarisation structure if the field is a photon etc).
So how do we relate the packaging of this story - in terms of scattering/correlators - to the question of whether the interaction is repulsive or attractive? Well if you calculate the leading contribution to $e^+e^-\rightarrow e^+e^-$ scattering, and consider the nonrelativistic limit, you get something that basically looks exactly what we considered before
\begin{align}
e^2j^\mu(p)\frac{\eta_{\mu\nu}}{k^2+i\epsilon}j^\nu(p')\sim\frac{e^2}{|p-p'|} \times \text{stuff}
\end{align}
where we won't be bothered explicitly writing what the $js$ are. Note the $\frac{\eta_{\mu\nu}}{k^2+i\epsilon}$ is the virtual photon line, and the $js$ come from the initial and final states of the electron/positron. But the leading contribution to the scattering amplitude in nonrelativistic quantum mechanics is given by the Born rule, which says basically
\begin{align}
T_{p'p}=-\tilde{V}(q)(2\pi)\delta (E_f-E_i)
\end{align}
where $T_{pp'}$ is the matrix element of the scattering matrix. So by comparison we see $\tilde{V}(p-p')=-e^2/|p-p'|$ i.e. an attractive Coulomb interaction$^3$.
The upshot is that virtual photons come from insertions of photon propagators, and as we saw in both examples they don't carry any information about the charges of the fermions. The attraction/repulsion came from the electron-photon vertex i.e. the factor of $\pm e$ due to the current operator i.e. what the Hamiltonian says a photon is supposed to do when it links up to a fermion. The Hamiltonian says like charges repel and opposite charges attract, and the virtual photon is told what to do by the Hamiltonian.
The final point I should return to is my earlier assertion that $W[J]$ somehow reflects the energy associated to the configuration $J$. Observe if we just take the source to be static, then $e^{iW[J]}=\langle 0| e^{-iHT} |0\rangle =e^{-iET}$ where $E$ is the energy of the two sources acting on each other. If you take the above expression for $W$ in real space, and let the sources be static then do the time integral you'll get $E>0$, which is what we were after$^4$.
$^1$In fact, if you fourier transform this back to real space, you'll find this expression equals the Coulomb interaction.
$^2$Incidentally you can run this exact same story through with gravity and discover that gravity is attractive! Without worrying too much about the exact form of the lagrangian for gravity, requiring the propagator be traceless, transverse and gauge invariant allows you to fix its structure on general grounds to be
\begin{align}
G_{\mu\nu,\lambda,\sigma}=\frac{\eta_{\mu\lambda}\eta_{\nu\sigma}+\eta_{\mu\sigma}\eta_{\nu\lambda}-\tfrac{2}{3}\eta_{\mu\nu}\eta_{\lambda\sigma}}{k^2+i\epsilon}
\end{align}
where again we've made the easiest gauge choice. Rather than couple to a vector current $J^\mu$, the graviton couples to a tensor current i.e. the energy momentum tensor $T^{\mu\nu}$, and two lumps of energy/mass pop up in the $T^{00}$ component of the energy momentum tensor. Running the same story through for $W[T]$ you'll find that like masses attract.
$^3$See for instance the discussion on page 125 of Peskin and Schroeder.
$^4$This entire story is run through (probably more clearly than I did) in Zee's cute book "Quantum Field Theory in a Nutshell", in one of the early chapters.
