This is a matter of convention.
You are totally right: the $Q$ operator you have written implies that $b$ annihilates a positive charge and vice versa.
The thing is that in QED one usually defines $Q$ in a slightly different way, namely:
$$Q=-\left| e\right| \int {d^3p \over (2\pi)^3(E_p/m)} \sum_s \{b^\dagger(p,s)b(p,s)-d^\dagger(p,s)d(p,s)\}$$
with $-\left| e\right|$ the electron's charge. Then, with this definition, $b$ annihilates states with negative charge $-\left| e\right|$.
The confusion comes from what it is arguably the worst convention in the history of physics (and possibly chemistry and engineering): the electron's charge was chosen negative! Since there are many more electrons than positrons in the universe and in the Earth (and for this reason the appropriate convention that the electron is the particle and the positron its antiparticle, and not in the other way around), the electron's charges should have been chosen as positive! —so the positron should have been named negatron or perhaps the Dirac or why not simply anti-electron and having negative charge ;-)
Anyway, it is just a convention.