Simply put, there are no stable charged particles beyond electrons, protons, and their antiparticles. This means that if you want a pair of oppositely charged particles, it needs to be $p^+e^-$ (hydrogen), $p^-e^+$ (antihydrogen), $e^+e^-$ (positronium), or $p^+p^-$ (protonium).
If you wish to use any "non-matching" particles, then at least one of the components must be unstable. The easiest example of this is muonium, $p^+\mu^-$, made from a proton and a muon. While the two won't annihilate, the muon will decay into an electron and a pair of neutrinos within a few microseconds. Moreover, that's about the longest that you can get any such scheme to work.