Is direct, complete conversion of matter to energy theoretically possible? I know that small amounts of matter can be converted to energy via chemical and nuclear reactions, and that complete conversion is possible if matter meets anti-matter. Other types of conversion might proceed more exotically, e.g., in a black hole collision. But I have a more theoretical question; is there any reason in principle that there couldn't be some way to directly turn 100% of a normal piece of baryonic matter into pure energy, without the trouble of first making anti-matter? (And, ideally, without most of it coming in the form of gamma rays, though with a large enough reactor one could convert this to heat and hence useful energy.) Obviously we don't know any practical way to do this--we're not even at fusion yet--I'm just wondering if there's some fundamental barrier to it.
Part of my interest is speculation on the Fermi problem; if large civilizations even in distant galaxies used the most readily-available source of energy we are aware of (stars) via, say, dyson swarms, we would notice a spectrum change tipping us off to that, and haven't. But if they've discovered some way to just convert matter directly to energy, stellar energy might be passe for them, and they could be flying around their stars using relatively invisible energy sources.
 A: Some theories do allow baryon number violation, but they do conserve $B-L$, or the difference in baryon number and lepton number.
Under such a physics, proton decay is possible:
$$ p \rightarrow e^+ + \pi^0$$
followed by:
$$\pi^0 \rightarrow \gamma\gamma$$
In bulk hydrogen, the position would then undergo annihilation with the atomic electrons:
$$ e^+ + e^- \rightarrow \gamma\gamma$$
(Note, there these are the main decay channels, and are not meant to be comprehensive).
If this reaction used atoms (or isotopes) other than $^1H$, then you'd be left with neutrons, and they would also decay:
$$ n \rightarrow p + e^- + \bar\nu_e$$
leaving more products with which to make hydrogen.
So, yes, there is speculative physics (but not speculative to the point that large experiments haven't been conducted to look for it).
If the proton is unstable, then its lifetime is extremely long, and there is no standard physics that catalyzes the reaction, so engineering challenges remain.
A: Conservation of baryon number might be a problem.
