I am having some conceptual difficulties with energy and momentum stored in the EM field.
Maybe that's because there's some conflation between field and force. See Minkowski's Space and Time:
"In the description of the field caused by the electron itself, then it will appear that the division of the field into electric and magnetic forces is a relative one with respect to the time-axis assumed; the two forces considered together can most vividly be described by a certain analogy to the force-screw in mechanics; the analogy is, however, imperfect."
See how he said the field? And then talked about electric and magnetic forces? Also see section 11.10 of Jackson's Classical Electrodynamics where he says "one should properly speak of the electromagnetic field Fμν rather than E or B separately". IMHO that's because E represents linear electric force, and B represents rotational magnetic force. And the important point is that there is no force density as some point in an electromagnetic field. The forces are only there when two electromagnetic fields interact. It takes two to tango.
So, my question is, what accounts for this remaining term?
I don't know. But since you have conceptual difficulties, maybe you need a conceptual approach. Start with Compton scattering, where some of the E=hf photon wave energy is converted into electron kinetic energy. Do another Compton scatter on the residual photon, and another and another ad infinitum. In the limit there’s no photon left. When you take all the energy out of a wave, it just isn’t there any more. It has been entirely converted into the motion of electrons. And yet in pair production, you can make an electron (and a positron) out of a photon. So in a way, the electron is made of motion. Or energy if you prefer. Or better still, energy-momentum.
Note that you can diffract electrons. And note that in atomic orbitals electrons "exist as standing waves". Think of the photon as something akin to a seismic wave which displaces you 1m left then 1m right. If you could make it go round a spin ½ Dirac's belt path, it would be a static 1m displacement. (Get a feel for this by cutting out 1-wavelength sine-wave paper strips and wrapping them into Möbius strips). You end up with a standing field rather than a field variation, and the momentum is hidden. See the static field section of the Wikipedia Poynting vector article? See this picture by Michael Lenz?
Poynting vector in a static field, where E is the electric field, H the magnetic field, and S the Poynting vector.
Well there is no electric field E, and there is no magnetic field H. Or B. It's the electromagnetic field, and to even begin to depict it you have to combine the typical radial and circular depictions of E and B. See this answerthis answer where I did this, and think of the hidden momentum as something like frame-dragging. It doesn't stay hidden when electron-positron annihilation occurs. Then your p=hf/c gamma photons are off like a shot, at c, from a "standing" start. A standing wave looks motionless, but it isn't. The motion is hidden. The momentum is hidden. The energy-momentum is hidden. And if it wasn't, the standing electromagnetic field wouldn't be there.