More specifically, would the distortion from a sphere of the planetary body lead to shifts in angular momentum for the moon.
If you are dealing with a closed system (a planet and its moon only), then the orbital motion is due to gravitational force, which is a central force. The total angular momentum of the system will remain constant.
The angular momentum may be transferred back and forth between the planet and the moon, depending on the spherical aberrations of each, but the angular momentum about the center of mass of the system will be constant.
So, yes, it's possible that the angular momentum of the moon could change and its orbit about the center of mass of the planet be non-elliptical. But it would be a repetitive pattern.
This is actually how we have mapped the spherical aberration of Earth using a pair of satellites, one behind the other. Their relative velocities are mapped as they travel in a common orbit. NASA's version was/is called GRACE.