What is the symmetry group of a particle interacting with external fields?

Question

I am following along with Ballentine's (in his Quantum Mechanics: A Modern Development) construction/identification of symmetry generators as operators representing the standard observables (observables here being used in the sense of a physical concept which have operators representing them) and I find myself a bit lost in the argument (I have attached the full argument at the end here in case).

My confusion is as follows. In the two cases involving isolated particles before this step, Ballentine uses that an isolated particle is invariant under the full set of Galilean symmetries. This invariance under the full set of Galilean symmetries implied some specific commutators which were then used in the construction of the operators ($\textbf{J},\textbf{G}, \textbf{P}, H$) representing key observables. Ballentine then goes on to say that in the case of a particle interacting with external fields we can no longer keep the commutators involving $H$. My question has to do with this last statement:

1. Where does it sneak that the particle is isolated in developing the commutators between $\textbf{J},\textbf{G}, \textbf{P}, H$? It seems like these commutators simply emerge from observing that there is a correspondence (homomorphism) between Galilean transformations on spacetime and on $\mathcal{H}$ which we enforce and thus arrive at the commutators? Nowhere I don't think do we say that the physics is invariant under these transformations? As far as I can tell we only enforce that when we talk about how the position operator $\textbf{Q}$ transforms. (This part of the question is not related to what's pictured. Hopefully I've given a sufficiently self-contained explanation of my issue though.)
2. Why can't we keep the commutators involving $H$ when the interactions are time-independent?

I think both of these questions may have something to do with Ballentine's paragraph beginning "One may ask..." but I'm not entirely sure I follow that.

Answer 1

• The presence of external fields ${\bf A}({\bf Q}),W({\bf Q})$ breaks the Bargmann algebra commutators (3.35g,h,i) involving the dynamical time-evolution operator $H$ in eq. (3.60).

• The other generators ${\bf G}, {\bf P}, {\bf J}$ generate a semidirect product between the 3D Heisenberg group of ${\bf G}, {\bf P}$ and the 3D rotation group of ${\bf J}$, cf. OP's title question.

• Ballentine argues that the other generators ${\bf G}, {\bf P}, {\bf J}$ are not affected by the external fields ${\bf A}({\bf Q}),W({\bf Q})$ because ${\bf G}, {\bf P}, {\bf J}$ essentially act as purely geometric operations within the Cauchy surface $$\Sigma~=~\{t\!=\!0\}.$$