The physical space is a space of orbits under $H$. That is, you start with some huge initial space $X$ but a little later you realize that many states represent the same physical state - any two of these are connected through the action of $H$. So, the physical space is $X / H$. Now, the physical space can be acted on by a symmetry group $G$. Now, to extend this to an action of $G \rtimes H$ on $X$ we need a connection.
Why is that? Well, we know how $H$ acts on all of $X$ and we know how $G$ acts on orbits of $X$ but we have no idea what does $G$ to the points in the orbit (besides the fact that they get all mapped to points in some other orbit). Connection is precisely the tool that gives us this identification of points between various orbits.
In the language of topology (all your sets and groups are finite and so we can give them discrete topology), $X \to X / H$ is a fibre bundle with vertical action of $H$. We are also given an action of $G$ on the base. And we'd like to lift it to a horizontal action of $G$ on all of $X$ but horizontal structure on the bundle is the same thing as connection. Now, once we have actions of both $G$ and $H$ on $X$, we can combine these two in an obvious way by twisting the horizontal action by the vertical action (similarly to how one can twist translations by rotations to get a full group of symmetries of Eucidean space).