I probably should have searched more extensively before posting here, I did eventually find this Hamiltonian in Exceptional and regular spectra of a generalized Rabi model by Michael Tomka, Omar El Araby, Mikhail Pletyukhov, and Vladimir Gritsev. The context is different from quantum optics (which is probably why I could not locate it earlier), it is a model of a two-dimensional electron gas with parabolic confinement in one direction, with a Zeeman field and Rashba and Dresselhaus spin-orbit couplings. The three-Pauli-matrix Rabi Hamiltonian is also referred to as the anisotropic Rabi Hamiltonian (arXiv:1401.5865).
In my answer to the Mathoverflow question I made a conjecture for the ground state energy in the special case that $\Delta=2g_xg_y$:
The generalised Rabi Hamiltonian $$H_0=\begin{pmatrix} a^\dagger
a+\alpha^2&\alpha a+\beta a^\dagger\\ \alpha a^\dagger+ \beta
a&a^\dagger a+ \beta^2 \end{pmatrix}$$ has a doubly degenerate ground state at zero energy
for any $\alpha,\beta\in\mathbb{R}$.
I now see that this conjecture follows from the Bethe Ansatz solution of the generalized Rabi Hamiltonian given in the Tomka et al. paper, section II.B.1. The case $\Delta=2g_xg_y$ corresponds to $\lambda_-=\delta$ in their notation, which is the "exceptional case"$^\ast$ in which the Rabi Hamiltonian has an eigenvalue $E_0=-\lambda_+=-(g_x^2+g_y^2)$. This corresponds to a zero eigenvalue for $H_0$, because the spectrum of $H_0$ is shifted relative to the Rabi spectrum by $(\alpha^2+\beta^2)/2=g_x^2+g_y^2$.
The zero eigenvalue is the ground state, because $H_0$ is positive definite,
$$H_0=Q^\dagger Q,\;\;\text{for}\;\;Q=\begin{pmatrix} \alpha&a\\ a&\beta \end{pmatrix}.$$
The two ground state wave functions, such that $Q\Psi_\pm=0$, are $\Psi_\pm={{\sqrt\beta}\choose{\mp\sqrt\alpha}}|\!\pm\!\!\sqrt{\alpha\beta}\rangle$, in terms of the coherent state $|\gamma\rangle$ with $a|\gamma\rangle=\gamma|\gamma\rangle$.
$^\ast$ In arXiv:1407.5213 the symmetry is identified that enforces the doubly degenerate ground state.