The reason for this is the origin of the gauge connection. Whenever you are setting up a theory of geometric phases you build yourself a fiber bundle equipped with a connection.
If the fiber bundle is the projective Hilbert space + U(1) gauge freedom you will end up with the usual Berry phase formula with the usual Berry connection only if you enforce the adiabatic assumption. Otherwise the story gets more complicated. The origin of this phase is therefore intrinsic to the dynamics.
In the case of the Aharonov-Bohm effect however, the gauge potential is not directly linked to the dynamics. It is some extra ingredient to the story. The U(1) gauge associated with the electromagnetic field does not rely on the adiabatic assumption as the electron dynamics does not alter the gauge connection.
To clarify the last point: Let $\Psi_0(\mathbf{r},t)$ be the freely propagating Schrödinger wavefunction in the absence of any electromagnetic fields. When we switch on the EM field, the Hamiltonian changes, in fact you now have
$$ H = \frac{(\mathbf{p} - q \mathbf{A})^2} {2m} + q \phi, $$
where $\mathbf{A}$ and $\phi$ are EM vector and scalar potential. A solution to the Schrödinger equation with EM field is then given by $\Psi(\mathbf{r},t) = \exp\left(\frac{i}{\hbar} \zeta(\mathbf{r},t) \right)\Psi_0(\mathbf{r},t),$
where
$$\zeta(\mathbf{r},t) = \int_\Gamma A^\mu(x') \text{d} x'_\mu .$$
Here I have used the four-vector notation $A = ( \phi / c, \mathbf{A})$, x = $ ( ct, \mathbf{r})$, and $\Gamma$ denotes a path from some $x_0$ to $x$. This transformation is the origin of the Aharonov-Bohm phase. At no point any adiabatic assumption had to be forced on the solution. The interpretation however is similar. You can view the four-vector potential as a connection in a fibre bundle just like the Berry connection.