The Berry's or Zak's phase is given as
\begin{align*} \gamma & =\oint_\mathrm{BZ}d\mathbf{k}\mathcal{\mathcal{\mathcal{A}}}(\mathbf{k})\ \ \mbox{mod }2\pi\\ & =i\oint_\mathrm{BZ}d\mathbf{k}\langle u_{n}(\mathbf{k})|\nabla_{\mathbf{k}}u_{n}(\mathbf{k})\rangle\ \ \mbox{mod }2\pi\\ \end{align*}
which can take any value between $[0,2\pi]$.
Now we want to learn the implications of the symmetries on this Berry's phase first assume that our system has only inversion symmetry such that the hamiltonian obeys
$$U_{p}H(\mathbf{k})=H(-\mathbf{k})U_{p}$$
then we have \begin{align*} U_{p}H(\mathbf{k})U(\mathbf{k}) & =E(\mathbf{k})U_{p}U(\mathbf{k})\\ H(-\mathbf{k})U_{p}U(\mathbf{k}) & =E(\mathbf{k})U_{p}U(\mathbf{k})\\ H(\mathbf{k})U_{p}U(-\mathbf{k}) & =E(-\mathbf{k})U_{p}U(-\mathbf{k}) \end{align*}
such that $E(k)=E(-k)$ and $U_{p}U(-\mathbf{k})=e^{i\phi(\mathbf{k})}U(\mathbf{k})$ where $U_{k}$ is the vector representation of $|u(k)\rangle$. Now the Berry connection is
\begin{align*} \mathcal{A}(\mathbf{k}) & =i(U_{p}U(-\mathbf{k})e^{-i\phi(\mathbf{k})})^{\dagger}\nabla_{\mathbf{k}}e^{-i\phi(\mathbf{k})}U_{P}U(-\mathbf{k})\\ & =i(U(-\mathbf{k}))^{\dagger}e^{i\phi(\mathbf{k})}\nabla_{\mathbf{k}}e^{-i\phi(\mathbf{k})}U(-\mathbf{k})\\ & =-\mathcal{A}(-\mathbf{k})+\nabla_{\mathbf{k}}\phi(\mathbf{k}) \end{align*}
and the Berry's phase is for a 1d $\mathrm{BZ}$ for simplicity
\begin{align} \gamma & =\intop_{-\pi}^{\pi}dk\,\mathcal{A}(k)\\ & =-\intop_{-\pi}^{\pi}dk\,[\mathcal{A}(-k)+\nabla_{k}\phi(k)]\\ & =\left.\phi(k)\right|_{-\pi}^{\pi}-\underbrace{\intop_{-\pi}^{\pi}dk\,\mathcal{A}(k)}_{\gamma}\label{eq:-18}\\ & =2\pi n-\gamma\label{eq:-17} \end{align}
thus we have \begin{align*} \gamma=\pi n \end{align*}
since $\gamma$ is well defined only up to $\mbox{mod }2\pi$ it can be $0$ or $\pi$ while obeying the last condition. My suspicion arises from the last step. The Berry phase should be gauge independent, but here it looks like gauge term determines its value.