The center symmetry in QCD comes from the
$$a\ \mathcal{P}\mathrm{exp}\left(ig_s \int_C dx^\mu \ A_\mu(x)\right) a^{-1} = \mathcal{P}\mathrm{exp}\left(ig_s \int_C dx^\mu \ A_\mu(x)\right),$$ where $C$ is a loop and $a \in \{e^{2\pi i k/3}I| \ k =0,1,2\} \simeq \mathbb{Z}_3$ is an element of the $\mathrm{SU}(3)_C$ center group. This makes the vacuum expectation value of the Wilson loop
$$W[C] = \mathrm{Tr}\left[\mathcal{P}\mathrm{exp}\left(ig_s \int dx^\mu \ A_\mu(x)\right)\right],$$ $\langle W[C] \rangle$ should be also center-invariant.
Then, when the center symmetry is broken (deconfinement), this implies that $W[C]$ is not more an element of $\mathrm{SU}(3)$ because it doesn't commute with center elements. This does make $\mathrm{det}(W[C]) \neq 1$?