The entire phase factor is a function of $\mathbf{k}$. The center of the wave packet is where the individual waves interfere the most constructively. In order to themaximize this constructive interference, the phase factors should be as close to each other as possible, and that occurs precisely at a stationary point, where the derivative of the phase factor with respect to $\mathbf{k}$ is zero. As we know from elementary calculus, there is no first-order change in a function at its stationary point.