Topological insulators with time-reversal symmetry: How to derive $\mathbb{Z}_2$-invariant from the zeros of $\langle u_i(k)|T|u_j(k)\rangle$?

Following the book ''Topological Insulators and Topological Superconductors'' by B. Bernevig (esp. chapter 10.1), I want to understand how to derive a Z2 invariant starting from the zeros of the off-diagonal element $$P(k)$$ of the matrix

\begin{align} \langle u_i(k)|T|u_j(k)\rangle = -\langle u_j(k)|T|u_i(k)\rangle = \epsilon_{ij}P(k) \end{align}

which gives the overlap between a state $$|u_i(k)\rangle$$ and it's time reversed partner $$T|u_j(k)\rangle$$ with $$i=1,2$$ the index of one of the two filled bands (minimal required number of filled bands due to Kramer's theorem) and $$T$$ with $$T^2 = -1$$ the time-reversal operator for spin-1/2 particles. $$P(k)$$ is in this case equal to the pfaffian of the matrix.

According to the book, two important subspaces of the Hilbertspace are the so called even and odd subspaces. In an even subspace, the spaces spanned by the $$|u_i(k)\rangle$$ and $$T|u_i(k)\rangle$$ are the same, while in the odd subspace they are orthogonal. Since $$T$$ reverses momentum, the eigenfunctions $$|u_i(k^*)\rangle$$ at time-reversal-invariant momenta $$k^* = -k^* + G$$, with $$G$$ a reciprocal lattice vector, will span an even subspace, while for all other $$k$$ the above matrix is zero, the spaces spanned by $$|u_i(k)\rangle$$ and $$T|u_i(k)\rangle$$ therefore orthogonal (they have different momenta) and we have an odd subspace. This is at least what I think.

In the book they talk about special points where $$P(k)$$, the off-diagonal matrix-element, is zero. One can then assign a vorticity to these points and derive the Z2 invariant (number of vortices in half of the BZ) from there, since an even number of vortices can always annihilate.

My problem is that as I understand it there should not be some special points where $$P(k) =0$$, but $$P(k)$$ should be zero everywhere except for TR-invariant points. Can you spot my error? Help is very appreciated!

The mentioned chapter of the book follows the article Z2 Topological Order and the Quantum Spin Hall Effect by Kane and Mele, 2005, by the way

I think your problem is you are assuming $$\langle u_k|u_\ell\rangle=0$$ if $$k\neq \ell$$. But this isn't true, at least in the convention Kane and Mele use. Their convention is that $$|u_k\rangle=e^{-ik\hat{x}}|\psi_k\rangle$$, where $$|\psi_k\rangle$$ is the eigenstate of the full Hamiltonian at quasimomentum $$k$$. Then $$|u_k\rangle$$ is just a wavefunction that has the same periodicity as the lattice. There's nothing stopping you from taking the inner product of this with $$|u_\ell\rangle$$, and you won't necessarily get zero.
I think you're confused by the fact that $$\langle\psi_k|\psi_\ell\rangle=0$$ when $$k\neq \ell$$. But this doesn't carry over to their corresponding $$|u\rangle$$s.
I'll add that there are also conventions where $$\langle u_k|u_\ell\rangle=0$$ if $$k\neq \ell$$ by definition. But this convention doesn't let you define the $$\mathbb{Z}_2$$ invariant, so they don't use it.
In addition to changing the phases of the individual wave functions, the wave functions can be mixed by a general $$U(2N)$$ transformation of the form $$\lvert u_{k,n}\rangle\rightarrow\sum_{m}U_{nm}(k)\lvert u_{k,m}\rangle \qquad(3.3)$$. After this transformation, $$\lvert u_{k,m}\rangle$$ need no longer be the individual eigenstates of the Hamiltonian, but rather should be interpreted as basis vectors spanning the space spanned by the 2N occupied eigenstates.