it seems to me that the equation $|n,k\rangle=e^{ikx}u_{n,k}(x)$ is a bit confusing. I would agree more if it were something like $\langle x|n,k\rangle=e^{ikx}u_{n,k}(x)$. Afterall, taking a quick look on the paper, they do not write down an equation like that anywhere. So, having said that (I will just offer the intermediate steps and the intermediate steps only, as I have no understanding of the subject!), one can write

$$\langle n,k|x|n',k'\rangle=\int dx \langle n,k| x|x\rangle \langle x|n',k'\rangle$$
and this should be equal to (after substituting $\langle x|n',k'\rangle$)

$$\langle n,k|x|n',k'\rangle=\int dx\,xe^{-i(k-k')x}u^*_{n,k}(x)u_{n',k'}(x)$$
Now, the variable $x$ can be written as $i\partial/\partial k$ and hence

$$\langle n,k|x|n',k'\rangle =i\int dx \frac{\partial}{\partial k} e^{-i(k-k')x}u^*_{n,k}(x)u_{n',k'}(x)$$
The last step involves integrating by parts (but be carefull! $u_{n,k}(x)$ also depends on $k$). So,

$$\langle n,k|x|n',k'\rangle=i\frac{\partial}{\partial k}\int dx  e^{i(k-k')x}u^*_{n,k}(x)u_{n',k'}(x)-i\int dx e^{i(k-k')x} \frac{\partial}{\partial k} u^*_{n,k}(x)u_{n',k'}(x)$$
Now, you can see that the second term is of the form given by what you denote as $X_{n,n'}$ and in fact you can use the form to calculate the normalization constant $N$. Moreover, the first term can be identified with the first term provided by you in your post (there is some sort of weird orthogonality relation, but this is for you to figure out)... If there are questions, please do not hesitate to comment.