The thermoelectric effects are indeed thermodynamically reversible. You are absolutely right that they are always accompanied by irreversible effects like Fourier's conduction law and the Joule effect. However one can focus solely on the reversible part, which encompasses all the thermoelectric effects, and this part is thermodynamically reversible. More precisely, as you say there is a Thomson heat that is generated or absorbed. However this heat is equivalent to the heat one should supply along the sample for it to stay with an unperturbed temperature profile when both a current and a temperature gradient are simultaneously present in the sample. This heat exchange between the sample and its surrounding does not produce entropy, it is merely a transfer of entropy. So yes, as you say, there are entropy fluxes due to the thermoelectric effects, but these do not increase the entropy of the system "sample + surroundings". They can either increase or reduce (or keep constant) the entropy of the sample itself, but they always maintain the whole system's entropy constant. It is in this way that the thermoelectric effects are fully thermodynamically reversible.
I find the Wikipedia sentence involving the ZT factor misleading, for it does not help to get to know why TE effects are thermodynamically reversible. Your references are more valuable for that purpose.
More precisely the following explanation is what they (Wikipedia) have in mind: when $ZT = \sigma S^2 T / \kappa$ is worth infinity, what they have in mind is that both $\kappa$ and $\rho = 1/\sigma$ are worth $0$ (note that this is mathematically not a necessity). In that case there is no heat conduction via Fourier's law and there is no Joule resistance. Then they assume that such a system can still produce a finite current (different from superconductivity which has $S=0$) solely due to the reversible TE effects. When all of these impossible conditions are happening all at once, the TE material can be used to make a engine having the efficiency of a Carnot engine, i.e. the engine based on the TE effects is thermodynamically reversible, hence the TE effects are thermodynamically reversible. But as pointed out, logically it does not fully make sense, and not only because finding a material with $ZT$ equal to infinity is unreachable.