Building on the excellent answer of HTNW (and also thanks to the comments of Ján Lalinský), I wish to elaborate a bit about inverting the primary and the secondary.
It is true that the primary and the secondary can be inverted while keeping the same currents, so that the same magnetic field is generated in both case. But regarding the electric field, it is also dependent upon the charge distribution on the surface of the wires, according to Jefimenko's equations (see note below); this charge distribution increases (more or less linearly in most cases) in the direction of the increasing potentials for the secondary of a transformer, and decreases in the direction of the increasing potentials for the primary. So, by inverting the primary and the secondary, keeping all currents the same, the charge distribution will revertinvert its direction in both the primary and the secondary. By symmetry, the electric field will be inverted, and
as a result, the Poynting vector will still be directed from the (new) primary to the (new) secondary. That explains the apparent paradox.
Note: Jefimenko's equations connects the EM field with the sources: $$\mathbf{E}(\mathbf{r}, t) = \frac{1}{4 \pi \varepsilon_0} \int \left[\frac{\mathbf{r}-\mathbf{r}'}{|\mathbf{r}-\mathbf{r}'|^3}\rho(\mathbf{r}', t_r) + \frac{\mathbf{r}-\mathbf{r}'}{|\mathbf{r}-\mathbf{r}'|^2}\frac{1}{c}\frac{\partial \rho(\mathbf{r}', t_r)}{\partial t} - \frac{1}{|\mathbf{r}-\mathbf{r}'|}\frac{1}{c^2}\frac{\partial \mathbf{J}(\mathbf{r}', t_r)}{\partial t} \right] dV',$$ $$\mathbf{B}(\mathbf{r}, t) = -\frac{\mu_0}{4 \pi} \int \left[\frac{\mathbf{r}-\mathbf{r}'}{|\mathbf{r}-\mathbf{r}'|^3} \times \mathbf{J}(\mathbf{r}', t_r) + \frac{\mathbf{r}-\mathbf{r}'}{|\mathbf{r}-\mathbf{r}'|^2} \times \frac{1}{c} \frac{\partial \mathbf{J}(\mathbf{r}', t_r)}{\partial t} \right] dV',$$ where $\mathbf r$′ is a point in the charge distribution, $\mathbf r$ is a point in space, and $$t_r = t - \frac{|\mathbf{r}-\mathbf{r}'|}{c}$$ is the retarded time.
From these equations, it is not obvious that the E-field should be inverted if the charge distribution is inverted in the coils, but it should be possible to show the expected symmetry by a technical derivation.