How do I understand different realizations of symmetry in the absence of fractionalization? To use a simple example to ask my question, consider the two dimensional toric code with a $Z_2$ global symmetry acting in two ways:


*

*The most boring trivial way possible.

*By permuting the charge and flux excitations.


Are the phases corresponding to the two situations (neither of which involves any `symmetry fractionalization') distinct? 
A more detailed version of my question- Consider a 2+1 dimensional topologically ordered system with a global symmetry $G$. If $\mathcal{C}$ is the Modular Tensor Category that describes the topological order and $\mathcal{A}$ is the group of Abelian anyons, the symmetry action is described by a map $\rho$
\begin{equation}
\rho: G \rightarrow \text{Aut}(\mathcal{C})
\end{equation}
Given a specific $\rho$, $G$ and $\mathcal{C}$, it was shown in the paper by Barkeshli et al that (provided an obstruction class valued in $H^3_{[\rho]}(G,\mathcal{A})$ vanishes) different symmetry fractionalization classes correspond to the elements of $H^2_{[\rho]}(G,\mathcal{A})$ and these also label different SET phases. My question is about how to understand the trivial element of $H^2_{[\rho]}(G,\mathcal{A})$ for different $[\rho]$- do they correspond to different phases of matter? 
 A: The reason I was confused about this question was because I was comparing the classification of Symmetry Enriched Topological (SET) phases with the classification Symmetry Protected Topological (SPT) phases. Consider, for example the classification of bosonic SPT phases with a unitary on-site action of a symmetry $G$. It is known that there exist SPT phases classified by elements of $H^{d+1}(G,U(1))$ meaning we can associate an element of $H^{d+1}(G,U(1))$ to each SPT phase. In particular, the trivial element of $H^{d+1}(G,U(1))$ corresponds to the trivial phase i.e with a ground state that can be adiabatically connected to the product state without breaking symmetry. This gave me the impression that in the case of SET phases, I should expect the trivial element of $H^2_{[\rho]}(G,\mathcal{A})$ to correspond to the same 'trivial fractionalization pattern' irrespective of what the $\rho$ is. This is wrong for two reasons.
1) Firstly, as @DominicElse mentioned, different actions on anyons cannot be interpolated and hence correspond to distinct phases.
2) Secondly, with a fixed action $\rho$ on the anyons, the different SET phases corresponding to different fractionalization classes are not labeled by $ H^2_{[\rho]}(G,\mathcal{A})$ rather they form a $H^2_{[\rho]}(G,\mathcal{A})$ torsor meaning (unlike the similar situation with the classification of SPT phases) there is no canonical way to associate an element of $H^2_{[\rho]}(G,\mathcal{A})$ to a fractionalization class rather different fractionalization classes are related to each other by an element in $H^2_{[\rho]}(G,\mathcal{A})$. So my intuitive understanding of what the trivial element of $H^2_{[\rho]}(G,\mathcal{A})$ corresponds to was wrong. 
All this is clearly explained in the paper by Barkeshli ei al.
