Why is the phase picked up during identical particle exchange a topological invariant? I've been wondering about the standard argument that the only possible identical particles in three dimensions are bosons or fermions. The argument goes like this:

Consider exchanging the positions of two identical particles in 3D. Because the particles are identical, $|\Psi|^2$ remains the same after having performed the exchange, so we must have
  $$\Psi \to \Psi e^{i\theta}.$$
  If we perform the exchange twice, the path described is homotopic to the trivial path. This is easiest to see by realizing that the double exchange of particles A and B is the same as having particle A encircle particle B completely, and this path can just be lifted into the third dimension and shrunk down to a point. Alternatively, it's because $\pi_1(S^2/\mathbb{Z}_2) = \mathbb{Z_2}$.
Because the phase picked up along a path is a homotopy invariant, and the path of a double exchange is homotopic to the trivial path,
  $$e^{2 i \theta} = 1$$
  which implies that under a single exchange, $\Psi\to\pm\Psi$, corresponding to bosons and fermions.

This argument sounds good, but it sneaks in the crucial physical input without justification: why should the phase be a topological invariant? Why can't it change under a deformation of the path? Not only do I not know how to prove this, it doesn't even appear to be true; for example, if there were a magnetic field, changing the path would change the magnetic flux through it, and hence the phase.
 A: The phase picked up during identical-particle exchange actually isn't a topological invariant. It comes from the combined contribution of many different factors:


*

*The standard dynamical phase $\sim e^{-i E t}$ resulting from Schrodinger time evolution and the fact that the exchange takes a positive amount of time

*The Berry's phase, which does not depend on the exchange duration but does depend on its specific geometric (not merely topological) path

*The instantaneous energy eigenbasis transformation (in the nonabelian case)

*Diabatic effects resulting from the fact that the exchange process is not infinitely slow

*Thermal effects if the system is at finite temperature

*Finite-size interaction effects from the fact that the anyons are not infinitely far-separated
These various factors do not always separate cleanly, but roughly speaking, the combined effects of the second and third contributions only depend on the braid's topology. For a gapped topological phase, the last two are exponentially small in the appropriate regimes, but they're always there. But for any physical system, there is actually only a window of exchange times, which is bounded in both directions, in which the topological phase dominates. If the exchange is too fast then diabatic errors mess things up, but if it's too slow than finite-size errors mess things up. In practice, the former issue is more problematic, because diabatic errors are generically only suppressed algebraically while finite-size errors are suppressed exponentially. Some of these phase shifts - particularly the dynamical one - can (hopefully) be ignored, because they are the same for every anyon that participates in the braid, and only phase differences are physically relevant.
For abelian anyons the topological phase is basically just an Aharanov-Bohm phase that the particle acquires (in addition to an ordinary dynamical phase) from circling magnetic flux, as you say. The key difference from the regular Aharanov-Bohm setup is that the flux is very tightly localized around the other anyon. So basically, if the other anyon is in the loop then the moving anyon circles all the flux and picks up the full exchange phase, and if it's out of the loop then it doesn't encircle any flux. In reality, for a system with a finite correlation length the anyons are not perfectly localized and some flux will leak into or out of the loop, but since the effective magnetic field around an anyon decays exponentially in a gapped phase, we can neglect this effect if the anyons stay separated much farther than the correlation length. Part of the reason why the importance of non-topological contributions to anyon phases isn't fully appreciated is that most people build up their intuition on the toric code, which is highly non-generic because it has zero correlation length, so anyons are "well-separated" even on adjacent lattice sites.
See the introduction to this paper for a discussion of these various subtleties.
In practice, theorists often sweep all these subtleties under the rug and only focus on the topological phase, although in reality the other contributions are always there. For example, topological quantum field theories can only capture the physics of the topological phase contributions, so any results derived from that formalism will necessarily fail to describe the other effects in a real system.
A: I think an answer can be found in this great paper. I'll write down the argument as I understand it. 
Consider 2 identical particles in 3D. The configuration space is $C=(\mathbb{R}^3 \times \mathbb{R}^3)/\mathbb{Z}_2$, where we quotient by $\mathbb{Z}_2$ since the particles are identical. By going to relative coordinates, $C=\mathbb{R}^3 \times (\mathbb{R}^3/\mathbb{Z}_2)$, where $\mathbb{R}^3$ is associated with the center of mass, and where the second is the space of relative coordinates $x\in \mathbb{R}^3$ which are identified with $-x$. This gives a singular point at $x=0$, and we remove it (we keep the particles apart). Thus, the relative part is $(\mathbb{R}^3-{0})/\mathbb{Z}_2=]0,\infty[\times (S^2/\mathbb{Z}_2)$. 
First focus on the part $S^2/\mathbb{Z}_2$. As I mentioned above in the question, closed loops in this space fall in two classes. If we think of the sphere $S^2$ before identifying antipodal points, these classes are classes of paths that 1. begin and end on the same point 2. begin and end on antipodal points. 
The paths in class 1, viewed in the total space $(\mathbb{R}^3-{0})/\mathbb{Z}_2$, do not encircle the removed point at 0 (or can be continuously deformed to a path that doesn't). The paths in class 2 encircle the origin once (or can be continuously deformed to a path that encircles it once). 
The actual argument can now be made using parallel transport of vectors along closed loops in the configuration space. For each $x$ we define a Hilbert space $h_x$ such that $\Psi(x)\in h_x$ is the value of the wave function at $x$. Thus we think of $\Psi(x)$ as a vector that we will parallel transport. Defining a basis $\chi_x$ for $h_x$, we can write $\Psi(x)=\psi(x)\chi_x$. We have a choice of basis, which is a gauge freedom. 
We then define a linear, unitary operator $P(x',x) : h_x \to h_{x'}$ which parallel transports vectors of $h_x$ to vectors of $h_{x'}$. It is assumed that it takes the form 
$P(x+dx,x)\chi_x = (1+i dx^k b_k (x) ) \chi_{x+dx}$ 
for infinitesimal parallel displacement. The $b_k$ are functions analogous to the vector potential $A_\mu$, and define gauge-invariant differentiation
$D_k = \partial_k -i b_k (x) $. 
In turn, this defines a "curvature" tensor $f_{kl} = i [D_k ,D_l]$ analogous to $F_{\mu \nu}$ in EM or $R_{\mu \nu}$ in GR. The idea, I think, is that parallel transport is trivial along paths which enclose no curvature, but that it can have an effect if the region enclosed is curved. 
In our case, $f_{kl}$ is chosen zero for all points in the configuration space, but it is not defined for the excluded point at the origin. 
Suppose we consider a path that does not encircle the origin. In that case, the enclosed "curvature" is zero, and we have $P(x,x)\equiv P(x) = 1$. This holds for any path that does not encircle the origin - deforming a path continuously will retain this property. If we consider a path that encircles the origin once, we write $P(x) = \exp[i\zeta]$, since $h_x$ is one-dimensional. Again, if we continuously deform such a path, we cannot change the enclosed "curvature", so the phase will be the same for all such paths.  
Then a path corresponding to a  "double exchange" yields $P(x) = \exp[2i\zeta]$. Now we continuously deform it back to the trivial path, which we can do without passing through the origin. The "curvature" enclosed for the "double exchange" path is therefore the same as that of the trivial path, namely zero (there should be some nicer way of looking at this). Since for all such paths $P(x)=1$, we have $\exp[2i\zeta]=1$. 
If we consider the 2D case, we would get infinitely many classes of paths, since the configuration space would be $\mathbb{R}^2 \times (]0,\infty[\times S^1/\mathbb{Z}_2)$. Yet, the same idea should apply. 
