If I'm reading your question right, I think you're having a relatively common issue. Feynman himself had the same issue when confronted with a creation operator for an electron. "How can an electron really be created? It violates the conservation of charge!"
The point is that not every operator needs to represent a physical realizable time evolution. Quite often, you just want to consider the formal properties of an abstract operator, regardless of whether it has any direct meaning itself. (In fact, at the level of quantum field theory, it certainly doesn't -- the particle exchange operator does nothing whatsoever, because the particles are identical. The simpler formalism of nonrelativistic quantum mechanics doesn't have that built in, which is why it's useful to consider the exchange operator.)
You've already seen this attitude earlier in quantum mechanics. For example, the position operator $\hat{x}$ is not a "physical thing", in the sense that if you have a state $|\psi \rangle$, $\hat{x} |\psi \rangle$ is not necessarily a sensible physical state. Instead $\hat{x}$ is useful because it represents an observable quantity. In quantum mechanics, operators are our basic tools. Some represent observables, some represent physical time evolutions, and some represent neither, and there's nothing wrong with those last ones.