What makes spin 1/2 anyons special?

If the spectrum of a TQFT contains a fermion, the theory becomes a spin-TQFT, and it depends on the spin structure of the manifold (cf. 1505.05856). On the other hand, if no such anyon exists, the TQFT can be formulated without specifying such structure.

This seems very unintuitive to me, because the spectrum of a generic TQFT, spin or otherwise, contains anyons whose spin take rational values from zero to one. In other words, we may find bosons (spin $0$), semions (spin $1/4$), fermions (spin $1/2$), and essentially any other rational value (e.g., $7/11$, $2/5$, etc.).

I understand that in $d\ge4$, the spin can only take integer and half-integer values, and therefore the fermionic particles are special. But in $d=3$, where all spins are allowed, fermions seem to lose their special status. Why would a theory containing spins $\{0,1/3,3/7\}$ be independent of the spin structure, but it suddenly depends on it if you include the number $1/2$? It seems to me that the number $1/2$ is the "least weird" one in the zoo of fractional spins. Yet it is precisely this number what changes the theory from a TQFT to a spin-TQFT. What makes spin 1/2 anyons special, as opposed to anyons of other rational spins?

Your premise is faulty. It is simply not true that a TQFT becomes a spin TQFT simply because it contains a fermionic excitation. There are many counterexamples -- the simplest would be $\mathbb{Z}_2$ gauge theory (the low energy theory of the toric code). However, note that these are emergent fermions in a system whose microscopic degrees of freedom are purely bosonic. (One consequence of this is that these fermions are never "transparent" -- there is always an excitation in the theory which braids non-trivially with the fermion).
Where spin TQFTs do arise in physics is in describing systems whose microscopic degrees of freedom are fermionic (for example, topological phases of electrons). The difference here is that the fermions are transparent -- there are no excitations which they braid non-trivially with. Note that fermions and bosons are the only particles which can be transparent, because if an Abelian anyon $x$ has topological spin $\theta$, then moving an $x$ particle all the way around another $x$ particle picks up phase $\theta^2$, and only fermions ($\theta = -1$) and bosons ($\theta = +1$) have $\theta^2 = 1$.