See Weiberg's QFT, Vol. I, section 10.3, the equation below 10.3.6. Cleaning up the notation a bit, and using the modern normalisation of spinors together with the mostly minus metric convention, the the pole of propagator for a general field is
$$
\Delta_{\ell\ell'}(p)=\frac{|Z|^2}{p^2-m^2+i\epsilon} M_{\ell\ell'} \tag{1}
$$
where $M$ is the usual projection matrix,
$$
M_{\ell\ell'}=\sum_\sigma u_\ell(p,\sigma)u^*_{\ell'}(p,\sigma) \tag{2}
$$
Using the fact that polarisation vectors are always orthogonal to each other (i.e., $u^\dagger v=0$), we see that
$$
Mv=0 \tag{3}
$$
On the other hand, using the fact that $u^\dagger u=|u|^2>0$, we see that
$$
Mu=|u|^2u \tag{4}
$$
Therefore, as the $u,v$ polarisation vectors are a basis, we see that $M\ge 0$ is a non-negative matrix (its eigenvalues are either zero or positive).
But there is a subtlety hiding in $|u|^2>0$: the polarisation vectors
$$
u(p)\equiv \langle 0|\psi(0)|p\rangle \tag{5}
$$
have positive norm if and only if $|p\rangle$ has positive norm. The same thing can be said about the polarisation vectors of antiparticles, defined as
$$
v(p)\equiv \langle \bar p|\psi(0)|0\rangle \tag{6}
$$
where the bar indicates antiparticle (opposite charge). The objects $u,v$ are the usual polarisation vectors one includes in the external lines of scattering amplitudes (in the case of spin $j=1$ particles, the usual notation is $\varepsilon^\mu$, but it is the same concept).
Therefore, the residue of the propagator $M$ is non-negative only if the physical sector has positive norm, that is, if the asymptotic states $|p\rangle$ have positive norm. If you have an external line with $|u|^2<0$, then the matrix $M$ becomes indefinite (it's no longer non-negative).
Note that in gauge theories, there are unphysical states as well, but these should never appear on external lines. These states are allowed to have negative norm. For example, in the case of spin $j=1$ particles, the polarisation vectors are space-like, but the longitudinal states $\varepsilon^\mu=p^\mu$ are time-like. But if we use $\varepsilon^\mu=p^\mu$ on an external line, the amplitude is zero (by the Ward identity).
Remark: the polarisation vectors $u,v$ are only a basis for the physical Hilbert space. Ghosts are orthogonal to the physical $u,v$, and they have negative norm, $u^\ell u_\ell^*<0$. As an example, consider once again the longitudinal polarisations for spin $j=1$ particles: the physical polarisation vectors are space-like, while the longitudinal polarisation is time-like. Together, these four vectors span $\mathbb R^4$, while the physical polarisations only generate space-like vectors.
