One often come across in QFT sentences like the following, for instance:
...under a Lorentz transformation $\Lambda$ implemented by the unitary operator $U(\Lambda)$, a Dirac field transforms as
$$ U(\Lambda)^{-1}\Psi(x)U(\Lambda) = D(\Lambda) \Psi(\Lambda^{-1}x),$$ where the matrix $D$...
What would be the other case, i.e. implemented by a non-unitary operator? Is $N(x)$ an example of such an operator?
$N(x):=\mathrm{i}x+x^2$.
Poincaré invariance is a fundamental requirement of relativistic (quantum) physics. In particular if $U_g : {\cal H} \to {\cal H}$ represents the (non-necessarily linear) action of a Poincaré transformation $g$ on (normalized) vectors $\psi$ of the Hilbert space $\cal H$ of the considered system, transition probabilities have to be preserved: $$|\langle U_g(\psi) | U_g(\phi) \rangle|^2 = |\langle \psi | \phi \rangle|^2 \quad \forall \psi, \phi \in {\cal H}\:, ||\psi||=||\phi||=1\:. \quad (1)$$ A celebrated theorem due to Wigner establishes that a (bijective) map $U : {\cal H} \to {\cal H}$ verifying (1) must necessarily be linear and unitary or anti linear and anti unitary, depending on the physical nature of the transformation.
Concerning representations of Poincaré group $\cal P$, by definition they have to satisfy, in addition to (1), ${\cal P} \ni g \mapsto U_g$ with $U_g U_h = U_{g\cdot h}$ ($^*$) and $U_e = id$, where $\cdot$ is the group product in $\cal P$, just in view of the definition of group representation. In principle each $U_g$ has to be unitary or anti unitary.
If $g\in \cal P$ belongs to the proper orthochronous subgroup ${\cal P}_+^\uparrow$, it can always be decomposed as $g= h\cdot h$ where $h$ still belongs to the same subgroup. Therefore $U_g= U_{h} U_{h}$, thus $U_g$ must be unitary (even if $U_h$ is anti unitary,as the composition of a pair of anti unitary operator is always unitary).
We conclude that the orthochronous Poincaré group ${\cal P}_+^\uparrow$ (and consequently the orthochronous Lorentz group $SO(1,3)^\uparrow$) can only be represented by unitary operators in quantum theories, when the action of the group is on states.
Non unitary representations arise dropping the last requirement. For instance dealing with Dirac or Weyl spinors.
($^*$) Actually a phase could take place, since states are represented by normalized vectors up to phases: $U_gU_h = e^{i\alpha(g,h)}U_{g\cdot h}$, however it does not change the result of the subsequent reasoning. As a matter of fact, it is possible to prove that continuous (projective) unitary representations of ${\cal P}_+^\uparrow$ are not affected by such phases, differently form representations of Galileo's group where those phases play a crucial role.
