A field in the $(A,B)$ representation of the Lorentz Group has a propagator that scales as $|p|^{2(s-1)}$ for $|p|\to\infty$, where $s=A+B$ is the "spin" of the field (Ref.1 §12.1). Therefore, the propagator is a decaying (or constant) function of $p$ if and only if $s=0,\,1/2,\,1$. Otherwise, the propagator grows in the UV and the theory is non-renormalisable (unless we have SUSY1, which in principle may allow you to go up to $s=3/2,2$). If fields of that type are actually realised in Nature, their effect is invisible in the IR (essentially, by dimensional analysis; more formally, by the standard classification of irrelevant interactions, cf. Ref.1 §12.3). That is why they have not been detected so far.
This leaves as the only options $(0,0),\,(\frac12,0),\,(0,\frac12),\,(1,0),\,(\frac12,\frac12),\,(0,1)$. All of these are used in the Standard Model but for $(1,0),(0,1)$ (these are the self-dual and anti-self dual anti-symmetric second rank representations, respectively). There is nothing intrinsically wrong about these representations; they just happen to be irrelevant for the Standard Model: no known particle is described by such a field. They are indeed sometimes used in toy models. Let me in fact quote a paragraph from Ref.1 §5.9:
Although there is no ordinary four-vector for massless particles of helicity $h=\pm1$, there is no problem in constructing an antisymmetric tensor $F_{\mu\nu}$ for such particles. [...] Why should we want to use a [gauge dependent vector field $A^\mu$] in constructing theories of massless particles of spin one, rather than being content with fields like $F_{\mu\nu}$ [which is gauge-independent]? The presence of derivatives in eq. 5.9.34 means that an interaction density constructed solely from $F_{\mu\nu}$ and its derivatives will have matrix elements that vanish more rapidly for small massless particle energy and momentum than one that uses the vector field $A^\mu$. Interactions in such a theory will have a correspondingly rapid fall-off at large distances, faster than the usual inverse-square law. This is perfectly possible, but gauge-invariant theories that use vector fields for massless spin one particles represent a more general class of theories, including those that are actually realized in nature.
Parallel remarks apply to gravitons, massless particles of helicity $\pm2$. [...] in order to incorporate the usual inverse-square gravitational interactions we need to introduce a field $h_{\mu\nu}$ that transforms as a symmetric tensor, up to gauge transformations of the sort associated in general relativity with general coordinate transformations. Thus, in order to construct a theory of massless particles of helicity $\pm2$ that incorporate long-range interactions, it is necessary for it to have a symmetry something like general covariance. As in the case of electromagnetic gauge invariance, this is archived by coupling the field to a conserved "current" $\theta^{\mu\nu}$, now with two spacetime indices, satisfying $\partial_\mu\theta^{\mu\nu}=0$. The only such conserved tensor is the energy-momentum tensor, aside from possible total derivative terms that do not affect the long-range behaviour of the force produced. The fields of massless particles of spin $j\ge3$ would have to couple to conserved tensors with three or more spacetime indices, but aside from total derivatives there are none, so highs-spin massless cannot produce long-range forces.
In short: most "non-standard" representations are fine but phenomenologically useless. The only non-trivial cases are $(1,0),\,(0,1)$, but they don't seem to be realised in Nature. A possible reason is that they mediate short-ranged interactions (but non-confining: it can be proven that confinement arises only if you have non-abelian gauge interactions) and are therefore not seen in actual experiments. If any such particle existed, we would need much larger accelerators.
References.
- Weinberg's QFT, Vol.1.
1: Fields of higher spin are always of the gauge type, because of the usual mismatch between field components and particle degrees of freedom. If you consider a bosonic field of arbitrary spin, you can always fix the gauge in such a way that its propagator is $\mathcal O(k^{-2})$ in the UV; similarly, a fermionic field can be gauge-fixed so that its propagator scales as $\mathcal O(k^{-1})$. Thus, it appears that any field is power-counting renormalisable. The catch is that the theory is gauge invariant if and only if you have a Ward-Takahashi-Slavnov-Taylor identity to control the unphysical degrees of freedom. You therefore need a conserved current which, as per Coleman-Mandula-Haag–Łopuszański–Sohnius-etc., is at most of the vector type if bosonic, or has spin $3/2$ if fermionic. In other words, you may only introduce $s=1$ fields if the symmetries form a regular algebra, or $s=2$ if you allow superalgebras. You cannot introduce any higher spin, simply because of the lack of a conserved current (cf. Weinberg's quote above).
Alternatively, if you don't want to fix the gauge (say, by working with massive particles and introducing Proca-like auxiliary conditions), the propagators always grow like $|p|^{2(s-1)}$, and the problematic large-$k$ behaviour is only cancelled if you have conserved currents at the vertices (so that the terms proportional to $k^\mu$ vanish, cf. [this PSE post](https://physics.stackexchange.com/a/383961/84967)). But, as in the previous paragraph, such currents can only be, at most, of the supersymmetric type, so no particle of spin higher than $s=2$ is allowed. See also [Weinberg–Witten theorem](https://en.wikipedia.org/wiki/Weinberg–Witten_theorem).
