The issue is the secondary radiation emitted after the neutron is absorbed.
Most medium-mass nuclei have a "neutron separation energy" of about 8 MeV. When a nucleus absorbs a neutron at rest (which is a good description of a milli-eV thermal neutron in a mega-eV interaction), the separation energy of the compound daughter nucleus must be emitted, usually as a cascade of several high-energy gamma rays. Cadmium is especially nasty. If you put a cadmium sheet with any macroscopic thickness in a neutron beam, there won't be any neutrons on the downstream side of it, but there will be a boatload of hard gamma rays. This gamma radiation is prompt and must be considered before you worry about whether the daughter nucleus is stable against beta-decay or whether it will contribute to "activation" of your neutron absorber.
Smaller compound nuclei are also more likely to decay by strong particle emission instead of photon emission. For neutron capture on boron the primary interaction is
$$
\require{mhchem}
\ce{ ^{10}B + n \to {}^7Li^{\star} + \alpha }
$$
with generally a soft, half-MeV photon from an excitation in the new lithium-7 nucleus, but a good chance (~15%) of no photon at all. The strongly-interacting lithium-7 generally does not escape the solid matrix which holds your boron powder; the alpha doesn't escape until long after it has cooled to room temperature and started to act like helium.
The gold-standard absorber is lithium-6, which has a large cross section for
$$
\ce{ ^6Li + n \to {}^3H + \alpha}
$$
in the presence of slow neutrons. Since neither the alpha nor the triton have any low-energy, photon-emitting excited states, lithium-6 basically turns a thermal neutron beam of any practical intensity into ... nothing. No electromagnetic radiation is emitted, and the gas atoms thermalize in whatever solid contains your lithium and then generally diffuse away into the air before accumulating in any chemically significant quantity.