First, I want to dispel the notion that there is a single general rule or pattern. Indeed, the diversity of interactions between matter and radiation allow for many interesting physical phenomena (and human inventions!) So, in short, the rules of radiation are more like a long list of exceptions.
There are many different mechanisms for absorbing or scattering light which are relevant in different situations. I provide some examples:
Rayleigh scattering, when the wavelength of light is longer than the particle size, but is stronger for shorter wavelengths (why the sky is blue).
Mie scattering, when the wavelength of light is shorter than the particle size, where an atom just acts like a solid ball.
Resonant absorption, when the photon has the right amount of energy to interact resonantly with a molecule or atom (this is where absorption lines/bands come from).
Photoionization, when the photon has enough energy to liberate an electron.
Compton Scattering, when the photon has so much energy that the atom holding the electron doesn't matter much, it just collides with the electron.
These mechanisms all have different cross sections and regimes where they operate. This makes it very difficult to pinpoint a single pattern or cause.
Visible light in the atmosphere experiences Rayleigh scattering, since a few hundred nanometers is larger than air molecules. Since air has very few molecular transitions in the visible light range, visible light passes through easily. The same holds for glass, which blocks IR but not visible.
Ultraviolet is easily absorbed by atoms with ultraviolet photoionization energies or excitation energies in the atmosphere, like ozone. It turns out that ~1-10 eV is the energy range of chemistry (and the energy scale of an electron in a Hydrogen atom, for example) and many materials interact strongly with UV in particular. But there are exceptions, like blacklight filters, that will block visible light but not UV.
Increasing to X-ray and gamma ray energies, we leave behind the energy range of chemistry, and now photons are primarily bouncing off of electrons as though the electrons are billiard balls. Material properties matter less and less. So lead, concrete, and water are all fine candidates for blocking gamma rays.