Why is the response of a material to an EM wave in the optical domain almost independent of magnetic effects? Why can we ignore the magnetization $\vec{M}$ but not the electric polarization $\vec{P}$ in first order when discussing the response of a material to optical fields?
Related to this: Why is the electric polarization $\vec{P}$ only dependent on electric and not the magnetic field in first order?
Summarized: Why is the magnetic contribution so much smaller?
 A: A few arguments can be made, broadly as follows

*

*Classical Electrodynamics

*Atomic Couplings

*Atomic Transitions

Note I use Gaussian units throughout.
1 - Classical Electrodynamics
This argument comes directly from the classic Electrodynamics Of Continuous Media by Landau and Lifshitz$^1$. I reproduce the argument here so the answer is self-contained. This is little more than a transcription directly from the text, with some extra text/equations to fill in where they reference earlier material.

Consider the magnetic moment per unit volume $\mathbf{M}=(\mathbf{B}-\mathbf{H})/4\pi$. The magnetic moment is given by
$$
\mathbf{m}=\int \mathbf{r} \ \times (\nabla \times \mathbf{M}) dV=\frac{1}{2c}\int\mathbf{r} \times \overline{\rho\mathbf{v}}dV\tag{1}\label{1}
$$
Where $\overline{\rho\mathbf{v}}$ is the mean value of the microscopic current density. From Maxwell's equations we have
$$
\nabla \times \mathbf{B}=\frac{4\pi}{c}\overline{\rho\mathbf{v}}+\frac{1}{c}\frac{\partial\mathbf{E}}{\partial t}
\quad \text{and} \quad
\nabla \times \mathbf{H}=\frac{1}{c}\frac{\partial\mathbf{D}}{\partial t}
$$
so that
$$
\nabla \times\mathbf{M}=\frac{\nabla \times\mathbf{B}-\nabla \times\mathbf{H}}{4\pi}=\frac{1}{4\pi c}\left(
4\pi\overline{\rho\mathbf{v}}+\frac{\partial\mathbf{E}}{\partial t}-\frac{\partial\mathbf{D}}{\partial t}\right)
$$
giving
$$
\overline{\rho\mathbf{v}}=c\nabla \times \mathbf{M}+\frac{\partial\mathbf{P}}{\partial t}\tag{2}\label{2}
$$
It can be shown that equation $\eqref{1}$ can be written in the form $\mathbf{m}=\int \mathbf{M}dV$ iff $ \ \overline{\rho\mathbf{v}}=c\nabla \times \mathbf{M}$ and $\mathbf{M}=0$ outside the body. Thus Landau and Lifshitz argue that $\mathbf{M}$ is only physically meaningful when the term $\partial\mathbf{P}/\partial t$ in Eq. $\eqref{2}$ is negligible, i.e. there is no meaning in using magnetic susceptibility unless $|c\nabla \times \mathbf{M}|\gg|\partial\mathbf{P}/\partial t|$.
By definition, $4\pi \mathbf{P}=\mathbf{D}-\mathbf{E}=\mathbf{E}(\epsilon(\omega)-1)$ so $P\propto E\to \partial\mathbf{P}/\partial t\propto \partial\mathbf{E}/\partial t$. To be able to neglect $\partial\mathbf{P}/\partial t$ we therefore need small $E$ and/or a small body (to increase the space derivatives in $\nabla \times \mathbf{M}$)
For an EM wave, $E\sim H$ so small $E$ is not valid. Even if it were - say by placing the body inside a solenoid where $E$ is due only to induction by the variable magnetic field - then we can see the following.
$$\nabla \times \mathbf{E} = -\frac{1}{c}\frac{\partial\mathbf{B}}{\partial t} \quad \text{with} \quad \frac{E}{\ell}\sim\frac{\omega H}{c}\to E\sim\frac{\omega \ell H}{c}$$
where $\ell$ is the "dimension of the body". Putting in $\epsilon-1\sim1$ we see
$$
\frac{\partial P}{\partial t}\sim \omega E \sim \frac{\omega^2\ell H}{c} \quad \text{and} \quad \mathbf{M}=\chi\mathbf{H}\to |c \nabla \times \mathbf{M}|\sim \frac{c\chi H}{\ell}
$$
We need
$$
|\frac{\partial \mathbf{P}}{\partial t}|\ll |c \nabla \times \mathbf{M}|\to \frac{\omega^2\ell H}{c}\ll \frac{c\chi H}{\ell} \to \boxed{\ell^2 \ll \frac{\chi c^2}{\omega^2}}\tag{3}\label{3}
$$
for the concept of magnetic susceptibility to even be meaningful. But for optical frequencies we have $\omega\sim v/a$ and $\chi\sim v^2/c^2$, where $a$ is the "atomic dimension" and $v$ is the electron velocity. This means that the boxed inequality in $\eqref{3}$ becomes $\ell\ll a$ or the body must be much smaller than an atom. Therefore Landau and Lifshitz conclude

There is certainly no meaning in using the magnetic susceptibility from optical frequencies onward, and in discussing such phenomena we must put $\mu=1$.

2 - Atomic Couplings
The magnetic component of an EM wave couples to the magnetic dipole moments in the material while the electric component couples to the electric dipole moments, to first order. As a rough approximation, take the average magnetic moment of an atom in the material to be one$^2$ Bohr magneton $|\boldsymbol{\mu}|=\mu_B=e\alpha a_0/2\approx e a_0/275$ where $\alpha$ is the fine structure constant. We can ballpark the average electric moment to be $|\mathbf{p}|=qr=e a_0$ i.e. the moment of a proton and electron separated by a Bohr radius. So we see the magnetic coupling is hundreds of times weaker than the electric coupling. This means the timescales to magnetize are slower than to polarize, so if the field oscillates quickly enough - magnetization won't have time to occur before the field reverses direction.
To ballpark these time scales, let's do some terrible physics and hope no one yells at me. Let's assume that

*

*Magnetization occurs solely due to electron magnetic dipole moments

*Polarization occurs solely due to electrons moving away from their $+e$ charged nucleus

*The electrons are free

*The fields are constant

Subject to a magnetic field, each electron will experience a torque trying to align it with the field.
$$
\boldsymbol{\tau}=\mathbf{m}\times\mathbf{B}=\frac{d\mathbf{L}}{dt}
$$
Since we're ballparking, take their Gyromagnetic ratio to be $\gamma\sim \mu_B/\hbar$
$$
\mathbf{m}=-\gamma \mathbf{L}\approx -\frac{\mu_B}{\hbar}\mathbf{L}\to 
\frac{d\mathbf{m}}{dt}\approx-\frac{\mu_B}{\hbar}\frac{d\mathbf{L}}{dt}=-\frac{\mu_B}{\hbar}\mathbf{m}\times\mathbf{B}\approx -\frac{\mu_B^2}{\hbar}\hat{\mathbf{m}}\times\mathbf{B}
$$
From here we can look at the energy transfer rate $dU_B/dt$ in terms of $\mathbf{m}$ to get an idea of the rate the magnetic field is spending energy magnetizing the system$^3$.
$$
U_B=\mathbf{m}\cdot\mathbf{B}\to\frac{dU_B}{dt}=\frac{d\mathbf{m}}{dt}\cdot\mathbf{B}\approx -\frac{\mu_B^2}{\hbar}\left(\hat{\mathbf{m}}\times\mathbf{B}\right)\cdot \mathbf{B}\sim \frac{\mu_B^2}{\hbar}B^2=\boxed{\frac{e^2\alpha^2 a_0^2}{\hbar}B^2}
$$
Subject to an electric field, the electron will move away from the nucleus giving rise to polarization. A free electron will feel a force $F=eE$ and thus accelerate with $a=eE/m$. Obviously the electron isn't free, but as an order of magnitude ballpark, let's pretend it is and see what happens.
$$
p=er\to \frac{dp}{dt}=e\frac{dr}{dt}=ev\approx e a \Delta t=\frac{e^2 E \Delta t}{m}
$$
Let's make $\Delta t$ out of our basic legnth $a_0$ and speed$^4$ $\alpha c$. This gives $\Delta t=a_0/(\alpha c)$. Combining this with $U_E=\mathbf{p}\cdot\mathbf{E}\approx pE\to dU_E/dt \approx (dp/dt)E$ we see
$$
\frac{dU_E}{dt}\approx \frac{e^2 E a_0}{\alpha c m}E=\frac{e^2 a_0}{\alpha c m}E^2=\boxed{\frac{e^2 a_0^2}{\hbar}E^2}
$$
This means that the electric field is, ostensibly, doing $\alpha^2\approx 5\times 10^{-5}$ less work magnetizing than polarizing. This factor agrees with Landau and Lifshitz who above argued that in the optical frequency range we have $\chi_E=1$ and $\chi_B=v^2/c^2=(\alpha c)^2/c^2=\alpha^2$. I would guess my terrible approximations completely break down when the time scales are much longer than an optical period.
In one period of an optical EM wave, $\sim 10$fs, we'd expect a 1nT magnetic field to do on the order of $\mu_B^2*(1nT)^2/\hbar\approx 10^{-44}$J of work. To flip a spin takes $\sim \mu_B B\approx 10^{-32}$J of energy. So around the optical range the work done by the magnetic field over a period is well below that of the energy to flip an electron spin i.e. we don't impart enough energy fast enough to magnetize at all really.
On the other hand, we'd expect the corresponding $\sim 1$N/C electric field to do $\alpha^{-2}$ times more than the magnetic field, or $\approx 10^{-39}$J of work polarizing the atom. We can get a ballpark on the energy needed to polarize by looking at the Stark effect which predicts an energy shift of
$$\Delta E=\frac{1}{2}\alpha_p E^2\tag{4}\label{4}$$
We can approximate the polarizibility $\alpha_p$ by using the Clausius–Mossotti relation
$$
\frac{\varepsilon_\mathrm{r} - 1}{\varepsilon_\mathrm{r} + 2} = \frac{N \alpha_p}{3\varepsilon_0}\to \frac{1}{4} = \frac{\alpha_p}{3\varepsilon_0 \frac{4}{3}\pi a_0^3}\to \alpha_p\approx 4\times 10^{-42}\text{Fm}^2
$$
Plugging this in to $\eqref{4}$ we find $\Delta E\approx 10^{-42}$J which is well below the approximate energy imparted by the electric field. And so, unlike the magnetization, the polarization cannot be ignored in the optical frequency range.
3 - Atomic Transitions
The allowed magnetic dipole transitions (M1) require that wavefunction parity is conserved while angular momentum is changed. Electric dipole transitions (E1) require parity be flipped. This enforces selection rules that make M1 transitions much less probable than E1 transitions. One factor is that M1 induces transitions between states that have the same spatial wavefunction, and typical splitting between these states aren't in the optical frequency range. This answer is getting too long so I'll skip the math here but you can work out from Fermi's Golden Rule that E1 transitions are on the order of 100 times more probable than M1 transitions. This further slows the magnetization of the material by incoming radiation.
Footnotes
$^1$ Section 60 "The dispersion of the magnetic permeability".
$^2$ In most materials, much less than one.
$^3$ This is basically wrong. The torque above causes no rotation toward the field but rather gives rise to precession. The correct treatment relies on damping terms arising from interactions in the material which gives a torque in the right direction that goes roughly like $$\gamma \left|\frac{d\mathbf{B}}{dt}\right|\left(\mathbf{m}\times\frac{d\mathbf{m}}{dt}\right)$$
which should have a comparable order of magnitude. At least it has a comparable constant out front.
$^4$ By the virial theorem we expect
$$mv^2\sim \frac{e^2}{a_0} \to v\sim \frac{e}{\sqrt{m a_0}}=\alpha c$$
