If you are analyzing the inductor over such short time scales, the answer is that not only is the electric field important, but the whole electromagnetic field. Disturbances propagate at the speed of light through the circuit as electromagnetic waves - they cannot move faster than this (otherwise there would be faster than light signaling, which is impossible as far as everyone knows because it would gainsay special relativity, which is one of the most thoroughly experimentally checked frameworks of thought that there is).
At the short timescales you speak of, inductor actually looks from a circuit standpoint more like a distributed $LC$ ladder network like the one I have drawn as part of the system at the end of this answer. A complex system of bi-directionally running waves bounce back and forth through the system and eventually all these transients die down to establish the inductor's steady state behavior.
Actually, it won't take as long for the waves to propagate through the system and establish steady state as you think. The windings of the coil lie near to one-another, so the propagating waves will jump directly across from one bight of wire through its insulating, dielectric coating to the neigbouring loop. This direct coupling means that the propagation delay is more like the length of the wound coil, not the total length of the wire. So your inductor will work as an inductor at higher frequencies than you think.
If you see my answer describing how circuits work "so fast": this is how that answer ties in with this one. Each electron shoves its neighbours in the circuit, but, at the subatomic level, the "shove" between each neighbour and the next is being transmitted by an electromagnetic wave arising from the shift in each charge. Quantum field theory would see this as an exchange of virtual photons and the whole process is constrained by QFT so that the momentum is not transmitted at greater than the speed of light. In the coil, as above, neighbouring charges are also on the neighbouring bight of wire in the loops of the coil.
Lastly, my comments on special relativity should not be too mysterious. You may ask "how does special relativity step in and prevent faster than light signaling?" The answer is that Maxwell's equations, which wholly determine your system's behavior, are well known to fulfill all the principles of special relativity. Indeed, Maxwell's equations historically were the first known "relativistic" system and early researchers (notably H. A. Lorentz) originally derived the transformation laws of special relativity directly from Maxwell's equations before Einstein gave his more general interpretation of the Michelson Morley experiment.