I assume in my question that all resistors follow Ohm's law.
You can, of course, assume that for the purpose of the question as long as you realize that it is not necessarily true that all resistors follow Ohm's law.
As I understand Ohm's law can be explained such that if we connect a
resistor to a power source that is not time dependent then we have the
relationship $I=V/R$, the current will be proprtional to the voltage.
It is not necessary for the voltage source to be not time dependent (to be constant). You just need to specify what $I$ is (instantaneous, peak, rms, etc..)
We also know that when we connect many resistors together they will
still follow Ohm's law individually and we have the known formulas for
series and paralell resistors.
Yes, again assuming the resistance of the resistors is constant (not dependent on voltage, temperature, etc..)
But why do we know that when we connect resistors together in either
series or paralell they still follow Ohms'law individually...
As long as their resistances are constant, there is no reason to believe otherwise.
...and the voltage drop over them will not vary in time, and the
current over them will not vary in time?
As I already indicated, Ohms law is not restricted to constant power sources, so voltages and currents can vary over time (just not the resistance, if they are ideal), and still be related by Ohm's law.
Is this something that is known because it has been shown through
experiments?
Good question. As a matter of fact, Ohm's law is not really a fundamental law, per se. It is based on observations (experiments).
Or is there a theoretical explanation for the fact that when we
connect resistors together the voltage drop over each of them will not
vary in time, and they will individually follow Ohm's law?
Again, there is no reason why the voltage and current cannot vary in time, as long the the relationship between the two for a purely resistive (no inductance or capacitance) circuit follows Ohm's law. Ohm's law is not limited to a constant voltage source.
Hope this helps.