>found mathematical explanations speaking of orthogonal vectors and Maxwell's equations and vector products

This is true and these mathematical explanations are also consistent with what we find physically. If you assume that Maxwell's equations consistently describe electromagnetism and lead to electromagnetic waves (and they do), you can show that $$\bf E\cdot B=0$$ for an electromagnetic wave. That is, they are orthogonal not only as a mathematical consequence, but also corresponds to how they behave physically (for electromagnetic waves in a vacuum).


>in what sense, physically speaking, are the electric and magnetic fields perpendicular to each other?

In the sense that the $\bf E$ and $\bf B$ fields, _physically_  oscillate at $90^\circ$ to each other and at $90^\circ$ to the direction of propagation. That _is_ physically how it is (again, for electromagnetic waves in space).

>I have seen that electromagnetism is the result of relativistic effects at the quantum level. Is it that the rotation of an electron creates these effects, and magnetic effects are related

You do not need to go into relativity or quantum mechanics. You are adding a layer of technicality that is not needed in describing this aspect of electromagnetic waves.