found mathematical explanations speaking of orthogonal vectors and Maxwell's equations and vector products
This is true and these mathematical explanations are also consistent with what we find physically. If you accept that Maxwell's equations consistently describe electromagnetism and lead to electromagnetic waves (and they do), you can show that $$\bf E\cdot B=0$$ for an electromagnetic wave. That is, the $\bf B$ and $\bf E$ fields are orthogonal not only as a mathematical consequence, but this also corresponds to how they behave physically (for electromagnetic waves in a vacuum).
in what sense, physically speaking, are the electric and magnetic fields perpendicular to each other?
In the sense that the $\bf E$ and $\bf B$ fields, physically oscillate at $90^\circ$ to each other and at $90^\circ$ to the direction of propagation. That is physically how it is (again, for electromagnetic waves in space).
I have seen that electromagnetism is the result of relativistic effects at the quantum level. Is it that the rotation of an electron creates these effects, and magnetic effects are related
You do not need to go into relativity or quantum mechanics. You are adding a layer of complexity that is not needed for describing this aspect of electromagnetic waves.
Edit: As pointed out in the comments below, in the general case of electric and magnetic fields, these fields are not orthogonal.