>found mathematical explanations speaking of orthogonal vectors and Maxwell's equations and vector products

This is true and these mathematical explanations are also consistent with what we find physically. If you accept that Maxwell's equations consistently describe electromagnetism and lead to electromagnetic waves (and they do), you can show that $$\bf E\cdot B=0$$ for an electromagnetic wave. That is, the $\bf B$ and $\bf E$ fields are orthogonal not only as a mathematical consequence, but this also corresponds to how they behave physically (for electromagnetic waves in a vacuum).


>in what sense, physically speaking, are the electric and magnetic fields perpendicular to each other?

In the sense that the $\bf E$ and $\bf B$ fields, _physically_  oscillate at $90^\circ$ to each other and at $90^\circ$ to the direction of propagation. That _is_ physically how it is (again, for electromagnetic waves in space).

>I have seen that electromagnetism is the result of relativistic effects at the quantum level. Is it that the rotation of an electron creates these effects, and magnetic effects are related

You do not need to go into relativity or quantum mechanics. You are adding a layer of complexity that is not needed for describing this aspect of electromagnetic waves.

**Edit**: As pointed out in the comments below, in the _general_ case of electric and magnetic fields, these fields are not orthogonal.