I am trying to understand what magnetism really is, beyond just a field, an attractive or repulsive force. I would like to know: how do flowing electrons (e.g. electricity) generate a magnetic field?
What kind of “how” answer are you imagining? There are charged particles and equations for the fields that they produce, depending on their motion, and most physicists are completely content with that. There is no “mechanism” behind this, just mathematics and key physical concepts like Lorentz invariance. This approach to physics has been sufficient to construct modern civilization. It is a powerful and satisfying way to understand nature. We don’t need mechanisms like gears and wheels, vortexes in the ether, etc.
Instead all we need is the field concept. Accelerated motion of an electron causes a “kink” in its electromagnetic field to propagate outward at the speed of light. The field “carries” information about how the electron is moving to other particles that are going to interact with it. Note that in this description there are only particles and fields. But the point of the field is to be a transport mechanism across space and time which has local dynamics at every spatial point and temporal instant.
In addition to carrying information about the motion of the charged particle, its EM field of an accelerating charge also carries energy, momentum, and angular momentum! In short: the field is the “mechanism” that explains how electromagnetism “works”.
The field of a charged point particle extends to infinity. It has two parts: a part that falls off like $1/r^2$, which it has even when it isn’t moving, and a part that falls off like $1/r$, which it has only when accelerating. You can see the expressions for the fields here and an animation of the “kink” produced by sudden acceleration here.
When you have a positive and a negative particle bound together, as in a hydrogen atom, the field falls off more quickly... exponentially fast for an atom in its ground state.
The field explains everything you need to know, and does so in a satisfyingly local way, where the changing influence of the charged particle, as it changes its motion, propagates outward from point to nearby point.
Well, this is not an easy question as there are not really an answer. There are fundamental physical laws that governs the universe like the Newton's laws or the Biot-Savart law which states that a current (or flowing point charges) creates a magnetic field. These are experimental facts that are observed, and asking why they exist could be more a metaphysical question than a physical one.
One could always refactor these laws into another more abstract theory but that would only push back the question of the origin of these laws to a more abstract level. For example, one could say that the Newton's laws are a consequence of the Principle of least action. On the contrary we could say in a completely equivalent way that the latter is a consequence of the Newton's laws, and none could argue which is the more 'fundamental'.
We could generalize your question to: why the laws of the universe are the way we see them? In other words, why do these laws in particular governs our universe instead of different ones? For instance, why does a flowing current create a magnetic field instead of a gravitational field, or why do they create a magnetic field instead of nothing?
Hawking in his book The Grand Design tries to give an answer to this question. Basically, he states that if we take the Feynman's path integral formalism to the letter, then our universe and the laws that governs it could be only one of the multiple 'path' of a 'greater universe'. That means that there could be a great number of 'other universes' with other physical laws and all the universes would interfere each other to give the universe we have today. In other words, according to this hypothesis, the reason why the Biot-Savart law is like this would be purely random! We are then just 'lucky' to have a universe governed by a randomly selected sample of laws among all possible physical laws.