Which mathematical operation does the right hand rule for current come from?

Where does it come from mathematically that when you point with your thumb in the direction of the current, your curved fingers will point in the direction of the $B$-field? In other words, which mathematical operation does it come from? Probably it has to do with some vector product or Stokes theorem application, but I am not quite sure about it.

-
Hint: for a left-handed coordinate system, you would use the left-hand rule. mathworld.wolfram.com/Left-HandedCoordinateSystem.html – Alfred Centauri May 27 '14 at 0:01
It's the curl operator: "The direction of the curl is the axis of rotation, as determined by the right-hand rule, and the magnitude of the curl is the magnitude of rotation." en.wikipedia.org/wiki/Curl_%28mathematics%29 which is related to Stoke's theorem: en.wikipedia.org/wiki/Stokes%27_theorem#In_electromagnetism – Alfred Centauri May 27 '14 at 0:15

It comes from the cross product. Every situation in which you have to use the right-hand rule corresponds to some mathematical equation that involves a cross product.

In this case, the relevant equation is the Biot-Savart law,

$$\vec{B} = \frac{\mu_0}{4\pi}\int \frac{I\,\mathrm{d}\vec{l}\times\vec{r}}{r^3}$$

If you use the right-hand rule - the version you know to use for cross products - to compute the cross product of $\mathrm{d}\vec{l}$ and $\vec{r}$, like this (apologies for the crude drawing):

you will get the same result as from the curling-fingers form of the right-hand rule that you've shown in your question. That's by design, and in fact the curling-fingers form of the rule is just a shortcut for an infinite number of applications of the cross-product form.

In case you're curious, there is a deeper reason the right hand rule is needed for cross products. When you take the cross product of a vector and another vector, you get a slightly different mathematical object called a pseudovector or axial vector. The magnetic field at a point is the best-known example of a pseudovector. Despite looking just like a vector, pseudovectors actually represent a magnitude and an oriented plane, whereas an ordinary vector represents a magnitude and direction. Now, if you have a plane, and you want to represent it with an arrow, in a sense you can do that by picking the arrow to be perpendicular to the plane, and then your convention is that an arrow represents the plane that is perpendicular to it. But there are two arrows perpendicular to the plane; how do you choose which one to use? That's where the right-hand rule comes in. It plays a role in mapping a pseudovector (the thing that e.g. magnetic field really is) to a vector (the thing we use to represent e.g. magnetic field).

-
The choice of right-hand vs. left-hand rules can be understood as a choice of orientation for 3d space. Given the orientation of the magnetic field bivector, different choices for the orientation of 3d space yield different normal vectors for planes. – Muphrid May 27 '14 at 3:35

The interesting thing about right-hand-rules is that they always manifest in pairs.

To find the direction of precession for a gyroscope you use a RHR to get the direction of the angular momentum and then use a RHR find the direction of the torque. Finally, you apply that torque as the time-rate-of-change of the angular momentum.

If you adopted the LHR for both applications you'd get the same resulting precession. In other words there is a arbitrary convention at work here.

There is a similar situation with magnetic fields, but it is less obvious that it is a pure convention. The direction of the magnetic field and the direction of the Lorentz force both involve a handed-rule and the physical observable is the force.

You could replace the RHR everywhere with a LHR and all out experiments would still work. Well, until you start describing weak decays.

-