There is no asymmetry, for any observable quantity, because you'll always use the right hand rule twice to get an observable answer.
For example, we might want to compute the acceleration of a charged particle near this wire. The force on the particle uses it once, since $\mathbf{F} \propto \mathbf{v} \times \mathbf{B}$, and the magnetic field is given by the Biot-Savart law,
$$\mathbf{B} \propto \int \frac{I d\mathbf{s} \times \mathbf{r}}{r^3}$$
If you instead used your left hand, $\mathbf{B}$ would be flipped, but $\mathbf{F}$ would stay the same. So the right hand rule is asymmetric but ultimately arbitrary.
You might not believe this -- after all, can't we figure out which way $\mathbf{B}$ really points by using a compass? Nope. We know which way the force points. Then we arbitrarily paint one side of the needle red, using the same freedom we had to choose either the right or left hand rule.
One more thing: why do we have to use asymmetric math to describe a symmetric situation? Here, the asymmetry comes from the cross product. It turns out we don't need the cross product at all -- we can instead use a mathematical construct called the wedge product, which takes two vectors and outputs a 'bivector'. Bivectors require no handedness rules to construct; geometrically, they're essentially the parallelgram that the two vectors make.
By a lucky coincidence, because we live in three dimensions, it's possible to associate every bivector with a vector -- but doing so requires one arbitrary choice of direction. We don't actually need to use a vector at all, but doing so prevents textbooks from having to introduce extra math.