I'd try to elaborate on what I'm saying right now, this is how I understood it.
First of all, have a look at vector cross products if you aren't so sure what's going on inside Biot-Savart rule and related stuff. Then you'll be presented again with the right hand rule. By saying right hand rule, I'm NOT talking about the right hand grip rule(which is identical to what this content elaborates on), but the rule discussing about three perpendicular vectors. Now why? Why does the right hand rule work? Let me tell you, it does so only in the realm/paradigm of physics we all have agreed upon. It's purely a definition. There is NO actual direction of the perpendicular vector obtained by vector cross product, its an arbitrary choice, but we the humans, have defined that it happens to be in a certain direction. But in reality there is no real direction(such as popping into/out of the plane) for that vector obtained. We use the right hand rule which will agree upon all the derivations and results obtained by those who came before. To be precise, those derivations and results also place their foundation upon these rules.
If this still seems absurd, think of the direction of the magnetic field. Have you ever seen one? Have you ever witnessed one? You may have witnessed, but when you say that you feel that magnetic field, it's actually a magnetic force that you're feeling. Will you be able to witness magnetic field alone, without a magnetic force? No! But we treat the magnetic field as a vector to facilitate our understandings. Magnetic field is something that makes magnetic force happen. There is no meaning to magnetic field without a magnetic force. It's purely virtual which we've used to describe everything better.