At some level your question just boils down to conventions: there are several sets of sign conventions that all give the right answers, so you just need to find one that you find conceptually satisfying and stick with it.
It might be helpful to point out that you can only ever directly measure polar vectors. Axial vectors can be thought of as just mathematical abstractions that only appear as intermediate steps in a physical process. For example, you are correct that in the Lorentz force law ${\bf F} = q{\bf v} \times {\bf B}$, the force is a physical quantity whose direction should not change sign under coordinate transformations. But remember that the magnetic field itself is physically determined by the Biot-Savart law (in the magnetostatic approximation; things get more complicated in the dynamic case but the parity transformation properties don't change):
$${\bf B}({\bf x}) = \int d^3x'\ \frac{{\bf J({\bf x}')} \times \hat{\bf r}}{r^2},$$
where ${\bf r} := {\bf x} - {\bf x}'$. Therefore, the magnetic field is an axial vector and can't be directly measured; you can only ever observe its effect by taking another cross product to get a force via the Lorentz force law.
The punch line is that you can always expand out any physically measurable vector as being determined by an even number of cross products. For example, you can write the magnetic force on a particle as
$${\bf F}({\bf x}) = q{\bf v} \times \int d^3x'\ \frac{{\bf J({\bf x}')} \times \hat{\bf r}}{r^2}.$$
So there are two different but physically equivalent ways to conceptualize a change in the handedness of your coordinate system. You can think of all "right-handed" cross products becoming "left-handed" cross products, in which case all axial vectors (like the magnetic field) will physically switch direction - but since they can't be physically measured, this has no consequences on observable physics. Or you can, as you prefer, think of cross products as being "physically" determined by the right-hand rule, in which case they don't switch direction because they don't care about your choice of coordinates. In this framework, the magnetic field does not switch direction under a coordinate inversion. Both conceptualizations lead to identical observable physics: since any directly observable vector is made up of an even number of cross products, it either picks up no minus signs or an even number of minus signs under a coordinate inversion. In either case, its direction is unchanged.