I am confusing myself about where physical quantities become mathematical objects. Where does one end and the other begin?
E.g. displacement $\vec{s}$. A displacement is a physical quantity that can be measured. In contrast to a mathematical vector which is an element of a vector space, which is a set V over a field F (with accompanying properties(+, $\cdot$ ) and axioms). In this case, are the elements of the set V the physical displacements? In that case, the (chosen) basis vectors $\vec{b}_i$ would also displacements and a general displacement can be written as a linear combination of basis vectors as
$$ \vec{s} = s_i\vec{b_i} $$
The scalars are then elements of the field F which must be the set of real numbers. If the basis vectors are orthonormal (cartesian) then the norm/length of the vector is
$$ |\vec{s}|^2 = s_i s_i $$
but this is just a number with no units. So the units are hidden in the basis vector $\vec{b_i}$. So maybe we should write this as
$$\vec{b_i} = (b \cdot [1\text{m}])\hat{e_i}$$.
Here $[1\text{m}]$ represents our unit of measure. But now $\hat{e_i}$ is a vector in a different vector space (the set of orientations, i.e. not a displacement). And $(b \cdot [1\text{m}])$ can't be a scalar of the vector space of displacements because we already said that was the set of real numbers So the actual "measurement" $(b \cdot [1\text{m}])$ should maybe be thought of as a map between two vector spaces (orientation and displacement)?
And you can't bake in the units in the field F either because if you scale a vector with a scalar twice (which should be allowed in a vector space) you get the unit squared. For the same reason you can't bake it into the scalar field used for the vector space of orientations.
So when we write
$$ \vec{s} = (0.1 m)\hat{x} + (0.2 m) \hat{y} $$
maybe we should think about the thing in the parentheses as a map between two vector spaces (displacement and orientation) and the number there is a product of the scalar field F (no units) and the length of the basis vector (with units). (This makes the covariance of the units and the contravariance of the components somewhat intuitive also....maybe).
In summary: confused!
In summary #2: Maybe this boils down to: If you want to think about physical quantities in a the language of mathematical vector spaces, where should you build in the units?