We know that the space of all displacements is a vector space.
The vector space is defined as a mathematical object $(V,k,+,\cdot)$ such that it satisfies the 8 properties, where $k$ is a field.
We notice that the field must not contain any unit, because it has to be closed under multiplication. for example, if we have a number "$5 \mathrm{m}$" in a field, then we must have the number "$25\mathrm{m}^2$" as well, which makes the space contain not only displacements.
Then the basis vectors of the space should contain the unit and the coordinates should contain only the scaling factors. The only way of writing coordinates that actually make sense should be like $$(1,2,3)_{(\mathrm{right}1\mathrm{m},\mathrm{front}1\mathrm{m},\mathrm{up}1\mathrm{m})}$$ instead of like
$$(1\mathrm{m},2\mathrm{m},3\mathrm{m})_{(\mathrm{right},\mathrm{front},\mathrm{up})}$$ while the latter seems to be what many people tend to write.
Now, we might stop caring that much about linearity. After all, as we
- frequently use polar coordinates which are not linear anymore;
- have generalized coordinates in theoretical mechanics that care even less about linearity;
- have GR which states that the world is not linear at all.
2 and 3 both points to the concept of a manifold. and we consider that the coordinate is just a map from part of the manifold to a set of stuff, might it contain units or not. Here we can have our "$(r,\theta)$" coordinates (one has units and one has not) back.
But this has another problem, because we need the concept of a tangent space, and the tangent space should have a basis of $\frac{\partial}{\partial q^i}$. if $q^i$ have the unit of $\mathrm{m}$, then the basis should have $1/\mathrm{m}$. Yet in lagrangian mechanics, we need $\dot{q_i}$ which have unit $\mathrm{m}/\mathrm{s}$ live in the tangent space.
Also, the definition of manifold might ban units in the coordinates as well, since the definition requires a "homeomorphism to open subset of $\mathbb{R}^n$".
Should we just take the units away from all physical objects to make them work properly in math, or is there as way to make units function well in physics?