It’s because it doesn’t make any sense to add different units. Each unit measures a specific dimension. You give example of a m+J. But what does that mean?
It’s like trying to add 4 oranges and 5 fire trucks. What’s the sum of that? Well, it’s 4 oranges + 5 fire trucks, because those are two totally different things. The same is true for m+J: meter measures distance, the Joule measures energy, so it makes no sense to add them.
It does however make sense to multiply units because of the geometry of areas and volume. A joule is a $N \cdot m$, because it is a force being applied (integrated) through the distance. The energy is the area of a curve determined by the force $F(r)$ as integrated over a range of $r$.
In this way it is possible to have “vectors” like you mention, depending on the units you use, and if they have qualities of linear independence. Newtons, meters, and joules could not be a basis for a unit vector space. However, meters, kilograms, seconds could. But physics is not formulated this way.