>do units have to make sense in Physics?

Absolutely. Also one can use natural unit system, where some value is the ratio of a quantity with some known constant. I.e. imagine that you are expressing speed in terms of $c$ - spaceship is moving at $1/2~c$, or - sphere has $200~e$ charge.
You can supply absolute value for $c,e$ and you will know exact measurement value in **SI**, **CGS** system or whatever. Or you can simply assign $c=1,e=1$ and just express every measurement as the ratio to some **base unit**.

>if you can cancel out all units, then you can give it any units you want in the end

Really ? So you say, that if I measured that concrete lightning-bolt has struck with a charge of $1000\times$ (no units) of an average lightning-bolt power in terms of charge,- then you can say that it has struck with $1000\times$ of an average milk fat content ? What's the point of such nonsense ?