This is a little bit of a pet peeve of mine, so I’ll chime in with my two cents / rant.
Torque and work don’t actually share the same units, torque actually has units of $\frac{\mathrm{N\,m}}{\mathrm{rad}}$. Dimensionally speaking, the distinction is irrelevant because radians are dimensionless, however most formulas for rotational dynamics fail if you are measuring angles in degrees, rather than radians. Basically measuring torque in $\mathrm{N\,m}$ is akin to measuring angular velocity in $\mathrm{s}^{-1}$ rather than $\frac{\mathrm{rad}}{\mathrm{s}}$.
You already have an example in the answer above: work being $W = \tau\theta$ only works if $\theta$ is in radians, meaning $\tau$ itself is “intrinsically” linked to radians. Another, more long-winded, example: $U = \frac{1}{2} I \omega^2$, in order to get the correct units for energy without “dropping” radians, you would need $I$ to be measured in $\frac{\mathrm{kg\,m^2}}{\mathrm{rad}^2}$. Then you have $\tau = I \alpha$, where you can see only one of the radians cancels out, giving the aforementioned $\frac{\mathrm{N\,m}}{\mathrm{rad}}$.
How do we get this to “match” with the way torque and moment of inertia are defined in terms of force, mass and position vectors? It can be taken as a result of the presence of cross products in their definition. The usual definition is $\boldsymbol{\tau} = \mathbf{r} \times \mathbf{F}$ which can be re-written as $\boldsymbol{\tau} = A_\mathbf{r} \mathbf{F}$, using the cross product matrix of $\mathbf{r}$. $$A_\mathbf{r} = \begin{pmatrix} 0 & -z & y \\ z & 0 & -x \\ -y & x & 0 \end{pmatrix}$$
But, you might say, there is no cross products in the definition of $I$. Well, the moment of inertia is actually a tensor quantity and, in tensor terms, is given by $I = m A_\mathbb{r} {A_\mathbb{r}}^\mathsf{T}$, two occurrences of our cross product matrix, meaning two occurrences of radians.
The same is true for angular momentum, which should have units of $\frac{\mathrm{J\,s}}{\mathrm{rad}}$.