It is not uncommon for units of a different physical entity to be used to measure a related physical entity. e.g. distance is generally measured in meters; but it is also measured in light years which is the distance traveled by light in a year. The important thing is that there should be a consistent way to convert one unit to another.

Someone pointed out that Torque is a vector (defined as a cross product) while Work is a scalar (defined as a dot product). However, that can't be "the (only) reason" for different units. Units are defined for "magnitude of a vector", which by itself is a scalar. So, the reason you can't use Joules for torque is because there is no consistent way of converting Newton-meters to Joules and vice versa.

There are 2 types of units viz., the basic/elementary units for mass, distance and time and the compound/derived units such Newton, Joule, etc for physical phenomenon that are derived from the basic units.

So, 1 Newton is the amount of Force required to increase the velocity of 1 Kg of point mass by 1 m/sec in 1 sec, in the direction of the change in velocity. 1 Joule is the amount of work done when a force of 1 Newton moves any point mass by a distance of 1m.

For a unit of Joule to be used for a unit of Torque, you would need a unit of Torque to always perform 1 Joule of work, which is not true.