I think that it's probably best to think of temperature as being distinct from energy because of a pseudo-unit. You noted that $T = \frac{\partial U}{\partial S}$. But what are the units for $S$? Well, if you read off the Boltzmann formula for entropy $S = k_B\ln W$ then $S$ must have whatever units $k_B$ has. What are the units for $k_B$? Well, $k_B 1.38... \times 10^{-23} \,\mathrm{J}\,\mathrm{K}^{-1}$. The problem with this, though, is that it's basically the same as the old practice of defining the Coulomb in terms of Amps as $1\,\mathrm{C} \equiv 1 \mathrm{A}\,\mathrm{s}$: it's phenomenologically convenient, and probably even the most numerically accurate, but it's conceptually shaky.
Because $k_B$ isn't a measured quantity the way $S$ and $W$ are, in principle, we can actually just throw its units out (the same way we can make velocity unitless). Then the entropy starts to look more like the information theory entropy (which stole it's inspiration from the Gibbs formula for thermodynamic entropy) which has a pseudo-unit (exactly like radians). In information theory they like to use bits $S = -\sum_i P_i \log_2 P_i$, digits $S = -\sum_i P_i \log_{10} P_i$, or similar because there the concern is counting how many symbols need to be in a message to encode it in the most efficient way (the goal then being to add redundancy in a way that is uniformly robust to random errors from noise). In astronomy we'd describe entropy that uses a base-10 logarithm as a "decade" or "dex".
Being physicists, though, we like to take derivatives, and the what makes the natural logarithm natural is the way it makes taking derivatives simple (same for why we like to measure angles in radians; in isolation cycles are actually a far more natural unit for thinking about angles, and that's why we normally describe angles as $x\pi\,\text{radians}$, because $x$ is then in semi-cycles). So, what unit would entropy have then? If were were talking about the logarithm of the ratios of powers or squared field strength in waves, then the unit of choice would be the Neper, but that's not what we're doing here. Here we have ratios of numbers (probabilities), so we might as well go for $e$-folds as the unit for entropy (i.e. an $e$-fold is a factor of $e$).
Then temperature would have the unit Joules per $e$-fold, or $\mathrm{J}\,e\text{-fold}^{-1}$, just like period is technically $\mathrm{s}\,\mathrm{cycle}^{-1}$ and not just seconds.
The problem with doing this, though, is that it is extremely numerically inconvenient. A Kelvin is a useful unit of temperature, and a Joule is a useful unit of energy, that's why we use that $1.38\times 10^{-23}$ factor. Not using it would be akin to talking about number of atoms instead of moles in chemistry (another pseudo-unit, like "millions"). If we convert that to a base for the logarithm, you can get an idea for how stupendously large the combinatorics in thermodynamics are:
\begin{align}
S &= -\sum_i k_B P_i \ln P_i \\
&= -\sum_i \frac{R}{N_A} P_i \ln P_i \\
&= -\sum_i P_i \frac{\ln P_i}{N_A / R} \\
&= -\sum_i P_i \frac{\ln P_i}{\ln\left(\exp\left(\frac{N_A}{R}\right)\right)} \\
&= -\sum_i P_i \frac{\ln P_i}{\ln\left(10^{\log_{10}\left(\exp\left(\frac{N_A}{R}\right)\right)}\right)} \\
&= -\sum_i P_i \frac{\ln P_i}{\ln\left(10^{\frac{N_A}{R}\log_{10}e}\right)} \\
&\approx -\sum_i P_i \frac{\ln P_i}{\ln(10^{3.15\times10^{22}})}.
\end{align}
In other words, to get temperature to have a conceivable scale we need to use a logarithm with base with a little more than $3\times 10^{22}$ digits, and it's all because probabilities tend to multiply when combined and we deal with Avogadro's number of particles at a time.