When deriving a result (whether physically or mathematically), it can be helpful to first find out the limits of the result. Then if your alleged explanation "explains" the result in regions where it doesn't hold, you know your explanation is faulty.
So, the Boltzmann factor for the canonical ensemble. We have to assume that the particle number isn't changing, and we have to assume the volume isn't changing. Having both of these is actually quite restrictive. For instance you could have a gas of hydrogen, but to preserve particle number they need to be cool enough to avoid any possibility of fusion (even through tunnelling, which in the sun is how it happens since even in the sun it is cold relative to fusion being classically achievable just from KE overcoming PE). But to preserve volume you have to have room for all your hydrogen, which means they all have to be near the ground state, because Rydberg atoms (highly excited but not quite ionized hydrogen) can get very very very large if the principle quantum number is insanely large. If you want this to never ever ever happen you'd have to have so little energy available that even if every atom was low in kinetic energy, that there isn't enough energy left over to use as internal energy within a hydrogen to ionize even one single hydrogen. So technically we'd only expect Boltzmann to hold for very very cold hydrogen.
So looking at $TdS=dU+PdV-\mu dN$, we can see that in those situations, $dV=0$ because no change in volume and $dN=0$ because no change in particle number, so $dS=dU/T$, now we are most of the way there.
So since we don't expect it to hold exactly, the real question is why does it work well at all. It must be that the effects of the violations are small, transitory, or their net effects cancel. So for instance if the hydrogen atoms only overlap a little bit (in space and time) we might be able to ignore it. If there is an equilibrium amount of free electrons, ionized hydrogen, and neutral hydrogen then we might be able to ignore some of them if the effects are small. If we have a population of hydrogen at different states, and even though some are large on average there is plenty of room, we can get that.
One way to see this physically is to imagine a whole bunch of roughly identical regions each with some gas, enough that there are a very large number of hydrogen atoms in the many regions all put together. Then we can try to find an equilibrium for how many electrons, ions, and hydrogen, how many are large on average. Then we can try to take those typical values and see how realistic it is to ignore certain abilities.
In the end, taking no change in particle number, no change in volume, and a relationship between entropy and probability leads to a Boltzmann factor. So if you want to see it physically, focus on the relationship between entropy and internal energy in the absence of change in volume or particle number.