Typically, in the derivation used for resistor noise, I see a hypothetical voltage source (the equivalent noise source) with a source resistance of R, connected to a transmission line of characteristic impedance R, terminated in a resistor of value R. So in the below figure, Vg is the hypothetical equivalent voltage noise, Zg is R, and Zf is also R.
We can then argue that this transmission line, being a transmission line, has electromagnetic modes that can be occupied by a number of photons. The occupancy of a mode of frequency $\omega$ is just given by the Bose-Einstein occupancy factor (photons are bosons) $$\frac{1}{e^{\hbar \omega / \tau}-1}$$ where $\tau=k_B T$, and $k_B$ is Boltzmann's constant. At high temperatures, this occupancy factor tends to $\hbar \omega / \tau$, and the average energy per mode is just equal to $\tau$. This all makes sense to me. Where things start to get confusing is usually people then start talking about the modes of the transmission line. They usually make a boundary conditions argument, that the field must go to zero at the boundaries, in which case you'll get the modes have frequencies $f=\frac{v}{2L}*m$, where v is the velocity of the mode (nearly the speed of light), L is the length of the transmission line, and m is a positive integer. You then interpret the frequency spacing of the modes as the bandwidth, remember to account for the initial voltage division on the line, and viola, you have that the noise voltage power is $4k_B T R \Delta f$.
There's a couple of things that bug me about this. First of all, for a matched line, modes wouldn't exist on the line. We can't apply boundary conditions, as the line isn't shorted or open at either end - it's perfectly matched. Applying these boundary conditions are inconsistent with the model used. Traveling waves going either direction are certainly allowed, but these don't have a particular mode spacing, as any value of $f$ would be allowed. Second, even if this was a valid thing to do, most physical resistors have dimensions of ~mm, which makes the first mode frequency in the ~GHz, but Nyquists' formula works well down to <Hz. This tells me that the transmission line argument is somewhat hacky and doesn't make sense. In order for it to work we need to take $L \rightarrow \infty$, but this isn't justified because resistors can be as physically small as we please and this should still work. $L$ is clearly not infinity. I get that the transmission line is meant to be an intermediate step, but it's an intermediate step I can't see having any physical justification.
A similar argument is made in the context of blackbody radiation in 3D (an artificial cavity is introduced which is then made to be the size of the universe), and for semiconductors, but this argument does seem to hold water because usually we can take the cavity to be the actual universe. What am I missing here?