The walls of the cavity are not made of conductor, since in this case they would be reflecting, instead of absorbing all the radiation.
In statistical physics one usually neglects the interactions leading to the establishment of the equilibrium. For example, the Maxwell-Boltzmann distribution does not depend on the collisions between the atoms, which are responsible for establishment of the equilibrium. Using developer's language, this is not a bug, but a feature - the statistical mechanics is based on logical reasoning, which allows to obtain very general results without sinking in gory details.
If course, in reality there are processes that lead to the establishment of the thermal equilibrium. E.g., if we start with a gas of two-level atoms, with transition frequency $\omega$, and no radiation - we would expect, in the first approximation, that only the radiation modes of this frequency will come into equilibrium with atoms. For full equilibrium we need to account for higher order processes, such as, e.g., the Raman scattering. Thus, this may take longer for the Planck's distribution to establish, but it will eventually be achieved - we believe in this, as we believe in energy conservation.
Remark
As one can see from the answers here (and from a discussion around a concurrent question), some confusion results from different ways one can define the black body radiation (BBR):
- BBR is a photon gas in thermal equilibrium If the number of photons in mode $\mathbf{k},\lambda$ is described by canonical distribution,
$$
p(n_{\mathbf{k},\lambda})\propto e^{-\beta \hbar\omega_{\mathbf{k},\lambda}n_{\mathbf{k},\lambda}},
$$
Planck's formula readily follows. In this case the radiation does not necessarily have to be in a contact with a black body - the role of the body/material is to mediate the energy exchange between the photon modes, for the thermal equilibrium to establish. This is the point of view adopted above. A perfect metal reflects all the radiation, and cannot lead to thermodynamic equilibrium. On the other hand, a metal with finite conductivity can do so (although not very efficiently) - the filament of an incadescent lamp could be discussed in this context. Black body is defined here as a body that emits radiation that is already black.
- BBR is the radiation emitted by a black body Here one postulates the properties of a black body - an object in thermal equilibrium that absorbs all the radiation incident at it. One can then calculate the radiation emitted by this object, which will be described by Planck's formula. This approach was taken historically, and presented in most introductory QM books, which is why many people stick to it. It's advantage is that one does not really need a cavity - the radiation is already black, which is how one applies Planck's formula to the radiation emitted by stars and other thermal sources, which are clearly non-equilibrium situations. (The cavity does appear in this approach, as a way to model a black body.) As I pointed out above, a metal (even a metal with a finite conductivity) cannot serve as a black body, because it does not absorb all the radiation incident on it.