Several things contribute to conductivity. I think two are particularly relevant to the question. First concerns band structure. Metals can conduct easily because there are empty states very close in energy to the Fermi surface, so that it is easy to promote an electron to a higher-energy level. In an electric field, the newly-occupied states all (loosely speaking) have momenta in the direction of the field, so there is net movement of the electrons in the direction of the field. How well this works depends on the density of states at the Fermi level. The more states available, the better the conductivity. I've looked for these data for the noble metals but haven't located them.
The second concerns the dominant loss mechanism at room temperature, the interaction of the electrons with the lattice. In a simple model of a solid the positions of the ion cores are taken to be fixed. In reality they are not, and a passing electron will excite lattice excitations (phonons) and lose energy. How well this works depends on the strength of the electron-phonon coupling in the solid, and that's complicated business.
Having said all that, I don't have data at hand to make an argument about the noble metals. But I think it should be clear that your picture of conductivity needs to be augmented, and that might be enough to answer your question.
Update I found data on density of states, and the order of densities of states is Ag, Cu, Au, from highest to lowest.