Looking at reflectance spectrum of silver, given e.g. here, we can say that more than about $95\%$ of light is reflected by silver. Similarly for white notebook paper, the spectrum is given here, and looks like the paper reflects about $60\%$ of light.
I guess if you put a piece of silver on a paper, you'll see that silver indeed looks darker than paper. My thought is that it's because silver's reflection is specular while paper's one is diffuse, which makes you see something like "average illuminance" color of paper while only color of more or less directly reflected objects in silver surface.
But then it seems somewhat counter-intuitive that if one roughens the surface of silver, it'll be even whiter than paper's surface. Is it actually true? Are there any photos doing such a comparison? Or is my explanation incorrect, and the reason for gray color of silver is another?