This is a very nice question! It's simple and still has so much physics under the hood. So what I'll write here is maybe a starting point or partial answer as I don't know all the physics for a more rigorous analysis.
I would say if you just want to minimize losses (inelasticity), you should choose a material that responds most linearly wrt to stress/strain, and has low dissipation, something like a single crystal, diamond cut into a sphere. This will probably bounce back quite highly from a perfectly rigid and smooth floor. But being rigid with high elastic moduli, it will emit some delta-like sound upon reflection (with substantial power around 2-4 kHz).
We could also note that the drop height/impact velocity does not affect the time scale of the contact with the floor, as it's just a half-cycle of the harmonic oscillator that can be thought of, having some effective mass and spring constant. Just the amplitude gets scaled. Thus there'll be a characteristic "finger-print" sound for every material. (Of course this is only true if the drop height is not so large that the material breaks under stress. This height can be small for brittle materials. But could be a starting point for the discussion.)
Foam tennis ball will be much lossier, but will move the peak of the sound emission spectrum to much lower frequencies (as it has such a long impact, having such low elastic moduli). And I guess you don't care about loss, as long it is loss into heat, not into sound? Foam is soft and also attenuates internal vibrations which for a rigid object would have a much larger efficiency of being radiated off as sound from the surface (like in monocrystalline sphere case).
So bottom line, from my point-of-view, for a given mass:
Take the lowest elasticity material, whose mechanical loss you can accept. If you have two optimization criteria it depends where you chose the trade-off. Minimum mechanical loss: Single crystal. Good compromise: Rubber bouncy ball. Minimum sound at appreciable loss: Foam tennis ball.
But I guess the problem is quite deep and I've only touched the surface. The expansion of an infinitesimal surface deformation into normal modes of the sphere will always be similar, but deformation magnitude will decrease with elastic modulus. At the same time monocrystalline material will ring for a long time, also at audible frequencies, and radiation of sound waves into the surrounding air can be an appreciable loss channel. The fact that the foam ball is very good at attenuating vibration also plays a huge role in it being a winner choice.
