Now generally, it is true that light incident upon an object is slightly shifted in frequency, but when we come to point of your question there are other factors weighing in.
In these cases it is better to look at the phenomena in Maxwell's wave model of light. When light falls upon a body, it energizes the lattice to vibrate (this can be thought of as transfer of momentum in photon model). Now it so happens that for a certain set of frequencies (dependent on the object) the lattice vibration is almost at the exact same frequency of response, thus the energy is released as another photon of the corresponding frequency, while all the other frequency is taken as heat by lattice vibrations.
So when a range of frequencies are incident upon an object, only a range is vibrated back at almost the same frequency, this we sense as color of the object. Thus the question of shifted frequency is not really in relevance here.
Now back to first part, I said generally because, say in case of Bragg's scattering there is no shift of frequency in the incident x-ray. And this is true even in vision, if the dye reflects exactly 1 wavelength, then there is no frequency shift.
I hope this was kinda satisfactory....