The perception of 'colour' is loosely based on a transfer of energy from photons to photoreceptive proteins in the retina. This suggests that frequency is responsible for perception of colour since the energy of a photon is directly proportional to frequency.
The frequency does not change because the energy does not change.
This would imply that there is no change in colour with refractive index, even though the speed and wavelength of the light wave changes.