TL;DR: The frequency of a light wave does not change from medium to medium while the speed of light (and thus wavelength) does. By knowing the frequency of an EM wave you know it's color in any medium.
Building on prior answers, the facts are: Color is determined by the energy of the EM Wave that reaches your eyeball. Energy is defined as $E = hf$, where $h$ is Planck's constant and $f$ is the light's frequency.
Thus, the color of an EM Wave is defined by its frequency.
In other words, measuring the frequency of an EM Wave is sufficient to identifying the color of light or the type of EM Wave that it is. This is opposed to measuring the wavelength, which would require knowledge of what the refractive index of the medium that the wavelength was measured in is to determine what color of light or type of EM wave the EM wave is.
Although $f$ can be defined by $v/l$, where $v$ is the speed of an EM wave in a medium and $l$ is the the wavelength in a medium, upon changing mediums the only constant is the frequency of the wave.
An example of why frequency is the defining factor:
When you throw a red brick in a pool, the wavelength of the EM Wave carrying the color of the object varries. If you were to measure the wavelength carrying the color of the brick, that information would be useless or misleading in identifying the color of the brick unless you knew the refractive index of (speed of EM Waves in) the medium that you were measuring in.
On the other hand - measuring the frequency of the EM Wave carrying the color of the brick anywhere would be sufficient in determining the color of the brick is Red, for it does not change regardless of what medium the EM Wave is found in.
From this, we can conclude that the color we see is dependent on the frequency of the EM Wave. (The Wave just happens to have an certain wavelength at that speed of EM wave determined by the medium the wave is in.)