Are colors grounded in physics or are they a matter of human perception? My father was colorblind, and I always wondered if colors were a matter of physics or if different colors are just a human way of describing and differentiating our visual perception of the world?
For example, is the color blue by nature blue, or is it just what we see?
 A: Our eyes are receptors, and have evolved so that they are most sensitive for visible wavelength photons, something called tricolor vision, with three types of receptors, each for Red, Green, and Blue wavelength. Now our receptors have naturally evolved for Sunlight, and Sunlight is made up of a combination of several different wavelength light (including non visible too, but our eyes are only sensitive for visible wavelength photons).
The receptors in our eyes, are sensitive for these three types of RGB wavelength photons. Each receptor sends information to the brain, and the brain perceives the combination of these photons as certain color light.

The normal explanation of trichromacy is that the organism's retina contains three types of color receptors (called cone cells in vertebrates) with different absorption spectra.

https://en.wikipedia.org/wiki/Trichromacy
In this sense, the answer to your question is that color is a perception in our brain. Of course in a physical sense, the color of light (based on natural Sunlight) is a combination of different wavelength photons. Thus, certain color light (the one that our brain perceives as a certain color) can be produced many ways. It can be made of just certain wavelength photons, or it can be made up of different combinations. Yes, two different combinations can sometimes combine into light that our brain could perceive as (approximately) the same color.

https://en.wikipedia.org/wiki/Color_vision
Now in the visible wavelength, we made arbitrary (based on how our brain would perceive those photons alone) decisions to call certain wavelengths certain color.
In your case blue color light can be produced in different combinations, certain shades of blue can include other wavelength photons, but we call them all blue based on our brain's ability to perceive them all as shades of blue.
Yes, theoretically, there can exist blue light that is made purely of blue wavelength photons, and our brain would see that kind of light blue as well. In that case, in our eyes, only the blue receptors (the receptors that are sensitive for blue wavelength photons) would be activated.

White is not a spectral color. It's a perceived color.

How much red, blue, and green does white light have?
In case some of the receptors in our eyes are not sensitive enough for certain wavelength photons, the color vision will be different, because our brain can only work with information it actually receives, but if the receptors do not send certain information (they are not sensitive for certain wavelength photons) to the brain, the brain perceives only the information it receives, and that will create a different color vision.
A: Physics dictates certain wavelengths for reflections of light but does not dictate a particular band of wavelengths as blue, red, green, etc. That is dependent on our biological perception. In other words, color is a function of the human visual system and not an intrinsic physical property. Objects do not give off color, they give off particular wavelengths in the electromagnetic spectrum (light) which appears to have a color. 
Color is only in the eye of the beholder. This depends on both your brain (processor) and your eyes (sensors). Your brain, particularly the occipital lobe, processes the frequencies that your eyes receive and gives you what color the object is. The eyes which first receive the light also need to work properly. Eyes have rods and cones cells. The former detects only light and dark and are very sensitive to low light levels. The cones, on the other hand, detect color and are concentrated near the center of vision. For your father, certain cone cells may be non-existent, not working or detecting a different color than usual. Mild color blindedness exists when a cone cell in a group of 3 does not work properly and hence the person sees a different color than usual.
A: It depends on the sense in which you are using the word colour.
In physics, the real phenomena which corresponds most closely to "colour" is electromagnetic frequency
However, the eye itself does not simply detect the specific electromagnetic frequency of what it can see. Rather, it has a set of so-called cones which respond to a range of frequencies. The response of each cone is strongest at an ideal frequency, and attenuates the further the frequency is off from the ideal. 
In most humans, there are three types of these cones, which have ideal frequencies that roughly correspond to red, green, and blue. In the "colour blind", one or more of these cone types (typically the red or the green type) are missing. Exceptionally in humans, some have four types of cone which extends the ability to perceive colours, and animals can have a significantly different configuration to humans.
Any given colour tends to activate all the cones to varying degrees according to the frequency. A red light will activate the red cone to a high degree, the green less so, and the blue less so again. A yellow light will tend to activate red and green to similar degrees, but the blue less so. The brain compares the relative strength of signals from all three to determine the colour.
The effect of missing cones is that the ability to precisely differentiate between different frequencies is lost. The person can still see broadly as normal, but some examples of colours that strike the normal person as obviously and vibrantly different (like red and green) begin to look similar - one colour will merely seem like a duller version of another colour, since a person will have fewer types of cone from which to draw the distinction.
Another issue is the status of the colour "white". Humans often tend to perceive or conceive this as a specific colour, but at the physical level it is a colour chord consisting of multiple or all colour frequencies across the visible spectrum. 
This colour chord system is also the reason why colourblind people struggle to distinguish, since more than one physical frequency profile is capable of producing the same perceptual response - in practice, normal humans can be made to perceive white light simply by showing them pure mixtures of red, green, and blue, and they can be made to perceive yellow by showing them either pure yellow or by showing them a chord of red and green.
Black, too, is special in that it corresponds to the absence of detectable light, not to the frequency of any light.
Also, the naming and categorisation scheme for colours is conventional or specific to normal human biology, and doesn't closely correspond to any objective physical schema.
A: This question is better asked on a philosophy forum.
If by "physics" you mean naturalist physics then you are basically asking how the naturalist treats qualia. The answers above go some way to detailing that position, though to that it should be added that learnt language has been shown to directly affect the subjective experience. A popular idea is that the qualia for turquoise are not generated until the word turquoise is learnt.
Panpsychism is a philosophy that seems to be creeping into the physics mainstream.  In this view, and others, qualia have direct causal ability. Perhaps one could treat electromagnetic waves as being merely a medium for the proliferation of qualia. You could, for example, argue that because photon frequency is not produced until observation that the concepts of qualia and electromagnetic waves are inseparable.
So in short, the answer to your question is up to you and what underlying philosophy you feel is ultimately the best for you, eg what you might believe will end up being most pragmatic. Personally I feel the idea of pan-consciousness as being the most attractive, where mental phenomena are considered to be in the same realm as the physical and have causal ability.
By the way, the question as it is currently posed could be a false dichotomy. For example, if you treat qualia as having causal ability, then they are natural phenomena and therefore "blue by nature" regardless 
A: Both!
Human color vision starts with the 3 different pigments in the retina that have different spectral responses (common case, there are a lot of complications and these spectral responses overlap a lot). We get 3 color-related signals that we can call roughly red, green and blue.
These signals are heavily processed both before reaching the brain (in the retina) and then in the brain. In order to be able to recognize a color (or an object) in a different lighting conditions and in different visual contexts, a great deal of analysis is applied - using the whole visual field and some previous knowledge, a 'white balance' and general intensity calibration is applied.
One could easily fool the white balance when no applicable reference points exist - that's how "yellow/blue dress" exists.
Intensity calibration is also a rather easy target - the Moon's ability to reflect the light is comparable to a pile of coal and then again it looks white and not black when high in the sky.
Then again, in most cases human color perception is pretty good. We precisely recognize our everyday object's colors in bright sunlight, when cloudy (1/100 of the bright sunlight), in different artificial light (even less light with different spectra) and so on.
The color blindness adds some more complexity. One can have some of the 3 retina pigments missing or altered by a genetic mutation to have a different spectral response. The most frequent color-blindness is the "red" and the "green" spectral response overlapping more than they should. The rest of the vision tract adapts (to the extent possible) to the compromised input signal.
A: Colours are "grounded" in physics but how we perceive them depends on how the human visual system works
A physicist–at least one with access to a photometer than can measure specific wavelengths of light–can fully describe the makeup of any light source. She will be able to plot the intensity of each wavelength in the light source. The relative intensity of each wavelength in the visible portion of the spectrum is a meaningful characterisation of the "colour" of that light. In this sense, colour is grounded in physics.
But human perception of colour has an extra complication because of the way the human visual system works. They eye does not detect all wavelengths like the physicist's photometer: it has, usually, only 4 types of receptor each with the ability to detect only specific ranges within the visual spectrum (one basically detects brightness across the spectrum, especially in low-light situations, the other three detect specific ranges of wavelengths often described as red, green and blue.) And, on top of that, the cornea filters out some near UV wavelengths that some of the receptors could detect.
The visual system also does a fair amount of computation before sending signals to the brain. In effect, the human perception of colour is driven by the differential intensity of the signals from each receptor in the eye. The perception of colour is based on those signals. While that perception does a very good job helping people describe the colours of the real world, it does not map perfectly onto what the physicist's photometer sees.
For example, the light in the natural world is almost always driven by the broad spectrum illumination of the sun. But many unnatural sources of light (in some computer monitors, fluorescent lights, LEDs and other sources) has narrow bands of light not a broad spectrum covering all possible wavelengths. Many fluorescent phosphors in CRTs and fluorescent lighting use narrow-band europium-based emitters to create "red". This excites receptors in the eye sensitive to red wavelengths and, correctly combined with emitters exciting the blue and green human receptors, give the impression of white light. A physicist with a photometer (or a simple spectrometer) can tell them apart but the human visual system cannot. What matters for perception is the relative signals created by the different human receptors and these can be excited by narrow bands of light in a way that matches the expected signal from sunlight.
This is the basis for colour photography, television and movies: the film (or digital sensor) does not record all wavelengths of light, just the intensity of light roughly corresponding to the eye's three colour receptors). As long as the reproduction of the colours (on a screen) cause the same excitation in the eye, the perception of colour will be preserved.
But some people have mutated receptors sensitive to slightly different ranges of wavelengths. The commonest form of colour blindness isn't cause by a lack of one colour receptor but by too much overlap between the red and green receptors (one way to improve colour perception for this group filters a narrow range of wavelengths at the point of overlap reducing the range of wavelengths that excite both red and green receptors thereby improving the differential signals being fed to the brain and increasing the ability to perceive the difference between red and green).
In short, colour is grounded in physics. But our perception of it is complicated by the way the human visual system works. And some people have mutations that alter their ability to process incoming light and distort their perception of colour.
A: As all the other answers already given here point out, the way we perceive colors is subjective. The brain processes the physical information, the wavelength of the light does play a role in the way it interacts with the photo-receptors that are sensitive to different frequencies of light. But in the end it's all a matter of how this information is processed by the brain.
Our perception of color is not just subjective, it's also something that the brain constantly adapts to the ambient lighting conditions. This necessitates us to adjust the white balance of our cameras. The degree to which the colors of a digital picture are off when using the wrong white balance is a good measure of the degree to which our brains adjust the color perception. 
The way we perceive color is also influenced by our upbringing, in particular the way colors are referred to in the language we learn. Some populations speak a language that lacks a word for the color blue, for example the Himba people have many words for different shades of green, but they lack a word for the color blue. In a color perception test, they struggled to pick the odd one out when given the following choices:

but they had no difficulties picking the odd one out from this set:

A: Colors are subjective, for instance, the rainbow does not have color bands, it is our brain that classify a continuous spectrum into discrete colors. In addition, there are colors like brown that do not correspond to any specific spectral band. There are cultures who do not distinguish green from blue, or orange from red.  The drive to classify the world into categories, even if the physical stimulus changes continuously, is ingrained in the human brain. 
And there is nothing "blue" in the light. We might all be calling blue to the color of the sky, even if the subjective feeling of the blue for you could be different than the one for me, that is, my blue could be more like your red, but we do not have any way to know if that is the case.
A: The different colors that exist in the visible spectrum are governed by the physics of electromagnetic waves. In that sense, colored light exists even if there were no human eyes to detect it with. 
The specific manner in which our eyes detect light, turn it into nerve impulses, and send those impulses to the brain involves physiological optics and the neuroscience of vision. In this context, there is nothing inherently "blue" about light with a certain wavelength, its "blueness" just happens to be the way our eyes and brains respond to that particular wavelength of light. 
A: The human eye contains three different types of cone cells, which respond to different bands of light wavelength, known by convention as "red", "green" and "blue". There is some overlap, for example wavelengths between the peaks for red and green stimulate both red and green cone cells and the colour is perceived as yellow. The human eye is unable to distinguish between a single "yellow" wavelength of light and a blend of "red" and "green" wavelengths. 
Other animals, such as goldfish, have more types of cone cells and are therefore believed to be able to differentiate better. Conversely many animals have only one type of cell and are believed to be colourblind. Also, some animals' eyes are able to percieve light which humans cannot, what we call infrared or ultraviolet.
For the response curves for the three types of cell in the human eye, see the link below.
https://en.wikipedia.org/wiki/Color_vision#/media/File:Cone-fundamentals-with-srgb-spectrum.svg
A: Colors can be described at three different levels.
The first two; electromagnetic frequency and neural signalling, have been well addressed in other answers. These are both objective measures of color, and it is even becoming possible to measure what color someone is seeing - or even imagining - in realtime, via non-invasively monitoring the pattern of their brain waves.
The third aspect or level is subjective. Why does blue feel "blue" and red "red"? Why don't they feel the other way around? Do all of us "feel" the same colors when similar neural signals arise? What about color-blind people, or other sentient creatures with different color vision, do they still see our blue as blue, or what? The problem is, it is impossible to find out. No physical theory, be it quantum physics, relativity, statistical mechanics, or of any emergent properties of matter arising from these, has anything at all to say about the subjective qualities of conscious experience. It is no longer the domain of physics but of philosophy, and even there it is simply known as "the hard problem".
