Do the "colors" live in a 3-dimensional vector space? Do the "colors" form a 3-dimensional vector space?
Colors are just different frequencies of electromagnetic lights, from the low frequency (infra-red, red) to the high frequency (blue, purple, ultraviolet).
So using the value of frequency $f$ or the wavelength $\lambda$ to label the color is 1-dimensional:
$$\lambda \in  [400 nm, 700nm].$$
However,  people usually use RGB or CMYK systems with 3 coordinates
$$(r,g,b)$$
$$(c,m,y)$$
to label the colors. Hence the colors seem to live in a 3-dimensional vector space? But we need to define the addition between two vectors, and the multiplication on the vector, if that makes sense. More precisely, how are colors being labeled mathematically?

 A: As What is Gray, from a physics POV? says, colors are only in the mind. Never the less, there is physics behind it, or at least color science.
While colors do form a 3D space of a sort, as thorimur's comment said, it is not a vector space. A vector space contains vectors that can be added together and multiplied by numbers. Let us talk about adding light by combining beams of colored light.
If red was a vector, $3*$red would be too. Also $100000*$red. This would mean a redder red, not a beam of light with more energy in it. There is a limit to how red things can get, because there is a maximum sensation your eye can generate.
Also there would be $0$ vector. Adding $0$ to any vector would leave the vector unchanged. It would have to be black, the absence of light. Given red, there would have to be a "negative red" that you could add to red to get $0$, or black. Light does not add this way.
Typically, mixing beams of light leads to white. See Can you create white light by combining cyan wavelengths (490-520nm) with red wavelengths (630-700nm)?. White is not the $0$ vector. It is more like the "biggest" color, the color you get by adding up all the others.
So it doesn't really work like a vector space. Never the less, color spaces have been invented, as the last link and Ruslan's answer describe.
A: 
Colors are just different frequencies of electromagnetic lights

If you follow this definition, you'll only ever get pure spectral colors, which your monitor isn't even able to display (unless it's a laser projection display). Going over spectral colors with one luminance, you'll only draw a one-dimensional curve in the whole visible color gamut.

the colors seem to live in a 3-dimensional vector space?

The set of all colors is normally not a vector space: it's more like a subset of a vector space, because there are limits to values of the color components. E.g. some values would yield imaginary colors, while others (too high) would simply destroy the retina. The latter is important because color is in principle a product of perception, and it doesn't make sense to abstract them from the limits of detectors.
But yes, the set of all* colors is well represented by a 3-dimensional space called color space. One of the most basic color spaces is the CIE 1931 color space, which is the base to derive other color spaces like RGB (with multiple varieties like CIE RGB, sRGB and others), CIELab and others.
The CIE 1931 color space allows to add the colors as vectors, yielding the color that one would get by mixing corresponding light sources. In others addition may be not as simple in meaning, e.g. due to nonlinear transfer function (sRGB), perceptual color distance function (CIELab), etc..

*There are caveats: in scotopic mode there's only one dimension—luminance—and its luminosity function differs from the photopic one. In between there's mesopic vision, where the color space is 4-dimensional, since rods and cones are active simultaneously.
A: I'll put this in another way, perhaps it will help someone to finally grasp it.
THERE ARE NO COLORS
There are just rays of light of different frequencies. And there's an infinite variety of frequencies available.
Where colors appear are in our eyes and in our minds. In our eyes we have the photoreceptor cells, and our minds interpret the signals from these cells. Furthermore, you probably already know that there are 4 types of these cells in our eyes - one senses just brightness, the other three sense ROUGHLY red, green and blue wavelengths. Although it's really more complicated. Check this out:

As you can see, each cell doesn't just respond to a single frequency, but rather to a large range of frequencies, and these ranges overlap A LOT.
In the end, the color that we perceive is determined by the RATIO of the stimulation of all three types of cells. You'll also note that some colors like white or brown aren't even on this chart, because they require more than one frequency of light rays at the same time.
Other animals can have more or less than 3 types of cells, and the colors they perceive will be drastically different from ours. Check out the mantis shrimp which has 12 to 16 types of color receptor cells!!!
Interestingly, there are even some humans with 4 types of cells, although a fully functioning tetrachromacy is very rare.
So back to RGB and CMYK and HSL and whatnotelse. These notations are used because they are convenient ways to describe most colors we humans can perceive. Enough for practical purposes*. But none of them actually cover ALL of the colors that we can perceive. Similarly, I also don't think we've created a device that could create all the possible colors that humans can perceive (but I could be wrong on this one).
And if we wanted to create a mathematical representation of all the colors that a mantis shrimp can see, we'd need 16 components instead of 3 components.
As for treating colors as 3D vectors - uhh, sure? You can do that, mathematically. What that will give you however, I don't know.
*Added: as noted in comments by Edgar Bonnet, there is the CIE 1931 XYZ color space which does cover all possible colors that humans can see.
A: It depends on how you use the word "color".
I. You can take three additive primary colors as the basis of a vector space.  The combinations of these colors (with positive coefficients) fill out not the entire vector space, but the first quadrant of that space.
But as you move out along any line through the origin, the points all represent the same color, just with different intensities.  So if you consider a standardly-imbedded simplex (i.e. the triangle below, with vertices at the standard basis vectors), there is a one-to-one correspondence between colors and points of that simplex.

If you take that simplex and round off the corners, you'll get the "color wheel" that you probably encountered in elementary school.
II. Note that every possible pure frequency of light determines a color, and that you can combine these any way you want.  Since there are infinitely many pure frequencies, the possible combinations form an infinite dimensional vector space, which you might want to think of as the space of "colors".
There is a linear map from the infinite dimensional space of colors in sense II to the three-dimensional space of colors in sense I (and from there, by projection along lines through the origin, to the color simplex/wheel), given by mapping a combination of pure frequencies to the way the human eye perceives that combination.  Obviously this map cannot be one-to-one, so that for any given color X in sense I there are a vast number of colors in sense II that we perceive as X.
A: Our perception of colour is reasonably modeled as a 3 dimensional vector with no negative elements.  2*Red is "brighter red", in the limit the Red kills you.
You can actually perceive a limited kind "negative red" using optical illusion tricks; basically, look at a bright red object for a while, then look away at a white object.  You'll see an afterimage that is a negative image of the bright red object you saw.
Like all things in physics, the mathematical model has non-universal applicability.
The 3 dimensional colour perception is linked to reality because much of our colour perception comes from 3 different pigments in the eye.  These pigments react to light of various frequencies in different amounts, and their differences and intensities are translated (to a first approximation) into that 3 dimensional colour space you are used to.
Light itself is an infinite dimensional vector, where all values are restricted to be positive.  Our perception of it, to a first approximation, is a convolution of that infinite dimensional vector space with the pigment frequency response curves.
There are a lot of things that don't exactly match this model.  For example, you can take one person and have them mix multiple frequencies to produce a colour that exactly matches a pure frequency, a different person often disagrees with what mixture matches the pure frequency!  What more, you can test that this disagreement is "real", in that given two people, one will be able to tell the difference between the mixture and the pure frequency, while the other will not.
This is because there are actually 4 different pigments that are common in the human population; two of them are close mutations of each other.  Typically someone has 3 of them, but which of the two variants they have is somewhat random, and (as it lives on the X chromosome) some women have all 4.  Those with all four are known as tetrachromats.  What's more, almost everyone has this at low light intensities (as our low-light vision is an additional pigment in effect).  It is not proven if this results in a different experience of colour vision.
But what you have to do is stop thinking about colours as frequencies as far as human perception is concerned.  We rarely see pure frequency light (rainbows and lasers mostly).
A: The way we represent colours is just that: a representation.
When I first worked with Tektronix colour terminals in the late 1970s, we used the HLS system, which was represented as a double cone. The Wikipedia article about HSL/HSV systems shows this model. Similarly, RGB, CMY, HSV and various other systems all use 3 parameters, so they can be represented by a 3D model. The K in CMYK just adds some extra capability; the main reason for having it is that we use so much black by itself.
The reason for using 3 parameters is the fact that our eyes have 3 different colour receptors, the retina's cones, with sensitivities in the red, green and blue wavelength ranges. The peaks of sensitivity are at around 440, 545 and 565 nm. However, the cones have sensitivities over a broad wavelength range and especially the read and green cones have a large overlap.
There is nothing 3-dimensional about the colours themselves. A colour is no more than a mixture of many different wavelengths. The cones in the retina translate those frequencies into different signals for each of the 3 cones. The brain then recombines them into something that we interpret as a certain colour.
In summary, while it is useful to think about colours in a 3D space, there is nothing physical about that, it is just a representation.
A: I wish to expand on Vilx's answer. First read this post on how we see colour. Next observe that the colour normal humans perceive in a small region depends entirely on the relative intensity of photons that excite those 3 types of retinal cones in the corresponding region on the retina.
So it is more or less correct that colours live in a 3-dimensional space. Moreover, this space is locally quite similar to a vector space, in that the average excitation frequency of cones of each type within a (not too small) region changes roughly linearly with a small change of the incident rate of photons of the same wavelength.
However, as with most things in the natural world, this linear relation will break down for very low or very high photon intensity reaching the eye. After all, when the rate of photons reaching the eye is too low, too much of them get absorbed by the stuff from the cornea to the blood vessels in the retina, so the rate of photons incident on the retina would fluctuate from zero to some low amount, and low signal levels from the retina are actually ignored by the brain (to filter out noise). And when the rate of photons reaching the eye is high, a greater fraction of them will not get absorbed by the cones (because there are only finitely many cones and each has a nonzero latency), so the signal that goes to your brain is not linearly proportional.
With all that said, you should see that it is unclear what it even ought to mean by colours living in a vector space.
