The Hertzsprung-Russell diagram that categorizes star types uses the letter codes O, B, A, F, G, K, and M to indicate a star's temperature/color. Hottest (blue) is O and coolest (red) is M.
What do the letters themselves relate to?
For example, why is "O" used for hottest/blue?
When astronomers started to get spectra of stars and began classifying them, the initial classification was based on the strength of the Balmer absorption lines in the spectra. The Balmer lines are created by electons in hydrogen atoms that are currently in the second energy level (N=2) absorbing energy and jumping up to higher levels. The stars with the strongest lines were given a designation of A, the second strongest B, etc. on down the line.
Later, the association with temperature was made and many of the classes were combined and reordered based on the temperature of the star. However, the main letter designations were kept even though they were now "out of order". So the letters actually correspond to the relative strength of the Hydrogen absorption lines.
The reason the O stars, which are hottest, are so far down on the scale is that they are so hot that Hydrogen is fully (or very nearly fully) ionized. If the atom has been ionized, it by definition doesn't have any electrons in the N=2 level since it doesn't have any electrons at all. Since there are no electrons in the proper level to form the line, the line is weak as it is only created when a chance recombination occurs.
The cooler M stars, at the other end of the main sequence, also show very week Hydrogen lines. The reason for this is that they are too cool to even get the electrons excited enough to get up to the N=2 energy state. If there are no electrons in N=2, there is no way to get the electron to absorb energy and jump higher and form the Balmer lines. What hydrogen lines are seen here are due to the few higher velocity atoms on the long tail of the velocity distribution. When they collide, they have enough energy to get their electrons up to N=2.
As the temperature increases, the energy available to excite the atoms increases as well. As you move up from M class stars you get to spectral types K, G, F, and then A, each one increasing in strength of the lines. The A stars, at about 10,000 K, are just the right temperature to push almost all the hydrogen atoms into the higher N=2 state where they can then be knocked higher and produce the Balmer series. Since there are a lot of electrons in N=2, you get strong lines. The next spectral class up is B, which has weaker lines because at this point you've started to ionize the hydrogen atoms and therefore have fewer atoms with electrons in the N=2 state. And then comes O stars which are nearly fully ionized and so are very week in the Balmer series.
The spectral classes were originally developed by spectroscopists, before stellar astrophysics was nearly as developed as it is today. So they were based on the presence, absence, or relative strength of spectral lines, rather than the overall blackbody spectrum, which would make more sense. Unfortunately, since most lines first become more prominent and then less so as the temperature rises, ordering the spectral classes in the order of their strength produces a peculiar order of temperatures. The original scheme was rather complicated but I think the strength of the H-alpha line was paramount.
Also, over time many of the spectral classes have been combined, leaving us with OBAFGKM on the main sequence, and a few more classes of sub-main sequence almost-stars.
PS The standard mnemonic for remembering the order is "Oh, Be A Fine Girl/Guy/Gal/Grad Student, Kiss Me!"
The letters used are not significant as they were assigned arbitrarily before anyone really knew anything about the subject. See the Wikipedia article here for more about it.
