Why aren't rainbows blurred-out into nothing after they are produced? I understand how a prism works and how a single raindrop can scatter white light into a rainbow, but it seems to me that in normal atmospheric conditions, we should not be able to see rainbows.

When multiple raindrops are side-by-side, their emitted spectra will overlap. An observer at X will see light re-mixed from various originating raindrops. The volume of rain producing a rainbow typically has an angular diameter at least as wide as the rainbow itself, does it not?
So why can we still see separate colours?
EDIT: To emphasise the thing I am confused about, here is a rainbow produced from a single raindrop...

...here are the rainbows produced by two raindrops, some significant distance apart...

...so shouldn't many raindrops produce something like this?

I will accept an answer which focuses on this many-raindrops problem, I will not accept an answer which goes into unnecessary detail as to how a single raindrop produces a rainbow.
 A: This isn't quite how rainbows work. The standard explanation is that light bounces around inside each droplet, and getting reflected once, and exiting at an angle:




Image source

However, the real picture is a little bit more complicated. When sunlight hits a water droplet, the rays will

*

*refract when they come in,

*(partially) reflect back when they hit the back of the droplet, and then

*(partially) refract on their way out.

For each droplet, though, there are a bunch of rays hitting the droplet at different locations, and each of them will bounce around differently and exit at a different angle, so that the end result looks like this:

Because there is a reflection inside the droplet, the light is mostly sent backwards, and because there are two steps where refraction happens, the angles are a bit wonky. But here's the important thing: the angle at which the light exits increases, has a maximum, and then decreases again, a fact which is clearly visible by following the dots as they go down from the negative-$x$ axis, stop, and then go back up again.
This means that if the relative angle between the Sun, the droplet, and your head is smaller than a certain maximal angle $\theta_\mathrm{max}$, usually equal to about $\theta_\mathrm{max}\approx 42°$, then the droplet will appear bright to you (and, since this isn't an individual droplet but a misty conglomerate, the mist will have a diffuse glow), and if the angle is larger than that, then there will be no extra light going towards your eyes from those droplets.
In other words, then, this process will produce a disk that's bright, centered at the anti-solar point (i.e. where your eyes receive the on-axis reflections in the diagram above) and with diameter $\theta_\mathrm{max}\approx 42°$, and this is precisely what's observed, particularly when the rainbow happens against a darker background:

Image source
Notice, in particular, that the inside of the (primary) rainbow is much brighter than the outside.
Moreover, notice that the brightness of this disk increases as you go from the center to the edge: this is caused because the rays cluster at the turning point at $\theta_\mathrm{max}$ (notice in the ray diagram that there's many more dots in that region than there are near the axis). This clustering means that, for each color, the disk of light has a particularly bright edge, called a caustic.
So what's with the colors?
Although your diagram's geometry is off, as you correctly note, the standard diagram (the first figure in this answer) is kind of misleading, because for it kind of implies that for every red ray that hits your eyes, there will be another droplet at another angle sending a yellow ray (or green, blue, orange, indigo, and so on) on the same path ─ and that is indeed correct! This is what happens inside this disk of light.
The thing with this process, though, is that the maximal angle of aperture of the cone of light that's reflected by each droplet depends very sensitively on the refractive index of the water that makes up the droplet, and this refractive index also depends on the wavelength of the light, so that the size of the disk increases with the wavelength, with the red disk being the largest, then the orange, yellow, green, blue, indigo and violet being successively smaller.
This means that, at the edge of the disk produced by the red light, where it is the brightest, there is no light of other colours to compete with it, so the light looks red there.
A bit closer in, at the edge of the orange disk, there is no light of yellow, green, or blue colors, since those disks are smaller ─ and, also, the light from the red disk is fainter, because it's not at the maximal-brightness edge and the orange disk does have its maximum shine there. Thus, at that location, the orange light wins out, and the light looks overall orange.
And so on down the line: for each color in the spectrum, the edge of the disk is brighter than the larger disks, and the smaller disks don't contribute at all, so the edge of each disk shines with its respective color.

For further reading on the creation of rainbows see e.g. this excellent previous Q&A.

And finally, to address the subquestion: why aren't the different colours blurred together once they reach the retina? Basically, because in the human eye the retina is not exposed directly to the air $-$ the human eye is a fairly sophisticated optical re-imaging system, which uses a lens at the front of the eye to focus the incoming light onto the retina:

If this lens was not present (say, if the retina was where the dashed gray line is, and the lens had no effect) then you would indeed have light of different colors hitting every cell of the retina, and the retina would report a big jumbled uniformly-coloured mess to the brain.
Luckily, of course, the lens is present, and the effect of the lens is to re-focus the light, so that (at least, when the eye is focused at infinity) light coming in collimated from different angles will be focused at different lateral positions in the retina. Since the different colors are coming in at different angles, collimated from the rainbow which is effectively at infinity, this means that all the red light will be focused onto certain retina cells, and the blue light will be focused onto different retina cells at a different location, and so on.
It's extremely important to note that this has nothing to do with the fact that what you're seeing is a rainbow, and this re-imaging scheme coming from the focusing by the lens at the front of the eye (and the potential blurring problem we'd have if the lens wasn't present) is universal to seeing any objects at all, colored or not, rainbows or not.
For more details of how the eye works, see your favourite optics textbook.
A: Your picture shows that an observer at X will see both the strongly scattered red light and the strongly scattered blue light, but he will see it coming from different directions.
That's the same way you usually see things: different amounts or colors of light reaching your eyes from different directions, and thereby creating an image on your retina.
A: To address your edit: you are mistaken, a single raindrop does not produce a rainbow from the observer's point of view.
A droplet occupies a given point in the field of view, and is visible as an infinitesimally small dot. Emilio's answer gives a detailed explanation of underlying phenomena, but the net result is the same as in case of dispersion - light of different colors travels alongside different paths. Suppose the observer is located at such an angle that the color traveling form that droplet to their eye is red. In that case, the observer will see a red dot.
A second droplet some distance to the right splits the light exactly in the same way as the first one. However, since it's located at a different angle w.r.t the observer, a different color will arrive to their eye, let's say, orange, while the red ray from that same droplet will miss the eye. The observer then sees an orange dot some distance to the right from the red one. Repeat this for the whole field of view, and you'll get your rainbow.
To sum it up: each droplet produces all colors at once, but the corresponding colored rays travel in different directions. If we only consider directions from a droplet to the observer, then each droplet produces only one color.
Now, light from individual droplets indeed mixes when it reaches ground. Imagine Christmas lights: if you take a whole bunch of them and point to the wall, the color of that mixed light will be more or less white. However, your eyes can still see individual lights and tell the colors apart. That's because individual lights are located at different angles, just like droplets in a rainbow.
A: The picture in your question represents a halo rather than a rainbow: the rainbow is seen when the Sun is behind you, while halos appear when the Sun is in front of you.
The actual mechanisms producing colours vary between the phenomena, but the basic idea is the same: if light of certain colours comes to you from different directions, your eye will distinguish those colours.

If you remove the lens from the picture, the colours will indeed blend, but that's the whole point: your eyes can't see without the lens.
A: Your confusion arises from the fact that you think a single raindrop would produce a rainbow. It doesn't. As answers to this and previous questions explain, the incident sun light creates what looks like a specular highlight on each individual drop, with maxima for the various colors at slightly different angles. See the animation in this excellent answer to "What do individual rainbow-forming droplets look like?"
You will only see that colored highlight if you look at a given raindrop from the correct angle. The incident sun beams are all parallel, so as the observer looks across the sky, all the same-color raindrops form what, from the observer's point of view, appears to be an arc, of which the antisolar point is the center. Note that, as the observer moves, so does that arc, and the constituent raindrops are then at different locations. (This is why you can never get to the end of a rainbow.)
Similar questions have been asked and answered that illustrate this further:


*

*Shape of the rainbow

*Why is rainbow always circular?

*Why is a rainbow curved in shape?
A: The key point is that you need to distinguish between what would be seen on a screen and what gets seen by your eyes. This is a common confusion when people start studying optics. 
Light can hit a screen at many different points. At each point, light coming in from every direction gets blurred together.
To a rough approximation (i.e. good enough to get the concept), light can only enter your eye at one point. But at that point, light coming in from each direction gets focused to a different point on the retina. So unlike a screen, you might miss some light entirely, but you can see where the light you do capture comes from. This is a concrete tradeoff that was made early in our evolutionary history.
A single raindrop would produce a very faint, complete rainbow on a screen. And indeed, multiple raindrops would cause rainbows to pile up on the screen, blurring it out.
A single raindrop does not produce a rainbow in your eye. In fact, unless you're standing right in the tiny "rainbow cone" produced by that raindrop, you won't see much of anything. If you're in the cone, you see a glint of one color, the color emitted in the direction that hits your eye, coming from where the drop is. And if there is an adjacent raindrop, you'll see a glint of a different color coming from a different direction. The combination of many drops creates one rainbow that you perceive. 

Getting the screen/eye distinction right is one of those key insights you need to understand optics. For example, you should be able to understand why you can watch a movie from the back of a room, but you can't watch it by looking at the screen's reflection on the back wall. You should also be able to understand why two people standing near to each other see completely "different" rainbows, in the sense that different water droplets are involved.
A: Your reasoning is correct for answering why rainbows don't appear as visible illumination on objects, like you would get if you shone a light source onto a large glass sphere. The scattering of light from all raindrops averages out with the end result that the sunlight passing through a rainbow and hitting a surface does not appear coloured, merely slightly darker.
However, a human eye is not just a patch of photoreceptor cells measuring the total illumination hitting them, but uses its shape and lens to allow for angular resolution, that is, knowing where the light is coming from.
And the light from each direction is colourful, because all raindrops along that sight line produce the same colour and thus reinforce each other. The light of a different colour comes from a different direction and is thus also seen from a different direction.
This is what makes rainbows so fascinating: They are an optical phenomenon that can only be observed by methods allowing for angular resolution, like looking or photography.
A: You're right tthat to our eye (located in point X) arrive rays of light of different color. But as your own picture shows, they arrive from different directions. And because of the optical properties of the eye the rays coming from different directions are eventually mapped onto different points of the retina. This makes it sdo that the colors originating from from different points don't mix and blur.
