From my line of thinking, if we had a room with a bunch of prisms
randomly placed about and had a light enter from one side, the walls
would be quite colorful.
That would depend on the size of the prisms. If the prisms are large enough, then they will throw macroscopic patches of color on the walls. The light path reaching two spots on the wall a millimeter apart may travel through the same prism in almost the same way. This patch is large enough to map over several cones in your retina, allowing color detection.
If the prisms you threw up were randomly oriented and microscopic, then there will be no large patches of color anywhere. If the overlap of the color patches projected onto your retina is such that each cone is getting about the same distribution of wavelengths, then the color effect is lost and you just perceive white.
...if that logic applies to clouds, then why not rainbows?
Clouds are microscopic water drops. Most of the light is scattered from the surface of drops with no preferred direction. You also get multiple scattering events. The end result is all the wavelengths going in all directions.
Rain drops are much larger. The surface to volume ratio is much smaller, so less light is scattered from the surface than with the same mass of water as condensation. Instead of multiple scattering events from the surface, you can get a significant amount of light reflected from the interior of the drop. Combined with incident light of one angle, and drops all relatively near each other, you get the effect of oriented prisms.