If the wavelength of a neutron is much shorter than the distance between atoms in a material, then the approximation that the neutron is interacting with the entire crystal collectively doesn't really hold any more. That means fast neutrons are insensitive to the details like crystal structure and phonon spectrum that make graphite different from diamond. For fast neutrons, graphite and diamonds are both just clouds of carbon atoms.
The reasons to use diamond instead of graphite in a fast-neutron environment would be if the material needed to be stronger, or harder, or if the two materials aged differently in the presence of radiation. Another possibility is that diamond might be less likely to contain non-carbon impurities. Supposedly, one of the reasons that the Allied nuclear program got ahead of the Axis during World War Two was that the Allies had better access to low-boron graphite.
I have a vague recollection about diamond being used as a surface treatment in some ultra-cold neutron experiments, but that's a the complete opposite end of the energy spectrum from a fission moderator.