The photons from the CMB were emitted when the universe was a few hundred-thousand years old. That was the amount of time it took for the universe to become transparent: at earlier times, the density of stuff (plasma) was too high, and photons were absorbed and scattered around instead of "free-streaming" out, and eventually to us.
Gravitational-Waves (GW) don't have the same obstacle. In particular, because the coupling between GW and matter is so weak, the universe becomes transparent to them incredibly quickly. I think the particular time they decouple is model-dependent, but suffices to say when the universe is (far) less than a second old. This is why people are excited about primordial GW observations1 --- for their window into the earliest stages of the universe from which we'll never see light.
1: There is a stochastic Gravitational Wave Background (GWB) which usually refers to the background of gravitational waves from unresolved astrophysical sources (like binaries) all added up. The Cosmic Gravitational Wave Background is different, and what we're talking about there.