If anything, the diffusion rate of cold air would likely be slower. We can get a characteristic diffusion time by considering the mean free path of molecules in the gas and its speed of sound. For two gases that are chemically identical (i.e. they differ only in $T$ and $p$), we have:
$$
v_\text{sound}\propto \sqrt{T}
$$
and
$$
d_\text{mfp}\propto\frac{1}{p}\sqrt{T}
$$
giving
$$
t_\text{diffusion} \propto p
$$
So, the diffusion rate, to a first approximation, does not depend on temperature. If we apply the ideal gas law $p=\rho R T$ to a closed box of air, then $p \propto T$, meaning colder air is associated with lower pressure (generally true in the midlatitudes) and hence we would expect warmer air to diffuse more quickly.