Consider an ensemble of electrons which all experienced a collision at time $t=0$. Let $n(t)$ denote the number of electrons in this ensemble.
Assume that the number of electrons $\mathrm{d}n$ from this ensemble experiencing collisions in a time $\mathrm{d}t$ is proportional to $n$, i.e. $$\mathrm d n = -an(t) \mathrm d t,$$ for some consant of proportionality $a$.
Edit. Also assume that at $t=0$ an electric field is switched off.
Why is the relative change in drift velocity $\frac{\mathrm d |\langle \mathbf v \rangle|}{|\langle \mathbf v \rangle|}$ equal to the relative change in the number of electrons which have not yet experienced a collision$-\frac{\mathrm d n}{n}$?
Edit. Noting that the thermal drift velocity is always zero, the contribution of an electron, which experience a collision after $t=0$, to the average velocity will vanish as soon as it experiences a collision. Because then it loses the drift velocity that was induced by the electric field.