Let us consider the following equation:
$$i=ev_dAn$$
where $i$ is the current passing through the material, $e$ the amount of charge on the charge carrier (here it's the charge on an electron), $v_d$ is the drift velocity, $A$ is the area of cross section of the material and $n$ is the number of charge carriers per unit volume of the material.
For the sake of simplicity, let's assume the material's dimension doesn't change on increasing the temperature. So $A$ is a constant. Also $e$ is invariant of temperature. In a conductor, we know that on increasing the temperature, the current passing through it decreases when the terminals are maintained at a constant potential difference. This is because its resistance increases.
Let's replace the constant terms in the above equation by $k$:
$$i=kv_dn$$
As $i$ decreases, the product $v_dn$ must also decrease to maintain the equality. In a conductor, the value of $n$ remains fairly constant and hence the decrease in current is due to decrease in the drift velocity. And hence the drift velocity decreases with increase in temperature. This is just a reverse reasoning based on our observations from experiments.
An interesting note on semiconductors: As we increase the temperature, the value of $n$ increases in a semiconductor. Even if $v_d$ decreases with increase in temperature, the increase in $n$ outshines it and hence the product $v_dn$ increases on the right hand side and finally the current increases with increase in temperature.
For further reading: Doris Jeanne Wagner and Rensselaer Polytechnic Institute