Let us begin by taking a real life example.
You are in your room and at the door, there is a large block that blocks almost the entire way( the space between the ions is very small ). Say it is motionless. Now, you would not find it hard to pass the block. All you have to do is to squeeze a bit to get through.
Now, let us say that the block is vibrating. It will be difficult for you to get through. You've got to go extremely fast in order to have a chance to get through. Remember, electrons drift very slowly, slower than you could imagine. They travel approximately 1 meter every hour.
You might say that as a result of thermal agitation, the electrons gain velocity and so the gain and loss balance each other. The question is, in what direction does it gain speed? Thermal agitation results in randomness. So, it is not right to say that the electrons gain speed in a particular direction. Electrons can gain speed in any random direction due to thermal agitation. There is a chance for an electron to gain speed in a particular direction but, there is also a chance for another electron to slow down( equal amount ) in that direction. So, the average velocity as a result of thermal agitation remains zero. Furthermore, the electric field created by the voltage source stabilizes everything and the electrons drift in a particular direction with an average velocity of 1 m/hr( approx ).
In summary, the number of collisions increases as the temperature increases. This is due to the lattice vibrations. The electric field tends to set the electrons in drift motion against thermal agitation. The vibrational kinetic energy of the ion increases which results in a greater number of collisions.