But if you take the pivot point to be one end of the rod, then we expect rotation.
If you take the pivot point to be one end of the rod, you will find that there is a net torque on the rod at that point. $\tau_{net} > 0$. You might expect that means that there is rotation (or at least that there is a change in the rotation). But that isn't the case.
In linear motion we equate a net force with a change in momentum. In this case we can equate a net torque with a change in angular momentum. When the center of mass is not accelerating, there must be a change in angular velocity. But in the case of your falling rod, there isn't one.
Instead, we can calculate the angular momentum of the rod about the same axis. The angular momentum of a point mass is $L = m v d$. For $m1$, the distance to the axis is zero, and it has no angular momentum. But $m2$ is located $d$ distance away, and it has downward velocity. In freefall, this velocity is changing, so the total angular momentum of the system is changing as well.
The net torque on the end of the rod is describing a system with a changing angular momentum. It's just that it is not associated with a changing angular velocity.
If you calculate the torque about the center of mass, then the contribution of all the masses cancel out and any change in angular momentum will be associated with a change in angular velocity.