I think it is correct because: v = displacement/time, if the direction changes, the displacement changes so the velocity is not constant anymore. Please correct me if I'm wrong
Yes you are correct$^1$, to define an object's velocity we have to specify in which direction it is travelling and how fast - this defines a vector quantity. If we say the vector is unchanging, its length (the object's speed) and its orientation (which direction the object is travelling) must be constant in time which corresponds to an object travelling in a straight line at a constant speed.
$^1$ Ignoring complications for example the presence of a gravitational field.