If no energy is wasted, it will take no energy to move the object.
You started by saying "at constant speed". This must mean that the object has speed at the start of your time interval - otherwise it would have to accelerate and speed is not constant. Ditto at the other end - it can't decelerate at the end.
And in between there is no dissipation ("waste") of energy.
The conclusion is that this is an idealized scenario in which Newton's first law applies: absent any forces, an object continues in a straight line at constant speed. And if there are no forces, then no work is done.
Note that the last part of your question "does it matter whether the object accelerates or decelerates..." is inconsistent with the conditions of the first paragraph ("at constant speed"). If the object decelerates, it is actually giving up energy to "something": if it is not "wasted", it must be transferred to something else (for example a spring is being compressed). In that case, energy is extracted from the object, and the moving "required negative energy". Similarly, if it accelerates (perhaps a loaded spring gave it a push) then it will have more energy at the end than at the beginning, and it took work to move it. The amount of work in either of these cases would be
$$W = \frac12m(v_2^2 - v_1^2)$$