What happens to the energy of a body when it is brought down some height?

I initially hold an object of mass m at a height, say h, above the ground. The potential energy is $mgh.$ Now I slowly bring the object down and stop the motion at a height $h'$ above the ground. Assume there is no air.

Now initially the potential energy of the object was $mgh$. As I bring it down, the object possesses some kinetic energy which would compensate for the decreasing potential energy and conserve the initial energy of $mgh$. But at the point $h'$ above ground, the object comes to rest. So there is no kinetic energy. So how is energy conserved? We need to have $mgh=mgh'+\text{some energy}$. But what is that some energy? Any help is very much appreciated.