The magnitude of the work lifting the mass 1 m is exactly the same as that in lowering it 1 m. In both cases the force you exert to either lift or lower the mass at constant velocity is simply the wieght of the object, a constant. The distance is also the same in both cases, therefore the magnitude of Fd is identical.
What's different is the sign of the work. On the way up, F and d are in the same direction so the work is positive. The result of this is a gain in the mass's gravitational energy. When the mass is lowered, the force you are exerting is upward but the direction the mass is moving is downward. The product of these is now negative, the result is a decrease in the gravitational energy of the mass.
You are not exerting the only force on the mass, the pull of Earth's gravity is also acting. In this case, as you raise the mass the work done by the Earth on the mass is negative. It is equal and opposite to that of you lifting the mass making the net work done on the mass zero. This implies, according to the Work-Energy Theorem, there is no change in kinetic energy. Likewise on the way down.
As an aside; it doesn't matter that the mass is lifted at constant velocity as long as it is at rest (or the same velocity) at both ends of the problem. Any amount of greater work done in accelerating the mass at the start will be compensated by the lesser work in decelerating at the end.
In the case of lifting, you have done positive work on the mass-Earth system thereby increasing its gravational energy. In the case of lowering the mass, you perform negative work on the system lowering its gravational energy. In both cases the work done by the Earth on the mass is within the system and doesn't affect the gain or loss of gravitational energy of the system. In fact, within the system, any work done by the Earth on the mass is canceled by work done by the mass on the Earth.