Is it equally difficult to lift something as it is to lower it? This just seems a bit counterintuitive cause lifting stuff seems harder, but if I do them at the same, constant speed, I will apply the same force opposing gravity, just over opposite displacements, resulting in equal (but opposite) work. Is this correct?
 A: First of all note that human muscles require energy even when doing no work, so be a bit cautious about analysing any situation involving humans manipulating objects. See Why does holding something up cost energy while no work is being done? for more details.
If we replace the human by some form of mechanical cantilever then you are quite correct that the force is the same whether you are going up or down, so the magnitude of the work done is the same. However when lifting the arm is doing work on the object i.e. the energy of the arm decreases and the energy of the object increases. When going down the object is doing work on the arm i.e. the energy of the arm increases and the energy of the object decreases.
So the same amount of work is done in both cases, but the energy flows in a different direction.
If you want to see how the maths describes this then we start with the equation for the work:
$$ W = \int_{x=a}^{x=b} \mathbf F\cdot\mathrm d \mathbf x $$
Where $a$ is the value of the position $\mathbf x$ when the motion starts and $b$ is the position when it ends. In the 1D case where $\mathbf x$ is just distance up and down and the force $\mathbf F$ is the constant $mg$ the integral becomes:
$$ W = mg(b - a) $$
When we are going up $b > a$ so $W_\text{up}$ is positive, and when we are going down $b \lt a$ so $W_\text{down}$ is negative:
$$ W_\text{up} = - W_\text{down} $$
