You are right, the work applied is not necessarily equal to $mgh$. To see this, let's look at energy conservation:
$$K_0+U_0+W_{\text{n.c.}}=K_f+U_f$$
where $K_0$ and $K_f$ are the initial and final kinetic energy, respectively, $U_0$ and $U_f$ are the initial and final potential energy, respectively, and $W_{\text{n.c.}}$ is the work done by any non-conservative forces.
Assuming the only potential energy in the system is gravitational potential energy $U=mgh$ where $h=0$ is at the starting position, and saying $W_{\text{n.c.}}=W_\text{you}$, then if $W_{\text{n.c.}}=mgh$ we get
$$K_0+0+mgh=K_f+mgh\to K_0=K_f$$$$K_0+0+mgh=K_f+mgh$$ $$K_0=K_f$$
In other words, the object needs to be moving at the same speed at the start and at the end of the path (although it can change in the middle). Otherwise, $W_{\text{n.c.}}\neq mgh$
As an aside, note that if we did have $W_{\text{n.c.}}= mgh$, that wouldn't necessarily mean the applied force is equal to $mg$, although certainly in the case where the forces are equal we would have $W_{\text{n.c.}}=mgh$