No that's not right. a work can be calculated for each force individually. you are mixing it with the equation between work and energy that says:
W = Delta(K) (K is kinetic energy)$$W = \Delta K \quad \text{K is kinetic energy}$$
but this work is actually the work of the overall force on the object. the work done by earth gravity force and your force is zero. But each is doing some work individually. you are doing some positive amount of work and the earth is doing it in the same amount but in negative sign.