Why doesn't the potential energy of any object equal 0

Consider a particle on the ground. This particle is raised by a force of magnitude $mg$ to a height $h$ above the ground. At this point, the work done on the particle by the force is $mgh$, which is equal to the potential energy of the particle. But, during this period, the force of gravity also acts on the particle and is displaced by $h$, and so does a work of $-mgh$ on the particle. Shouldn't the two cancel and no net work should be done on the particle?

If they don't cancel, then where did the energy that came from the work done by the force of gravity go?

-
Note that work is a scalar quantity. Will vector addition laws apply? :) – mikhailcazi Feb 14 '14 at 11:51
Note that the definition of work $W=\int_{\gamma} {\bf F}\cdot\mathrm{d}{\bf r}$ depends on which force ${\bf F}$ it refers to, e.g. gravitational force, friction force, net force, etc. Gravitational potential energy by definition only refers to the work of the gravitational force, and not any other force. – Qmechanic Feb 14 '14 at 11:57
@Qmechanic: Could you elaborate? – Gerard Feb 14 '14 at 12:22
@mikhailcazi: I don't believe I used vector addition. – Gerard Feb 14 '14 at 12:23
The change in energy is calculated by doing that; it's not really net work. If you remember the work-energy-theorem: $W_{conservative} + W_{non-conservative} + W_{other} = \Delta K.E.$ – mikhailcazi Feb 14 '14 at 12:57

Think about the work-kinetic energy theorem, which states that the net work done on an object is equal to its change in kinetic energy: $$W_{net}=\Delta\mathrm{KE}.$$
You are right that when lifting an object of mass $m$ by a height $h,$ in a uniform gravitational field, the work you do is $W_{you}=mgh$ (assuming, as you said, that you're applying a force of $mg$), and for that same displacement, the work done by gravity is $W_{grav}=-mgh.$ The fact that these two cancel out ($W_{net}=W_{you}+W_{grav}=0$) means that the kinetic energy of the object after being lifted is 0. So the work done by gravity went to sucking energy out of the object that you were adding, thereby converting it to gravitational potential energy. (If it did not get sucked out, then the object would gain kinetic energy--just imagine a case where you apply the same force to the object as you did when you lifted it, but this time there is no gravitational field. Then, since there will be a net force on the object in that case (or, net work will be done (by you)), the object's KE will increase.) Meanwhile, the change in gravitational potential energy of the object is $\Delta U = -W_{grav}$.