Suppose I am sliding a block very slowly on a rough surface. If the block has travelled $d$ distance then work done by me is $W_1=\mu mg d$ and that by friction is $W_2=-\mu mg d$.
Now the energy transferred from me to block is $\mu mgd$ and that taken by friction from block is $\mu mgd $, The net energy of block remains same but the energy taken by friction evolves as heat and that is equal to my chemical energy consumed, so total energy of $block + me$ system remains constant.
Now If I pull a block of mass $m$ slowly towards up to a height $h$, then work done by me is $W_1=mgh$ (assuming $h$ is much less than radius of earth) and that by gravity is $W_2=-mgh$. Therefore $mgh$ goes from me to block and $mgh$ from block to earth, So here also energy of block doesn't change, then why do we say that potential energy of block increases.
I know I am lacking something here, as the total energy of the system would not be conserved if the block's energy doesn't change and my energy decreases.
Please help me in understanding where I am wrong.