Actually the force which you are applying is not constant.
Let the force be $f$ and displacement be $d$, then when you draw a graph according to Hooke law (I.e. force is directly proportional to deflection, where $k$ (stiffness) is constant, it will look like a square, in that at any point you can find the amount of force require to get desired displacement or deflection.
Now, consider half of the square area I.e. a triangle ( remember area=force time deflection), and in that total work done by the spring is converted into potential energy which is stored in the spring itself while half of the triangle shows the resistive force which is resisting the work done by you in compression and half of the same force is actually energy to regain its shape.
This is your answer where you have gone wrong. The work done on the spring is actually the work by you and is k*x² , and the work done by the spring is 1/2 kx² ( this is the actual energy you transfer to spring ) so this is the work energy produced in the spring. Other half is wasted in recovering the shape