> If it takes $2 \text{ N}$ of force to displace a spring by $2 \text{ m}$ ...

That is a mistake right there. The force exerted by the spring is not constant, but it linearly depends on the elongation, as in

$$\boxed{F_e = -k x} \tag 1$$

where $k$ is the spring constant and $x$ is the elongation. The minus sign defines that the spring (force) always acts in the direction opposite to the elongation.

Since force is not constant over some distance, we cannot simply multiply the final force value with the total displacement. We have to integrate the force over the distance (geometrically, this means finding an area under the force curve):

$$W_e = \int_{x_1}^{x_2} -k x \cdot dx = \left. -\frac{1}{2} k x^2 \right|_{x_1}^{x_2} = -\frac{1}{2} k x_2^2 + \frac{1}{2} k x_1^2 = -\Delta U_e \tag 2$$

where $U_e = \frac{1}{2} k x^2$ is the *elastic potential energy*, and $\Delta$ denotes a difference (final minus initial value). If you are not familiar with calculus and integrals ($\int$ and $dx$ in the above equation), what it means is that you sum the spring force ($-kx$) for every elongation $\{x,x+dx,x+2dx,...\}$ between $x_1$ and $x_2$, where $dx$ is *infinitesimally small increment*.

> The change in potential energy is the work done on a spring.

This is correct when it comes to "work done on a spring". But the work that spring does equals negative of the change in elastic potential energy! This *negative sign* is really important, do not ever forget it! The same applies to the *gravitational potential energy*. Check Eq. (2) if you do not understand why.

In this context we can define an *average force* that the spring exerts over some distance

$$W_e = \bar{F_e} \cdot (x_2 - x_1) \quad \text{or} \quad \boxed{\bar{F}_e = -\frac{1}{2} k (x_1 + x_2)}$$

The force $\bar{F}_e$ is now constant, but notice the difference compared to the definition in Eq. (1).