The work is basically the amount of energy that is used to make something move. So first some math to gain insight how work works:
In the case of constant force work is defined as $$W=F s,$$ where $W$ is work, $F$ is the applied force and $s$ is the distance the object traveled in the direction of the force. The force is defined as $$F=m a,$$ where $m$ is the mass of the object and $a$ its acceleration. For constant force we have constant acceleration, which can be computed as $$a=\frac{v_2-v_1}{t},$$ where $v_2$ is the end velocity, $v_1$ is the starting velocity and $t$ is the time that passed during slowing down from $v_1$ to $v_2$. We also need the distance that the object traveled, which is: $$s=v_1 t +\frac{at^2}{2}=v_1 t +\frac{v_2-v_1}{2}t=\frac{v_2+v_1}{2}t,$$ where we plugged in our formula for acceleration. Now to put it all together we get: $$W=m\frac{v_2-v_1}{t}\frac{v_2+v_1}{2}t=m\frac{v_2^2-v_1^2}{2}=E_2-E_1,$$ where $E_2$ is end kinetic energy and $E_1$ is starting kinetic energy of the object.
So why is this not proportional to velocity difference but to velocity squared distance? That is simply because the force applied is proportional to the velocity difference through the acceleration being proportional to the velocity difference. That makes sense doesn't it? To slow down your car your force need to be bigger the bigger the velocity difference is, if your are to take it the same amount of time.
But this force you need to multiply by the distance traveled and that distance depends on your initial velocity. The bigger your initial velocity, the bigger the distance you travel to slow down by the same amount of speed with the same acceleration, which seems pretty intuitive to me. So once you multiply the force, that is proportional to the velocity difference, by something that is bigger the bigger your initial velocity is, your resulting work must be bigger the bigger the initial velocity is, if your are to have the same velocity difference. Just as your computation suggests.