To see why the exhaust speed is important, let's do a calculation.
Let's start with a rocket of mass $m$ going at speed $u$. (We measure all speeds with respect to some inertial reference frame.)
Now, suppose it exhausts a tiny amount of propellant of mass $\delta m$ and the propellant is traveling at speed $u_P$. After it exhausts that fuel, the rocket now has mass $m-\delta m$ and is now going at speed $u+\delta u$.
Conservation of momentum says:
$$ (m - \delta m)(u +\delta u) + \delta m\ u_P = m u$$
Simplifying the above and keeping only first order terms, we obtain:
$$ m \delta u = \delta m (u-u_P) $$
In other words, for a given amount of mass, the increase in speed of the rocket, $\delta u$, depends only on the difference between the speed of the rocket, $u$, and the speed of the propellant, $u_P$. The absolute value of either speed is irrelevant.
The difference between the rocket speed and the propellant speed is called the exhaust speed.
For the best chemical rockets, the exhaust speed is around 3,000 meters per second. When electric propulsion is used, exhaust speeds can be up to 20,000 meters per second or more.