Co-efficient of restitution Co-efficient of restitution is defined as 

$\displaystyle\frac {Relative \ velocity \ after\  collision}{Relative \ velocity\  before\  collision}$

so lets consider two bodies A and B moving with constant velocities $\vec u_{a} \  and \ \vec u_{b}$ in the same direction where $|\vec u_{a}| < |\vec u_{b}|$. After some time B collides with A and they get final velocities $\vec v_{a} \ and \ \vec v_{b}$ in the same direction as before.
So according to the definition it should be $\displaystyle \frac{\vec v_{a} - \vec v_{b}}{\vec u_{a} - \vec u_{b}}$ but it actually is the previous value with a negative sign. But why is there a negative sign ?
 A: Your definition is incorrect. The coefficient of restitution, $e$, is not defined as you stated. Let $u_1$, $u_2$, $v_1$, and $v_2$ be the initial and final velocities of objects $a$ and $b$ respectively.
The way to correctly remember the coefficient of restitution is defined as the velocity of separation divided by the velocity of approach. Alternatively, you can remember it is the negative of the relative velocities. 
$$e=\frac {v_2 - v_1}{u_1 -u_2}=-\frac {v_2 - v_1}{u_2 -u_1}$$
It is very important to note you cannot write these as vectors ($\vec u_1$ and so on). As bemjanim pointed out in the comments, you can't divide vectors. Instead, the velocities we use here are the components of the actual velocities along the line of force, or the head-on velocity components of the collision. Of course, if you're considering a collision between point objects in $ℝ^1$, all collisions are head on.
It's useful to note what velocity of separation and velocity of approach actually mean, since these terms can be a little confusing. Check this question for further clarity.
As a side note, one should be careful while using Wikipedia as a source of scientific formulae. Moreover, if you scroll further down in the same page, you will see that the derivation of this formula is done by equating kinetic energies for an elastic collision. Try proving this to yourself by doing the derivation. 
A: There seems to be some confusion here on how to get relative velocities from vector quantities.
You need to project the velocity vectors $\vec{v}_a$, $\vec{v}_b$ along the contact normal direction $\vec{n}$ to arrive at the law of contacts.
$$ \left( \vec{n} \cdot \vec{v}_a - \vec{n} \cdot \vec{v}_b \right) = -\epsilon \left(\vec{n} \cdot \vec{u}_a - \vec{n} \cdot \vec{u}_b \right) \tag{1}$$
Here $\vec{n}$ must be a unit vector.
where $\epsilon$ is the coefficient of restitution. Notice the negative sign that is there to signify the bounce that happens, and that the sequence a minus b occurs the same on both sides of the equation.
Equation (1) is often stated also as
$$ \boxed{ \epsilon = - \frac{ \vec{n} \cdot \left( \vec{v}_a - \vec{v}_b \right) }{ \vec{n} \cdot \left( \vec{u}_a - \vec{u}_b \right) } = - \frac{v_{\rm rel}}{u_{\rm rel}} } \tag{2} $$
So if you are consistent with your conventions (coordinate system) and keep the minus sign there the physics works out correctly, regardless of which velocity has the highest magnitude.
A: Reading form the Wikipedia page Coefficient of restitution, the quantity you are looking for is the ratio of the moduli of the initial relative velocity and the final relative velocity, so using your notation
$$
e = \frac{|\vec{v}_a-\vec{v}_b|}{|\vec{u}_a-\vec{u}_b|}
$$
Therefore the coefficient of restitution is always a positive dimensionless quantity, in particular 


*

*$e=0$ for a perfectly inelastic collision,

*$e=1$ for a perfectly elastic collision,

*$0<e<1$ for real-world inelastic collision,

*$e>1$ if some (chemical) energy is released in the collision, e.g.in an explosion. 

A: I guess the formula written by you must be corrected,
$$\boxed{Coefficient of restitution = -\displaystyle\frac {Relative \ velocity \ after\  collision}{Relative \ velocity\  before\  collision}}$$

Even in the formula given in wikipedia this formula is stated with a negative sign.
