Update as a result of a comment.
There are two ways of answering the question.
The first is to say that the magnitude and direction of the force on a unit test charge is the electric field.
If this was the definition then the direction of the force would differ as to whether the test charge was negative or positive,
$\vec E= \dfrac {\vec F}{q\,(\text{no sign assigned)}}$
A positive test charge would have a force on it in one direction and a negative test charge would have a force on it in the opposite direction.
However that definition is not the one that is used.
When using a unit test charge there must be included in the definition of the electric field the sign of the unit test charge.
$\vec E= \dfrac {\vec F}{\pm q\,(\text{sign assigned)}}$
As you have pointed out when this definition is used it matters not what the sign of the test charge is, the direction of the electric field is always the same.
If the direction of the force on a positive test charge is positive then the electric field is in the positive direction.
The force on a negative charge at the same position would be in the negative direction but the direction of the electric field would be positive as there would be a negative sign in both the denominator and numerator of the equation which defines electric field.