Regardless of the positive or negative, doesn't the number determine the total displacement and not the sign in front of the numbers?
Before solving kinematics equations, a standard is usually set for what directions are positive and negative. For example, north and east are positive, therefore, south and west are negative. In this case, if an object moves $3\ m$ west, its displacement is $-3\ m$ horizontally.
Also note that displacement is a vector quantity, meaning it consists of a magnitude and direction (determined by the sign or an angle). Distance on the other hand is a scalar and is the magnitude of the resultant displacement vectors, which is always positive. So in the same example, the object would have traveled $3\ m$, direction is not specified.
Wikipedia - A displacement is a vector whose length is the shortest distance from the initial to the final position of a point. It quantifies both the distance and direction of an imaginary motion along a straight line from the initial position to the final position of the point.
For simplicity assume that $\hat d$ is the unit vector in the downward direction and that a displacement can only be up or down.
A downward displacement $\vec d$ is a vector quantity and therefore has both a magnitude $|\vec d| = d$ and a direction $\hat d$ so it can be written as $\vec d = d\,\hat d$.
What is the meaning of a displacement $-\vec d$?
$\vec d+(-\vec d) = \vec 0$ and so one can describe the displacement $-\vec d$ in one of two ways:
$(-d)\,\hat d$ where (-d) is the component of the vector $\vec d$ in the downward direction $\hat d$.
$d\,(-\hat d)$ where $d$ is the component of the vector $\vec d$ in the direction which is opposite to downwards ie upwards with $(-\hat d) = \hat u$.
Suppose a change of position of $3\,\rm m$ in the upward direction.
The magnitude of the displacement is $3\,\rm m$, always a positive quantity.
The component of the displacement is $-3\,\rm m$ in the downwards direction and $+3\,\rm m$ in the upwards direction.