Well, say that you are looking at a man lifting boxes. Each box weighs 10kg. At first you look at him standing, but since just looking at him made you tired, you decide to lie down. Now, from your horizontal position, the scene looks different, but is the man doing more, less, or the same amount of work per box?
The word “scalar” is often used to just mean “number”, but it actually has a technical meaning: a scalar is a quantity that does not change when the system is transformed following one of the symmetries of the theory.
In this case, I think you are talking about classical mechanics, and the transformations involved are rotations in 3D space. As you know, the dot product of two vectors is invariant under rotation, that is why it's called “the scalar product”. The cross product, also called the vector product, transforms as a vector.
Now there is a final bit of the puzzle. The magnitude of a vector is also a scalar. Rotating the vector does not change its length.
So in fact we have two ways to obtain a scalar from two vectors $\vec F$ and $\vec s$: taking the scalar product $\vec F \cdot \vec s$ or the magnitude of the vector product $|\vec F \times \vec s|$. How do we chose one?
Well, we know that when the force and the displacement are aligned, the work is maximum, and when they are perpendicular the work is minimum. This allows us to chose one alternative.
