The dot and scalar products of vectors are defined as follows.
\begin{align}
\mathbf a \cdot \mathbf b &= |\mathbf a||\mathbf b|\cos\theta \quad \fbox{dot} \\
\mathbf a \times \mathbf b &= \mathbf{\hat n}|\mathbf a||\mathbf b|\sin\theta \quad \fbox{cross}
\end{align}
Where the angle between the vectors $\mathbf a$ and $\mathbf b$ is $\theta$ and $\mathbf n$ is a normal vector perpendicular to the two vectors according to the handedness of the coordinate system.
Since the dot product is a product of three scalars (the length of the two vectors and the cosine of the angle between them) it is itself a scalar (hence its alternative name "scalar" product).
On the other hand, the cross product is the product of three scalars (the length of the two vectors and the sine of the angle between them) and a vector (a normal perpendicular vector). The three scalars multiply to give a scalar, but this scalar times the vector gives a vector, so the result of the cross product is a vector (hence its alternative name "vector" product).
Your misunderstanding in the first half of your question was down to your definition of the quantities. Your definition for the scalar product did not include the normal vector so you concluded that it was multiplying by the $\sin\theta$ term that gave you the vector result. This is of course incorrect, it is the inclusion of a vector in the equation that makes the result a vector in this case.
Hopefully you can now see the answer to your last question: the quantity $|\mathbf a| |\mathbf b| \cos\theta$ is a scalar, but if you were to add some vector term, let's call it $\mathbf c$, into the expression to give $\mathbf c |\mathbf a| |\mathbf b| \cos\theta$, the result would now be a vector.