Why is the direction of cross products of two vectors perpendicular to the plane? Why is the direction of cross products of two vectors perpendicular to the plane? How is that possible?
 A: A cross product of two vectors actually is not a vector but a second rank antisymmetric tensor. In 3D this has 3 components so it is usually called an axial "vector". An axial vector is invariant under space inversion while a true vector changes sign.
The antisymmetric tensor formed by two vectors is
$$\begin{pmatrix}
0 & x_1 y_1 & x_1 z_1 \\
-x_1 y_1 & 0 & y_1 z_1 \\
-x_1 z_1 & -y_1 z_1 & 0
\end{pmatrix}$$
The object $$\begin{pmatrix}
y_1 z_2 - y_2 z_1 & z_1  x_2 - z_2  x_1& x_1 y_2 - x_2 y_1
\end{pmatrix}$$
Transforms as a vector under rotation and translation, but not inversion. Its (pseudo)scalar products with $\left( x_1~~y_1~~z_1\right)$ and $\left( x_2~~y_2~~z_2 \right)$ are zero, so the axial vector form of the cross product of two vectors is perpendicular to both vectors.
A: Reading your question I can think of two applications at the moment:

*

*When an electron moves through a magnetic field, the electron is deflected sideways. Empirically (through observations and measurements) it has been found that the direction of the deflection is always perpendicular to the surface spanned by the direction of the electron's movement and the direction of the magnetic field. If the direction of motion is parallel to the magnetic field the deflection of the electron is zero and in the mathematical view the area between the two directions (or better vectors) is zero too. In all other cases the Lorentz force is perpendicular to the plane.


*When a model aircraft is connected to an axle by a rod, the repulsion turns the axis. If the rod is long enough, even a small aircraft generates a large torque. Empirically, the cross product between the vector of repulsion (its force and its direction) and the vector of the rod (the length and the direction between the axis and the plane) is proportional to the torque. Again, the resulting parameter can be considered as perpendicular to both input variables (and the sine between them).
So for physics, the vector product is a useful mathematical construct for calculations.
A: How is that possible?
It's only possible because we consider 3 dimensions.
It wouldn't be possible on a 2D-plane for example. If you have 2 non-collinear vectors in 2 dimensions: you couldn't find a third vector perpendicular to both of them. You're stuck inside the plane defined by those 2 vectors.

But if you consider 3 dimensions, it's perfectly possible to define a new vector which is perpendicular to the others. E.g. $\vec{k}$ in this diagram:

Why?
There are many useful examples for the cross-product. Here are 2 basic ones.
Basis of 3 vectors
The cross product makes it really easy to find the third vector in the above diagram, e.g. if you want to define a basis:

*

*If you have $\vec{i}$ and $\vec{j}$, their cross-product gives you $\vec{k}$

*If you have $\vec{j}$ and $\vec{k}$, their cross-product gives you $\vec{i}$

*If you have $\vec{k}$ and $\vec{i}$, their cross-product gives you $\vec{j}$
It might seem to be an overkill if the vectors are well-defined and parallel to the axes:
$\begin{bmatrix}1\\0\\0\end{bmatrix}$
$\begin{bmatrix}0\\1\\0\end{bmatrix}$
$\begin{bmatrix}?\\?\\?\end{bmatrix}$
But if your vectors have been rotated, it becomes much harder to calculate the 3rd vector:
$\begin{bmatrix}-0.10978507 \\ -0.49005325 \\  0.86475144\end{bmatrix}$
$\begin{bmatrix} 0.80521637 \\ 0.46621439 \\ 0.36642971 \end{bmatrix}$
$\begin{bmatrix}?\\?\\?\end{bmatrix}$
Calculating the cross-product of the first two vectors gives you the answer directly:
$\begin{bmatrix} -0.58272964 \\  0.73654053 \\  0.34341552 \end{bmatrix}$
Axis of rotation
If some object is rotating and you know the velocity vectors at two different points, you can get the axis of rotation directly by calculating their cross-product.

A: Cross products can be understood from the perspective of Geometric Algebra, which defines the product of two vectors as the sum of a scalar and a 'bivector', a new sort of object that represents planes and areas in the same way that a vector represents lines and lengths. The scalar part is just the scalar product. Because the bivector part produces an object that vector algebra can't cope with, a trick is applied to turn it into a vector. This is done by multiplying it by a constant, the unit trivector, which has the effect of constructing the perpendicular object. So in 3D a vector switches places with the bivector perpendicular to it.
This works in 3D, but not in any other number of dimensions. In 2D, you have one basis scalar, two basis vectors in x and y axes, and the basis bivector xy, representing the plane. There is no vector perpendicular to this plane, since we only have two dimensions to play with. In 4D, two vectors define a plane, the perpendicular to which is another plane! Any vector in this plane will do - the answer is not unique. So cross products don't work in 4D and higher dimensions.
But the Geometric Algebra product does. The product of two vectors a and b is |a| |b| ([unit scalar] Cos(θ) + [unit bivector in a-b-plane] Sin(θ)), where θ is the angle between them. In 2D it produces the Complex numbers. In 3D it produces the Quaternions. In 4D, the bivectors have six dimensions, representing the six basis planes xt, yt, zt, yz, xz, xy. You can't identify a unique 4-vector to represent it. So a different trick is used to let vector algebra handle it, which is to multiply the result by the unit t vector. In xt, yt, zt, the t is cancelled out and you get a vector x, y, z. With yz, xz, xy you get the trivector basis yzt, xzt, xyt (a 3-space hyperplane in 4D) which can now be turned into a vector using the original 'perpendicular' trick to become x, y, z. So a 4D bivector field (like the electromagnetic field) is turned into two 3D vector fields: the electric field which comes from xt, yt, zt, and the magnetic field which comes from yz, xz, xy. It's why the magnetic field is an 'axial' vector (also known as a 'pseudovector') where the electric field is a 'polar' vector. The split depends on the chosen t axis, which depends on your frame of reference. Change your basis, and electric fields turn partly into magnetic fields and vice versa.
Vector algebra is 'broken'. The products are non-invertible, one of them is not generalisable beyond 3D, the other mixes types. You have to fudge things by using tricks like taking the perpendicular to fit everything you need into its restricted framework. Geometric Algebra combines them into a single product which is (usually) invertible and generalisable to any number of dimensions. The bivector part of the product is always in the plane of the vectors being multiplied.
A: In their modern form, vectors appeared late in the 19th century when Josiah Willard Gibbs and Oliver Heaviside (of the United States and Britain, respectively) independently developed vector analysis to express the new laws of electromagnetism discovered by the Scottish physicist James Clerk Maxwell.
Quaternions were introduced by Hamilton in 1843. Important precursors to this work included Euler's four-square identity (1748) and Olinde Rodrigues' parameterization of general rotations by four parameters (1840), but neither of these writers treated the four-parameter rotations as an algebra. Carl Friedrich Gauss had also discovered quaternions in 1819, but this work was not published until 1900.
The fact is a vector is more useful and easy to understand while quaternions are tougher to understand. The Maxwell's equations itself were first more in quaternions than in vector form.
So Cross product and dot products are mears a part of the quaternion product.See for more, here.
The simplest answer is probably "because physicists historically have found that definition useful". To reduce quaternion into vector concept which is easy to understand in daily life.
A: There are several areas to which cross-product can be linked, including wedge products, axial vectors etc, but it is simple enough to be treated on its own.
Below I will show that cross-product naturally arises if one seeks a bi-linear transformation of two 3d vectors, that gives rise to a third, perpendicular 3d vector.
So let us define a quantity $\epsilon_{\alpha\beta\gamma}$, such that given two vectors $\mathbf{a}=a_x\mathbf{\hat{x}}+a_y\mathbf{\hat{y}}+a_z\mathbf{\hat{z}}$ and $\mathbf{b}=b_x\mathbf{\hat{x}}+b_y\mathbf{\hat{y}}+b_z\mathbf{\hat{z}}$, we can create a third vector $\mathbf{c}$:
$$
\begin{align}
c_x=&\epsilon_{xxx}a_xb_x+\epsilon_{xxy}a_xb_y+\epsilon_{xyx}a_yb_x+\dots+ \epsilon_{xzz}a_zb_z\\
c_y=&\epsilon_{yxx}a_xb_x+\epsilon_{yxy}a_xb_y+\epsilon_{yyx}a_yb_x+\dots +\epsilon_{yzz}a_zb_z\\
c_z=&\epsilon_{zxx}a_xb_x+\epsilon_{zxy}a_xb_y+\epsilon_{zyx}a_yb_x+\dots+\epsilon_{zzz}a_zb_z\\
\end{align}
$$
Which we can write as:
$$
c_\alpha=\sum_{\beta\gamma=\{x,y,z\}}\epsilon_{\alpha\beta\gamma} a_\beta b_\gamma
$$
This is the most general way we can do bilinear transformation of $\mathbf{a},\:\mathbf{b}$ into $\mathbf{c}$.
If $\mathbf{c}\perp\mathbf{a},\mathbf{b}$ then:
$$
\mathbf{a}.\mathbf{c}=\sum_{\alpha\beta\gamma}\epsilon_{\alpha\beta\gamma}a_\alpha a_\beta b_\gamma=0
$$
Next we can swap the order of $a$-coefficients (since order does not matter in multiplication), and then re-label dummy indices:
$$
\mathbf{a}.\mathbf{c}=\sum_{\alpha\beta\gamma}\epsilon_{\alpha\beta\gamma}a_\alpha a_\beta b_\gamma=\sum_{\alpha\beta\gamma}\epsilon_{\alpha\beta\gamma}a_\beta a_\alpha b_\gamma=\sum_{\beta\alpha\gamma}\epsilon_{\beta\alpha\gamma}a_\alpha a_\beta b_\gamma=0
$$
Therefore:
$$
\sum_{\alpha\beta\gamma}\epsilon_{\beta\alpha\gamma}a_\alpha a_\beta b_\gamma+\sum_{\beta\alpha\gamma}\epsilon_{\beta\alpha\gamma}a_\alpha a_\beta b_\gamma=0=\sum_{\alpha\beta\gamma}\left(\epsilon_{\alpha\beta\gamma}+\epsilon_{\beta\alpha\gamma}\right)a_\alpha a_\beta b_\gamma
$$
This has to work, with the same $\epsilon$, for any vectors. By considering, for example, $\mathbf{b}=\left(1,0,0\right),\:\left(0,1,0\right),\:\left(0,0,1\right)$ one can see that $\epsilon_{\beta\alpha\gamma}=-\epsilon_{\alpha\beta\gamma}$ is necessary. You can follow the same line of reasoning to get $\epsilon_{\alpha\beta\gamma}=-\epsilon_{\gamma\beta\alpha}$.
Finally you may note that if the two vectors you are starting with: $\mathbf{a}$ and $\mathbf{b}$, are equal or co-linear ($\mathbf{b}=constant\cdot\mathbf{a}$) then there can be no single vector perpendicular to them, in essence there are infinitely many vectors perpendicular to both $\mathbf{a}$ and $\mathbf{b}=constant\cdot\mathbf{a}$. Mathematically it is convenient then to choose that the 'perpendicular' vector in such case is zero
EDIT: Lets make it a proper assumption. Assuming the product of co-linear vectors is zero.
EDIT2: Pretty sure this assumption can be handled by demanding that the result of the transformation ($\mathbf{c}$) would lie in a single irreducible representation of the rotation and parity groups (hello axial vectors), but this is way outside the scope of the question
$$
\sum_{\beta\gamma}\epsilon_{\alpha\beta\gamma}a_\beta a_\gamma=0\:\mbox{for all }\alpha
$$
Which gives $\epsilon_{\alpha\beta\gamma}=-\epsilon_{\alpha\gamma\beta}$.
So in the end you have:
$\epsilon_{\alpha\beta\gamma}=-\epsilon_{\beta\alpha\gamma}=-\epsilon_{\gamma\beta\alpha}=-\epsilon_{\alpha\gamma\beta}$
Which fully defines all 27 components of $\epsilon$ up to a single constant. We then normally choose $\epsilon_{xyz}=1$ and call this quantity Levi-Civita, and the bi-linear operation
$$
c_\alpha=\sum_{\beta\gamma=\{x,y,z\}}\epsilon_{\alpha\beta\gamma} a_\beta b_\gamma
$$
becomes the conventional cross-product.
In summary: In 3d space cross-product is the only possible bi-linear way of creating a vector perpendicular to two other non-co-linear vector up to a choice of a single constant, assuming the product of co-linear vectors is zero

EDIT In response to comment. Why does cross-product produce a vector perpendicular to two input vectors. Taking the short way of writing down the cross-product of vectors $\mathbf{a},\,\mathbf{b}$, and taking the dot-product with, e.g. second input vector
$$
\begin{align}
\mathbf{b}.\left(\mathbf{a}\times{\mathbf{b}}\right)=&\sum_{\alpha\beta\gamma}b_\alpha\,\epsilon_{\alpha\beta\gamma}a_\beta b_\gamma=\frac{1}{2}\sum_{\alpha\beta\gamma}a_\beta\cdot b_\alpha b_\gamma \left(\epsilon_{\alpha\beta \gamma}+\epsilon_{\alpha\beta \gamma}\right) \\
=&\frac{1}{2}\sum_{\alpha\beta\gamma}a_\beta\cdot b_\alpha b_\gamma \left(\epsilon_{\alpha\beta \gamma}-\epsilon_{\gamma\beta\alpha}\right)
\end{align}
$$
Now simply re-label the dummy indices $\alpha\leftrightarrow\gamma$ in the last term to get zero:
$$
\begin{align}
\mathbf{b}.\left(\mathbf{a}\times{\mathbf{b}}\right)=&\frac{1}{2}\sum_{\alpha\beta\gamma}a_\beta\cdot b_\alpha b_\gamma\epsilon_{\alpha\beta \gamma}-\frac{1}{2}\sum_{\alpha\beta\gamma}a_\beta\cdot b_\gamma b_\alpha\epsilon_{\gamma\beta \alpha} \\
=&\frac{1}{2}\sum_{\alpha\beta\gamma}a_\beta \cdot b_\alpha b_\gamma\epsilon_{\alpha\beta \gamma}-\frac{1}{2}\sum_{\alpha\beta\gamma} a_\beta \cdot    b_\alpha b_\gamma \epsilon_{\alpha\beta \gamma}=0
\end{align}
$$
Basically the anti-symmetry, with respect to index exchange, of $\epsilon_{\alpha\beta\gamma}$, is the immediate reason for output vector being orthogonal to the input vectors. Having said that, I would say that cross-product gives orthogonal output vector by construction - I have constructed it be that way.
Final remark. To connect the $\epsilon$-based representation with more conventional expression for cross-product note that due to anti-symmetry the only terms of Levi-Civita that are not zero are $\epsilon_{xyz}$ and various permutations thereof. All three indices have to be different. So if I want:
$$
c_x=\sum_{\beta\gamma}\epsilon_{x\beta\gamma}a_\beta b_\gamma
$$
I know that the only two non-zero options are $\epsilon_{xyz}=1$ and $\epsilon_{xzy}=-\epsilon_{xyz}=-1$. Hence:
$$
c_x=a_y b_z - a_z b_y
$$
etc
