Why do we use cross products in physics? We can define cross products mathematically like if we take two vectors, we can find another vector with certain properties but why do we use it in physics, if we consider a hypothetical physical quantity like force which is equal to cross product of certain vectors?

For example, the force exerted on a charge in motion in an uniform magnetic field.

Why is it so? Why does that force have to be a cross product of two vectors?
Is it possible to come up with them when what we do is just observe the nature?
 A: This is a great question. The dot and cross products seem very mysterious when they are first introduced to a new student. For example, why does the scalar (dot) product have a cosine in it and the vector (cross) product have a sine, rather than vice versa? And why do these same two very non-obvious ways of "multiplying" vectors together arise in so many different contexts?
The fundamental answer (which unfortunately may not be very accessible if you're a new student) is that there are only two algebraically independent tensors that are invariant under arbitrary rotations in $n$ dimensions (we say that they are "$\mathrm{SO}(n)$ invariant"). These are the Kronecker delta $\delta_{ij}$ and the Levi-Civita symbol $\epsilon_{ijk \cdots}$. Contracting two vectors with these symbols yields the dot and cross products, respectively (the latter only works in three dimensions). Since the laws of physics appear to be isotropic (i.e. rotationally invariant), it makes sense that any physically useful method for combining physical quantities like vectors together should be isotropic as well. The dot and cross products turn out to be the only two possible multilinear options.
(Why multilinear maps are so useful in physics is an even deeper and more fundamental question, but which answers to that question are satisfying is probably inherently a matter of opinion.)
A: It's really much simpler than the other answers so far have made it out to be.  We use the cross and dot products (and all the other math) because they allow us to create fairly simple mathematical models (that is, the laws of physics) that accurately represent what the universe actually does.
A: Cross products are often used with pseudovectors (aka axial vectors). Less with vectors (aka polar vectors). Understanding the difference between axial and polar vectors helps here. 
Both axial and polar vectors are what mathematicians would consider a vector. Both are a set of 3 coordinates. They are often drawn as arrows. They can be added together and multiplied by numbers like arrows. 
Physicists require something more to consider a quantity to be a vector. They must represent a physical quantity that transforms in the right way when you change the basis. 
Polar vectors represent quantities like distance, velocity, acceleration, and force. These can describe motion of a point particle with a magnitude and direction.
Axial vectors represent a different set of quantities, like angular velocity and angular momentum. These describe things like rotary motion in a plane. They are a magnitude and orientation of the plane. This is equivalent to motion around an axis. They are often represented by an arrow, where the arrow is parallel to the axis and perpendicular to the plane. Plane orientation include the idea of clockwise vs counter clockwise. This is represented by putting the arrow on one side or the other of the plane as dictated by the right hand rule. 

Axial vectors often arise as the product of two perpendicular polar vectors. $\vec\omega = (\vec r \times \vec v)/r^2$. 
For a rigid object fixed to an axis, each point can only move with $v$ perpendicular to $r$. But a free particle can move an any direction. For this case, the cross product picks out the component of $v$ that is perpendicular to $r$, the component that contributes to rotation around the axis. The result is an vector perpendicular to $v$ and $r$ in accordance to the right hand rule. 

Magnetic field is an axial vector. See Why is the B-Field an axial Vector? for more. This means a current generates a $B$ field around it, described by magnetic field lines. For a straight line current, the field lines are planar and circular. For more complex currents, they are always closed curves. At any point, the field line is the "axis" that is perpendicular to the plane of the magnetic field. 
Magnetic force is generated when a charge moves in the plane of $B$. That is, when a charge moves perpendicular to the "axis" of B. This is captured by $\vec F = q\vec v \times \vec B$.
A: Cross products are inherently useful when describing rotations.
First, let's look at two different ways of describing rotations in $\mathbb{R}^{3}$.
The first way to do this is to give the axis of rotation, which is given by a line, $L$, in $\mathbb{R}^{3}$, and a magnitude (representing the angle), which is given by a number, $\theta$, in $\mathbb{R}$. If I pick a vector of length $\theta$, along my line $L$, I get a vector, say $x \in \mathbb{R}^{3}$.
Another good way to do this, is to give the plane in which I'm rotating, which I can represent by two perpendicular lines in $\mathbb{R}^{3}$ and a magnitude (representing the angle), which is again a number in $\mathbb{R}$. I encode these things by picking two vectors $v,w \in \mathbb{R}^{3}$, and say that the magnitude is encoded by the product of the lengths $\|v \| \|w\|$.
This means that lots of different pairs of $v,w \in \mathbb{R}^{3}$ give the same rotation, but that's ok.
(I can even allow more different pairs, by not assuming that $v$ and $w$ are perpendicular, but then I have to replace their product by the area of the parallelograms spanned by them.)
Now, the cross product gives us a way to translate between these different ways of encoding rotations.
To be precise, if $x \in \mathbb{R}^{3}$ and the pair $v,w \in \mathbb{R}^{3}$ describe the same rotation, then $x = v \times w$.
(The fact that lots of different pairs $v,w \in \mathbb{R}^{3}$ describe the same rotation means that $x$ can be written as the cross product in lots of different ways, i.e., there are lots of $v',w' \in \mathbb{R}^{3}$ such that $v' \times w' = v \times w = x$.)
Now, why this turns up in physics doesn't have such a clear cut answer, except that both these two different ways of representing rotations have their uses.
For instance, in your example talking about a charge moving in an electric field, I would say that this is just a fact of nature that was established experimentally.

An interesting aside is that rotations can be composed, i.e., given two rotations I can first do one and then the other to get a third rotation.
It might be interesting to try to figure out how this works in either of the pictures I have given above.
A: The cross product is the representation of the so(3) Lie Algebra. This means infinitesimal rotation are represented by the cross product.
A: A cross product is highly related to another concept, the exterior product (or wedge product).  An exterior product is a very natural product which occurs in algebra.  The exterior product of two vectors is a bivector, whose directions are very natural (while torque as a vector is at right angles to the force and the lever arm, in exterior product it's simply a bivector defined by two directions -- the force and the leve arm).
Unfortunately, exterior products are difficult to teach early on.  They take a lot of math.  Cross products are much easier to explain.  And, as it turns out, in 3 dimensions, cross products and exterior products are isometric.  They transform in the same ways.  If you do the math with cross products, you get the same answer as if you did them with exterior products.  This doesn't work in all dimensions (cross products are a 3 dimensional thing, while exterior products can be done in any number of dimensions), but it does work in 3, and lots of physics is done in three dimensions!
A: I'm not sure how advanced you are mathematically, so it's hard to know how much to add, verbally.  Besides, I'm posting from a tablet, so typing is cumbersome.
There is no single answer, but the cross product involves some kind of rotation about an axis.  Whether that is a physical rotation, or a mathematical displacements depends on the circumstance.
One place where the cross product is fairly easy to understand is in the relationship between angular momentum, rotational kinnetic energy, and torque.
Let me know if you can follow the math, based on the diagram.  I'm talking about the dervations in the boxes.  The stuff below that is incomplete.

A: I am focusing on the geometry of cross products
Cross products are used when we are interested in the moment arm of a quantity. That is the minimum distance of a point to a line in space.


*

*The Distance to a Ray from Origin. A ray along the unit vector $\boldsymbol{e}$ passes through a point $\boldsymbol{r}$ in space.
$$ d = \| \boldsymbol{r} \times \boldsymbol{e}  || \tag{1}$$ 
$d$ is the perpendicular distance to the ray (also known as the moment arm of the line).

*The moment arm of Force (Torque Vector). A force $\boldsymbol{F}$ along $\boldsymbol{e}$ causes the following torque about the origin
$$ \boldsymbol{\tau} = \boldsymbol{r} \times \boldsymbol{F}\;\; \rightarrow \| \boldsymbol{\tau} \| = d\, \| \boldsymbol{F} \| \tag{2}$$

*The moment arm of Rotation (Velocity Vector). A rotation $\boldsymbol{\omega}$ about the axis $\boldsymbol{e}$ causes the a body to move at the origin location by 
$$ \boldsymbol{v} = \boldsymbol{r} \times \boldsymbol{\omega}\;\; \rightarrow \| \boldsymbol{v} \| = d\, \| \boldsymbol{\omega} || \tag{3}$$

*The moment arm of Momentum (Angular Momentum). A classical particle with momentum $\boldsymbol{p}$ along $\boldsymbol{e}$ has angular momentum about the origin 
$$ \boldsymbol{L} = \boldsymbol{r} \times \boldsymbol{p} \;\; \rightarrow \| \boldsymbol{L} \| = d\, \| \boldsymbol{p} \| \tag{4}$$
