What is the purpose of differential form of Gauss Law? I am learning the differential form of Gauss Law derived from the divergence theorem. 
$${\rm div}~ \vec{E} =\frac{\rho}{\epsilon_0}.$$
So far in my study of math and physics, the word "differential" has been associated with a infinitesimal change in a value (e.g. $dv$, $dr$...etc)
I do not really understand what the physical interpretation of the Gauss Law in differential form is. In general, what is the purpose of putting an equation into differential form? 
 A: This is a good example of a procedure that happens in many areas of physics. In general, physical laws - and particularly conservation laws - tend to be most naturally phrased in integral form, or even in mixed integro-differential form. For an example of the latter, consider the integral form of Faraday's law:
$$
\oint_{\partial S}\mathbf{E}\cdot\text d\mathbf{s}=-\frac{\text d}{\text dt}\int_S\mathbf{B}\cdot\text d\mathbf{A}.
$$

In general, these laws of physics are easier to formulate directly as they deal with finite quantities and their rates of change, whereas differential laws involve concepts which are harder to visualize. Thus, the integral form of the continuity equation,
$$
\frac{\text d}{\text dt}\int_\Omega\rho\,\text dV
=-\int_{\partial\Omega}\mathbf{j}\cdot\text d\mathbf{A},
$$
says that the total mass of fluid within some finite volume $\Omega$ increases at the rate that fluid flows in, and the left- and right-hand sides of the equations are precisely those quantities. On the other hand, the differential form of the law,
$$
\frac\partial{\partial t}\rho=-\nabla\cdot\mathbf{j},
$$
involves rather more abstract concepts. The density of fluid is OK, but if you're really honest then any reasonable attempt at providing an intuitive picture of the divergence $\nabla\cdot\mathbf{j}$ will be more like the flow out of finite volumes, in the limit when those volumes go to zero.

The problem with integral laws of physics, on the other hand, is that they are very difficult to actually solve. Suppose, for example, that you have a given charge density $\rho$ and current density $\mathbf{J}$, which possibly change over time, and you want to find out the electric and magnetic fields they produce under appropriate boundary conditions. Mathematically, the integral set of Maxwell's equations,
$$
\left\{\begin{align}
\int_{\partial\Omega}\mathbf{E}\cdot\text d\mathbf{A}&=\frac{1}{\epsilon_0}\int_\Omega\rho\,\text dV
&
\int_{\partial\Omega}\mathbf{B}\cdot\text d\mathbf{A}&=0
\\
\oint_{\partial S}\mathbf{B}\cdot\text d\mathbf{s}-\mu_0\epsilon_0\frac{\text d}{\text dt}\int_S\mathbf{E}\cdot\text d\mathbf{A}&=\mu_0\int_S\mathbf{J}\cdot\text d\mathbf{A}
&\!\!\!\!\!\!
\oint_{\partial S}\mathbf{E}\cdot\text d\mathbf{s}+\frac{\text d}{\text dt}\int_S\mathbf{B}\cdot\text d\mathbf{A}&=0
\end{align}\right.
$$
(for all possible volumes $\Omega$ and surfaces $S$ within your domain) are a well-determined system with a unique solution. However, it is tremendously hard to solve them directly, and no one seriously attempts it (and definitely not in front of students). The fundamental reason for this hardness is the fact that the electric and magnetic fields at any one point $\mathbf{r}$ always appear in equations which involve a whole host of other points that can be and usually are rather far away.
To actually solve this mathematical problem, then, the usual practice is to transform the equations into their differential form, which is done by taking the limit of infinitesimally small volumes and surfaces. If you do that, you get the differential Maxwell equations,
$$
\left\{\begin{align}
\nabla\cdot\mathbf{E}&=\frac1{\epsilon_0}\rho
&\quad
\nabla\cdot\mathbf{B}&=0
\\
\nabla\times\mathbf{B}-\mu_0\epsilon_0\frac{\partial \mathbf{E}}{\partial t}&=\mu_0\mathbf{J}
&\quad
\nabla\times\mathbf{E}+\frac{\partial \mathbf{B}}{\partial t}&=0.
\end{align}\right.
$$
These are to be taken at each point, instead of over nonlocal volumes and surfaces, and they connect the values of the fields at each point to the sources and, through the spatial derivatives, to the values of the fields at neighbouring points, but no further.
This is a much easier problem, because it imposes a local constraint of the fields and it is easier to solve for the functional behaviour of fields at specific localized regions. Once you have that, you can begin 'gluing' together different far-away regions, essentially by imposing the appropriate boundary conditions. But the crucial step - finding the possible functional forms of the fields that are allowed by the equations - is most easily done locally because there are no longer an infinity of different possible volumes and surfaces of integration to consider.

On the other hand, differential forms for physical laws are a lossy tool, and there are certain situations, like point charges or line currents, jump discontinuities and the like, which they handle rather badly. To deal with those situations, you essentially have to go full circle round to the integral forms of the laws. There exists a large and established formalism to deal with the  differential equations in those cases, and it involves Green's functions that are actually distributions instead of functions, but it essentially boils down to the concept of a weak solution of a differential equation. (See also this question.) Wikipedia describes the concept quite well:

a weak solution (also called a generalized solution) to an ordinary or partial differential equation is a function for which the derivatives may not all exist but which is nonetheless deemed to satisfy the equation in some precisely defined sense

and this "precisely defined sense" turns out to be exactly that the corresponding integral law be satisfied.
A: For each problem it may be more useful to use the integral or the differential form of equation. The two sets are equivalent, of course, but often one will be better then the other. 
For the case of Gauss's law. The differential form is telling you that the number of field lines leaving a point is space is proportional to the charge density at that point. If you have an expression for the electric field then you can use the differential form to find the charge density.
A: This point (why differential) does not stop or start at Gauss law. 
Why should we write second law of Newton in differential form? 
The answer is (and that is what Newton considered as his best discovery) that many laws of nature look best (simple) if they are written as relations between differentials. Not all of them -- but many. 
In this particular case Gauss law tells you what kind of VECTOR field the electrical field is. By putting a special constrain on it. Not all vector fields have this property. Naively speaking (which is always good) this differential formula tells you that electrical field is a "radial" field with a collection of POINTLIKE sources. It tells you that sources of the field can be infinitesimal points. Field starts at the source or ends. Like water -- comes or sinks.
Contrary - In magnetism points can NOT be sources, and only lines can be sources of the magnetic field, that's why magnetic field depends on CURRENT which is a LINE/curve (if you think geometrically/topologically) in space. Thus magnetic field is a totally different kind of vector field.
Ps: Vector field - an infinite collection of vectors assigned to each point in space, like speed of each water molecule in water stream.
A: Many a times integration becomes tedious because if more than two variables are involved we do not have any formula as such to solve the integral.
Under such situations the differential form of gauss law might prove to be useful.
