How natural is the number $e$? The definitions of Shannon entropy and Thermodynamic entropy bear a close resemblance to each other.
Trying to take the analogy further we see that, in the definition of Shannon entropy, the base of the logarithm is 2, the number of distinct states that can be taken by the source producing "information" (in a naïve sense). Similarly, in physical processes we notice that the exponents and logarithms are in base $e$. (In general, if we have $D$ distinct states to a system we can take base $D$ to the logarithm.)
Could this frequent occurrence of $e$ in nature be a hint at how information is coded (in a sense, dimension of nature; look at this paper) naturally, or is it just due to mathematical definition that leads to it in physical formulae?
 A: Short version: it is not $e$ but $e^x$ that is natural, and it is natural because it is invariant under differentiation.
The "naturalness" of numbers is maybe a philosophical, maybe an aesthetic question, but also does show up in many theoretical physics discussions. When a dimensionless constant shows up in your theory, if it is a very large or small compared to unity people say it doesn't look natural - maybe you cheated by fine-tuning it to make the theory fit the data.
However, $e$, $\pi$, $\sqrt{5}$ and similar purely mathematical factors work differently. The reason they are present is usually the mathematical symmetries and relations between variables in the theory. If something is related to circles, spheres, angles or even integer values of the Riemann zeta function $\pi$ tends to show up. If you do geometry with 5-fold symmetry there will be $\sqrt{5}$s all over the place, helped by the invariance properties of the golden ratio under integer powers.
$e$ is natural because $y(x)=e^x$ is the unique solution of $y'(x)=y$ with $y(0)=1$, essentially the simplest nontrivial differential equation and a building-block for solving many others. Again, the invariance properties of the function makes it recur again and again since anything involving derivatives will leave it unchanged.
In particular, linear  first-order differential equations of the form $y' + f(x)y = g(x)$ have the general solution $$y(x) = \frac{\int\mu(x)g(x) dx+c}{\mu(x)}$$ where $\mu(x)=\exp\left(\int f(x) dx\right)$. Since this is often a decent local linear approximation to more complex equations, exponentials will crop up. Linearizing a multi-variable differential equation around a critical point $\mathbf{y}'(\mathbf{0})=\mathbf{0}$ produces $\mathbf{y}'=\mathbf{Jy}$ where $\mathbf{J}$ is a constant matrix (the Jacobian) and the solutions have the form $\mathbf{y}(t)=\sum_n c_n\mathbf{\Lambda}_ne^{\lambda_n t}$ where $\mathbf{\Lambda}_n$ represents eigenvectors of  $\mathbf{J}$ and $\lambda_n$ their corresponding eigenvalues. And so on.
The value of $e$ is in many ways less important than the function: you will relatively rarely find explicit $e$ factors in your physics formulas. When $e$ shows up it is often acting as a scale factor converting one kind of measure to another, as in logarithms or $e$-foldings of an exponential process.
