# Shannon Entropy (Information Entropy) [closed]

Assuming $10^{10}$ gas particles are inside a volume of $1~\mathrm m^3$. How much bits does it takes to store the initial conditions?

My approach:

We know Shannon Entropy is given by,

$$S_\textrm{inf} = - \sum_i p_i \cdot \log_2 (p_i)$$

also, the probability distribution for the gas particles is given by (Gaussian Distribution),

$$p(v) = C \cdot v^2\cdot e^{\beta \cdot v^2}$$

then,

$$S_\textrm{inf} = - C \sum_i v^2\cdot e^{\beta \cdot v^2} \cdot \log_{2}\left(C \cdot v^2\cdot e^{\beta \cdot v^2}\right)$$

Now, how do I proceed it from here?

## closed as off-topic by Jon Custer, user36790, John Rennie, glS, heatherNov 5 '16 at 16:48

This question appears to be off-topic. The users who voted to close gave this specific reason:

• "Homework-like questions should ask about a specific physics concept and show some effort to work through the problem. We want our questions to be useful to the broader community, and to future users. See our meta site for more guidance on how to edit your question to make it better" – Jon Custer, Community, John Rennie, glS, heather
If this question can be reworded to fit the rules in the help center, please edit the question.

• Did you almost have the answer in your expression? Just to use the property of log function to separate the product? – Xiaodong Qi Nov 3 '16 at 20:17

First of all, you are neglecting the initial conditions regarding the position of the particles.

Moreover I feel like some data are missing: do you not know the temperature of the system?

This is what my answer would be, assuming you have $N$ particles of ideal gas of mass $m$ in a volume $V$ at temperature $T$ (so that $\beta={1\over k_B T}$).

The probability distribution of the particles' positions and momenta is Boltzmann's $$P(x, y, z, p_x, p_y, p_z)={N\over V}\left({\beta m\over2\pi}\right)^{3/2}e^{-\beta{p_x^2+p_y^2+p_z^2\over 2m}}$$

It is very similar to the $P(v)$ you have mentioned but

1-takes into account the probability distribution of the position (the $N/V$ part)

2- the second part takes into account the three components of the momentum separately instead of the magnitude (your $P(v)$ was the distribution of the magnitude of $v$, not of its components).

Now, we just have to compute Shannon Entropy using this distribution. The definition you wrote $S=-~\displaystyle\sum_i p_i\log(p_i)$ works if we have a discrete set of states ${i}$ each one with probability $p_i$. In our case we have a continuous set of state described by $P(x, y, z, p_x, p_y, p_z)$ so we have to integrate over all possible state instead of summing. Since the states are labelled by the positions and momenta of the particles we get for Shannon Entropy

$$S=-~\int\int P(x, y, z, p_x, p_y, p_z)\log(P(x, y, z, p_x, p_y, p_z))~\mathrm dx~\mathrm dy~\mathrm dz~\mathrm dp_x~\mathrm dp_y~\mathrm dp_z$$ i.e. $$S=-~\int\int {N\over V}\left({\beta m\over2\pi}\right)^{3/2}e^{-\beta{p_x^2+p_y^2+p_z^2\over 2m}}\log\left({N\over V}\left({\beta m\over2\pi}\right)^{3/2}e^{-\beta{p_x^2+p_y^2+p_z^2\over 2m}}\right)~\mathrm dx~\mathrm dy~\mathrm dz~\mathrm dp_x~\mathrm dp_y~\mathrm dp_z$$

It is the same thing you were trying to do but it has the right distribution (Boltzmann's distribution) and is integrated rather than summed as the set of states is continuous.

You have to solve that integral which is quite doable if you consider

1-that the distributions are normalised so that $\displaystyle\int \left({\beta m\over2\pi}\right)^{3/2}e^{-\beta{p_x^2+p_y^2+p_z^2\over 2m}}~\mathrm dp_x~\mathrm dp_y~d\mathrm p_z=1$ and that $\displaystyle \int N/V\; ~\mathrm dx~\mathrm dy~\mathrm dz=N$

2- the properties of the logarithm so you can decompose the integral in simpler parts and so that you can us $\log a^B=B\log a$.

3- that $\displaystyle\int P(p_x, p_y, p_z) (p_x^2+p_y^2+p_z^2)/2m ~\mathrm dp_x~\mathrm dp_y~\mathrm dp_z$ is the mean value of the kinetic energy of a particle which is $3/2k_B T$.

Then you are done. Try to solve it on your own.

The only thing that troubles me is the dependence of my solution on the Temperature T which you have not mentioned.. (but it was present also in your attempted solution in the beta-factor...)