# Entropy and data redundancy

I have a quick question about the redundancy of data and the information that it represents. A means of evaluating the amount of information contained in the data is the notion of entropy.

1) For example, if in my data set, I have only constant values ​​(all more or less equal), Then the entropy (of Shannon?) Will be low, is it the case?

In this case, we are talking about redundant data, are we not?

2) How to prove it mathematically (that the entropy is low)? :

Can I use the following formula:

$$H_ {b}(X) = - \mathbb{E} [\log_{b} {P(X)}] = \sum_{i = 1}^{n} P_{i} \log_{b} \left ({\frac{1}{P_ {i}}}\right) = - \sum_{i = 1}^{n} P_{i} \log_{b} P_{i}$$

where $$P_{i}$$ is the probability that the random variable $$X$$ takes the value $$X = x_{i}$$ among $$n$$ possible values.

In the case where I have only equal values ​​($$x_{i} = \text{constant} \Rightarrow\,P_{i} = 1 \quad \text{for all i}$$), then have a zero Shannon entropy, that is, zero information.

3) On the contrary, if the data are really different and scattered, I would expect a large entropy: is this also correct?

4) In the universe, we say that entropy only increases with the homogenization of matter: we say this because there will be a lot of possible configurations with this homogenization, but which parameter do we take into account when we talk about this entropy (which is not the entropy of Shannon) ? :

At first sight, we could think that if the homogeneity is complete, the position of each galaxy is uniformly distributed and therefore represents redundant data . But issue is that their position is not a constant value contrary to my previous question: they will be evenly distributed but not equal to each other.

If you could help me clarify this point? It seems that I am confused between the possible number of configurations and an eventual fixed distance between 2 galaxies (which is redundant from a Shannon's entropy point of view). I should maybe consider the distance between each couple of galaxies ??

5) In conclusion, are these little problems well understandable, I hope so.

I'm interested in this because I'm working on Fisher's formalism which is a kind "entropy".

Any help would be great, Regards

• What do you mean by saying the the entropy "will be weak"? Do you mean low? – user137661 Nov 19 '19 at 0:37
• @SV Yes I wanted to say "low entropy" since all the distances between couple of galaxies is the same, so a kind of redundant information. I am going to try to understand the first answer below of Hadrien. Regards – youpilat13 Nov 19 '19 at 0:42
• You say that $P_i=1$, that would mean that either you have only one datum, or all your data has the same value. This is my interpretation... but don't you intend to say that you have $N$ data points and each has its own value $x_i$ but all of them are different? – user137661 Nov 19 '19 at 0:47
• @SV that's the key point, like said Hadrienin below answer : there sould be a difference between what we call a redundant information (a data set contains the same values or relatively closed such as $P_{i} = P(X=x_{i}) \simeq 1$ so total Shannon's entropy is closed to zero : That's the Shannon point of view. Now, I was wondering if there were similarities with the classical Entropy (of Boltzmann) for Universe : – youpilat13 Nov 19 '19 at 22:59
• if one tends to uniform density, one tends to a maximal entropy which is I think equal to $S=k_{B}\ln\Omega$ with $\Omega$ the number of configurations : but what do we call in this case the number of configurations ? That would mean that none galaxy will be distinguishable object ? – youpilat13 Nov 19 '19 at 22:59

I think that you have a confusion between redundant information and uniform probability distribution. To answer 1) and 2) The error here is that the probability distribution is not the one you think. You only have one possible outcome (assuming that only microstate $$j$$ is observed) so you have a Dirac distribution $$$$P_i = \left\{ \begin{array}{l} 1 \text{ for } i=j \\ 0 \text{ otherwise} \end{array} \right.$$$$

In the case 3), in the contrary, you have $$P_i= 1/\Omega$$ whatever $$i$$ (where $$\Omega$$ is your number of states), and this is a uniform probability distribution, if you compute the Shannon entropy in this case, you will get $$H_b=\ln \Omega$$ and you can shown that this is the maximum of the Shannon entropy when you maximize over possible probability distribution (see Wikipedia https://en.wikipedia.org/wiki/Entropy_(information_theory)#Maximum for a demonstration)

This uniform distribution is then the opposite of redundant data since you cannot determine the rest of the data from a subset of them.

4) In this case, we are speaking of the Shannon entropy at time $$t$$ :$$-\sum_i P_i(t) \ln P_i(t)$$, but with the current probability of observing microstate $$i$$ that is not necessary equal to the microcanonical probability of uniform outcome. Now the evolution with time of this quantity can only grow according to the second law of thermodynamics and the current probability of observing microstate $$i$$ will eventually converge towards the equilibrium uniform probability.

Edit to answer the question of the mutual distance of the galaxies: From your comment above, I now see why you want to consider a dataset of mutual distances as a redundant dataset. However, it would be a redundant dataset if all distances would have been exactly the same, i.e. if I give you the position of one of the galaxy then you can determine with no error the position of all others one. But in the actual dataset having the position of one of the galaxy only give approximately the position of the others ones and this errors is precisely quantify by the Shannon entropy.

A configuration here is the actual position galaxies, so $$(\vec{x}_{G1},\vec{x}_{G2},\ldots, \vec{x}_{GN})$$, with discernable galaxies, so exchanging the position of two galaxy will give you a different configuration.

Assume that I give the position of one the galaxy, from the set of mutual distance, you know that the average distance between galaxy is a certain number so you can give the position of another galaxy with a certain uncertainty, but now if I try to compute the position of a third galaxy from the second one, I will have again an uncertainty that will add to the first uncertainty, so if I continue to add more galaxy by putting them at constant distance the one from each others, the actual configuration (so the set of positions of all galaxy) will be very off from the computed one. That is very different of being able to give with full certainty the position of all galaxies.

So the number of configurations here would be the number of possible positions for the set of all galaxies, times the number of permutations of galaxies as they are discernible.

• Thanks, you said This uniform distribution is then the opposite of redundant data since you cannot determine the rest of the data from a subset of them.: I interpret it like "uniform distribution $\rightarrow$ $S = max$ and "redundant dataset $\rightarrow$ $S = 0$, don't I ? Can we consider a dataset of mutual distances between galaxies less or more equal like a redundant dataset from a Shannon point of view ? . Regards – youpilat13 Nov 19 '19 at 23:13
• you can see my last comment above : coud you have any confirmation about what I have written ? Thanks – youpilat13 Nov 20 '19 at 17:13
• So you do have "uniform distribution" $\rightarrow S=max$ and "redundant dataset" $\rightarrowS=0$. – Hadrien Nov 21 '19 at 0:16
• Sorry for the misprints. I put more in my answer about the mutual distance dataset. – Hadrien Nov 21 '19 at 0:37
• If you consider discernable particles, then permutatiing two particles without touching anything else will give another configuration. Whereas if the particles where indiscernable, the configuration will be the same. Then when counting the number of possible congifuration, the discernibility will matter (it add usually a factor $N!$ to the number of configuration with $N$ the number of particles – Hadrien Nov 21 '19 at 15:18