Entropy and data redundancy I have a quick question about the redundancy of data and the information that it represents. A means of evaluating the amount of information contained in the data is the notion of entropy.
1) For example, if in my data set, I have only constant values ​​(all more or less equal), Then the entropy (of Shannon?) Will be low, is it the case?
In this case, we are talking about redundant data, are we not?
2) How to prove it mathematically (that the entropy is low)? :
Can I use the following formula:
$$ H_ {b}(X) = - \mathbb{E} [\log_{b} {P(X)}] = \sum_{i = 1}^{n} P_{i} \log_{b} \left ({\frac{1}{P_ {i}}}\right) = - \sum_{i = 1}^{n} P_{i} \log_{b} P_{i} $$
where $ P_{i} $ is the probability that the random variable $X$ takes the value $ X = x_{i} $ among $ n $ possible values.
In the case where I have only equal values ​​($ x_{i} = \text{constant} \Rightarrow\,P_{i} = 1 \quad \text{for all i}$), then have a zero Shannon entropy, that is, zero information.
3) On the contrary, if the data are really different and scattered, I would expect a large entropy: is this also correct?
4) In the universe, we say that entropy only increases with the homogenization of matter: we say this because there will be a lot of possible configurations with this homogenization, but which parameter do we take into account when we talk about this entropy (which is not the entropy of Shannon) ? :
At first sight, we could think that if the homogeneity is complete, the position of each galaxy is uniformly distributed and therefore represents redundant data . But issue is that their position is not a constant value contrary to my previous question: they will be evenly distributed but not equal to each other.
If you could help me clarify this point? It seems that I am confused between the possible number of configurations and an eventual fixed distance between 2 galaxies (which is redundant from a Shannon's entropy point of view). I should maybe consider the distance between each couple of galaxies ?? 
5) In conclusion, are these little problems well understandable, I hope so. 
I'm interested in this because I'm working on Fisher's formalism which is a kind "entropy".
Any help would be great, Regards
 A: I think that you have a confusion between redundant information and uniform probability distribution.
To answer 1) and 2)
The error here is that the probability distribution is not the one you think.
You only have one possible outcome (assuming that only microstate $j$ is observed) so you have a Dirac distribution
\begin{equation}
P_i = \left\{ \begin{array}{l} 1 \text{ for } i=j \\
0 \text{ otherwise}  \end{array}  \right.
\end{equation}
In the case 3), in the contrary, you have $P_i= 1/\Omega$ whatever $i$ (where $\Omega$ is your number of states), and this is a uniform  probability distribution, if you compute the Shannon entropy in this case, you will get $H_b=\ln \Omega$ and you can shown that this is the maximum of the Shannon entropy when you maximize over possible probability distribution (see Wikipedia https://en.wikipedia.org/wiki/Entropy_(information_theory)#Maximum for a demonstration)
This uniform distribution is then the opposite of redundant data since you cannot determine the rest of the data from a subset of them.
4) In this case, we are speaking of the Shannon entropy at time $t$ :$-\sum_i  P_i(t) \ln P_i(t)$, but with the current probability of observing microstate $i$ that is not necessary equal to the microcanonical probability of uniform outcome. Now the evolution with time of this quantity can only grow according to the second law of thermodynamics and the current probability of observing microstate $i$ will eventually converge towards the equilibrium uniform probability.
Edit to answer the question of the mutual distance of the galaxies:
From your comment above, I now see why you want to consider a dataset of mutual distances as a redundant dataset. However, it would be a redundant dataset if all distances would have been exactly the same, i.e. if I give you the position of one of the galaxy then you can determine with no error the position of all others one. But in the actual dataset having the position of one of the galaxy only give approximately the position of the others ones and this errors is precisely quantify by the Shannon entropy. 
A configuration here is the actual position galaxies, so $(\vec{x}_{G1},\vec{x}_{G2},\ldots, \vec{x}_{GN})$, with discernable galaxies, so exchanging the position of two galaxy will give you a different configuration.
Assume that I give the position of one the galaxy, from the set of mutual distance, you know that the average distance between galaxy is a certain number so you can give the position of another galaxy with a certain uncertainty, but now if I try to compute the position of a third galaxy from the second one, I will have again an uncertainty that will add to the first uncertainty, so if I continue to add more galaxy by putting them at constant distance the one from each others, the actual configuration (so the set of positions of all galaxy) will be very off from the computed one. That is very different of being able to give with full certainty the position of all galaxies.
So the number of configurations here would be the number of possible positions for the set of all galaxies, times the number of permutations of galaxies as they are discernible.
