I have a quick question about the redundancy of data and the information that it represents. A means of evaluating the amount of information contained in the data is evaluated with a notion of entropy.

1) For example, if in my dataset, I have only constant values ​​(all more or less equal), Then the entropy (of Shannon?) Will be weak, is it the case?

In this case, we are talking about redundant data, are not we?

2) How to prove it in a mathematical way (that the entropy is weak)? :

can I use the following formula:

$$ H_ {b}(X) = - \mathbb{E} [\log_{b} {P(X)}] = \sum_{i = 1}^{n} P_{i} \log_{b} \left ({\frac{1}{P_ {i}}}\right) = - \sum_{i = 1}^{n} P_{i} \log_{b} P_{i} $$

where $ P_{i} $ is the probability that the random variable X takes the value $ X = x_{i} $ among $ n $ possible values.

In the case where I have only equal values ​​($ x_{i} = \text{constant} \Rightarrow\,P_{i} = 1 \quad \text{for all i}$), then have a zero Shannon entropy, that is, zero information.

3) On the contrary, if the data are really different and scattered, I will wait for a big entropy: here too a confirmation?

4) In the universe, we say that entropy only increases with the homogenization of matter: we say that because there will be a lot of possible configurations with this homogenization but which parameter we take into account when we talk about this entropy (which is not the entropy of Shannon) ? :

At first sight, we could think that if the homogeneity is complete, the position of each galaxy is uniformly distributed and therefore represents redundant data . But where there is concern is that their position is not a constant value contrary to my previous question: they will be evenly distributed but not equal to each other.

If you could help me clarify this point?

5) In conclusion, are all my little problems of understanding where I would like to get help?

I'm interested in that because I'm working on Fisher's formalism which is like entropy.

I thank you in advance


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.