I am referring to one of the first examples in Shannon's famous paper. I (think I) understand the concept of average entropy of a system, but let's say we are interested in a specific sequence of letters that system generates, like 'bed'. I guess by saying that word, we reduced 'uncertainty' or 'surprise', but how do we measure this? As a delta between the average of the system and the one for this specific sequence of letters? thanks!

  • 2
    $\begingroup$ I'm not sure how this is a physics question rather than, say, a Mathematics or Computer Science question. Might it be better suited on one of these sites? $\endgroup$
    – ACuriousMind
    Jul 12 '17 at 9:04

Entropy isn't defined without reference to an underlying probability distribution.

If the underlying probability distribution is something like "things that people speaking English say", then Shannon found experimentally about two bits of uncertainty per English letter.

But the probability distribution could be defined in terms of anything else. For example, we could calculate the entropy under the distribution which says that letters are drawn independently at random and there's a $\frac16$ chance of "b" and a $\frac1{30}$ chance of any other English letters (assuming we don't expect to see spaces or other punctuation). Under that assumption, "bed" would have entropy of $\log(\frac16 * \frac1{30} * \frac1{30})$.

This is beyond the scope of your question, but the field of algorithmic information theory attempts to study natural probability distributions which can be defined in terms of fundamental things about computation.

  • $\begingroup$ thanks that's very interesting. But let's say we are in the case described by Shannon, (example 'D'), where there is a language with 5 letters and 16 words, where the probability of 'bed' is 0.04, whereas 'deed' is 0.15. Does this mean that 'bed' is more 'informative' as it reduces the entropy more than 'deed'? $\endgroup$
    – magnolia1
    Jul 12 '17 at 7:09
  • $\begingroup$ It's not defined to talk about how 'informative' words are, because we're just talking about a probability distribution over letters. At the start of a word, a 'B' is certainly more surprising than an 'A' (again using the example from Shannon's paper). And given that the first letter was a 'B', an 'A' would be more surprising than a 'E' as the next letter. $\endgroup$ Jul 12 '17 at 7:16
  • $\begingroup$ Dan Styer has a wonderful treatment of this distinction in his notes on Statistical Mechanics. Section 2.6.2. $\endgroup$ Jul 12 '17 at 13:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.