# How much entropy is in the word 'bed'?

I am referring to one of the first examples in Shannon's famous paper. I (think I) understand the concept of average entropy of a system, but let's say we are interested in a specific sequence of letters that system generates, like 'bed'. I guess by saying that word, we reduced 'uncertainty' or 'surprise', but how do we measure this? As a delta between the average of the system and the one for this specific sequence of letters? thanks!

• I'm not sure how this is a physics question rather than, say, a Mathematics or Computer Science question. Might it be better suited on one of these sites? – ACuriousMind Jul 12 '17 at 9:04

But the probability distribution could be defined in terms of anything else. For example, we could calculate the entropy under the distribution which says that letters are drawn independently at random and there's a $\frac16$ chance of "b" and a $\frac1{30}$ chance of any other English letters (assuming we don't expect to see spaces or other punctuation). Under that assumption, "bed" would have entropy of $\log(\frac16 * \frac1{30} * \frac1{30})$.