It appears to be defined by probability; however, does it have some unit that indicates its 'information level' in terms of probability?
$\begingroup$
$\endgroup$
4
asked Jun 18, 2015 at 15:13
-
2$\begingroup$ Did you even search for units of information beforehand? $\endgroup$– ACuriousMind ♦Commented Jun 18, 2015 at 15:14
-
$\begingroup$ a nybble is a unit of information. A nybble is half a byte. $\endgroup$– JimCommented Jun 18, 2015 at 15:17
-
$\begingroup$ @ACuriousMind I'm searching for a unit of information as defined by information theory. $\endgroup$– HyperLuminalCommented Jun 18, 2015 at 15:19
-
$\begingroup$ some speak about half-bits but it is another thing. Simply the bit ? $\endgroup$– user46925Commented Jun 18, 2015 at 15:32
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
It depends on which logarithm you choose. To demonstrate (but not prove) the information theory equation: the definition $S = \log W$ (uncertainty is the logarithm of multiplicity, Boltzmann's famous relation) becomes equivalent, if $p_i = 1/W$ ($W$ equally probable outcomes) to $$S = -\sum_{i=1}^W p_i \log p_i.$$But since $\log_a(b) = \log_c(b) / \log_c(a)$ we are always a multiplicative constant away from choosing whatever base we want.
The easiest way, common in computer science circles, is to choose the base-2 logarithm. Then the uncertainty is measured in bits, and so is information (which is a reduction in uncertainty).
So if you have a system with ten distinguishable coins, there are 1024 possible outcomes and all of them are equally probable. If you know nothing more about the system, then your uncertainty is $\log_2 1024 = \text{10 bits}$. Someone needs to send you 10 bits to tell you what state the system is in. But if you have a system with ten indistinguishable coins, there are only 11 different possible outcomes (none are heads up, one is heads up, ... all ten are heads up) which we could easily encode as a 4-bit number. Information theory says that these various probabilities corresponding to the distribution of [1,10,45,120,210,252,210,120,45,10,1] allow us to reduce the problem to $-\sum_i p_i \log_2 p_i = 2.706\dots\text{ bits}$. Probably to actually attain this you'd need a "lossy compression" and you'd evaluate how good it is with a "square-root of the sum of squared errors" in what it reproduces, but the simplest case is to round 0, 1, and 2 to 2, and 10 to 9, and then you're typically only off by 0.34.
