In the communication context, every book i've started to read says:

if you want to send a datum about a random experiment $X$, let it be a fair coin toss, that is $$p(tails)=p=0.5; p(heads)=q=0.5$$ you first create an alphabet consisting of two symbols, like {0.1voltage; 1.0v} pulses. Then toss a coin and sent one pulse.

This is all good and well, but then the authors say something like this:

Each symbol received carries 1 bit of information.

And I say: What?! According to Shannon entropy the entropy of the whole experiment is $$H(X)=-p\log(p)-q\log(q)$$, where $-p\log(p)$ is info content of a single symbol, right? So reading $0.1v$ or $1.0v$ in the wire means that I get only $-p\log(p)$ amount of Shannons. Which in the context of a coin toss means that I get only 1/2 a Shannon bit of information and not the whole bit!

Looks like authors mean something different by saying "you get one bit from one symbol" and looks like there are many types of "bits" (physical, informational etc.) which are somehow all mixed together and make up my confusion.

Can anybody help me with this bit vs bit confusion? Thanks.

P.S. some quotations which I'm struggling with:


For example, a high pulse may represent a 1-bit and a low pulse (or no pulse) may represent a 0-bit (or vice versa). In this case, the voltage pulses are the symbols, and each pulse carries 1 bit of information. source >>>


In the case of the H-T stream $p$ and $q$ are both $1/2$, so $1/p$ and $1/q$ are both 2. The logarithm (to base 2) of 2 is 1, so the formula gives the value 1. This means that each symbol, H or T, conveys one bit of information.

source >>>

  • $\begingroup$ Your second image says that 1000 bits are needed to store information of 1000 measurements of the variable $X$, this is the case $N=1000$ in my answer and the total information is $NS$. For a fair coin $S=1$, so it needs 1000 bits. $\endgroup$ Sep 26 at 11:01
  • $\begingroup$ So, again one pulse\symbol of a wave carries 1 bit of information. But one pulse or a wave always represents a SINGLE value, and in case of a coin toss, single value has 0.5 shannon bits of info... I'm confused. $\endgroup$
    – coobit
    Sep 26 at 11:08
  • 2
    $\begingroup$ Please do not post images of text (or mathematics). The preferred method is to use a combination of text and Mathjax. Note that the site engine cannot usefully search imagesso text and Mathjax are much more useful. $\endgroup$
    – StephenG
    Sep 26 at 12:42
  • $\begingroup$ The information content associated to an event $E$ occurring with probability $p$ is $-\log(p)$ and not $-p \log p$, as you've written. See also the example in the linked Wikipedia page. By the way, are you sure this question belongs to physics SE? I mean of course the concepts of information theory have applications in physics etc., but this question does not deal with any physics at all. $\endgroup$
    – Jakob
    Sep 26 at 16:57

The information content of a single outcome is just $-\log p$. To make it concrete, if all probabilities are powers of two, then the length of the Huffman code for a signal of probability $p$ is exactly $-\log p$.

$-p\log(p)-q\log(q)$ is the expected information content of a trial with two possible outcomes. Only the sum is meaningful, and only if $p+q=1$.

  • $\begingroup$ Yeah, looks like i've inforamtional content of an event and informational entropy of the whole random value mixed together in my brain. Thanks. $\endgroup$
    – coobit
    Sep 27 at 6:06

The information stored in a random variable $X$ is defined as: $$S=log_b(N) $$ where $N$ is the number of different outcomes we get when we measure $X$. Here the choice of $b$ is not really important and changing $b$ is same as multiplying $S$ with some constant.

The common choices for $b$ are:

  • bit for $b=1$
  • trit for $b=3$
  • nat for $b=e$

Check here for more info. Clearly, if you have a fair coin then there are two different states and $S=log_2(2)=1$ bit. So it has 1 bit of information.

For a random variable $X$ with 2 possible outcomes, $A$ with probability $p$ and $B$ with probability $1-p$. The number of possible outcomes is approximately: (assuming the message will consist very nearly of $pN$ occurrences of $A$ and (1 − p)N occurrences of $B$) \begin{align*} \frac{N!}{(pN)!((1-p)N)!} \approx \dfrac{N^N}{(pN)^{pN}((1-p)N)^{(1-p)N}}=2^{NS} \end{align*} $$\implies S=-p\log(p)-(1-p)\log(1-p) $$ where $NS$ is the information in $N$ successive measurements of the variable $X$.

Also check arXiv:1805.11965.

  • $\begingroup$ I'm sorry, but how does it explain the contradiction between "One symbol of a 2 symbol alphabet carries 1 bit of information about X random variable when shannon clearly states that it is not 1 but −qlog(q) (which is 1/2 bits in case of fair coin)? $\endgroup$
    – coobit
    Sep 26 at 10:26
  • $\begingroup$ @coobit "One symbol of a 2 symbol alphabet carries 1 bit of information" this is clearly wrong. Can you provide a source that says it? $\endgroup$ Sep 26 at 10:30
  • $\begingroup$ > In this case, the voltage pulses are the symbols, and each pulse carries 1 bit of information. source: usna.edu/ECE/ec312/Lessons/wireless/… page 1, paragraph 1 $\endgroup$
    – coobit
    Sep 26 at 10:46
  • $\begingroup$ @coobit In the PDF you mentioned each pulse can either show $0$ or $1$ and they both have the same probability $p=\frac{1}{2}$. So each pulse has 1 bit of information. By symbol I don't think they are referring to specifically $0$ or $1$, they are referring to the pulse itself which has information of both $0$ and $1$. $\endgroup$ Sep 26 at 10:56
  • $\begingroup$ Still dont' get it. IF bit value 0 has $p=1/2$ of occuring then it's shannon entropy equals 1/2log(1/2) which is 0.5 shannon bits, where does whole 1 bit comes from? Yes, to store "value of bit 0" you need 1 physical bit of memory, but I guess it's not that kind of a bit which is being spoken here. $\endgroup$
    – coobit
    Sep 26 at 11:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.