What is the amount of information carried by a $X$-bit system?

Question

Please consider me a beginner and not an expert by any means. My question below will thus be very very simple as that of an uninitiated. Let me consider four simple physical systems with ${\rm N}_S$ states. With those, I'll try to explain my current state of knowledge/understanding at this point and ask the question.

• First, one coin. It has two possible states- a 'head' and a 'tail' i.e., ${\rm N}_S=2$. Each state can be denoted by $0$ and $1$. This is an example of a $1$-bit system.

• Second, consider $N$ identical coins. Clearly, the number of states (or configurations) is now $N_S=2^N$. Each state of the full system can again be represented or encoded by a distinct string of '$p$' zeros and '$q$' ones such that $p+q={\rm N}$. This is an example of a classical ${\rm N}$-bit system.

• Third, consider a dice where the number of states $N_S=6$. By definition, this is a $\log_2 6\approx 2.585$-bit system.

• Fourth, consider $N$-dies so that the number of states is $N_S=6^N$. It is, by definition, a $\log_2(6^N)\approx 2.585N$-bit system.

Therefore, irrespective of whether each 'microscopic' constituent is a $1$-bit system (e.g., a coin) or not (e.g., a dice), the quantity $X=\log_2N_S$ is used define a $X$-bit system. I have tried to explain that with my four examples above.

Given the above set up, my question is, if we have a $X$-bit system, what is the amount of information carried by that system?

Answer 1

The amount of information carried by a system is given by entropy function $ H(X) $ which is defined as:$$ H(X) =-\sum p(x) \log_{2}(p(x)) $$ where $ p(x)$ is the probability of random variable $X$, where the sum is over support of $p(x) $.

For example: Now if you have biased coin with $p({\rm heads}) = 0.8$ and $p({\rm tails}) = 0.2$. You can calculate the $H(X) $ which will be less than one $1$-Bit. Hence it requires less information than a fair coin to describe it.

The amonut of information carried by system is a measure of amount of uncertainty present in the system.