# What is the amount of information carried by a $X$-bit system?

Please consider me a beginner and not an expert by any means. My question below will thus be very very simple as that of an uninitiated. Let me consider four simple physical systems with $${\rm N}_S$$ states. With those, I'll try to explain my current state of knowledge/understanding at this point and ask the question.

• First, one coin. It has two possible states- a 'head' and a 'tail' i.e., $${\rm N}_S=2$$. Each state can be denoted by $$0$$ and $$1$$. This is an example of a $$1$$-bit system.

• Second, consider $$N$$ identical coins. Clearly, the number of states (or configurations) is now $$N_S=2^N$$. Each state of the full system can again be represented or encoded by a distinct string of '$$p$$' zeros and '$$q$$' ones such that $$p+q={\rm N}$$. This is an example of a classical $${\rm N}$$-bit system.

• Third, consider a dice where the number of states $$N_S=6$$. By definition, this is a $$\log_2 6\approx 2.585$$-bit system.

• Fourth, consider $$N$$-dies so that the number of states is $$N_S=6^N$$. It is, by definition, a $$\log_2(6^N)\approx 2.585N$$-bit system.

Therefore, irrespective of whether each 'microscopic' constituent is a $$1$$-bit system (e.g., a coin) or not (e.g., a dice), the quantity $$X=\log_2N_S$$ is used define a $$X$$-bit system. I have tried to explain that with my four examples above.

Given the above set up, my question is, if we have a $$X$$-bit system, what is the amount of information carried by that system?

• Could you clarify what exactly you mean by "amount of information"? I mean, how are you characterizing the amount of information? – David Z May 20 at 3:15
• @DavidZ Very naive. Colloquially, if one knows the exact 'microstate' i.e., which coin is head and which con is tail, then one has more information about the system compared to if one does not observe or measure. – mithusengupta123 May 20 at 3:40
• @DavidZ Also see my comment at the answer below. – mithusengupta123 May 20 at 3:52

The amount of information carried by a system is given by entropy function $$H(X)$$ which is defined as:$$H(X) =-\sum p(x) \log_{2}(p(x))$$ where $$p(x)$$ is the probability of random variable $$X$$, where the sum is over support of $$p(x)$$.
For example: Now if you have biased coin with $$p({\rm heads}) = 0.8$$ and $$p({\rm tails}) = 0.2$$. You can calculate the $$H(X)$$ which will be less than one $$1$$-Bit. Hence it requires less information than a fair coin to describe it.
• Therefore, the amount of information carried by a fair coin (my first example) is $-\frac{1}{2}\log_2(1/2)=1/2$? I used $p(\rm head)=1/2$. Also, the last phrase of your answer suggests that the 'amount information carried' is more when the ignorance about the system is more and knowledge is less. Is that true? – mithusengupta123 May 20 at 3:52
• sum is over the support of probability distribution in other words all possible outcomes with non zero probability. So information carried by a fair coin will be $-0.5 log_2(0.5)-0.5 log_2(0.5) = log(2) = 1$ – Blaze May 20 at 4:13