15
$\begingroup$

Although it seems simple, I can't get the derivation correct. Here is my reasoning:

$P(S)=P(A)P(B)$

Where P is the probability and S, A, and B denote different systems.

$S_A=-P(A)\ln P(A)$ and $S_B=-P(B)\ln P(B)$

Then $$S_{S}=-P(S)\ln P(S)=-P(S)\ln P(A)P(B)=-P(S)\ln P(A)-P(S)\ln P(B)$$

The problem is, since $P(A)\neq P(S)$ and $P(B)\neq P(S)$, how can entropy be additive?

Improve this question
$\endgroup$
  • 3
    $\begingroup$ I'm pretty confused, because it looks like you start by assuming that $P(S)=P(A)P(B)$ and then show that it contradicts an assumption that $P(S)\ne P(A)P(B)$, which of course shouldn't be surprising. Have I misunderstood what you're trying to show? $\endgroup$ – Nathaniel Mar 1 '16 at 5:41
  • $\begingroup$ Entropy is only additive in systems without interaction, so your assumption is wrong, already. $\endgroup$ – CuriousOne Mar 1 '16 at 6:53
  • $\begingroup$ @Nathaniel if I assume P(S)=P(A)P(B) (I don't know if it's a correct statement) then entropy cant be additive because $-P(S)lnP(A)-P(S)lnP(B) \neq -P(A)lnP(A)-P(B)lnP(B) $. That's what I tried to say. $\endgroup$ – SaudiBombsYemen Mar 1 '16 at 18:27
  • $\begingroup$ @SaudiBombsYemen ah, I see. But when you sum over A and B, they are equal after all. (It that isn't clear, let me know and I'll post an answer.) $\endgroup$ – Nathaniel Mar 2 '16 at 3:46
  • $\begingroup$ @Nathaniel hmm... I think the problem is that my level of statistics and calculus is still highschool one (last year) so yeah, it'd be perfect if you explained me what you mean :) $\endgroup$ – SaudiBombsYemen Mar 2 '16 at 15:47
17
$\begingroup$

If you consider a system $C$ and two subsystems $A,B$ with associated probability distributions $p_C,p_A,p_B$ and want the entropy to add, you must assume the subsystems are independent in the sense that $p_C(X) = p_A(Y)p_B(Z)$ where $Y$ and $Z$ are a partitioning of the variables $X$ belonging to $C$. Then, the total entropy is \begin{align} S_C & = \langle -\ln(p_C)\rangle\\& = -\int p_C(X)\ln(p_C(X))\mathrm{d}X \\ & = -\int p_A(Y)p_B(Z)(\ln(p_A(Y))+\ln(p_B(Z))\mathrm{d}Y\mathrm{d}Z \\ & = -\int p_A(Y)p_B(Z)\ln(p_A(Y))\mathrm{d}Y\mathrm{d}Z - \int p_A(Y)p_B(Z)\ln(p_B(Z)\mathrm{d}Y\mathrm{d}Z \\ & = S_A+ S_B \end{align} because $\int p_A(Y)\mathrm{d}Y = 1$ and likewise for $B,Z$.

The assumption of statistical independence is crucial. Entropy is, from the information-theoretic viewpoint, the expected amount of information encoded in a system. If two systems are not independent, obviously the information encoded in them together will be less than the sum of the information you can extract from each without knowing the other. That is, classical entropy is subadditive, but only additive if the systems you are adding together are statistically independent.

Improve this answer
$\endgroup$
  • $\begingroup$ This was Shannon's 4th axiom used to find Shannon entropy $\endgroup$ – innisfree Jun 2 '17 at 23:52
  • $\begingroup$ @ACuriousMind in the step going from X to Y and Z why is there no involvement of dZ, and how does it come in the next step? $\endgroup$ – SK Dash Dec 10 '19 at 3:25
  • $\begingroup$ @SKDash I'm afraid I don't understand the question - can you be more specific? $\endgroup$ – ACuriousMind♦ Dec 10 '19 at 7:14
  • $\begingroup$ @ACuriousMind first you wrote $S_c$ as integral of $p_c(X)ln(p_c(X))dX$, that is in terms of X, in the next step, there is no involvement of dZ, and in the step after that there is an involvement of day and dZ, these three steps, I am not able to understand $\endgroup$ – SK Dash Dec 10 '19 at 7:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.