In Jaynes' Probability Theory: The Logic of Science, there is a quick derivation of continuous entropy in chapter 12. He does so by taking the discrete definition of entropy and combines it with intuitions from the definition of an integral by Riemann sum. I've filled in the gaps as best I could:

$$ \lim_{n \rightarrow \infty} \sum_{i=1}^n f(x_i) \Delta x = \int_a^b dx\ f(x)$$ $$f(x)=f(x_i)\Delta x$$ $$ \Delta x = \frac{b - a}{n} $$
$$\Delta x = \frac{1}{nm(x_i)}, \ \text{where} \ m(x) = \frac{1}{b-a} $$ $$p_i=f(x_i)\frac{1}{nm(x_i)} \tag{2.6}$$ $$H=-\sum_1^n p_i \log[p_i] \tag{2.2}$$ $$H^c= \lim_{n\rightarrow \infty} \left [-\sum_1^n \frac{f(x_i)}{nm(x_i)} \log \left[ \frac{f(x_i)}{nm(x_i)} \right ] \right]$$ $$H^c= \lim_{n\rightarrow \infty} \left [-\sum_1^n \Delta x f(x_i) \log \left [\frac{f(x_i)}{nm(x_i)} \right ] \right ]$$ $$ H^c = \int dx\ p(x)\log\left [\frac{p(x)}{nm(x)}\right ] \tag{2.7}$$

I think everything seems to make sense, except I cannot figure out why there is no negative sign in equation 2.7. What have I missed?

  • 1
    $\begingroup$ ... a typo ? see for example mtm.ufsc.br/~taneja/book/node13.html $\endgroup$ – GiorgioP Jun 6 at 21:34
  • $\begingroup$ I think you are right. The final form that $H^c$ takes does have a negative sign in the text, but there are some additional steps of subtracting out log(n). Revisting the math with this hypothesis led me to discover an error in my proof which corrects for this typo. $\endgroup$ – Davey Jun 6 at 22:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.