I'm trying to solve a exercise about Shannon entropy. I've tried some things but I can't solve it, and couldn't find information on the internet.

Let's see. We have the Shannon entropy, and a system defined by a probability distribution $p(n)$:

$$S = \sum _{n\in N} p(n) \log (p(n)) $$

Where $N$ can be a finite set of probabilites or a infinite one, and $n$ is integer.

Now let's define the generating function of information (not sure if this is its name in English):

$$G(u) = \sum _{n \in N} p^u(n)$$

The question is, how can I write $S$ in terms of $G(u)$? Also it would be great to have a physical interpretation of this function (but the exercise doesn't ask for it)

My attempts to get the solution: I tried to write $G(1)$ and insert it into the expression of $S$, but I cannot get rid of the logarithm. Also I've tried to make a Taylor expansion of the logarithm, but I cannot write the series around $x=0$ because the logarithm is not analytic in this point. Around $x=1$,

$$\log(x) = - \sum _{k=1} ^{+\infty} \dfrac{(-1)^k (x-1)^k}{k}$$

But it looks like I can't substitute the expression of $G(u)$ inside this using $x\equiv p(n)$.

Any advice or hint is well received. As I've said, I haven't found any information about the function $G(u)$ on the Internet.

  • 1
    $\begingroup$ And what do you get if you expand G(u) around u = 1? $\endgroup$ – Count Iblis Mar 21 '16 at 19:33
  • 1
    $\begingroup$ Try using partial derivarives with respect (say, for example, $\frac{\partial G}{\partial u}$, then evaluate the function at $u=1$, your initial idea). That is the way most generating functions work. $\endgroup$ – AGEscovar Mar 21 '16 at 19:45

It wasn't. After a recalculus I've realized that if you take the derivative of $G(u)$:

$$\dfrac{dG(u)}{du} = \sum _{n\in N} p^u(n)\log(p(n))$$

using that the derivative of $a^x$ is $a^x \log(a)$. Now, taking $u=1$ in the above expression, you find the Shannon entropy, so

$$S=\left [\dfrac{dG(u)}{du} \right ]_{u=1}$$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.