0
$\begingroup$

I have already asked this question in math.SX but here might be more proper. So I decided to put a copy here and delete the one which is not the one that got an answer:

I know that for one dimensional differential entropy of a density function $p(x)$, one has the following formula by change of variables:

$$H(p)=\int p(x)\log(p(x))dx=\int\limits_{0}^{1}\frac{d}{dp}[\log F^{-1}(p)]dp$$.

Where $F$ is the cumulative distribution function of $p$. To extend the idea for multivariate density functions my idea was to try to integrate on level sets of $p(x)$ to simulate somehow what happens in one dimension.

As a results I took a look at this question and topics relevant to pullback method in differential geometry and co-area formulation but couldn't come up with anything by myself. Does anybody know of any method to get a similar formula for differential entropy when $p$ is a multivariable density function?

EDIT: As it has been clarified by "NowIGetToLearnWhatAHeadIs" CDF does not have a unique generalisation. What I thought to be the natural one is:

$$F(x_1,...,x_n)=\int\limits_{-\infty}^{x_1}\int\limits_{-\infty}^{x_2}...\int\limits_{-\infty}^{x_n}p(x_1,...,x_n)dx_ndx_{n-1}...d_2d_1$$

$\endgroup$

1 Answer 1

1
$\begingroup$

Its unclear to me what you are actually asking. There isn't a generalization of cumulative distribution function to higher dimensions that I know of.

Here is one thing I thought of though. I am not sure if you already found this formula or if this is something you are looking for.

So we have some measure space $X$ with pdf $p$. Then we can define a function $\lambda$ by

$\lambda(p) = \mu(\{x \in X | p(x) < p\}).$

Then we obtain the expression for the entropy $-H = \int p \log p\, \, dx = \int_0^\infty p \log p \lambda'(p) d p .$

Basically this just says that you can look at all the places in $X$ where $p(x)$ is between $p_0$ and $p_0 + dp$. Each of these places gives a contribution of $p_0 \log p_0$ to the integral.

The total contribution is then the contribution from each point times the measure of the set of points contributing; that is, $(p_0 \log p_0) \mu(\{x \in X | p_0 < p(x) < p_0 + dp\})=(p_0 \log p_0) (\lambda(p_0 + dp) -\lambda(p_0 ) ) =(p_0 \log p_0)\lambda'(p_0 ) dp. $

Thus the value of the integral is the sum of the contributions form each interval $(p_0, p_0+dp)$. That is, the integral has the value given above: $\int_0^\infty p \log p \lambda'(p) d p .$

$\endgroup$
1
  • $\begingroup$ Thank you very much for this interesting reply. What I see as the natural extension of CDF for high dimensions is: $$F(x_1,...,x_n)=\int\limits_{-\infty}^{x_1}\int\limits_{-\infty}^{x_2}...\int \limits_{-\infty}^{x_n}p(x_1,...,x_n)dx_ndx_{n-1}...d_2d_1$$ What seems a bit strange to me is the fact that in your formulation one cannot get the one dimensional case that I have written above; first of all, $\lambda(p)$ does not coincide with 1-dim CDF and secondly there is no log-derivative expression as it is the case in the formula that I have written. $\endgroup$
    – Cupitor
    Jun 11, 2014 at 14:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.