0
$\begingroup$

I am trying to solve a Lagrange multiplier problem for the following equation

$$ L= - \int_{-\infty}^\infty \rho(x) \ln \frac{\rho(x)}{q(x)} dx + \alpha \left( 1- \int_{-\infty}^\infty \rho(x) dx \right) +\beta \left( \overline{O} - \int_{-\infty}^\infty O(x) \rho(x) dx \right) \tag{1} $$

for $\frac{\partial L}{\partial \rho(w)} =0$.

Where the first term is the relative entropy, the second term is the constraint that it sums to one, and the last term is the constraint of an observable involving its average value. The problem solves for the probability measure that maximizes the relative entropy of a continuous parametrization x.


An interesting property of the relative entropy is that its equations remains invariant with respect to a change in variable. Indeed,

$$ \int_{-\infty}^\infty \rho(x)\ln \frac{\rho(x)}{q(x)} dx \to \int_{-\infty}^\infty \rho(y(x)) \left|\frac{\partial y}{\partial x}\right|\ln \frac{\rho(y(x))\left|\frac{\partial y}{\partial x}\right|}{q(y(x)) \left|\frac{\partial y}{\partial x}\right|} dx = \int_{-\infty}^\infty \rho(y)\ln \frac{\rho(y)}{q(y)} dy \tag{2} $$

and

$$ \int_{-\infty}^\infty \rho(x)dx \to \int_{-\infty}^\infty \rho(y(x)) \left|\frac{\partial y}{\partial x}\right| dx = \int_{-\infty}^\infty \rho(y) dy \tag{3} $$


However, the last term isn't. Indeed:

$$ \int_{-\infty}^\infty O(x) \rho (x) dx \to \int_{-\infty}^\infty O(y(x)) \left|\frac{\partial y}{\partial x}\right| \rho (y(x)) \left|\frac{\partial y}{\partial x}\right| dx = \int_{-\infty}^\infty O(y) \rho (y(x)) \left|\frac{\partial y}{\partial x}\right| dy \tag{4} $$


How can I modify the integral that contains O so that it is coordinate invariant?

What is the expression for the average of a function O(x) over a continuous parametrization x, such that the average value is coordinate invariant?

$\endgroup$
3
  • $\begingroup$ crossposted: math.stackexchange.com/questions/4657570/… $\endgroup$
    – Anon21
    Mar 12, 2023 at 16:38
  • $\begingroup$ Why do you add an extra factor $|dy/dx|$ when you go from $O(x)$ to $O(y)$? This is the cause of absence of invariance, and I don't think it should be there $\endgroup$
    – LPZ
    Mar 12, 2023 at 17:26
  • $\begingroup$ @lpz If I change $O(x) \to O(y(x) |dy/dx| $ and $\rho(x) \to \rho(y(x)) |dy/dx|$, I assumed changing both entailed $O(y(x)) |dy/dx| \rho(y(x)) |dy/dx| $? Comparatively, I have to multiply each change of variable on the relative entropy (equation 2) for it to be invariant. This is explained here: en.wikipedia.org/wiki/… $\endgroup$
    – Anon21
    Mar 12, 2023 at 17:29

1 Answer 1

2
$\begingroup$

lpz is correct that you are including an erroneous factor of $|dy/dx|$ in your transformation of $O(x)$. You may be getting misled by the transformation of the density, which uses the same symbol for the density of both $x$ and $y$ in your notation. Your conclusion in the comments that all functions must transform the same way under a change of variables is incorrect.

The transformation is more accurately written $$\rho_X(x) = \rho_Y(y) \left|\frac{dy}{dx}\right|$$ where $\rho_X(x)$ is the probability density of $x$ and $\rho_Y(y)$ is the probability density of $y$; these two densities are different functions. They are related here under the assumption that we may view $x$ as a function of $y$ or vice versa. The derivative here comes from the Jacobian of the integral under the change of variables, which is then combined with $\rho_X(x)$ to define $\rho_y(y)$.

The transformation of $\int dx~O(x) \rho_x(x)$ should therefore be $$\int dx~O(x) \rho_x(y) = \int dy \left|\frac{dx}{dy}\right| O(x(y)) \rho_X(x(y)) = \int dy \frac{1}{\left|\frac{dy}{dx}\right|} O(x(y)) \rho_X(x(y)) = \int dy~ O'(y) \rho_Y(y),$$ where I used the fact that $dx/dy = 1/(dy/dx)$ and defined $O'(y) = O(x(y))$, with $x(y)$ is the variable $x$ interpreted as a function of $y$. Note that $O(x)$ and $O'(y)$ will not have the same shape unless $x(y) = y$, and this is similar to the fact that $\rho_X(x)$ and $\rho_Y(y)$ are not the same function.

The values of the integrals $\int dx~O(x) \rho_X(x)$ and $\int dy~O'(y) \rho_Y(y)$ will be the same.

$\endgroup$
8
  • $\begingroup$ I have to confess I am a bit sceptical. The link en.wikipedia.org/wiki/… shows that a quotient is required to eliminate |dy/dx| in the relative entropy. It then states that without this elimination, the relative entropy would not be coordinate invariant. From your suggestion, they could have easily claimed the differential entropy is coordinate invariant $\int \rho(x) \ln \rho(x) dx \to \int \tilde{\rho}(y(x)) |dy/dx| \ln \tilde{\rho}(y(x)) |dy/dx| dx = \int \tilde{\rho}(y) \ln \tilde{\rho}'(y) dy$. But they do not. Can you clarify? $\endgroup$
    – Anon21
    Mar 18, 2023 at 22:04
  • $\begingroup$ As an alternative, I was leaning towards $ \overline{O/A} = \int \rho(x) \frac{O(x)}{A(x)} dx$, where it is the ratio between two observables that is invariant. Thoughts? $\endgroup$
    – Anon21
    Mar 18, 2023 at 22:09
  • $\begingroup$ The differential entropy is not coordinate invariant because if you evaluate the integrals $\int dx~\rho_X(x) \ln \rho_X(x)$ and $\int dy~\rho_Y(y) \ln \rho_Y(y)$ you would find they have different values (because you're missing the Jacobian factor inside the ln). In the $D_{KL}$, $\int dx~\ln(p_X(x)/q_X(x))$ has the same value of $\int dy~\ln(p_Y(y)/q_Y(y))$ when both $p$'s and $q$'s are interpreted as probability densities, even though $p_X(x) \neq p_Y(y)$. If you want $\int dx~O(x) \ln \rho_X(x) = \int dy~O(y) \rho_Y(y)$ for the same O on each side but different rho, then ... $\endgroup$
    – bbrink
    Mar 19, 2023 at 21:24
  • $\begingroup$ you are going to be restricted to a limited class of observables. For example, any $O$ that is a function of ratios of quantities that transform like probability densities would work, $O(x) = f\left(p_X(x)/q_X(x)\right)$, but then your observable is just of the form of an F-divergence in that case. Perhaps in a multivariate setting you could restrict to functions satisfying certain symmetries you could have more options. It might help if you clarify why you want your Lagrange constraint to be coordinate-invariant. $\endgroup$
    – bbrink
    Mar 19, 2023 at 21:29
  • 1
    $\begingroup$ For Eq. (2) it might help if you do the derivation a different way, again by a change of variables: $\int dx p_X(x) \ln\left(\frac{p_X(x)}{q_X(x)}\right) = \int dy f'(y) p_X(f(y)) \ln \left(\frac{p_X(f(y))}{q_X(f(y))}\right) = \int dy~ \left\{f'(y) p_X(f(y)) \right\} \ln \left(\frac{p_X(f(y))f'(y)}{q_X(f(y))f'(y)}\right) = \int dy~ p_Y(y) \ln \left( \frac{p_Y(y)}{q_Y(y)}\right)$, where $p_Y(y) \equiv p_X(f(y)) f'(y)$ and $q_Y(y) \equiv q_X(f(y)) f'(y)$. Here, I introduced $1 = f'(y)/f'(y)$ in the logarithm so that I could combine them with $p_X$ and $q_X$ to form proper probability densities. $\endgroup$
    – bbrink
    Mar 21, 2023 at 15:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.