All Questions

Filter by
Sorted by
Tagged with
0
votes
0answers
23 views

Proof of continuous Shannon entropy

In Jaynes' Probability Theory: The Logic of Science, there is a quick derivation of continuous entropy in chapter 12. He does so by taking the discrete definition of entropy and combines it with ...
2
votes
0answers
29 views

Fisher Information in Statistical Mechanics

I am studying the canonical ensemble and it seems to me there is an analogy between derivatives of the partition function, which can extract energy momenta for the system and Fisher score /information....
0
votes
0answers
25 views

Maximizing mutual information with Restricted Boltzmann Machines and Monte-Carlo sampling

So I've been reading through Koch-Janusz and Ringel's, "Mutual Information, Neural Networks, and the Renormalization Group" (check it out here). I'm currently trying to reimplement some results from ...
0
votes
1answer
59 views

Is physical entropy opposite to information entropy? [duplicate]

I am currently taking a class on Decision Theory where I was introduced to the notion of "entropy" in an information theoretic setting while we were studying decision trees. I was given the following ...
2
votes
2answers
84 views

E. T. Jaynes' subjectivism vs measurement of distributions

In his paper, E. T. Jaynes argues that entropy is a measure of our ignorance about a system. As such, the probability distribution of states $\{p_k\}$ has to be chosen in the most unbiased way, thus ...
1
vote
1answer
107 views

What is the physical interpretation of the action integral, without the stationary action principle?

I'm still wondering about the physical interpretation of the action integral of some mechanical system (classical theory here, to simplify things): \begin{equation}\tag{1} A = \int_{t_1}^{t_2} L(q, \, ...
3
votes
0answers
58 views

Is it necessary to assume that equivalent microstates cannot get transformed into inequivalent microstates to derive the Landauer principle?

The Landauer erasure principle states that to erase a bit of information from a system, the entropy of the environment will be increased by at least $k_B\log2$, or equivalently, it costs at least an ...
1
vote
2answers
89 views

How to handle bra-ket in logarithm?

$\newcommand{\ket}[1]{\left| #1 \right>}$ $\newcommand{\bra}[1]{\left< #1 \right|}$ Say the following two equations: $$ S = - k_B \text{Tr} (\rho \ln \rho) $$ $$ \rho = \sum _\epsilon \ket{\...
0
votes
1answer
62 views

Information to Energy equations?

Are there are any known formulas or equations that can calculate information from energy or energy from information? One of them, I believe, is the Bekenstein bound which is the maximum information ...
3
votes
2answers
72 views

Why is it necessary to irreversibly erase a memory?

I know that the most accepted resolution of the Maxwell's demon paradox was proposed by Landauer and revolves around the fact that the demon's memory is finite and will have to be erased at some point....
2
votes
5answers
222 views

How does the lack of information increase as temperature increases?

Suppose one knows nothing about the concept of entropy. How can we argue that the lack of information/ignorance about the system typically increases with the increase in the temperature using the ...
4
votes
0answers
146 views

What are some good articles on trend to equilibrium?

I am interested in studying systems out of equilibrium that are trending to equilibrium. Trend to equilibrium, entropy production, etc. seem to be very tricky topics. Any suggestions will be ...
3
votes
1answer
216 views

Entropy: subjective lack of knowledge that leads to objective conclusions

There is something I really don't get about entropy. Let's consider a classical system (not quantum mechanics here). We can compute the entropy of a system via the formula $$S=-\sum_l P_l Log(P_l)$$ ...
1
vote
0answers
41 views

Proving that Shannon entropy is maximal for the uniform distribution using convexity

I need to show that $-\sum_i{p_i \log{p_i}}$ is maximal iff $p_i=p_j$ for all $i\neq j$ using the convexity inequality: $\phi (\frac{\sum{a_i}}{N})\leq \frac{\sum{\phi (a_i)}}{N}$ I tried expanding ...
4
votes
2answers
431 views

Deriving the Boltzmann distribution using the information entropy

I was going through my lecture notes and I found something I could not quite understand. First, it starts by deriving an expression for the information entropy (as used in physics?): Let $p_i$ be ...
3
votes
0answers
191 views

Path integrals, Ensembles, and Information theory in QFT and statistical mechanics (POV of a mathematician)

I apologize in advance for any stupid/wrong/enraging remarks in the question. I am a mathematician and not a physicist. Consequently this question was written entirely from the perspective of a ...
4
votes
2answers
472 views

Connection between Different Kinds of Entropy (Boltzmann, Volume, and Surface Entropies)

On Wikipedia for Microcanonical Ensemble it says, $ \omega $ is said to be an energy width. I am not entirely sure what that means, but I believe it means that it is a variable stuck in to annihilate ...
5
votes
0answers
169 views

Is it possible to derive Liouville's Theorem purely from maximum differential entropy?

Typically in physics (at least the way I learned mechanics), this is derived using the multi-dimensional divergence theorem on the $2N$-dimensional phase space i.e. $0=\partial_t \rho + \sum\limits_{...
8
votes
1answer
585 views

Maxwell-Boltzmann Distribution (speed) as a Maximum Entropy Distribution and Its Interpretation

I am aware of the typical "physics" way of deriving the Maxwell-Boltzmann Distribution for speed $v$: $p(v) = \sqrt{\left(\frac{m}{2\pi k_B T}\right)^3} 4\pi v^2 \exp\left(-\frac{mv^2}{2k_BT}\right)$ ...
-2
votes
1answer
73 views

“Computing force” in the universe: What? Where? [closed]

When we ask a child what $2 \times 3$ is, we expect him/her to spend time thinking and only give us an answer after some energy has been spent solving the question. When we swipe in an app on our ...
1
vote
0answers
37 views

Interpretation of quantifying information after quantum measurement

Given initial quantum state $\rho$ of a system and $\tilde{\rho}_n$ one of the possible final states. We then define the Groenwold information after measurement as $$I := \langle \Delta S \rangle = S(\...
1
vote
0answers
44 views

How Fast Can (Large) Semiclassical Black Holes Evaporate?

In "Entropy in Black Hole Pair Production" (arXiv:gr-qc/9306023), Strominger et al. notes The issue of whether (1.2) can be taken literally has bearing on the vexing question of what happens to ...
14
votes
0answers
252 views

Renormalisation and the Fisher-Rao metric

The renormalisation group (I'm talking about classical, statistical physics here, I'm not familiar with field theory too much) can be though of as a flux in a space of possible Hamiltonians for a ...
48
votes
1answer
4k views

Does entropy depend on the observer?

Entropy as it is explained on this site is a Lorentz invariant. But, we can define it as a measure of information hidden from an observer in a physical system. In that sense, is entropy a relative ...
-3
votes
1answer
144 views

Why does a liter water contain more information than all the information on the whole internet? [closed]

In this (probably well known) picture you can see that a litre water contains much more information than all the information on the whole internet. Now I understand that the total number of possible ...
-3
votes
1answer
87 views

When my body is decomposed into all it's elementary particles, do these particles possess more information than my whole body?

If the elementary particles out of which my body is built up are all disconnected, do these particles contain more information than when they were part of my body? In other words, is the number of (...
0
votes
1answer
329 views

How is “a bit of information” defined in physics?

I keep encountering the phrase a bit of information but I don't know precisely what this means. I have an intuition of a system that can be in 1 of 2 possible states (in the classical realm that is) ...
4
votes
1answer
252 views

How to compute entropy of networks? (Boltzmann microstates and Shannon entropy) [closed]

I also asked in SO here a few days ago, thought it may be also interesting for physics-related answers. I would like to model a network as a system. A particular topology (configuration of edges ...
1
vote
1answer
176 views

Shannon Entropy (Information Entropy) [closed]

Assuming $10^{10}$ gas particles are inside a volume of $1~\mathrm m^3$. How much bits does it takes to store the initial conditions? My approach: We know Shannon Entropy is given by, $$S_\textrm{...
2
votes
1answer
403 views

Driven harmonic oscillator with thermal Langevin force. How to extract temperature from $x(t)$?

Suppose you have driven harmonic oscillator (parameters: mass,gamma,omega0) by a deterministic force Fdrive (a sine wave say). Now suppose that you add stochastic Langevin force FL which is related to ...
1
vote
0answers
28 views

Is Boltzmann's H the information or the negative of information?

I noticed that Boltzmann's H is defined as $$H_\mathrm{Boltzmann} = +\left<\ln p\right>$$ (where brackets indicate the expectation value) while the usual definition of $$H_\mathrm{...
2
votes
1answer
156 views

How can “information” be a useful physical quantity given that its value is model-dependent?

From @Humble's answer to "What is information?": Information contained in a physical system = the number of yes/no questions you need to get answered to fully specify the system. That is, however, ...
5
votes
2answers
328 views

What is the black hole information paradox really? [closed]

Preliminaries What is the black hole information paradox really? Is it a sophisticated way to ponder and debate the existence of an operator on the boundary that can tease out the interior of a ...
5
votes
2answers
667 views

Books on entropy [closed]

What books introduce entropy in a intuitive, elementary way (at most, for a person with undergraduate physics studies)? The book should not necessarily introduce entropy in relation only to ...
1
vote
1answer
122 views

Does conservation of information mean that the direction of causality is arbitrary? [duplicate]

If it is the case that the information content of the universe is conserved, and the past can be constructed from a complete knowledge of the future just as easily as vice versa, then is there any ...
1
vote
0answers
162 views

What are some good books on foundations of Thermodynamics [duplicate]

I am not looking for some introductory texts, graduate text, etc. I am looking for something that addresses to foundational problems in thermodynamics and statistical mechanics.
1
vote
1answer
112 views

Reference for statistical mechanics from information theoretic view

I am interested in knowing if some one here knows book/notes for statistical mechanics from the information theoretic viewpoint.
40
votes
6answers
5k views

Is there a thermodynamic limit on how efficiently you can solve a Rubik's cube?

Suppose I build a machine which will be given Rubik's cubes that have been scrambled to one of the $\sim 2^{65}$ possible positions of the cube, chosen uniformly at random. Is it possible for the ...
9
votes
4answers
3k views

Exorcism of Maxwell's Demon

I am possessed! Yes, with the thinking that if there is actually a Maxwell's Demon, then it would open the negligible weighted door which would ultimately make the second law invalid. But really can ...
4
votes
4answers
520 views

Why the self-information is $-\log(p(m))$?

Why is self-information given by $-\log(p(m))$? Shannon derived a measure of information content called the self-information or "surprisal" of a message $m$: $$I(m) = \log \left( \frac{1}{p(...
6
votes
2answers
368 views

How can the microstates be measured with zero energy expenditure?

James P. Sethna. Statistical Mechanics. Exercise 5.2: What prevents a Maxwellian demon from using an atom in an unknown state to extract work? The demon must first measure which side of the ...
5
votes
2answers
255 views

Definition of Information in Information Theory

I am not sure in which SE site I have to put this question. But since I have learnt Shannon Entropy in the context of Statistical Physics, I am putting this question here. In the case of Shannon ...
5
votes
1answer
2k views

Should entropy have units and temperature in terms of energy? [duplicate]

I've been thinking about entropy for a while and why it is a confusing concept and many references are filled with varying descriptions of something that is a statistical probability (arrows of time, ...
5
votes
0answers
1k views

An explanation for the Landauer's principle

Has anyone understood the Landauer's principle? What is the current status? In specific, is there a theoretical derivation of the Landauer's Principle?(not the heuristic one based on Salizard's ...
1
vote
1answer
461 views

Do laws of thermodynamics have a place in Theory of Everything? [closed]

I am having a difficulty understanding why second law of thermodynamics is still a valid universally accepted concept. I understand it works on paper for describing isolated heat systems. However, I ...
5
votes
2answers
383 views

Entropy: two explanations for the same quantity?

I studied thermodynamics and I saw the following definition for entropy: $$ \Delta S = \int_1^2 \frac{\text{d}Q}{T} $$ that we use to calculate $\Delta S$ for different types of transformations. In ...
14
votes
3answers
13k views

What is the relationship between Energy, Entropy, and Information?

What is the relationship between Energy, Entropy, and Information? I read this - What Is Energy? Where did it come from? - and the top answer says that 'energy' is an abstract number that is a ...
3
votes
1answer
605 views

Thermodynamics and cross entropy

I am facing with the concept of cross entropy. I would like to know the thermodynamic and statistical meaning of cross entropy (if exists)?
25
votes
3answers
3k views

Ignorance in statistical mechanics

Consider this penny on my desc. It is a particular piece of metal, well described by statistical mechanics, which assigns to it a state, namely the density matrix $\rho_0=\frac{1}{Z}e^{-\beta H}$ (...
8
votes
1answer
535 views

What is the information geometry of 1D Ising model for a complex magnetic field?

Consider the one-dimensional Ising model with constant magnetic field and node-dependent interaction on a finite lattice, given by $$H(\sigma) = -\sum_{i = 1}^N J_i\sigma_i\sigma_{i + 1} - h\sum_{i = ...