# Does the act of organizing information (e.g. categorization) reduce entropy?

I am fascinated by the relationship between entropy and life. From the Wikipedia article of that name, to the science fiction series "Three Body Problem" characterizing human-like lifeforms across the universe as simply 'low entropy beings'. However, entropy can be a tricky concept to grasp my mind around (I am an earth scientist, not pure physicist, by training). So I come to ask: does the act of organizing information (e.g. categorizing or identifying patterns) reduce entropy?

My understand is yes, because 'organizing information' is creating information about information, and the creation of information (or 'knowledge'?) reduces the randomness of (or at least uncertainty about) the states of things.

This question could seem ambiguous or nonsensical since entropy is used in a thermodynamic sense and an information science sense, and I'm asking about changes in information but am not necessarily asking about information entropy only. Citing sci-fi might not help, but I think this has a sound basis even if it's far out: consider the concept of humanoids being 'low entropy beings'. Humans in particular and life in general does seem to reduce entropy locally, at the least by simply keeping a warm body and therefore working to prevent the entropy of one's body from increasing. In that context - thermodynamic entropy in the physics of life - does the act of gaining knowledge (organizing information) reduce entropy? Again I'd think yes, in a simplest sense because when I have information organized I can spend more of my energy on actually doing work efficiently.

Sensing and learning information clearly reduces entropy in the observer, who pays a corresponding or bigger physical entropy cost to perform these actions.

Imagine a system (a robot, human, alien) observing another system (a rock, say). This paper shows that the sensor device cannot be at equilibrium in order to detect things. Indeed, it needs to produce some entropy to work, because otherwise it will just fluctuate uselessly. Now, the information gained by this process may reduce entropy in the memory of the observer - a probability estimate representing whether there is a rock there or not goes from a uniform distribution to a tighter distribution (or, you can imagine a discrete RockIsThere bit being set to a 0 or 1 value, for a more rigid design). Having a memory can also increase the ability of the sensor to acquire information (with some trade-offs).

So when the observer knows something, it has reduced the entropy of its internal probability distributions (epistemic entropy, what Jaynes was talking about), or we can say it has literally physically reduced the entropy of the RockIsThere bit by setting it to a definite value. In the later case there was the Landauer cost $k_B T \ln(2)$ J that had to be paid in the form of waste heat - the local entropy reduction had to be balanced with increasing global entropy. I suspect one can show that even the epistemic entropy reduction calculation will require a Landauer cost, since it erases past uncertain knowledge with more certain knowledge, but things can get messy if one has extra memory to store old results in (essentially it acts as a cold heat bath one can dissipate entropy into "for free" until it fills up).

Categorisation is trickier to analyse. It basically corresponds to searching for short descriptions that efficiently describe large amounts of data. It turns out that one can erase unwanted information more cheaply if one has knowledge of its internal structure, so the observer that figures out that X,Y,Z are actually one thing A can now (in principle) free up memory at a lower cost than was originally used to write X,Y,Z into their memory - the entropy "win" will be this compression efficiency minus the entropy cost of coming up with A (which might be small if the observer is a reversible computer; for such an agent interacting with the world is the main entropy cost).

I would not say organising information decreases entropy in the large - there is a lot of computation and writing/erasing memory going on - but locally inside the organising agent memory it seems like it does.

• Interesting points, and helpful analogies (e.g. memory serving as a cold heat bath we can store heat loss in until it 'fills up' and is just dissipating heat to the universe). It is quite complicated and I don't think I understand the "cost of coming up with A" well enough to know: does categorizing information have a net reduction on the universe's entropy? It is safe to say it reduces entropy for the agent of knowledge, but what is it doing to the universe? Is it possible that low-entropy beings such as ourselves can counteract the universe's tendency to greater entropy, or do we hasten it? – cr0 Mar 21 '18 at 13:51
• I will follow up with your links and see if I can understand the entropic cost of gaining knowledge a little better, to understand how our gains of information come with trade-offs of universal increases in entropy. It makes sense that there must be such tradeoffs, but that's about where my intuition ends for now. – cr0 Mar 21 '18 at 13:53
• @cro - Yes, this is confusing. Note that if I spend a lot of energy and entropy coming up with A, I can also communicate this to you, saving you the effort. You can now compress XYZ, gaining a bonus reduction in entropy. – Anders Sandberg Mar 21 '18 at 15:57
• Right, communication allowing for cheap spread of knowledge is an important part of this. David Deutsch discusses a lot of the 'physics of knowledge' if I can call it that, and he argues that once knowledge (information, that the bit is 1 or 0 for example or that XYZ are really just A) is gained, it has a tendency to never be lost and never needs to be remade again. That further suggests to me that the act of creating new knowledge could have a net reduction on entropy in the universe. Perhaps that idea is too far out to be sure of on such a broad scale. – cr0 Mar 21 '18 at 18:08
• @cro - Sure: "yes, but...". Consider humanity as a species of Maxwell's demons. By understanding physics we create machines that lowers entropy nearly arbitrarily (fridges, optical cooling), we can in principle build Dyson spheres around all stars within gigaparsecs, remodel galaxies, and run near-reversible computers made from a sizable fraction of all baryonic matter up to proton decay. That is a lot of reduced entropy! ...except that to build it we will dissipate enormous amounts of waste heat over the next trillions of years, adding to the entropy of the universe. There is no free lunch. – Anders Sandberg Mar 21 '18 at 18:53

Entropy is anthropic.

In a famous paper, E.T. Jaynes shows that entropy represents human information. To him, it is known since 1875 with Gibbs that entropy has, his terms, an "anthropomorphic nature": it is not a purely objective property of a physical system: it is a measure of the information we human have about the system that help us get work from it. We have to be able to distinguish thermodynamic degrees of freedom to make use of them.

So in a sense you are right, there is a deep link between classification and entropy: entropy as a measure needs classification. And if the classification changes, in Jaynes' paper when imaginary superkalic elements are discovered, then entropy also changes along with our capacity to make new kind of tools, superkalic pistons, and effectively use the newly acquired knowledge, while on the objective physical side of the situation, nothing changed.

When new qualitative distinctions are available for the description of a system, its entropy can be seen to decrease, because effectively we can see more order than before.

• That last sentence sums up the anthropomorphic nature of entropy nicely: its can entropy can be seen to decrease, because effectively we can see more order than before. Thus, more work is able to be done with a substance given a fixed amount of energy. – cr0 Mar 21 '18 at 13:45