10
$\begingroup$

Maximum entropy is equivalent to minimum information in statistical mechanics and quantum mechanics and the universe as a whole is tending towards an equilibrium of minimum information/maximum entropy. What seems to be part of the canonical picture as well is that all the structure formation we see on earth is the result of the earth being an open system, a thermodynamic machine so to say, that is driven by the energy of the sun, and so the apparent violation of the second law of thermodynamics on earth is only an illusion that is resolved by including the rest of the universe.

But, what I have asked myself just yesterday:

is it legitimate to consider the information that is stored in the brain a part of the thermodynamic information/negentropy? Or is this a completely different level of information?

The discrepancy between these two aspects of information probably becomes more apparent if I ask the same question about the information that is stored in a computer. Because the computer is completely deterministic, and hence it seems to me that it is not valid to consider it as part of a statistical mechanical or quantum mechanical system (at least unless the computer is destroyed, or an electric discharge changes one of its storage flip-flops or something along these lines).

Or to take it yet another step further to absurdity: do the books in the US library of congress, or the art shown in the Louvre in Paris contain any substantial fraction of thermodynamic negentropy, or isn't it rather irrelevant thermodynamically if a page of a book or a painting contains a black or green or red spot of paint that is part of a letter "a" or the eye of the Mona Lisa, or if it contains no paint whatsoever?

Of course, I know that we can define Shannon's information for computer memory straightforwardly as well as we can define the entropy of the brain as a electrochemical-thermodynamic system. But my question is, are these concepts actually the same, or is one included in the other, or are they completely disjoint, complementary or whatever unrelated?

Edit: this question (thanks to Rococo) already has several answers, each of which I find pretty enlightening. However, since the bounty is now already on the table, I encourage everybody in providing his own point of view, or even statements conflicting to the ones in the linked question.

$\endgroup$
5
  • 1
    $\begingroup$ Why make the brain special? It's just a bunch of neurons firing through biochemical reactions. You want to consider the net entropy of all those reactions, i.e. life, not just one small part of it. Don't be fooled by the notion of "information entropy" into thinking that the non-brain parts of biology do not contain information. $\endgroup$ Apr 21 at 12:58
  • 3
    $\begingroup$ Similar, if not duplicate, of physics.stackexchange.com/q/263197 $\endgroup$
    – Rococo
    Apr 21 at 13:45
  • $\begingroup$ @AtmosphericPrisonEscape: I might repeat part of my comment to Vadim's answer. Does thermodynamic entropy allow to explain, why the original manuscript of Newton's principia would gain a higher bid at Sotheby's than an equivalent book with empty pages (or, say, one with random black dots in it). Or is this only due to human ignorance which misses the many precious secrets captured between the lignin fibers of all these pages? (No offence in this irony, just to make the contrast sharper) $\endgroup$
    – oliver
    Apr 21 at 19:21
  • 1
    $\begingroup$ @Rococo: thanks for the link. Indeed this is almost the same question differently phrased. Very interesting answers. I tend most to knzhou's answer, which, to my interpretation, implies that Boltzmann entropy deals with the big changes in probability and ignores the tiny ones (which is why calorimetric measurements yield the same for a printed book as for an empty book), while Shannon entropy kind of "subtracts the background" Boltzmann entropy and only considers the tiny probability changes due to Shannon entropy (which is also rescaled because of abstraction from paint dots to letters). $\endgroup$
    – oliver
    Apr 21 at 20:05
  • $\begingroup$ @oliver: You might want to look at the entropy of an anti-ferromagnet at 0K ... $\endgroup$ Apr 21 at 22:49
2
+50
$\begingroup$

in short:

Yes, information and negentropy are measuring the same thing, and can be directly compared and just differ by a constant scale factor.

But this introduces a problem when talking about the valuable information in a book, brain, or computer, because the valuable information is overwhelmed by the relatively numerically gargantuan entropy of the arrangement of the mass that is storing that information.  This problem, though, is often easily solved by asking information questions that are phrased in a way to select the information of interest.   So in a computer, where this separation is more obvious, for example, it's important to differentiate between the entropy of a transistor (a crystalline structure with very high information) and the information contained in its logical state (a much much lower information, but usually more interesting).

Therefore, the question ends up being only, do we understand the system well enough to determine what information we are interested in? Once we know that, it's usually possible to estimate it.

are negentropy and information measuring the same thing?:

Yes. This is spelled out very clearly by many people, including this answer on PSE, but I'll go with an old book by Brilluoin, Science and Information Theory (i.e, this is the Brilluoin of Brilluoin Zones, etc, and also the person who coined the term "negentropy").

The essential point is to show that any observation or experiment made on a physical system automatically results in an increase of the entropy of the laboratory. It is then possible to compare the loss of negentropy (increase of entropy) with the amount of information obtained. The efficiency of an experiment can be defined as the ratio of information obtained to the associated increase in entropy.

information vs valuable information:

Brilluoin also distinguishes between "information" and "valuable information", and says that a priori there's no mathematical way to distinguish these, although in certain cases we can define what  we consider to be the valuable information, and in those cases we can calculate it. 

We completely ignore the human value of the information. A selection of 100 letters is given a certain information value, and we do not investigate whether it makes sense in English, and, if so, whether the meaning" of the sentence is of any practical importance. According to our definition, a set of 100 letters selected at random (according to the rules of Table 1.1), a sentence of 100 letters from a newspaper, a piece of Shakespeare or a theorem of Einstein are given exactly the same information value. In other words, we define “information” as distinct from “knowledge,” for which we have no numerical measure. We make no distinction between useful and useless information, and we choose to ignore completely the value of the information. Our statistical definition of information is based only on scarcity. If a situation is scarce, it contains information. Whether this information is valuable or worthless does not concern us. The idea of “value” refers to the possible use by a living observer.

So then, of course, to address the information in the Pricipia, the question is to separate information from valuable information, and note that a similar book with the same letters in a specific but random sequence will have the same information, but different valuable information.

In his book, Brilluoin provides many ordinary examples, but also computes some broader and more interesting examples that are closely related to some subtopics of this questions. Instead of a picture (as the OP's question suggests), Brilluoin constructs a way to quantify what is the information of a schematic diagram.  Instead of a physics text (as the OP's question suggests), he analyses a physical law (in the case, the ideal gas law), and also calculates its information content.  It's not a surprise that these information values are swamped by the non-valuable information in the physical material in which they are embodied.

a specific case, the brain:

Of the three topics brought up by the question the most interesting one to me is the brain.  Here, asking what is the information in the brain, creates a similar ambiguity as for a computer, "are you talking about the crystalline transistors or are you talking about their voltage state?"  But in the brain it is more complex for various reasons, but the most difficult to sort out seems to be that there is not a clear distinction between structure and state and valuable information.

One case where it's clear how to sort this out, is the information in spikes within neurons.  Without giving a full summary of neuroscience, I'll say that neurons can transmit information via voltages that appear across their membranes, and these voltages can fluctuate in a continuous way or exist as discrete events called "spikes".  The spikes are the easiest to quantify their information.  At least for afferent stimulus-encoding neurons where people can make a reasonable guess what stimulus they are encoding, it's often possible to quantify bits/spike, and it is usually found to be 0.1 to 6 bits/spike, depending on the neuron (but there's obviously some pre-selection of the neurons going on here).  There is an excellent book on this topic titled Spikes, by Fred Rieke, et al, although a lot of work has been done since its publication. 

That is,  given a model of 1) what's being being encoded (eg, aspects of the stimuli), and 2) what is the physical mechanism for encoding that information (eg, spikes), it's fairly easy to quantify the information. 

Using a similar program it's possible to quantify the information stored in a synapse, and in continuous voltage variations, although there's less work on these topics.  To find these, search for things like "Shannon information synapse", etc.  It seems to me not hard to imagine a program that continues along this path, and it if it were to scale to the large enough size, could eventually estimate information in the brain from these processes.  But this will only work for the processes that we understand well enough to ask the questions that get at the information we are interested in.

$\endgroup$
7
  • $\begingroup$ The distinction between information and "valuable information" is a very interesting point of view. This makes me think that the core of the problem is not really about how the information is physically represented (book/brain/"bunch of transistors"), but rather the process of information retrieval. Do you know if information science has a theoretical framework for this? I see connections to databases and cryptography. What looks like invaluable information as a cyphertext is actually valuable information as a clear text. Could reading a book be considered as a process of "decryption"? $\endgroup$
    – oliver
    Apr 25 at 5:43
  • $\begingroup$ BTW, a minor correction: the "the Brilliuon of Brilliuon Zones" is actually spelled "Brillouin". $\endgroup$
    – oliver
    Apr 25 at 5:45
  • $\begingroup$ In my above comment "invaluable" must be replaced by "worthless". $\endgroup$
    – oliver
    Apr 25 at 5:49
  • $\begingroup$ @oliver: 1) Information theory is more about a means of quantification than a means of discernment of value. The discernment must be done through knowledge about the system, and the system in relation to other systems. 2) Yes, often for information someone would consider valuable, its information content is independent of the medium. 3) I don't think retrieval is central in information theory, beyond the obvious encode/decode questions. 4) The next level key info theory I left out in this discussion are more around "mutual information", changes in distributions, etc. $\endgroup$
    – tom10
    Apr 25 at 15:44
  • $\begingroup$ I think I will read up some more about mutual information. My preliminary review of the corresponding wikipedia article looks like there is some insight to what knowledge is, theoretically. $\endgroup$
    – oliver
    Apr 25 at 17:32
1
$\begingroup$

There are many different ways to define information of the books in a library - depending on whether one views them as physical paper objects, or as the collections of letters/words or as something else.

If we focus on the information stored in the text, i.e., the specific arrangement of letters and words, then it can be viewed on two levels:

  • First of all, the arrangement of letters and words is not random, i.e., it is clearly not in the state of maximum entropy. As an amusing manifestation of this one could compare the frequency of letters used in a text written in English and another language using the same (or at least overlapping) alphabet - they are manifestly different, and in fact often serve as the first step in text recognition algorithms. Obviously, the real text is by far more complex than statistics of letters and words - since the letters and the words are correlated themselves via complex rules (Shannon's paper presents early discussion of such analysis, but there are more complex rules studied in mathematical linguistics under such terms as transformational grammar, generative grammar, etc.)
  • There is however even more to the texts than the statistics of symbols - their correlation with other statistical distributions, e.g., with the words, sentences and other data stored in a head of a person reading a book. Clearly, a monolingual Spanish speaker can get more of a book written in this language than a monolingual English speaker, even though the letter still gets more than a monolingual speaker of Chinese. In the information science such correlations are measured by Mutual information - i.e., the information measured in respect to a specific background/context.

From such a textual point of view, the loss of the information due to a spot or a missing page in a book depends on how much it distorts the statistics of the symbols in this book, and the correlations between them. Sometimes a blacked-out word can be easily guessed from the context, while in other cases it may render a whole phrase or a paragraph incomprehensible.

$\endgroup$
3
  • $\begingroup$ Thanks, Vadim. Although this is a nice recap of what entropy/information means in the context of information science, I am interested specifically in the question whether the thermodynamical negentropy (either the one you compute from statistical mechanics or from statistical quantum mechanics) contains the information in the sense of information science, or if these are different concepts applied in different contexts. $\endgroup$
    – oliver
    Apr 21 at 18:54
  • $\begingroup$ @oliver perhaps you could clarify your question: as is, it is not exactly clear what you are asking, and how specialized the answer should be. Knowledgeable people might miss it (I cannot really add much more beyond of what I have already written above.) $\endgroup$ Apr 21 at 18:59
  • $\begingroup$ I am unsure about what is unclear about the question. Maybe you can help me see it. E.g. why did you think this was only about shannon entropy and not also thermodynamic entropy? I think we agree that the original manuscript of Newton's principia would be paid much higher at Sotheby's than a similar book with all pages erased? While this value could probably be measured at least approximately by Shannon entropy, I can hardly imagine how thermodynamic entropy could do the same job, given that it only deals with temperature, pressure, particle types and so on... $\endgroup$
    – oliver
    Apr 21 at 19:13
1
$\begingroup$

Is it legitimate to consider the information stored in the brain as part of thermodynamic entropy?

Insofar as the brain itself is physical then we can.

However, the mind itself is not understood in this way, for example, it's not even clear what consciousness is. Nor is it clear what can be meant by information stored in the mind. Hence, it's better to steer away from thinking of mind by physical concepts.

$\endgroup$
1
  • $\begingroup$ I agree that the brain is not the best possible example for what I meant. However the question about the book does not suffer from these difficulties. $\endgroup$
    – oliver
    Apr 23 at 12:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.