What is entropy really?

Question

On this site, change in entropy is defined as the amount of energy dispersed divided by the absolute temperature. But I want to know: What is the definition of entropy? Here, entropy is defined as average heat capacity averaged over the specific temperature. But I couldn't understand that definition of entropy: $\Delta S$ = $S_\textrm{final} - S_\textrm{initial}$. What is entropy initially (is there any dispersal of energy initially)? Please give the definition of entropy and not its change.

To clarify, I'm interested in the definition of entropy in terms of temperature, not in terms of microstates, but would appreciate explanation from both perspectives.

Answer 1

There are two definitions of entropy, which physicists believe to be the same (modulo the dimensional Boltzman scaling constant) and a postulate of their sameness has so far yielded agreement between what is theoretically foretold and what is experimentally observed. There are theoretical grounds, namely most of the subject of statistical mechanics, for our believing them to be the same, but ultimately their sameness is an experimental observation.

1. (Boltzmann / Shannon): Given a thermodynamic system with a known macrostate, the entropy is the size of the document, in bits, you would need to write down to specify the system's full quantum state. Otherwise put, it is proportional to the logarithm of the number of full quantum states that could prevail and be consistent with the observed macrostate. Yet another version: it is the (negative) conditional Shannon entropy (information content) of the maximum likelihood probability distribution of the system's microstate conditioned on the knowledge of the prevailing macrostate;

2. (Clausius / Carnot): Let a quantity $\delta Q$ of heat be input to a system at temperature $T$. Then the system's entropy change is $\frac{\delta Q}{T}$. This definition requires background, not the least what we mean by temperature; the well-definedness of entropy (i.e. that it is a function of state alone so that changes are independent of path between endpoint states) follows from the definition of temperature, which is made meaningful by the following steps in reasoning: (see my answer here for details). (1) Carnot's theorem shows that all reversible heat engines working between the same two hot and cold reservoirs must work at the same efficiency, for an assertion otherwise leads to a contradiction of the postulate that heat cannot flow spontaneously from the cold to the hot reservoir. (2) Given this universality of reversible engines, we have a way to compare reservoirs: we take a "standard reservoir" and call its temperature unity, by definition. If we have a hotter reservoir, such that a reversible heat engine operating between the two yields $T$ units if work for every 1 unit of heat it dumps to the standard reservoir, then we call its temperature $T$. If we have a colder reservoir and do the same (using the standard as the hot reservoir) and find that the engine yields $T$ units of work for every 1 dumped, we call its temperature $T^{-1}$. It follows from these definitions alone that the quantity $\frac{\delta Q}{T}$ is an exact differential because $\int_a^b \frac{d\,Q}{T}$ between positions $a$ and $b$ in phase space must be independent of path (otherwise one can violate the second law). So we have this new function of state "entropy" definied to increase by the exact differential $\mathrm{d} S = \delta Q / T$ when the a system reversibly absorbs heat $\delta Q$.

As stated at the outset, it is an experimental observation that these two definitions are the same; we do need a dimensional scaling constant to apply to the quantity in definition 2 to make the two match, because the quantity in definition 2 depends on what reservoir we take to be the "standard". This scaling constant is the Boltzmann constant $k$.

When people postulate that heat flows and allowable system evolutions are governed by probabilistic mechanisms and that a system's evolution is its maximum likelihood one, i.e. when one studies statistical mechanics, the equations of classical thermodynamics are reproduced with the right interpretation of statistical parameters in terms of thermodynamic state variables. For instance, by a simple maximum likelihood argument, justified by the issues discussed in my post here one can demonstrate that an ensemble of particles with allowed energy states $E_i$ of degeneracy $g_i$ at equilibrium (maximum likelihood distribution) has the probability distribution $p_i = \mathcal{Z}^{-1}\, g_i\,\exp(-\beta\,E_i)$ where $\mathcal{Z} = \sum\limits_j g_j\,\exp(-\beta\,E_j)$, where $\beta$ is a Lagrange multiplier. The Shannon entropy of this distribution is then:

$$S = \frac{1}{\mathcal{Z}(\beta)}\,\sum\limits_i \left((\log\mathcal{Z}(\beta) + \beta\,E_i-\log g_i )\,g_i\,\exp(-\beta\,E_i)\right)\tag{1}$$

with heat energy per particle:

$$Q = \frac{1}{\mathcal{Z}(\beta)}\,\sum\limits_i \left(E_i\,g_i\,\exp(-\beta\,E_i)\right)\tag{2}$$

and:

$$\mathcal{Z}(\beta) = \sum\limits_j g_j\,\exp(-\beta\,E_j)\tag{3}$$

Now add a quantity of heat to the system so that the heat per particle rises by $\mathrm{d}Q$ and let the system settle to equilibrium again; from (2) and (3) solve for the change $\mathrm{d}\beta$ in $\beta$ needed to do this and substitute into (1) to find the entropy change arising from this heat addition. It is found that:

$$\mathrm{d} S = \beta\,\mathrm{d} Q\tag{4}$$

and so we match the two definitions of entropy if we postulate that the temperature is given by $T = \beta^{-1}$ (modulo the Boltzmann constant).

Lastly, it is good to note that there is still considerable room for ambiguity in definition 1 above aside from simple cases, e.g. an ensemble of quantum harmonic oscillators, where the quantum states are manifestly discrete and easy to calculate. Often we are forced to continuum approximations, and one then has freedom to define the coarse gaining size, i.e. the size of the discretizing volume in continuous phase space that distinguishes truly different microstates, or one must be content to deal with only relative entropies in truly continuous probability distribution models Therefore, in statistical mechanical analyses one looks for results that are weakly dependent on the exact coarse graining volume used.

Answer 2

The entropy of a system is the amount of information needed to specify the exact physical state of a system given its incomplete macroscopic specification. So, if a system can be in $\Omega$ possible states with equal probability then the number of bits needed to specify in exactly which one of these $\Omega$ states the system really is in would be $\log_{2}(\Omega)$. In conventional units we express the entropy as $S = k_\text{B}\log(\Omega)$.

Answer 3

Here's an intentionally more conceptual answer: Entropy is the smoothness of the energy distribution over some given region of space. To make that more precise, you must define the region, the type of energy (or mass-energy) considered sufficiently fluid within that region to be relevant, and the Fourier spectrum and phases of those energy types over that region.

Using relative ratios "factor out" much of this ugly messiness by focusing on differences in smoothness between two very similar regions, e.g. the same region at two points in time. This unfortunately also masks the complexity of what is really going on.

Still, smoothness remains the key defining feature of higher entropy in such comparisons. A field with a roaring campfire has lower entropy than a field with cold embers because with respect to thermal and infrared forms of energy, the live campfire creates a huge and very unsmooth peak in the middle of the field.

Answer 4

In terms of the temperature, the entropy can be defined as $$ \Delta S=\int \frac{\mathrm dQ}{T}\tag{1} $$ which, as you note, is really a change of entropy and not the entropy itself. Thus, we can write (1) as $$ S(x,T)-S(x,T_0)=\int\frac{\mathrm dQ(x,T)}{T}\tag{2} $$ But, we are free to set the zero-point of the entropy to anything we want (so as to make it convenient)¹, thus we can use $$S(x,T_0)=0$$ to obtain $$ S(x,T)=\int\frac{\mathrm dQ(x,T)}{T}\tag{3} $$ If we assume that the heat rise $\mathrm dQ$ is determined from the heat capacity, $C$, then (3) becomes $$ S(x,T)=\int\frac{C(x,T')}{T'}~\mathrm dT'\tag{4} $$

---

^{1 This is due to the perfect ordering expected at $T=0$, that is, $S(T=0)=0$, as per the third law of thermodynamics.}

Answer 5

You can set the entropy of your system under zero temperature to zero in compliance with the statistical definition $S=k_B\ln\Omega$. Then the S under other temperature should be $S=\int_0^T{\frac{dQ}{T}}$.

Answer 6

In classical thermodynamics only the change of entropy matters, $\Delta S = \displaystyle\int \frac{\mathrm dQ}{T} $. At what temperature it is put zero is arbitrary.

You have the similar situation with potential energy. One has to arbitrarily fix some point where the potential energy is put zero. This is because only differences of potential energy matters in mechanical calculations.

The concept of entropy is very abstract in thermodynamics. You have to accept the limitations of the theory you want to stick to.

By going to statistical mechanics one will get a less abstract picture of entropy in terms of the number of available states $\rho$ in some small energy interval, $S=k\ln (\rho)$. Still here we still have the arbitrary size of the small energy interval, $$ S = k\ln (\rho) = k\ln\left(\frac{\partial \Omega}{\partial E}\Delta E\right)= k\ln\left(\frac{\partial \Omega}{\partial E}\right)+ k\ln(\Delta E) $$ Here $\Omega(E)$ is the number of quantum states of the system with energy lower than $E$. The last term is somewhat arbitrary.

Answer 7

The definition of a physical concept can be a differential form but can’t be the difference of functions. $\Delta S=S_{\textrm{final}}-S_{\textrm{initial}}$ is an equation but not the definition of entropy. Thermodynamics itself now can hardly explain “what is the entropy really" , the reason please see bellow.

1.Clausius’ definition

\begin{align}\mathrm dS=\left(\frac{\delta Q}{T}\right)_\textrm{rev}\end{align}

Questions: 1) Since $\displaystyle \oint \delta Q/T\le 0$, $S$ cannot be proved to be a state function in maths, it can only depend on the reversible cycle of heat engine, this does not seem like a perfect foundation in the usual sense, and is an only exception as the definition of the state function both in mathematics and physics. As a fundamental principle, the state function changes must be independent of the path taken, why the definition of the entropy is an exception? 2) Clausius’ definition cannot explain the physical meaning of the entropy.

2. The fundamental equation of thermodynamics

\begin{align}\mathrm dS=\frac{\mathrm dU}{T}-\frac{Y~\mathrm dx}{T}-\sum_j\frac{\mu_j~\mathrm dN_j}{T}+\frac{p~\mathrm dV}{T}.\end{align}

Questions: 1) The equation includes the difference of functions, what is this difference? 2) The equation cannot explain the physical meaning of the entropy.

3) Boltzmann entropy

\begin{align}S=k\ln\Omega. \end{align}

Question: 1) $\Omega$ depend on the postulate of the equal a priori probability, but this postulate does not need to be considered in thermodynamics. In general, the postulate of the equal a priori probability cannot hold for mechanics potential energy and Gibbs free energy, a chemical reaction comes from the gradient in chemical potentials $\Delta \mu$ but not the equal a priori probability. The postulate can be applied to describe thermal motion but is not suitable for interactions.

Answer 8

First, you have to understand that Rudolf Clausius put together his ideas on entropy in order to account for the losses of energy that was apparent in the practical application of the steam engine. At the time he had no real ability to explain or calculate entropy other than to show how it changed. This is why we are stuck with a lot of theory where we look at deltas, calculus was the only mathematical machinery to develop the theory.

Ludwig Boltzmann was the first to really give entropy a firm foundation beyond simple deltas through the development of statistical mechanics. Essentially he was the first to really understand the concept of a microstate which was a vector in a multidimensional space (e.g. one with potentially infinite dimensions) that encoded all of the position and momentum information of the underlying composite particles. Since the actual information about those particles was unknown, the actual microstate could be one of many potential vectors. Entropy is simply an estimate of the number of possible vectors that actually could encode the information on the particle positions and momentums (remember, each individual vector on it own encodes the information about all the particles). In this sense entropy is a measure of our ignorance (or lack of useful information).

It is this latter use of entropy to measure our level of knowledge that led Claude Shannon to use the machinery of entropy in statistical mechanics to develop information theory. In that framework, entropy is a measure of the possible permutations and combinations a string of letters could take. Understanding information entropy is very critical to understanding the efficacy of various encryption schemes.

As far as defining Temperature in terms of entropy. These are general viewed as being distinct but related measures of the macrostate of a system. Temperature- entropy diagrams are used to understand heat transfer of a system. In statistical mechanics the partition function is used to encode the relationship of temperature and entropy.

Helpful Websites

This website is very useful; see eq 420, temp is embedded in definition of beta. This website explains Rudolf Clausius' definition of entropy. This website talks about Claude Shannon and his contributions to information theory. This website explains the history of entropy and some of the different definitions. This website talks about Ludwig Boltzmann's life and definition of entropy. This website further explains the relation between temperature and entropy.

Answer 9

A higher entropy equilibrium state can be reached from the lower entropy state by an irreversible but purely adiabatic process. The reverse is not true, a lower entropy state can never be reached adiabatically from a higher entropy state. On a purely phenomenological level the entropy difference between two equilibrium states, therefore, tells you how "far" away they are from being reachable the lower entropy state from the higher entropy one by purely adiabatic means. Just as temperature is a scale describing the possibility of heat flow between interacting different temperature bodies, entropy is a scale describing the states of a body as to how close or far apart those states are in the sense of an adiabatic process.

Answer 10

As a general rule, physics gets easier when the mathematics gets harder. For example, algebra-based physics comprises a bunch of seemingly unrelated formulae, each and every one of which needs to be memorized separately. Add calculus and wow! Many of those supposedly disparate topics collapse into one. Add mathematics beyond the introductory calculus level and the physics gets even easier. The Lagrangian and Hamiltonian reformulations of Newtonian mechanics are much easier to grasp -- so long as you can understand the mathematics, that is.

The same applies to thermodynamics, in spades. There used to be a website that provided 100+ statements of the laws of thermodynamics, the vast majority of which addressed the second and third laws of thermodynamics. The various qualitative descriptions were quite hair-pulling. Most of those hair-pulling difficulties vanish when you use the more advanced mathematics of statistical mechanics as opposed to the sophomore-level mathematics of thermodynamics.

For example, consider two objects at two different temperatures in contact with one another. The laws of thermodynamics dictate that the two objects will move toward a common temperature. But why? From the perspective of thermodynamics, it's "because I said so!" From the perspective of statistical mechanics, it's because that common temperature is the one temperature that maximizes the number of available states.

Answer 11

Since my contribution is not valued and appreciated. This will be my last post here.

• What really entropy is? Answer: Information or more precisely the inverse of information
• What is a major characteristic of entropy? Answer: The more even, the higher entropy (the less information).

Now, let's get to the rigorous part. This definition of entropy will unify both definitions from the most voted answer above.

• Given a fix encoding(basic description, axiomatic block) of a system, the inverse of entropy (i.e. information) is the minimal length of what you can create to fully describe the system.

In the physics world, we describe a system as an area of space with content in it. Let's fix the encoding for the description by having the following blocks:

1. The basic particles in the space
2. The basic shapes: lines, surfaces, etc... (which are mathematical equations)
3. The coordinate system (how you define space into cells)

If the space is completely even which means we can simply describe the space by saying: This is what a cell in this space looks like and it is like this everywhere else in the space. This is a short(est) description of the space which means this evenly distributed space has low information (can be described by short and easy string). And low information means high entropy. An example is a book with content containing only the letter 'b'. We can describe the book with just $(b \times 10,000)$ times, it is a short description as the books has low information and high entropy. If you know a bit about computer science, then you will recognize the $10,000$ times is a compression and the source of that $\ln$ part in the entropy formula

When the space is less evenly distributed, we can still describe the space with short description such as: it is how a typical cell's content looks like. It looks like this everywhere else except for the cells with the following coordinates [...] The exception part can also use the basic shapes of the encoding such as: A long this line, on this surface the cells have this kind of content. The main idea here is that the description keeps getting longer. This means the space has more information and lower entropy. Of course there are many ways to describe the same complicated space, but the length of the shortest description is the number to define information and entropy of the space.

Now we should be aware of a space with low information, but described by a long string. This does not mean the space has low entropy (high information). An example of this kind of space and description is an evenly distributed space with the letter b and the description of the space is "bbbbbbbbbbb..." repeated many times leading to long unnecessary/uncompressed description.

Now let's extend this to temperature in physics. Since temperature goes together with movement of particles in the space. We have to extend the coordinate system to account for time (since without time, we cannot describe motion and movement). This means adding another dimension to the coordinate system.

The same thing happens with even distribution characteristic. At low temperature, where the particles don't move, we can describe the space at one moment in time and say It is like this at all other time as well. Again, the description is short. This space has low information and high entropy. When there are movements, you have to add more description such as: "particles move with this mathematical pattern in space described by this equation". The minimal description length increases and the information level increases with harder to describe movements. You have to use more combination of basic mathematical equations to describe the movement.

The highest amount of information comes from space that cannot be described by the given encoding at the beginning. You have to describe it one by one for each cell at each moment in time.

My final note is that: closed space has no outside interaction. This space has no change in information and entropy. The movement patterns (if there are movement) are cyclical. You can describe it at each time of the cycle and say then it repeats. A description of the space might not be perfect, but if it is the core part of the shortest description, it can still describe the space imperfectly but still accurately. With more added to it, it becomes "more perfect".

Answer 12

The entropy plays a "complementary role" to what the internal energy does. The internal energy - or rather its change - measures the quantity of energy that a thermodynamic system possesses. The entropy - or rather its change - measures - in some sense - the quality of this energy. The lowest the entropy the higher the quality.

There is a molecular distinction between the energy transfer as work and heat. Energy transfer as work is done in an ordered way. During the raising of a weight the molecules move uniformly upwards. On the other hand heat is the energy transfer through the random collisions of molecules. That is why a formula such as $$\mathrm dS=\frac{\mathrm dQ}{T},$$ makes sense. If we want the entropy change to serve us as a disorder measure, it must be proportional to the disorder introduced to the system, the disordered energy transfer (aka heat) $\mathrm dQ$. Moreover if the system is already highly disordered (high temperature) then the relative increase in disorder is small. This explains why the temperature must be in the denominator (the correct power being determined only in a technical way). There is a nice analogy where $\mathrm dQ$ is represented by a sneeze and $T$ is related to the disorder of some environment. If we are in a quit library, the disorder is small and a sneeze will disturb the system so much that the disorder increases a lot. On the other hand if we are in a busy street, highly disordered, the same sneeze will correspond to a quite small increment of disorder.

As a concrete example of quality let us consider a heat engine operating between two thermal reservoirs of hot and cold temperature, $T_h$ and $T_c$, respectively. The total energy entering the engine is $|Q_h|$, the heat coming from the hot source. The work delivered is $$W=|Q_h|-|Q_c|,$$ where $|Q_c|$ is the heat rejected to the cold source. After a complete cycle of the engine, the entropy change of the system (engine+sources) is just the entropy changes of the sources, i.e. $$\Delta S=-\frac{|Q_h|}{T_h}+\frac{|Q_c|}{T_c}.$$ By the second law of thermodynamics this change cannot be negative so $$|Q_c|\geq\frac{T_c}{T_h}|Q_h|.$$ Plugging this into the expression for the work delivered by our engine we get $$W\leq|Q_h|\left(1-\frac{T_c}{T_h}\right),$$ i.e., the maximum work is delivered when the entropy change vanishes. That is what I meant by quality of the energy.

Answer 13

Entropy is simply the amount of disorder. Forget the ideas of temperature, because unstructured space (like prior to the Big Bang) has complete disorder where there is nearly 0 (classical) temperature. I say "classical" because saying there is a "temperature" is already imposing an order on it.

What is order, then? The amount of connectedness. This is what should be measureable with some specifier or unit as base-state reference (in the Greek sense: indivisible) and then entropy (or the amount of information that was contained within the mass, say) would be its inverse.

There is hardly any information in air, for example, so it has high entropy. On the other hand, there is a lot of information in a mass of lead (relative to the remainder of the multidimensional universe, somewhere on the order of m*c², I presume), so it has low entropy.

Answer 14

What is entropy really?

I want to answer(!) this question from a different point of view.

First off, I focus on your title and the phrase “really”. We don’t know what entropy is really. We don’t know what energy is really also, and any thing or concept else too. Entropy, like all other concepts created by humans, is a convention between some people to refer to the same(!) thought or sense.

I want to mention an example here. I want to ask “What is green color really?” The answer is same: “We don’t know.” But we usually talk about that with other people and if no one knows what green is, so how they can understand their aims? How has this concordance (or synchrony (sorry because of poor English)) between humans’ thoughts or senses been created? The answer is “Passing of time creates that concordance”. Passing of time helps us to understand each other without knowing that how we understand each other!

I think another intuitive example can help. I want to refer to ordinary education of children. No child knows why $1+1=2$ or $1$ plus $1$ is equal to $2$ (honestly, I don’t know also, even now!). They just see(!) some shapes like $1$, $1$, $+$, $=$ and $2$, and the cleverest ones those can analyze more than their classmates, say with themselves “When I see theses shapes $1+1=$, I must draw this shape $2$ after the $=$.” In fact, they don’t think about “What those shapes are” and what is important and wonderful here is that after some period (passing of time) and repetition they think that they have learnt addition and it was so simple! This process is occurring at high levels and ages too. One of my professors was saying (I don’t know it was true or not, I just quote): “If you ask from a Japanese engineer what stress is, he/she cannot answer you but they build nice bridges, machines, etc.” I think if they cannot talk about reality of stress, that is because they have passed the same process of education. When engineering students see the formula of stress $\sigma=\frac FA$, it is strange for them. When they write that formula themselves, its strangeness reduces a little bit. And after some period and repetition, they think that stress never has been strange for them while they don’t know what stress is even after passing of time and repetition.

Maybe you say that you have seen some people that are able to talk about stress for hours. You are right but even those people don’t know what stress is. Because for defining stress, they get help from other concepts or things and as I mentioned before, we don’t know all concepts and things. Their explanation is too much useful for decreasing the time that is necessary for creating the common thought or sense, but doesn’t remove the ignorance.

So, if you cannot understand the entropy, I should say: “No worry. You will understand it without that you know how you have understood it if you are patient and repeat. Passing of time will do her job well!”