In thermodynamics, entropy is defined for gases.
In thermodynamics, entropy is defined for a state of knowledge about the microscopic particle configurations in a system—the positions, velocities, etc., of every particle in the system—also called microstates, consistent with macroscopic averages of the system like its temperature, pressure, etc., or macrostate.
The Gibbs entropy of a thermodynamic system in some known macrostate is defined to be $$-k_B \sum_s P(s) \log P(s),$$ where $s$ ranges over all possible discrete microstates, and $P(s)$ is the probability that a system with the known macroscopic temperature, pressure, etc., is in the specific particle configuration of $s$.
In other words, $P$ represents our state of knowledge about the unknown microstate of the particle configuration, and the Gibbs entropy is a measure of uncertainty about what the microstate could be, knowing what the macrostate is.
Strictly speaking, the Gibbs entropy is defined for a probability distribution $P$ on microstates; at equilibrium with a set of macroscopic averages like temperature, pressure, etc., the thermodynamic entropy is the maximum of the Gibbs entropy over all probability distributions having those macroscopic averages.
And, of course, the space of microstates isn't really discrete (except in some sense at the quantum level)—either you treat this as an approximation, or you take the limit as an arbitrary choice of discretization gets finer, or, instead of summing over discrete microstates, you integrate over phase space volume elements.
History:
Clausius originally introduced the idea of entropy purely in terms of macroscopic heat transfer, without the microscopic perspective.
Boltzmann connected entropy to the microscopic perspective of particle configurations in statistical mechanics, giving rise to the Boltzman entropy $k_B \log \Omega$, where $\Omega$ is the number—or, really, phase space volume—of possible microstates, assuming all are equiprobable.
Gibbs generalized Boltzmann's formula to nonuniform distributions—Boltzmann entropy is just the special case where $P(s) = 1/\Omega$.
In information theory, we strip away the details of particle configurations—positions, velocities, etc.—and macroscopic averages—temperature, pressure, etc.—and generalize this to Shannon entropy, measuring a state of knowledge in any probability distribution supported on any domain:* $$-\sum_x P(x) \log P(x).$$
The base of the logarithm isn't important except to make entropy of two different probability distributions commensurate; in physics traditionally it is the natural log (or base $e^{1/k_B}$ in a sense), and in information theory and cryptography traditionally it is base 2 in which case we talk about entropy in units of bits.
Claude Shannon devised this notion of entropy as a general measure of uncertainty, with applications to compression and cryptanalysis, and with the property the entropy of two independent unknowns is the sum of their respective entropies—this will come in handy for cryptography.
Cryptography is the design and study of games that an adversary can play, in the sense of game theory, with guarantees on how high the adversary's success probability is even when the adversary knows how the game works and everything in it and the other players' strategies except for small secret keys known only to the other players.
"Success" here means, for example, guessing a secret bit in an encrypted message sent by a player, or forging a signature on a message and fooling a player into accepting it.
How do the players in cryptography choose their keys?
They could flip coins, roll dice, count ionizing events in a Geiger–Müller tube or use the events to sample a clock, or—more practical in mass-manufactured silicon computers—sample bits with stochastic circuits like drifting independent ring oscillators or metastable flip-flops affected by thermal noise (arising from unknown microscopic particle configurations in the thermodynamic system of the silicon die!).
An understanding of the physics behind these systems—and a crucial assumption that the adversary can't see the players' observations of these systems—tells us the entropy, in the adversary's state of knowledge, of the outcomes of observing them.
The component—whatever it is under the hood—for generating secret keys in cryptography is a random number generator.
[My laptop] contains a random number generator and I have seen the word ‘entropy’ being used in this context. Is this the same entropy?
Cryptography is actually usually concerned with a different but related notion of entropy: min-entropy, $$-\max_k \log P(k),$$ where $k$ ranges over all possible keys, because cryptography is all about making sure the adversary's largest success probability using the best strategy is still small.
(In contrast, in, e.g., compression algorithms, without an adversary, we care more about the average compressed message length, not just about the most probable message, so compression is usually concerned with Shannon entropy.)
Min-entropy and Shannon entropy are both instances of the Rényi entropy family, $$H_\alpha := \frac{1}{1 - \alpha} \log \sum_x P(x)^\alpha,$$ with Shannon entropy being the limit as $\alpha \to 1$ and min-entropy being the limit as $\alpha \to \infty$.
Min-entropy and Shannon entropy are related by theorems like the leftover hash lemma used in cryptography.
For practical and numerological reasons—based on thermodynamic limits on the adversary's energy budget!—we typically require the keys to have min-entropy of at least 128 or 256 bits, meaning that the adversary's probability of guessing them is at most $1/2^{128}$ or $1/2^{256}$.†
If consecutive samples of a physical system like coin tosses, die rolls, G–M-driven clock samples, etc., are independent (or mostly independent), we can keep taking more of them to get enough entropy for cryptography, because entropy of independent unknowns sums, and we can compress large collections of samples into small keys with a hash function.
And, for simplicity of design and of proving theorems in cryptography, we usually require keys to be uniformly distributed, in which case the min-entropy and Shannon entropy coincide—we get uniformly distributed keys from physical observations like clock samples by feeding the observations through hash functions, called conditioning components, to smooth out the distribution.
But how much entropy do we need for cryptography, you might ask?
In a cryptographic game, the players will do things like encrypt messages into ciphertexts or sign messages giving signatures which are then exposed to the adversary, revealing to the adversary deterministic functions of their secret keys like $\operatorname{SHA256}(k)$ or $\operatorname{AES}_k(123)$ or $m^3 \bmod p\cdot q$ for secret $k$, $m$, $p$, and $q$.
In principle, deterministic functions of a random variable cannot have higher entropy than the random variable itself.
So in theoretical terms, knowledge of $\operatorname{AES}_k(123)$ can narrow down $k$ to a small set of candidate 256-bit strings, meaning the conditional entropy of $k$ given $\operatorname{AES}_k(123)$ is much lower than 256 bits.
However, these functions are chosen to be one-way, that is, chosen to be difficult to invert: we don't know any algorithm that will find, with the energy resources humanity can expend, a single candidate 256-bit string $k$ given only the value of $\operatorname{AES}_k(123)$ and the knowledge that every possible key has probability $1/2^{256}$—or even given the values of $\operatorname{AES}_k(123),$ $\operatorname{AES}_k(124),$ $\dotsc,$ $\operatorname{AES}_k(n)$ for very large $n$.
Protocols like TLS are designed with theorems saying that:
- if an adversary can attain better success probability than so and so with such and such cost at breaking the security—guessing an unknown bit in a message, forging a message, etc.—
- then the adversary must have found a cheaper way to invert AES than a generic search (or the other players' secrets must have been leaked, e.g. via a side channel attack).
The result of these theorems—and of the decades of failure by clever cryptanalysts to find any techniques for inverting AES—is that cryptography really only requires a small amount of min-entropy for users, and can then simulate generating fresh observations with more min-entropy by using deterministic algorithms called pseudorandom number generators or deterministic random bit generators like AES-CTR_DRBG.
As far as the adversary is concerned, these "observations" might as well have been chosen by flipping coins, not by a deterministic algorithm—as long as the players keep their keys secret.
But you still need those first 256 bits of entropy, which is why your laptop most likely has a stochastic circuit wired up through an AES circuit as a conditioning function to a CPU instruction called RDRAND (or RNDRRS or DARN or …) used to kick off cryptography so you can log into stackexchange.com without anyone being able to impersonate you over the network.
* Any discrete domain, that is.
With a lot more mathematical chicanery this can also be generalized to continuous domains, using probability densities and Lebesgue integrals and a lot of nonsense invented principally to torment graduate students in mandatory classes and quals on measure theory.
† Some commentators suggest that Kolmogorov complexity, sometimes called "Kolmogorov entropy", is relevant to random number generators.
But it is not.
The Kolmogorov complexity of a string is always defined relative to a language, and it means: What is the length of the shortest program in this language that can produce the string as output?
For example, the Kolmogorov complexity of the first 1000 decimal digits of $\pi$ in FORTRAN 77 is the number of bits in the shortest FORTRAN 77 program (without libraries) that will print the first 1000 decimal digits of $\pi$.
This depends fundamentally on the choice of language; the Kolmogorov complexity of the first 1000 decimal digits of $\pi$ in Python 2.7 is likely to be very different from its Kolmogorov complexity in FORTRAN 77.
This has nothing to do with quantifying an adversary's uncertainty about secrets, and everything to do with code golfing, and thus figures only into very obscure corners of cryptography.
