A coin you cannot predict
Flip a fair coin and write down what you see: 1 for heads, 0 for tails. Before each flip you have no idea which digit comes next, and afterwards you know exactly. That change, from not knowing to knowing, is what information means in science and engineering. It has nothing to do with whether the message is important or true. It measures how much a message settles.
Now rig the coin so it lands heads nine times in ten. The tape fills with 1s, and you could guess most of it without looking. A flip you could have guessed tells you little. A coin that always lands heads tells you nothing at all: you knew the answer before it landed.
Surprise, measured in bits
Claude Shannon turned this into arithmetic in 1948. The surprise of an outcome that happens with probability p is −log₂ p bits. A fair coin’s heads has probability ½, so it carries exactly 1 bit: one yes-or-no question answered. On the rigged coin, heads (0.9) carries only 0.15 bits, while the rare tails (0.1) carries 3.32 bits. Rare events are big news; expected ones are small.
The base-2 logarithm is what makes the unit a bit. Two fair flips carry 2 bits, and four equally likely outcomes need two binary digits to tell apart. Surprises of independent events add.
Entropy is the average surprise
A rigged coin mostly produces small surprises with the occasional big one. Averaging over many flips gives the entropy of the coin:
H = −p log₂ p − (1 − p) log₂ (1 − p)
For the 90% coin that is 0.9 × 0.15 + 0.1 × 3.32 ≈ 0.47 bits per flip. The curve is a hump: 0 bits when the coin is certain either way, rising to its peak of exactly 1 bit at a fair coin, where you are least able to guess. A 75% coin still carries 0.81 bits per flip; a 99% coin, only 0.08.
Try it: step Measure and compress to Entropy, then slide Chance of heads up to 90%. The marker slides down the curve to 0.47 bits, and the readout shows how small the surprise of a head has become. Slide it to 100% and it drops to zero.
Predictable data compresses
Entropy is not just a definition. Shannon proved it is the smallest number of bits per flip that any code can use on average to record the flips exactly. That makes it a hard limit on compression, and it explains why predictable data shrinks.
The machine’s second tape shows a real compressor, a Huffman code, which David Huffman devised as a student and published in 1952. It reads the flips in blocks and gives each possible block a codeword, short for common blocks and long for rare ones, arranged so that no codeword is the start of another and the tape can be read back without gaps. For the 90% coin in blocks of four, all heads happens about 66% of the time and is written as a single bit. A block with one tail takes 3 or 4 bits, and four tails take 10. Most blocks shrink, a few grow, and on average the coin costs 0.49 bits per flip, against an entropy limit of 0.47. On one run of 100 flips the tape came to 48 bits.
A fair coin cannot be compressed at all. Every block is as likely as every other, so no block deserves a short codeword, and the lower tape is exactly as long as the upper one. Compression works by betting on what is likely; when nothing is more likely than anything else, there is nothing to bet on.
How close can a code get?
Slide Code block size down to 1 flip: the lower tape becomes a copy of the upper one, whatever the coin. A single flip has only two outcomes, and a codeword cannot be shorter than one bit, so coding flips one at a time saves nothing. Longer blocks give the code more to work with. For the 90% coin, blocks of 2, 3, 4 and 6 cost 0.65, 0.53, 0.49 and 0.47 bits per flip. A Huffman code over blocks of k flips always lands between the entropy and the entropy plus 1/k bits per flip.
The certain coin shows the same limit from the other side. Its entropy is zero, yet the code still spends one bit per block, a quarter of a bit per flip in blocks of four. Longer blocks, or the arithmetic coding used in modern compressors, approach zero.
On any short stretch of tape the count can land a little above or below the limit, because a lucky run of heads is cheap to write. The limit holds on average, and over long streams the counts close in on it.
What this model leaves out
Every flip here is independent of the others and has the same, known chance. Real data is rarely like that. In English text, a q is almost always followed by a u, and whole words repeat; general-purpose compressors such as those behind ZIP files find these patterns and repeats, not just an uneven mix of symbols, so they can do far better than a code that only knows how often each letter appears. The code here is also built knowing the coin’s true chance; a real compressor has to estimate it from the data. Huffman codes only approach the entropy for long blocks, and the machine stops at six flips. The coins’ spin, bounce and landing are drawn for clarity rather than computed from physics, and a real tossed coin is very slightly more likely to land the way it started.