Surprise measured in bits.

Step 1 of 5

What information is

A machine flips coins and prints each flip on a tape: $1$⁠ for heads, $0$⁠ for tails. Before each flip you do not know what the next digit will be, and after it you know exactly. That change, from not knowing to knowing, is what computer science calls information.

This definition, Claude Shannon’s from 1948, does not ask whether a message is important, true or interesting. It asks only how much uncertainty the message removes: how much you did not know before it arrived. A message you could have guessed in advance tells you nothing.

Step 2 of 5

One bit

Uncertainty is measured in bits. One bit is the answer to one yes-or-no question when both answers are equally likely: exactly what a fair coin gives.

Shannon measured the surprise of an outcome with probability $p$⁠ like this:

$-\log_2 p = \log_2 \frac{1}{p}$$-\log_2 p =$$\log_2 \frac{1}{p}$⁠

$\log_2 x$⁠ is the power you must raise $2$⁠ to in order to get $x$⁠. For example, $\log_2 8 = 3$⁠, because $2^3 = 8$⁠. Chance of heads is now fixed at 50%. Work out how much surprise a heads carries.

Step 3 of 5

A biased coin

Now the coin is biased, and lands heads nine flips out of ten. Watch the tape: it fills with 1s, and you could have written most of it without looking at the coin.

When an outcome is expected, it adds almost no knowledge. When it surprises, it adds a lot. Rare events are big news.

Step 4 of 5

Every halving is a bit

The formula $\log_2 \tfrac{1}{p}$⁠ says something simple: every time an outcome’s chance is halved, its surprise grows by one bit. An outcome with chance $\tfrac12$⁠ carries one bit, one with chance $\tfrac14$⁠ two, and one with chance $\tfrac18$⁠ three.

That is also the number of bits it takes to record the outcome: to tell four equal possibilities apart, two binary digits are enough, 00, 01, 10 and 11. And the surprises of independent flips add: two fair flips carry two bits.

Step 5 of 5

Rare news

Measure and compress is at Entropy: the glass board shows the surprise of each face of the coin. Change Chance of heads and watch the surprise of heads grow as it becomes rare, and shrink as it becomes expected.