Entropy.

Step 1 of 5

The average surprise

A biased coin produces mostly small surprises, and now and then a big one. To know how much information it produces on average, per flip, weight each surprise by its chance. That average is called entropy, written $H$⁠:

$H = p \cdot \log_2 \frac{1}{p} + (1-p) \cdot \log_2 \frac{1}{1-p}$$H =$$p \cdot \log_2 \frac{1}{p} + (1-p) \cdot \log_2 \frac{1}{1-p}$⁠

It is the same weighted average you take in probability, like an expected value: each value times its chance. Chance of heads is fixed at 90%, and Measure and compress is at Coins. Work out the entropy yourself.

Step 2 of 5

The curve

Measure and compress is now at Entropy. The curve on the glass board is $H$⁠ for every chance of heads from 0% to 100%, and the marker shows the current coin. The curve is a hump: it rises from both ends to a peak of exactly one bit in the middle.

Before you move anything, think about what will happen to the entropy if the bias flips.

Step 3 of 5

A certain coin

At both ends of the curve the coin is no longer really flipped: the outcome is known in advance. The tape still fills with bits, one per flip, but the length of a record is one thing, and what it tells you is another.

Step 4 of 5

In between

Between the fair coin and the certain one, the entropy falls slowly at first and fast at the end. A 75% coin still carries about four fifths of a bit per flip, and a 99% coin only about eight hundredths. For a coin to be really predictable, it has to be very biased.

Step 5 of 5

Random noise is full of information

High entropy means the next symbol is hard to guess. So a fair coin’s tape, which looks like a meaningless jumble of 0s and 1s, is exactly the tape with the most information per bit. That sounds backwards, but it is the definition: information is what you could not have guessed. Compressed and encrypted files look like noise for the same reason.