Unit 6
Information and codes.
How much information a message holds: surprise in bits, entropy, the limit of compression, and codes that repair errors.
Before this unit
- Bits and data
- Probability Mathematics
The unit, lesson by lesson
Lessons
5 lessons · 26 practice questions · about 2 hours
Surprise measured in bits
What the word information means in computer science: how much uncertainty a message removes, why a fair coin flip carries one bit, and why a rare outcome is more surprising.
Entropy
A source's average surprise: how to work it out, why it peaks for a fair coin and falls to zero when the outcome is certain, and why random noise is full of information.
Compression
How Huffman's code shortens predictable data: short words for common blocks, why a fair coin does not compress, and why no code goes below the entropy.
The parity bit
A noisy line flips bits: how many blocks arrive wrong, how one parity bit catches a flip, and what it cannot catch.
Hamming's code
Three overlapping parity checks find which bit flipped and repair it: how to read them as a number in base 2, what happens when two bits flip, and what it costs.