A positive result
Suppose one person in a hundred has a certain disease, and there is a test for it that is right 90% of the time: it gives a positive result for 90% of the people who have the disease, and a negative result for 90% of the people who do not. You take the test, and it comes back positive. How likely is it that you have the disease?
Most people answer 90%, and many doctors do too when the question is put to them as a puzzle. The real answer is about 8%. The test has not failed: it did exactly what its accuracy says. What the quick answer leaves out is how rare the disease was before anyone was tested.
Count the crowd
Percentages hide the reasoning; counting people shows it. The Scene puts a thousand people on the floor of a sports hall, and everyone takes the test. Each person whose result is positive holds up an amber card. With the starting settings, 108 cards go up.
Now count who is really ill. One in a hundred of a thousand is 10 people. The test catches 90% of them, so 9 of them hold a card and 1 is missed. The other 990 people are healthy, and the test wrongly calls 10% of them positive: 99 people. So the 108 cards are 9 people who are ill and 99 who are not. A card means “ill” only 9 times in 108, about 8%.
Try it: step Sort the results to Who is ill. The ten people who really have the disease turn red, and everyone else fades to grey. Most of the cards are held by grey people. Step on to Positives sorted and every card holder walks to a mat: nine to Has the disease, ninety-nine to Healthy.
Counting like this, in whole people out of a round number, is called using natural frequencies. The chart beside the Scene draws the same counts as a tree: the thousand split into the ill and the healthy, and each of those split into positive and negative results. The answer is the left positive box divided by both positive boxes together.
The base rate decides
The trap is that the false alarms come from the healthy group, and the healthy group is enormous. A small error rate applied to 990 people makes far more wrong positives than a large success rate applied to 10 people makes right ones. How common something is before you look at the evidence is called its base rate, and ignoring it is the base-rate fallacy.
Try it: slide Test accuracy up to 99%. Now 10 ill people and 10 healthy people hold cards, and a positive is a coin toss. Slide How common the disease is down to 0.1%, 1 in 1000: one person is ill, and ten healthy people stand with them, so a positive is right 1 time in 11. Then slide it up to 10%, 1 in 10: 99 of the 108 card holders are ill. The test is the same in all three; only the base rate changed.
So “how accurate is the test?” is the wrong question on its own. A test that is excellent in a hospital ward, where many of the people tested are ill, can mislead when it screens a whole population for something rare. This is why screening programmes follow a positive result with further tests before anyone is told they are ill.
Bayes’ rule
The counting has a formula. Write P(ill) for the share of people who are ill before testing, P(+ | ill) for the chance an ill person tests positive and P(+ | healthy) for the chance a healthy person does; the bar is read “given”. The chance that someone who tests positive is ill is
The top is the share of everyone who is ill and positive, the left positive box of the tree. The bottom is the share who are positive at all, both positive boxes. With the starting numbers it is 0.9 × 0.01 ÷ (0.9 × 0.01 + 0.1 × 0.99) = 0.0083, the 9 in 108 the crowd shows.
This is Bayes’ rule, after Thomas Bayes, an English minister whose essay on it was published in 1763, two years after his death, by his friend Richard Price; Pierre-Simon Laplace worked out the general form independently and used it widely. Its parts have names. P(ill), the belief before the evidence, is the prior. P(ill | +), the belief after it, is the posterior. In medicine P(+ | ill) is the test’s sensitivity and 1 − P(+ | healthy) its specificity; this Scene sets both with one control.
Bayes’ rule is easiest to use in odds. Before the test the odds of being ill are 1 to 99. A positive result is 9 times more likely from an ill person (90%) than from a healthy one (10%), and that ratio, the likelihood ratio, multiplies the odds: 9 to 99, or 9 ill people for every 99 healthy ones among the cards.
Evidence adds up
A posterior can be the prior for the next piece of evidence. Turn on Retest the positives and everyone who tested positive takes a second, independent test; only people positive both times keep their cards. The odds are multiplied by 9 again: 81 to 99. In the crowd, 8 ill people and 10 healthy people hold cards, so a positive is now right about 45% of the time instead of 8%. With Test accuracy at 99% a second positive leaves only the 10 who are ill.
The second test helps only if its mistakes are independent of the first. If something about a healthy person fooled the first test, the same kind of test may be fooled again, so doctors confirm a result with a different kind of test. Note the cost, too: the retest misses 2 of the 10 ill people instead of 1, because each test can miss them.
Beyond medicine
The same reasoning applies whenever evidence bears on an idea. In the scientific method a surprising result counts for more or less depending on how likely the idea was to begin with, which is why a single experiment claiming something very unlikely is checked before it is believed. In court, a forensic match that would happen by chance only once in a million sounds damning, but in a city of ten million people about ten innocent people would match too; treating “one in a million” as the chance of innocence is called the prosecutor’s fallacy. Spam filters weigh each word of an email the same way, multiplying the odds that it is junk.
The probabilities themselves are the subject of probability and the bell curve. How sure a measured number is comes from models and uncertainty, and what a link between two measurements is evidence of from statistics and regression. An epidemic changes the base rate from week to week, so the same test means more at its peak than at its start.
What this model leaves out
- One accuracy for two numbers. Real tests have a sensitivity and a specificity that differ, often widely; a test may catch 99% of the ill but clear only 95% of the healthy. The Scene uses one control for both to keep the counting simple.
- Rounded, expected counts. Every number in the crowd is the expected count rounded to a whole person, so it can differ slightly from Bayes’ rule: with the disease at 0.1% (1 in 1000) and the test 99% accurate the crowd shows 1 in 11, about 9.1%, and the rule gives 9.0%. A real group of a thousand would vary by chance around these numbers.
- Who gets tested. People who see a doctor because they feel ill have a much higher prior than a random person, so the same positive result means more for them.
- Independent tests. The second test is assumed to make its own mistakes, unrelated to the first.
- Positive or negative only. Many real tests give a number, and where the line between positive and negative is drawn trades missed cases against false alarms.