A machine that guesses the next token
A large language model, the engine inside a chatbot, does one thing over and over: given some text, it gives a probability for every possible next piece of text. One piece is picked, added to the end, and the whole text goes back in to choose the piece after that. An answer of a hundred words takes well over a hundred of these steps.
The pieces are tokens. A tokenizer cuts text into a fixed vocabulary of tokens, usually some tens of thousands to a couple of hundred thousand of them. Common words are one token each, usually including the space in front of them; rarer words are split into pieces. A made-up word like zorblat splits into zor and blat. Each token in the vocabulary has a number, and the model never sees letters, only those numbers.
Each token number is then looked up in a table that gives it a long list of numbers, its vector: a few hundred to over ten thousand numbers in a large model. The Scene shows eight, as the column of lit cells behind each tile: blue for a positive number, amber for a negative one. The same token always starts with the same vector, which is why both copies of the in the first sentence have identical columns. Training has arranged these vectors so that tokens used in similar ways have similar vectors.
Try it: step Follow the attention to Tokens and read the tiles. The faint ␣ mark on a tile is the space the token carries, and the number under the word is its place in the vocabulary.
Attention: looking back at the words that matter
To guess what follows The keys to the cabinet in the hall, the model needs to know that the sentence is about keys. The nearest nouns, cabinet and hall, are the wrong ones: the next word is far more likely to be are than is. A model that only looked at the last word or two would get this wrong.
Attention is how a token gathers what it needs from the tokens before it. Every token makes three vectors from its own: a query (what am I looking for?), a key (what do I offer?) and a value (what I will pass on). The attending token compares its query with the key of every earlier token, and the closer the match, the higher the score. The scores are turned into shares that add up to 100% (the softmax), and the token takes that share of each earlier token’s value and adds the blend to its own vector.
In the Scene, the raised tile is the attending token. Each arc runs back to an earlier token and is as thick as the share of attention that token gets; the brass strip under the tiles prints the shares. The column of the attending token changes colour as the blend arrives: it now carries information from the words it looked at.
Try it: with Sentence on The keys… and Attending token slid up to 8, the last token, see hall send most of its attention past cabinet to keys. Then drag Attending token down one step at a time and watch the arcs regroup.
The smoked glass covers the tokens after the attending one. A language model that writes left to right may only look back: during training it must not peek at the very word it is learning to predict. This is called the causal mask.
One head, many jobs
The Scene shows one attention head. A real model runs many heads side by side in each layer, each with its own way of making queries, keys and values, so each learns to look for something different. The three sentences show three kinds of looking that researchers have found in trained models:
- Agreement. In The keys to the cabinet in the hall, a head that tracks the subject lets the model choose are, which agrees with keys.
- Recall. In The Eiffel Tower is in the city of, the last token gathers Eiffel and Tower. Attention only brings the clue together; the answer, Paris, comes from what the rest of the network learned about the world in training.
- Copying. In My pet zorblat sleeps. The zor, no model has ever learned the word zorblat. The head finds the earlier zor and looks at the token that came after it, so the model can copy blat. Heads that do this are called induction heads, and they are one reason models can use names and terms they meet for the first time in a prompt.
The other positions matter too. During training the model guesses the next token at every position at once, and is scored on how much probability it gave the token that really came next. Early in a sentence it cannot know much: after The the real next word, keys, gets under 0.1%.
Turning probabilities into a choice
After the attention and the rest of the network have done their work, the model ends with a score for every token in its vocabulary. The softmax turns those scores into probabilities. In the Scene they are the slices of the wheel: a large slice for a likely token, and a grey hatched slice for the long tail of every other token, a thousand slivers too thin to draw one by one.
The model does not have to take the likeliest token. A sampler spins the wheel. The temperature reshapes it before the spin: every score is divided by the temperature before the softmax. Below 1 the big slices grow and the small ones shrink, so the model plays safe and repeats itself more. Above 1 the slices even out, and because the tail holds so many tokens, it soon takes over and the text turns to nonsense. Many chat systems sample at a temperature between about 0.5 and 1, often also cutting off the tail first.
Try it: step Follow the attention to Next token and slide Temperature down to 0.2, then up to 1.6. The pointer stays at the same place on the wheel, but the slices move under it, and the pick shown on the glowing tile changes.
The chart beside the Scene lists the probabilities, marks the real next word with a star and says how much probability it got. That number is what training improves. The average surprise at the real next token, measured in bits, is the model’s cross-entropy, the same quantity as in information and entropy.
From one head to a large language model
A transformer, the design introduced by researchers at Google in 2017 and used by almost every large language model since, stacks this step many times. Each layer has many attention heads followed by a small network that works on each token separately, and each layer adds its result to the token’s vector, so the vector gathers more and more context on its way up. GPT-3, from 2020, had 96 layers of 96 heads and 175 billion adjustable numbers, its parameters. Those numbers start random and are tuned by gradient descent, as in neural networks, over trillions of tokens of text.
The result is a model that is very good at continuing text in a plausible way. That is not the same as being right. A model that has learned what a confident answer sounds like can produce one that is fluent and false, which is called a hallucination. Chat assistants are further trained on examples of helpful conversations and on people’s ratings of their answers, but underneath they still pick one token at a time.
Attention has a cost. Every token compares itself with every token before it, so doubling the length of the text roughly quadruples that work. This is why a model has a context window, a limit on how much text it can look back over at once, and why making it longer is hard.
What this model leaves out
The weights, probabilities, token numbers and vectors in the Scene are made by hand to look like what a large model does; they were not computed by one. A real model has many heads in each of dozens of layers, and what any single head does is often less tidy than these examples. Real vectors have hundreds to thousands of numbers, not eight, and each head compares and blends transformed versions of them rather than the raw columns shown. The Scene leaves out the position of each token, which real models add to its vector so that word order counts. The long tail of the wheel is drawn as a thousand equally unlikely tokens, while in a real vocabulary of tens of thousands their chances vary widely, and real samplers usually cut the tail off before spinning. The sentences are in English; Hebrew text is usually cut into more, shorter tokens.