A web of tiny sums
A neural network is built from simple units, loosely modelled on nerve cells, arranged in layers. Each unit does very little: it takes the numbers coming in from the layer before, multiplies each by a weight, adds them up with a constant of its own (the bias), and squashes the total into a fixed range. Its result, the unit’s activation, is passed on to the next layer.
The network in the experiment has two inputs, the x and y position of a point, two hidden layers of five units, and one output: its answer to whether the point is blue or orange. Each link’s thickness shows the size of its weight. Blue links add to the next unit’s total, orange ones subtract.
Learning is adjusting weights
At the start the weights are random, so the answers are guesses. Training shows the network the 80 example points with their true colours and measures how wrong it is, a number called the loss. Then it works out, for every single weight, whether a slight increase would make the loss bigger or smaller, and nudges every weight a little in the direction that helps. That is gradient descent: like walking downhill in fog by feeling which way the ground slopes.
Working out all those slopes at once is done by backpropagation: the error at the output is passed backwards through the layers, so each link learns its share of the blame.
Try it: start with Training steps at 0 and drag it slowly to the right. Watch links thicken and change colour, and the score climb towards 100%.
The boundary it learns
Choose On under Show the decision boundary. The board shows the network’s answer at every position: the surface rises and turns blue where it says blue, sinks and turns orange where it says orange. The white line is the decision boundary, where the network is exactly undecided.
Choose Two clusters and a straight boundary is enough, so the network learns it in under twenty steps. Corners (XOR) puts blue in two opposite corners and orange in the other two; no single straight line can split them, the problem that famously stumped the earliest one-layer networks in the 1960s. With hidden layers, the network bends its boundary into a band. For Ring it learns to draw a closed loop.
The hidden units are why: each one draws its own straight cut through the plane, and the next layer combines those cuts into curves.
What this model leaves out
This network is tiny, so you can see every weight. Image and language models have billions of weights, are trained on millions of examples in small random batches rather than all at once, and use extra techniques to keep learning stable. Doing well on the training points is not the same as doing well on new ones: a network can memorise its examples, called overfitting, so real training checks it on data it has never seen. Units in a neural network are a loose analogy for neurons; brains do not learn by backpropagation in this form. The link colours, glowing units and travelling beads show the numbers in the network; nothing in a computer actually lights up.