Two ways to draw a 3D world
A screen is a grid of pixels, and a picture of a 3D world is just a colour for each of them. There are two ways to work out those colours.
The first is rasterisation. Every object is described as a mesh of triangles. The computer takes each triangle in turn, works out where its three corners land on the screen, and fills in the pixels it covers. A depth buffer remembers how far away each pixel’s nearest surface is, so a triangle behind another one is drawn over and then hidden again. Each covered pixel is then shaded: a small program on the graphics card works out its colour from the surface, its angle and the lights.
Rasterising is fast. A graphics card can push billions of triangles a second through it, which is why almost every game, and every Scene on this site, is drawn that way: three.js hands the triangles to your browser’s WebGL, and the graphics card rasterises them up to sixty times a second. The studio around the camera in the experiment above is drawn exactly like this.
The second way is ray tracing. Instead of asking which pixels a triangle covers, it asks, for each pixel, what can be seen through it. It fires a ray from the eye through the pixel and follows it into the scene.
Follow the light backwards
In the real world, light leaves a lamp, bounces off things and a tiny part of it ends up in your eye or a camera. Following light forwards from the lamp would be hopelessly wasteful, because almost all of it lands somewhere the camera never looks. But a path of light runs the same way in either direction, so a ray tracer runs it backwards: from the camera, out through each pixel, until it finds where the light came from.
The geometry is old. In 1525 Albrecht Dürer published a woodcut of artists drawing a lute by stretching a thread from a fixed point on the wall, through a wooden frame, to a point on the lute: where the thread crossed the frame told them where to mark the drawing. The frame in front of the camera in the Scene is the same idea, the image plane, cut into pixels. In 1968 Arthur Appel used the computer to cast such rays and decide which surface each pixel shows and whether it lies in shadow.
Try it: step Trace the rays to Camera rays. Pale blue rays leave the camera’s lens, pass through the marked pixels on the frame and stop at the first surface each one meets: the red ball, the mirror, the floor or the wall. Each pixel’s colour comes from what happens there.
Shadows and reflections
Finding the first surface is only half the job. To know how bright that spot is, the tracer asks whether it can see the light. It sends a second ray, a shadow ray, from the spot towards the light. If the shadow ray gets there, the spot is lit, more strongly when it faces the light squarely and when the light is close. If something gets in the way, the spot is in shadow.
A mirror is handled just as simply. When a ray hits a mirrored surface it carries on in the reflected direction, at the same angle on the other side, and the pixel shows whatever that new ray finds. In 1980 Turner Whitted put these together into recursive ray tracing: reflected and refracted rays spawn their own shadow rays and reflections, and the pictures came out with sharp shadows, mirror spheres and glass that bent the scene behind it, things rasterisation could not do.
Try it: step Trace the rays to Shadow rays. Gold rays reach the light panel; a red ray stops where something blocks it, and its spot is dark. Then set Bounces to 0: rays may no longer change direction, so the mirror ball comes out black in the picture, and every shadow is pitch black.
Compare the mirror ball in the studio view with the one in the picture. The studio’s ball is rasterised: it reflects a stored picture of the studio’s lamps, wrapped round it like wallpaper, so it never shows the red ball beside it. The traced ball shows the red ball and the checkerboard bent across its surface, and the light panel as a bright square, because its rays actually went and looked. Rasterisers fake reflections and shadows with tricks like that stored picture and shadow maps, a depth picture taken from the light; ray tracing gets them by following light.
Light that bounces
Real light does not stop after one reflection. Light from the panel hits the floor, and some of it scatters up onto the underside of the red ball; the red ball throws a little red onto the floor beside it. Shadows are never quite black, because light arrives from everywhere that is lit.
In 1986 James Kajiya wrote down the rendering equation, which says that the light leaving any point is the light it gives off plus all the light arriving from every direction, scattered by the surface. Path tracing estimates it by following each ray through many bounces. At each matte surface it sends a shadow ray to the light, then picks one direction at random and carries on, the ray’s colour multiplied by the surface’s, so a ray that has bounced off the red ball carries red light onwards.
Try it: step Trace the rays to Bounces and set Bounces to 1, 2 and then 4. The paths grow, each bounce tinted by the colour it has picked up. In the picture, the mirror ball shows its neighbours from 1 bounce, and the shadows lighten as more bounces bring in light scattered off the wall and floor.
Why the picture starts speckled
The random choices are the price. A ray might sample a point on the light that happens to be hidden, or bounce into a bright wall rather than the dark studio, so two rays through the same pixel can bring back very different colours. One ray per pixel gives a speckled picture: this is noise. Averaging many rays per pixel settles each pixel towards its true colour, but slowly. The noise left falls as one over the square root of the number of rays, so four times the rays only halve it, and halving it again takes sixteen times.
Not every part of the picture is equally noisy. A large light makes soft shadows, because a spot near a shadow’s edge can see part of the light but not all of it, and a handful of random rays judges that share badly. A tiny light gives hard shadows that one ray already gets right; what remains is the stair-stepped edges of shapes, because a single ray through each pixel’s centre cannot tell that an edge crosses the pixel. Several rays spread through the pixel smooth those edges too, which is called anti-aliasing.
Try it: slide Rays per pixel down to 1, then Light size up to 2 m: the shadows fizz with noise. Slide Light size down to 0.1 m and set Bounces to 0, and one ray is nearly enough, apart from jagged edges. Now slide Rays per pixel up to 256: the screen on the camera counts up to 256 rays per pixel as the picture settles, and the chart beside the Scene shows how many rays it took and how long.
Ray tracing in films, games and this site
A feature film frame is a few million pixels, often traced with hundreds or thousands of rays per pixel and many bounces, through scenes of hundreds of millions of triangles. Testing every ray against every triangle would never finish, so renderers sort the triangles into a tree of nested boxes, a bounding volume hierarchy: a ray that misses a box skips everything inside it. Even so, a frame can take hours. Since the mid-2010s most animated films have been path traced.
Games have a sixtieth of a second per frame. Graphics cards gained circuits for testing rays against boxes and triangles in 2018, and many games now mix the two methods: they rasterise the picture, then trace a ray or two per pixel for shadows, reflections or bounced light, and clean up the speckle with a denoiser that smooths it while keeping edges.
This site stays with rasterisation. Its Scenes must run on phones, sixty times a second, through WebGL, which gives no access to ray-tracing circuits. The small picture in this Scene is traced by your computer’s processor (the CPU) in ordinary JavaScript, one ray at a time; the pixels it fills are the same grid of red, green and blue numbers described in Digital images and colour.
What this model leaves out
- Five shapes and one light. Real renderers trace millions of triangles; this one knows two balls, a box, the dais, the wall and the floor round it, so it needs no bounding boxes. It has no glass, so no refraction, although Whitted’s tracer had it. The camera, its frame and the light’s cables are not in the traced scene, so the mirror ball does not show them.
- Surfaces are perfectly matte or perfect mirrors. Real materials are mostly in between: glossy paint, brushed metal, skin that lets light in under its surface. Each needs its own rule for which way a ray goes next.
- Three colours, not a spectrum. Light is traced as red, green and blue, as a screen shows it, so effects that depend on wavelength, like a prism’s rainbow, are out of reach.
- Every pixel is averaged equally. Real path tracers aim their random rays where light is likely to come from and stop a path early when it carries little light, which cuts the noise for the same work. This one samples the light directly at each matte bounce but otherwise picks plainly at random, so its noise is honest but higher.
- The studio view is not the picture. The set around the camera is rasterised and lit by the studio’s lamps as well as the panel, and its shadows come from a shadow map; it shows what the picture is of, not how the picture is made.