How a machine learns

This is the first of four notes on how ChatGPT and programs like it work. ChatGPT writes by guessing what comes next, one small piece of text at a time. It learned to guess well by turning a lot of knobs a little, over and over, the way this note shows. Part 2 turns text into numbers, and part 3 shows the transformer blocks that ChatGPT is built from. Part 4 puts the pieces together and trains a tiny version of it in your browser.

Guess the fare

A taxi ride costs a starting fee plus a price for every kilometer. If the fee is 3 dollars and a kilometer costs 2, a 5 kilometer ride costs 3 + 5 × 2 = 13 dollars.

Now turn it around. You have eight old rides from another city. You know how far each one went and what you paid, but nobody tells you the fee or the price. Can you find a fee and a price that guess these fares well?

The chart has your eight rides. The line is your guess. Move the two sliders until the line runs close to every ride. Each square shows how far off the guess is for one ride, so a bad guess has big squares. No line touches every ride, because some of them sat in traffic. The number under the chart, next to the word loss, is the average area of the squares. Smaller is better.

That is all learning means here. There is a rule with a few knobs, and the machine looks for the knob settings that make its guesses close to what really happened.

The rule you just tuned is a model. It takes a distance and gives back a fare. The two sliders are its parameters, the knobs you turn to make it fit. The price per km is called a weight, because it scales the input. A longer ride gets more of it. The starting fee is called a bias, because it moves every guess up or down by the same amount. The fare the line gives for one ride is a prediction.

That average area is the loss, one number for how far all the guesses are from what the riders really paid. The machine's whole job is to make it small. However you set the sliders, it never gets below 0.33, because no line touches every ride.

It uses squares for two reasons. A miss of 4 then counts four times as much as a miss of 2, so big misses matter most. And an area is never negative, so a guess that is too high can't cancel one that is too low.

Downhill in fog

You could see the chart while you moved the sliders. The machine has to find the settings another way.

Every setting of the two knobs has a loss. Lay all the settings out as a floor, the starting fee one way and the price per km the other, and raise each spot by its loss. You get a valley, and the best settings sit at the bottom. The picture squeezes the tall parts so the whole valley fits. The cube stands at your current settings, the same ones the sliders set, and the small chart shows the line they make. The floor runs down to -$2 a km. No taxi charges that, but to the machine a price is just a number.

This taxi rule has a shortcut formula that finds the lowest loss directly. The machine could jump straight to those two settings. We make it walk here to show a method that also works when there is no shortcut.

A larger model can have millions of knobs. Its valley is usually lumpier, with no formula that takes it straight to the bottom. The machine feels the slope where it stands, takes a step downhill, and feels again. Think of a hill in thick fog. You can feel the ground under your feet, but you can't see where the lowest point is. You may reach a low spot with uphill ground all around and stop there, even though a lower spot lies beyond it. A downhill walk can find a low spot without finding the lowest one.

The machine never sees the whole picture we drew. Turn on the fog to see only what it can feel. You can also click a spot to put the cube there. If your browser draws the valley in 3D, you can drag it to turn it.

Press step a few times. Each step turns both knobs a little, in the direction where the ground drops fastest. The step size control under the valley sets how big the steps are, and that matters. Crawl takes tiny steps and needs hundreds of them. Good walks down the wall and then along the floor of the valley. Zigzag overshoots and lands on the far wall, again and again, so it wastes steps. Too big jumps across the valley and lands higher than it started. From where the cube starts, it does that on every step, so the loss grows instead of shrinking. For these eight rides, any step size above about 0.33 climbs out of the valley sooner or later, even when the first few steps look fine, unless it starts right at the bottom.

The slope under your feet is called the gradient. Each step moves every knob downhill by its slope times a number called the learning rate, so steeper ground or a bigger learning rate means a bigger step. The step size control sets the learning rate. Walking downhill one step at a time is gradient descent. Repeatedly adjusting the knobs to make the guesses closer to the examples is training. In this fare demo, walk stops when the ground is nearly flat.

Press walk, and once the cube stops, ask the line about a ride it has never seen. None of the eight rides went 1.6 km. Right now the line has a starting fee of 2.00 dollars and a price per km of 2.50, so for 1.6 km it says 2.00 + 1.6 × 2.50, which is about 6.00 dollars. Using a trained model to answer a new question like this is inference.

Which way is downhill?

Each step needs the slope for every knob. Here is how the machine gets it. Take one ride and work out its loss like a receipt, one line at a time. Distance times price per km. Plus the starting fee, which gives the guessed fare. Minus what the rider paid, which gives the miss. Then the miss times itself, which is this ride's loss.

Press nudge to add one cent to the price per km and watch the bottom line. It changes a little. That change divided by one cent is very close to the slope for this knob. It says how fast the loss moves when the knob turns, and which way. A negative slope means turning the knob up makes the loss smaller.

Nudging works, but a real model has millions of knobs, and nudging each one in turn would take forever. So the machine walks the receipt from the bottom line up instead, like tracing a wrong total back to the items that caused it. A line's slope says how much the bottom line moves for each unit that line moves, and each line works it out from the line after it.

The bottom line is the miss times itself. If the miss is 3, this ride's loss is 9. Make the miss 3.01 and the loss is 9.0601, about 0.06 more, which is 2 × 3 × 0.01. So the loss moves about two times the miss for each unit the miss moves, and the miss gets a slope of two times the miss. The guessed fare moves the miss one for one, so it gets the same slope, and so does the starting fee. The price per km moves its line only 0.3 times as much, because the ride was 0.3 km, so its slope is 0.3 times that. Press backward to watch it. The slope it gets for the price per km is almost exactly what the nudge gave, and the starting fee got its slope in the same walk. This walk backwards is backpropagation. Do it for all eight rides, average the slopes, and you have the gradient from the valley.

For readers who know JavaScript, here is Value, the class the receipt above runs on. A Value keeps its number in data, its slope in grad, and the values it was made from. Each of those comes paired with how much this value moves when that one moves by one. backward walks from the bottom line up, and each value passes its slope back to the values it was made from, times those numbers. It works for any calculation built from these methods, not only this receipt. A value used twice adds up the slopes from each use. The bottom line uses the miss twice, as miss times miss. Each use passes back the miss, so the miss gets two times the miss.

The code is TypeScript, which is JavaScript with the type of each value written after a colon. To run it, save it as value.mts, add a few lines at the end that use it, and run node value.mts. That needs Node 22.18 or later.

export class Value {
  data: number;
  grad = 0;
  inputs: [Value, number][];

  constructor(data: number, inputs: [Value, number][] = []) {
    this.data = data;
    this.inputs = inputs;
  }

  add(other: Value | number) {
    const that = other instanceof Value ? other : new Value(other);
    return new Value(this.data + that.data, [
      [this, 1],
      [that, 1],
    ]);
  }

  mul(other: Value | number) {
    const that = other instanceof Value ? other : new Value(other);
    return new Value(this.data * that.data, [
      [this, that.data],
      [that, this.data],
    ]);
  }

  pow(n: number) {
    const slope = n === 0 ? 0 : n * this.data ** (n - 1);
    return new Value(this.data ** n, [[this, slope]]);
  }

  exp() {
    const e = Math.exp(this.data);
    return new Value(e, [[this, e]]);
  }

  log() {
    return new Value(Math.log(this.data), [[this, 1 / this.data]]);
  }

  tanh() {
    const t = Math.tanh(this.data);
    return new Value(t, [[this, 1 - t * t]]);
  }

  relu() {
    const slope = this.data > 0 ? 1 : 0;
    return new Value(Math.max(this.data, 0), [[this, slope]]);
  }

  backward() {
    const order: Value[] = [];
    const seen = new Set<Value>();
    const visit = (node: Value) => {
      if (seen.has(node)) return;
      seen.add(node);
      for (const [input] of node.inputs) visit(input);
      order.push(node);
    };
    visit(this);
    for (const node of order) node.grad = 0;
    this.grad = 1;
    for (const node of order.reverse()) {
      for (const [input, slope] of node.inputs) {
        input.grad += slope * node.grad;
      }
    }
  }
}

When a straight line isn't enough

A new question. Is a day comfortable indoors? Each point is a day, temperature across and humidity up, filled if people said comfortable and hollow if not. The comfortable days sit in the middle. Not too cold, not too hot, not too dry, not too damp.

Take the taxi rule and give it two inputs, temperature and humidity, each with its own weight, plus a bias. Then squash the result so big sums level off. That rule draws one straight fence across the chart and says how far onto the yes side a day is. One fence can't pen in the middle. Wherever it runs, the side that holds all the comfortable days holds some uncomfortable ones too. A rule like this is one neuron.

So put several neurons side by side. That is a layer, and it draws several fences. Each neuron hands its number to every neuron in the next layer, which gives each one a weight, adds its bias, and squashes again. That is how a second layer reads those fences and joins them into a shape, and four fences make a box around the comfortable days. The last neuron squashes its answer to between 0 for no and 1 for yes. The squash is called the activation function. Without it, stacking fences only ever makes another straight fence. Layers in a row are a network.

For a yes or no answer, this playground scores its guesses a little differently from the squares, and part 2 explains how. It also remembers earlier slopes when choosing how to turn each knob. That method is called Adam. Because earlier slopes still count, a knob can sometimes move with its current slope rather than against it. The aim is to lower the loss over many updates, but an individual step can raise it.

Press train and watch the shape form. This network has two layers of eight neurons and one more neuron that gives the answer. Unlike the fare walk, the playground does not stop when the loss levels off. Press pause when you have seen enough.

The shade behind the points is its answer for every spot, stronger where it is surer the day is comfortable. Each neuron in the first layer gets a row of three squares, one for each input and one for its bias. A filled square is a positive number and a hollow one is a negative number. A bigger square is a number farther from zero. The layers after it decide what those numbers do to the answer. Point at a row to draw that neuron's fence.

For readers who know JavaScript, here is the network the playground trains, built on Value. A Neuron adds up each input times its weight, plus its bias, and squashes the sum with tanh. A Layer is a list of neurons that all read the same inputs, and an MLP is layers in a row. The last layer skips the squash and hands back a raw score.

export class Neuron {
  weights: Value[];
  bias = new Value(0);
  squash: boolean;

  constructor(inputs: number, normal: () => number, squash: boolean) {
    this.weights = Array.from(
      { length: inputs },
      () => new Value(normal() * Math.sqrt(1 / inputs)),
    );
    this.squash = squash;
  }

  call(x: Value[]) {
    const sum = this.weights.reduce((total, w, i) => total.add(w.mul(x[i])), this.bias);
    return this.squash ? sum.tanh() : sum;
  }
}

export class Layer {
  neurons: Neuron[];

  constructor(inputs: number, outputs: number, normal: () => number, squash: boolean) {
    this.neurons = Array.from({ length: outputs }, () => new Neuron(inputs, normal, squash));
  }

  call(x: Value[]) {
    return this.neurons.map((neuron) => neuron.call(x));
  }
}

export class MLP {
  layers: Layer[];

  constructor(sizes: number[], normal: () => number) {
    const last = sizes.length - 2;
    this.layers = sizes.slice(1).map((size, i) => new Layer(sizes[i], size, normal, i < last));
  }

  call(x: number[]) {
    let values = x.map((input) => new Value(input));
    for (const layer of this.layers) values = layer.call(values);
    return values[0];
  }

  parameters() {
    return this.layers.flatMap((layer) => layer.neurons.flatMap((n) => [...n.weights, n.bias]));
  }
}

Here is the training loop. crossEntropy is the yes or no score that part 2 explains. It turns the raw score into a chance with sigmoid and measures how far that chance is from the real answer, worked out so that a very sure guess never overflows. adam uses averages of current and earlier slopes, and their squares, to choose each knob's update. trainStep scores all 100 points, averages their losses, walks back with backward, and turns every knob once. The playground calls it repeatedly while it trains.

export type Point = { x: number; y: number; label: 0 | 1 };

export const sigmoid = (s: number) => 1 / (1 + Math.exp(-s));

export function crossEntropy(score: Value, label: number) {
  const s = score.data;
  const loss = Math.max(s, 0) - label * s + Math.log(1 + Math.exp(-Math.abs(s)));
  return new Value(loss, [[score, sigmoid(s) - label]]);
}

export function adam(parameters: Value[], rate = 0.01) {
  const m = parameters.map(() => 0);
  const v = parameters.map(() => 0);
  let t = 0;
  return () => {
    t++;
    parameters.forEach((p, i) => {
      m[i] = 0.9 * m[i] + 0.1 * p.grad;
      v[i] = 0.999 * v[i] + 0.001 * p.grad * p.grad;
      p.data -= (rate * (m[i] / (1 - 0.9 ** t))) / (Math.sqrt(v[i] / (1 - 0.999 ** t)) + 1e-8);
    });
  };
}

export function trainStep(model: MLP, points: Point[], step: () => void) {
  const losses = points.map((p) => crossEntropy(model.call([p.x, p.y]), p.label));
  const loss = losses.reduce((sum, each) => sum.add(each)).mul(1 / points.length);
  loss.backward();
  step();
  return loss.data;
}

The whole loop

Every demo here showed one part of the same loop. A rule with knobs makes its guesses. The loss says how far off they are, as one number. Backpropagation finds the slope for every knob. Training uses those slopes to adjust the knobs, aiming for a smaller loss over many steps. A step can raise the loss, and a downhill walk can stop in a low spot that is not the lowest. The fare walk stops when the ground is nearly flat. In the playground, you decide when to press pause. The fare rule had 2 knobs, and the playground's network has 105. ChatGPT learned with the same loop and far more knobs, from text instead of fares, and part 2 starts there.