How a machine learns

ChatGPT writes by guessing what comes next, one small piece of text at a time. That's the whole trick. It got good at guessing by turning a lot of knobs a little, over and over, and this note shows you how that works, with taxi fares instead of text.

This is the first of four notes on how ChatGPT and programs like it work. Part 2 turns text into numbers. Part 3 shows the transformer blocks ChatGPT is built from. Part 4 puts the pieces together and trains a tiny version of it in your browser.

Guess the fare

A taxi ride costs a starting fee plus a price for every kilometer. If the fee is 3 dollars and a kilometer costs 2, a 5 kilometer ride costs 3 + 5 × 2 = 13 dollars. Easy.

Now turn it around. You have eight old rides from another city. You know how far each one went and what you paid, but nobody tells you the fee or the price. Can you find a fee and a price that come close?

Try it on the chart below. The line is your guess, and the two sliders move it. Each ride gets a square as wide as your miss, so a bad guess has big squares.

The number under the chart is the average area of those squares, one number for how far the guesses are from what the riders really paid. Smaller is better. That number is the loss. You won't get a line through every ride, because some of them sat in traffic.

That's all learning means here. There's a rule with a few knobs, and the machine looks for the settings that make its guesses land close to what really happened.

What you just tuned has a name. The rule is a model: give it a distance, it gives back a fare. Each slider is a parameter, a knob you turn to make it fit. The fare it gives for one ride is its prediction.

The two knobs even have their own names. The price per km is a weight, because it scales the input, so a longer ride gets more of it. The starting fee is a bias, because it shifts every guess up or down by the same amount.

The machine's whole job is to make the loss small. Set the sliders however you like. It never gets below 0.33, because no line touches every ride.

Why squares? A miss of 4 counts four times as much as a miss of 2, so the big misses matter most. And an area is never negative, so a guess that's too high can't cancel one that's too low.

Downhill in fog

You could see the chart while you moved the sliders. The machine doesn't. It has to find the settings another way.

Every setting of the two knobs has a loss. Lay all the settings out as a floor, the starting fee one way and the price per km the other, and raise each spot by its loss. You get a valley, and the best settings sit at the bottom.

The cube is you. It stands wherever the sliders say, and the small chart shows the line those settings make. (I squeezed the tall parts so the whole valley fits.) The floor even runs down to -$2 a km. No taxi charges that, but to the machine a price is just a number.

This taxi rule has a shortcut formula that finds the lowest loss directly. The machine could jump straight to those two settings. I make it walk here, so you can see a method that also works when there's no shortcut.

A larger model can have millions of knobs. Its valley is usually lumpier, and there's no formula that takes it straight to the bottom. So the machine feels the slope where it stands, takes a step downhill, and feels again.

Think of a hill in thick fog. You can feel the ground under your feet, but you can't see where the lowest point is. You may reach a low spot with uphill ground all around and stop there, even though a lower spot lies beyond it.

The machine never sees the whole picture we drew. Turn on the fog to see only what the machine can feel.

Press step a few times. Each step turns both knobs a little, toward where the ground drops fastest. How big that step is matters. The step size control under the valley sets it.

Crawl takes tiny steps and needs hundreds of them. Good walks down the wall, then along the floor of the valley. Zigzag overshoots and lands on the far wall, again and again, so it wastes steps. Too big jumps across the valley and lands higher than it started. From where the cube starts, it does that on every step, so the loss grows instead of shrinking.

Here's the sneaky part. For these eight rides, any step size above about 0.33 climbs out of the valley sooner or later, even if the first few steps look fine. The only exception is starting right at the bottom.

That slope under your feet is the gradient. Each step moves every knob downhill by its slope times one number, the learning rate, which is what the step size control sets. Steeper ground or a bigger learning rate means a bigger step.

Walking downhill like this, one step at a time, is gradient descent. Doing it over and over, turning the knobs so the guesses get closer to the examples, is training.

Press walk. Once the cube stops, ask the line about a ride it has never seen. None of the eight rides went 1.6 km. Right now the line has a starting fee of 2.00 dollars and a price per km of 2.50, so for 1.6 km it says 2.00 + 1.6 × 2.50, which is about 6.00 dollars. Using a trained model to answer a new question like this is inference.

Which way is downhill?

Each step needs the slope for every knob. Here's how the machine figures that out. Take one ride and work out its loss like a receipt, one line at a time. Distance times price per km. Plus the starting fee, which gives the guessed fare. Minus what the rider paid, which gives the miss. Then the miss times itself, which is this ride's loss.

Press nudge to add one cent to the price per km and watch the bottom line. It changes a little. That change divided by one cent is very close to the slope for this knob. It says how fast the loss moves when the knob turns, and which way. A negative slope means turning the knob up makes the loss smaller.

Nudging works. A real model has millions of knobs, though, and nudging each one in turn would take forever. So the machine walks the receipt from the bottom line up instead, like tracing a wrong total back to the items that caused it.

A line's slope says how much the bottom line moves for each unit that line moves. Each line works its slope out from the line after it.

• Loss. If the miss is 3, the loss is 9. Make the miss 3.01 and the loss is 9.0601, about 0.06 more, which is 2 × 3 × 0.01. So the miss's slope is two times the miss.
• Miss. The guessed fare moves the miss one for one, so it gets the same slope.
• Starting fee. It moves the guess one for one too. Same slope.
• Price per km. It moves its line only 0.3 times as much, because the ride was 0.3 km, so its slope is 0.3 times that.

Press backward and watch it fill in. The slope it gets for the price per km is almost exactly what the nudge gave, and the starting fee got its slope on the same walk, for free. Do that for all eight rides, average the slopes, and you have the gradient from the valley. That walk backward is backpropagation.

For readers who know JavaScript. value is a line of the receipt: a plain object with the number in data, its slope in grad, and the values it was made from. Each input is paired with how much the result moves when that input moves by one. backward visits the inputs before the result, walks that order in reverse, and passes the slope back times the number on that connection. A value used twice adds up both slopes. The loss uses the miss twice, so the miss gets two times the miss.

Each block runs on its own in your browser, with nothing to install. Press Run to see this ride's loss and the slopes for the fee and the price.

function value(data, inputs = []) {
  return { data, grad: 0, inputs };
}

function add(left, right) {
  return value(left.data + right.data, [
    [left, 1],
    [right, 1],
  ]);
}

function multiply(left, right) {
  return value(left.data * right.data, [
    [left, right.data],
    [right, left.data],
  ]);
}

function backward(loss) {
  const order = [];
  function visit(node) {
    if (order.includes(node)) return;
    for (const [input] of node.inputs) visit(input);
    order.push(node);
  }

  visit(loss);
  for (const node of order) node.grad = 0;
  loss.grad = 1;
  for (const node of order.reverse()) {
    for (const [input, slope] of node.inputs) {
      input.grad += slope * node.grad;
    }
  }
}

const fee = value(2);
const price = value(2.5);
const charge = multiply(value(0.3), price);
const guess = add(fee, charge);
const miss = add(guess, value(-6.25));
const loss = multiply(miss, miss);
backward(loss);
console.log('Loss:', loss.data);
console.log('Fee slope:', fee.grad);
console.log('Price slope:', price.grad);

When a straight line isn't enough

A new question. Is a day comfortable indoors?

Each point is a day, temperature across and humidity up. Filled if people said comfortable, hollow if not. The comfortable days sit in the middle. Not too cold, not too hot, not too dry, not too damp.

Take the taxi rule and give it two inputs, temperature and humidity, each with its own weight, plus a bias. Then squash the result so big sums level off. That draws one straight fence across the chart and says how far onto the yes side a day is. One fence can't pen in the middle. Wherever it runs, the side that holds all the comfortable days holds some uncomfortable ones too.

That rule is one neuron.

So put several neurons side by side, and you get several fences. That's a layer.

Each neuron hands its number to every neuron in the next layer. That next neuron gives each number a weight, adds its bias, and squashes again. That's how a second layer reads the fences and joins them into a shape. Four fences make a box around the comfortable days.

The last neuron squashes its answer to between 0 for no and 1 for yes. Without that squash, stacking fences only ever makes another straight fence. The squash is the activation function.

Layers in a row are a network.

For a yes or no answer, this playground scores its guesses a little differently from the squares. Part 2 explains how. It also uses a method called Adam, which remembers earlier slopes when it turns each knob. So now and then a step goes uphill. That's fine. The aim is a smaller loss over many steps, not every single one.

Press train and watch the shape form. This network has two layers of eight neurons, plus one more neuron that gives the answer. Unlike the fare walk, it doesn't stop when the loss levels off. Press pause when you've seen enough.

The shade behind the points is its answer for every spot, stronger where it's surer the day is comfortable.

Each row of squares is one first-layer neuron's knobs. Filled means positive, hollow means negative, and bigger means farther from zero. The layers after it decide what those numbers do to the answer.

Point at a row to see the fence that neuron draws.

For readers who know JavaScript, a short forward pass. Two neurons in the first layer, one in the last. neuron adds each input times its weight, plus a bias. The first layer squashes its sums with tanh, and the last layer leaves a raw score, which we turn into a chance of yes. The score and the chance are each rounded to two decimal places.

function neuron(inputs, knobs, squash) {
  let sum = knobs.bias;
  for (let i = 0; i < inputs.length; i++) {
    sum += knobs.weights[i] * inputs[i];
  }
  if (squash) return Math.tanh(sum);
  return sum;
}

function forward(inputs, layers) {
  let numbers = inputs;
  for (let i = 0; i < layers.length; i++) {
    const next = [];
    const squash = i < layers.length - 1;
    for (const knobs of layers[i]) {
      next.push(neuron(numbers, knobs, squash));
    }
    numbers = next;
  }
  return numbers;
}

const layers = [
  [
    { weights: [1, -1], bias: 0 },
    { weights: [0.5, 0.5], bias: -0.5 },
  ],
  [{ weights: [2, -1], bias: 0.1 }],
];
const [score] = forward([0.2, 0.8], layers);
const chance = 1 / (1 + Math.exp(-score));
console.log('Score:', score.toFixed(2));
console.log('Chance of yes:', chance.toFixed(2));

Back to the eight taxi rides, for a short training loop. measure works out their average squared miss and the slope for each knob. These are the same slopes the backward walk finds, written directly for this one rule. step moves both knobs downhill. Over 100 steps at a learning rate of 0.1, the loss falls from 4.43 to 0.33.

const rides = [
  [0.3, 6.25],
  [0.5, 5.35],
  [0.8, 6.6],
  [1.1, 6.15],
  [1.4, 8.2],
  [1.8, 7.5],
  [2.1, 7.85],
  [2.4, 9.2],
];

function measure(knobs, rides) {
  let loss = 0;
  let fee = 0;
  let price = 0;
  for (const [km, paid] of rides) {
    const guess = knobs.fee + knobs.price * km;
    const miss = guess - paid;
    const slope = 2 * miss;
    loss += (miss * miss) / rides.length;
    fee += slope / rides.length;
    price += (slope * km) / rides.length;
  }
  return { loss, fee, price };
}

function step(knobs, rides, rate) {
  const slopes = measure(knobs, rides);
  const fee = knobs.fee - rate * slopes.fee;
  const price = knobs.price - rate * slopes.price;
  return { fee, price };
}

let knobs = { fee: 2, price: 2.5 };
for (let i = 0; i <= 100; i++) {
  if (i % 25 === 0) {
    const loss = measure(knobs, rides).loss;
    console.log('Step', i, 'loss:', loss.toFixed(2));
  }
  knobs = step(knobs, rides, 0.1);
}

The whole loop

So that's the loop. Guess, measure how wrong you were, find which way is downhill for every knob, take a small step, repeat. The fare rule had 2 knobs. The playground's network has 105. ChatGPT learned with the same loop and far more knobs, from text instead of fares. Text has to become numbers. That's where part 2 starts.