[Bartosz Jarocki](https://jarocki.me/index.md) · [notes](https://jarocki.me/notes.md) · [work](https://jarocki.me/work.md) · [about](https://jarocki.me/about.md)

2026.10.16 · ai, learning

# A tiny GPT, trained in your browser

This is the part where you train a GPT. A tiny one, on Shakespeare's plays, right here in this tab, and you get to watch it go from noise to familiar letter groups and some real words.

It's the phone keyboard from part 2 with a longer memory and a different reading list. It still guesses one letter at a time. Nobody tells it about characters, speeches, or spelling. All it gets is the text and one job: guess what comes next.

Everything from the first three parts is inside. It's a rule with knobs, tuned using the slopes of its mistakes. It reads letters as lists of numbers. And attention lets each letter read the ones before it.

## A handful of passages

Each turn, the trainer grabs a handful of short passages from the text. That handful is a **batch**. Reading `to be`, the model sees `t` and guesses `o`, sees `to` and guesses the space, and so on, and it scores all those guesses together. Part 3's no-peeking rule keeps each position from seeing its own answer, even though the computer works on many positions at once.

Then the trainer averages the model's surprise over the batch, works out every knob's slope, and adjusts the knobs. That whole turn is one **step**. Like part 1's playground, it uses Adam, which remembers recent slopes and their sizes. A single step won't always help. What counts is the trend over many of them.

## Practice and an exam

We hold back a tenth of the text and never train on it. Think of practice cards and an exam with different questions. A model that only memorized the practice cards can still flunk the exam.

The model's average surprise on practice text is its **training loss**. Its average surprise on the text it never trains on is its **validation loss**. Lower is better: it means the model gave the real next letters higher chances.

The chart calls these lines practice and exam. Both use the same fixed sets of passages at each check, so the comparisons are fair. The horizontal axis counts training steps. The chart counts steps, not seconds, so a slow phone draws the same lines, just later.

If the practice line keeps falling while the exam line flattens or rises, the model is learning details that don't carry over. That is **overfitting**. A single bump isn't enough to diagnose it. Look for the lines pulling apart over several checks.

Press **Train** to start from random knobs. **Pause** whenever you like, and the current numbers stay. If you don't want to wait, **Load the trained one** swaps in saved numbers from a longer run. Every 150 steps the model writes a new sample. The counter stalls briefly while it does, along with the check of both lines.

> **Interactive demo: Train a tiny GPT on Shakespeare.** The practice and exam losses fall as a small model learns from lowercase Shakespeare. Three recorded samples show how its writing changes. With JavaScript, train your own model or load the saved one.
>
> Try it at https://jarocki.me/notes/a-tiny-gpt

## Write with the current knobs

Writing doesn't train the model. It guesses a letter, picks one using the loaded die from part 2, then feeds that letter back in to guess again. The numbers stay fixed throughout. Its memory holds at most the last 64 letters.

Early samples are mostly noise. Later ones pick up familiar letter groups and some real words. Some lines look like a play and mean nothing at all.

## Keep a saved model

A saved set of knobs is a **checkpoint**, like a save file in a game. This one is the same model after 12,000 offline steps: 30,055 numbers in 120,220 bytes, plus a small file saying where each number goes. That's all it is. There's no Shakespeare in there, no list of replies, just numbers. It doesn't hold the trainer's memory of past slopes either.

Loading it pauses training and replaces the current model. Train again starts a fresh model so you can watch it learn from noise.

Everything runs in a background worker in this tab. Training pauses when the demo leaves the screen or you switch tabs. Nothing you train here gets uploaded, and leaving the page throws your run away.

## Look inside the model

The letter map from part 2 and the attention arcs from part 3 give two views of the learned numbers. The map shows the numbers assigned to each symbol before it reads a sentence. The arcs depend on the sentence as well as those numbers.

This model reads all 39 symbols in the lowercase text, including spaces, line breaks, and punctuation. Each symbol has 32 learned numbers. A screen can't show all those directions at once. We squeeze the 32 numbers into three directions that preserve as much of their spread as possible. This method is called principal component analysis, or PCA.

Squeezing 32 numbers into 3 loses a lot, so take the picture with a grain of salt. Two symbols can look close here and still be far apart in the full 32. The axes are mixtures of the original numbers, not named properties such as vowel or meaning. And if the cloud spins while the model trains, that's not progress by itself. The exam line is the real check.

Train or load the saved model above, then scroll down. These views use the current model in your tab. Training pauses while its controls are off screen. You can also train, reset, or sample from the map. Reset returns the shared model to its starting numbers. Without JavaScript, both pictures show the recorded saved model instead.

> **Interactive demo: Your model's 39 symbols in three PCA directions.** All 39 symbols from lowercase Shakespeare are shown. Each symbol's 32 learned numbers are squeezed into three PCA directions. The recorded picture uses the saved model. A middle dot marks a space and an arrow marks a line break.
>
> Try it at https://jarocki.me/notes/a-tiny-gpt

The attention heads answer a different question. At this point in this sentence, which earlier letters does each head read? A dark arc means a bigger share. It doesn't prove that the model understands the sentence, or that a head always has one job. The tiny model still can't reliably solve the bag-and-car example from part 3.

> **Interactive demo: Attention from your current model.** Eight heads read the sentence about the bag and the car. Select a letter to see what each head reads from that position. These arcs use your current model's complete weights. Without JavaScript, the arcs show the saved model's last letter.
>
> Try it at https://jarocki.me/notes/a-tiny-gpt

## How small is tiny?

Count every learned number in the model and you get its **parameter count**. These are the knobs training can turn, including the embeddings and the numbers inside each block. More knobs give a model room to learn more patterns, but it also needs enough examples to learn useful settings.

The table compares our saved model with the largest models in the GPT-2 and GPT-3 papers. A training token here means a next-token target processed during a training step. Reading the same passage again counts again. It isn't the number of different words in the reading list.

Learned numbers and text used for training

| Model | Parameters | Training tokens |
| --- | --- | --- |
| Our saved model | [30,055](https://jarocki.me/models/shakespeare.json) | [12,288,000 character targets](https://github.com/BartoszJarocki/jarocki.me/blob/503d79067107c5f5ac1c02c71bf66627c9c5a29a/src/lib/ml/gpt.ts#L569-L605) |
| GPT-2, largest | [1,542 million, as reported in table 2](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf#page=4) | [Not reported. The paper gives 40 GB of text](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf#page=3) |
| GPT-3, largest | [175 billion](https://arxiv.org/pdf/2005.14165#page=8) | [300 billion](https://arxiv.org/pdf/2005.14165#page=8) |

Our total is 12,000 steps times 16 passages times 64 next-letter guesses. It excludes the checks and generated samples, which don't update the knobs. A shorter run in your tab has processed fewer targets. The saved weights use four bytes per parameter, so 30,055 times 4 gives the 120,220 bytes mentioned above.

GPT-2's 40 GB is the size of its collected text, not a count of training tokens. The paper doesn't give the total number of tokens processed during training, so the table leaves it unknown. GPT-3 does report that total. Its reading list contains about 499 billion tokens, obtained by adding the dataset sizes in [table 2.2](https://arxiv.org/pdf/2005.14165#page=9). It trains on 300 billion draws from that mixture, repeating some sources more often than others. Dataset size and training tokens are different quantities.

Our tokens are single characters. GPT-2 and GPT-3 use pieces of words, so their token counts don't measure the same amount of text per token. The comparison shows the difference in size, not a fair contest of their losses. Increasing the model and its training together is what people mean here by **scale**.

## What we left out

Training on a large reading list to predict the next piece of text gives a model that can continue text. That first broad training is **pretraining**. Our Shakespeare run is a small example of it. It learns the shape of plays, not how to hold a helpful conversation.

You can then take an existing model and train its knobs further on a more focused set of examples. That is **fine-tuning**. One route to an assistant trains on example conversations, then uses people's rankings of different replies to guide more training. The [InstructGPT paper](https://arxiv.org/abs/2203.02155) describes this route. It's a concrete example of what separates a text continuation model from a system that tries to follow instructions.

An assistant such as ChatGPT has far more knobs and learns from far more text than this demo. Training on conversations and feedback also changes which replies it tends to choose. And none of that makes an answer true. A model can learn to produce a convincing mistake. The papers in the table describe published models, not the undisclosed size or training budget of today's ChatGPT.

We also skipped the engineering that makes big models practical: splitting work across specialized hardware, reusing calculations while writing, tokenizers for more languages than our little alphabet, and the hard work of choosing and checking the training text. That last one matters, because a model can repeat the errors and prejudices in what it reads.

Still, you just ran the real thing. A passage supplies the next-letter answers. The model turns its current knobs into guesses, measures its surprise, and works backward to find slopes. Training adjusts the knobs and tries again. Writing uses the resulting guesses without adjusting them. That's the loop from the taxi fares in part 1, now with a reading list, a longer memory, and attention. And the numbers it learned are sitting right there in your tab.

## words from this note

- batch · A handful of practice passages whose next-letter guesses are scored together
- step · One turn of guessing, averaging surprise, finding slopes, and adjusting the knobs
- training loss · The model's average surprise on its practice text
- validation loss · The model's average surprise on exam text that never trains its knobs
- overfitting · Learning practice-text details that do not carry over, so the practice and exam lines pull apart
- checkpoint · A save file of learned knob settings, ready to load without repeating training
- parameter count · The number of learned knobs, including embeddings and the numbers inside each block
- scale · The size of the model and how much training it receives
- pretraining · The broad first training that teaches a model to continue text
- fine-tuning · Further training of an existing model on a more focused set of examples

## the ai series

- part 1 · [How a machine learns](https://jarocki.me/notes/how-a-machine-learns.md)
- part 2 · [Words as numbers](https://jarocki.me/notes/words-as-numbers.md)
- part 3 · [Paying attention](https://jarocki.me/notes/paying-attention.md)
- part 4 · [A tiny GPT, trained in your browser](https://jarocki.me/notes/a-tiny-gpt.md)

---

- [← all notes](https://jarocki.me/notes.md)
- [← part 3](https://jarocki.me/notes/paying-attention.md)