Capstone · Module 7 of 7

Build Your Own Model

Everything in this course — the training loop, data and tokenization, the architecture, scaling, fine-tuning — is about to run in your browser. You are going to train a real language model from scratch, watch its loss fall in real time, and generate text from what it learned. No libraries. No cloud. Just you, some nursery rhyme text, and gradient descent.

By the end of this capstone you will have

Trained a character-level neural language model entirely in your browser
Watched the loss curve fall as gradient descent optimises the model's weights
Generated new text by sampling from the trained model
Traced every step in the pipeline back to a concept from the course

What You Are Building

You are going to train a character-level language model: a small neural network that learns, from a short training corpus, which character is likely to follow any given sequence of characters. Given the sequence "mar", a trained model should assign high probability to "y", because that pattern appears in the training text.

The model is a simple two-layer neural network: an input layer that encodes the last three characters as a one-hot vector, a hidden layer with 64 neurons and a tanh activation, and an output layer that produces a probability distribution over the entire vocabulary. It has about 5,700 parameters. A frontier language model has hundreds of billions. The mathematics, however, are identical: forward pass, loss calculation, backward pass, weight update. What you are training is a miniature version of the same process that produced the models you use every day.

3context window (chars)

64hidden neurons

~5.7Ktotal parameters

600training epochs

The Training Corpus

The model will train on a short nursery rhyme: "Mary Had a Little Lamb" (traditional, public domain). The full text is 228 characters long. Every character — letters and spaces — is a token. The model sees three characters at a time and predicts the next one. This connects directly to Module 3: every training example is a (context, target) pair derived from the corpus.

How the Training Loop Runs

When you click Start Training, the browser runs 600 epochs of full-batch gradient descent. Each epoch is one complete pass through all the training examples. For every example, the model makes a prediction (forward pass), the loss measures how wrong it was, and backpropagation computes how each weight contributed to the error. Then every weight is nudged in the direction that reduces the loss. After 600 epochs, you should see the loss fall from around 3.0 (random guessing over the vocabulary) to below 1.4 — the model has learned the character patterns in the training text.

The loss curve below is drawn in real time as training progresses. Watch it fall. When it flattens, the model has largely converged: the easy patterns have been learned and further gains require more data or a bigger model. That flattening is the same phenomenon you read about in Module 5 — it just happens much faster at this scale.

Live Training — Character-Level Language Model Ready

Click Start to train the model. The loss curve updates every 20 epochs. Training takes 3 to 6 seconds depending on your device. After training completes, the text generator will appear.

Epoch 0 / 600

Training loss over epochs — directional illustration of gradient descent convergence on this corpus

Generate text from your trained model

Temperature: 0.8

Your generated text will appear here.

Lower temperature = more predictable. Higher = more varied. The model learned character patterns only from the training text above.

What Just Happened

If you watched the loss fall and then generated some text, you just ran every major concept from this course in sequence:

Module 1 — What a Model Is: you built one. A set of 5,000 numbers (weights) that, together, map an input (three characters) to an output (a probability over the next character). Before training, those weights were random. After training, they encode the character patterns of your corpus.
Module 2 — The Training Loop: you ran 600 iterations of forward pass, loss calculation, backward pass, and weight update. The loss curve is the visual record of that loop working.
Module 3 — Data and Tokenization: the training corpus was tokenized at the character level. Each character is a token. The training examples are (context, target) pairs: every consecutive group of four characters in the text becomes one example. That is exactly the sliding window approach described in Module 3.
Module 4 — The Transformer Architecture: this model has no attention mechanism. It cannot learn long-range dependencies. A model trained on the same corpus with a Transformer architecture would learn richer patterns — but would also need more data to justify its complexity. The Transformer was invented to solve exactly the limitation you just bumped into.
Module 5 — Scaling and Evaluation: your model has about 5,700 parameters and roughly 229 training characters. Scaling up both — more parameters and far more data — is what produces frontier models. The shape of the loss curve you saw is the same power-law relationship described by Kaplan et al. (arXiv:2001.08361), just at a much smaller scale.
Module 6 — Fine-tuning and Alignment: the weights you trained are the equivalent of a pre-trained model. If you wanted this model to specialise further — say, to generate only rhyming continuations — you would fine-tune it on a smaller dataset of rhyming pairs, starting from the weights you just learned, not from random initialisation.

The whole pipeline in one browser tab A corpus gives you data. Tokenization converts that data into sequences the model can process. The training loop optimises the model's weights to predict those sequences. Evaluation (loss) tells you how well the model is doing. The trained model can then generate new sequences by sampling from what it learned. You have now seen every step. The only difference between this model and a frontier language model is scale.

Capstone Quiz · 4 Questions

1. In the model you just trained, what did the loss of approximately 3.0 at the start of training represent?

2. The model you trained uses the last three characters to predict the next one. Why would a Transformer architecture produce better results on the same corpus?

3. Your model trained on 228 characters of nursery rhyme text. Why would adding more training text (while keeping the model size the same) improve the model's generated output?

4. Suppose you wanted to adapt your trained model to generate text in a specific rhyming style. Which approach best describes what you would do, and why?

← Module 6: Fine-tuning and Alignment Course Home →