LLMs from Scratch
Home› AJ University› LLMs from Scratch› Module 3
Advanced 35 min Module 3 of 6
Module 3 of 6

Pretraining

Pretraining is where an LLM becomes capable. Not smart. Not aligned. Capable. The model reads trillions of tokens from the web, books, and code, and learns to predict what comes next. That single task, run long enough on enough data, forces the model to learn grammar, facts, reasoning patterns, and world knowledge as instrumental tools for accurate prediction. This module explains exactly how that happens and what the Chinchilla scaling laws say about how to do it efficiently.

By the end of this module you will be able to

The Training Objective: Predict the Next Token

Pretraining has one job: given a sequence of tokens, predict the next token. That is it. No labels, no human annotations, no task specification. Just text, and the task of predicting what comes next in that text.

This is called autoregressive language modeling, and it has a remarkable property: the right answer is already in the data. If the training text says "The capital of France is Paris," then after seeing "The capital of France is," the correct next token is "Paris." The model does not need to be told what Paris is or that it is a capital. It just needs to predict accurately. Accurate prediction of "Paris" in that context requires representing the fact that France has a capital and that it is called Paris.

This is why pretraining on text produces models with broad world knowledge: to predict accurately across billions of sentences covering history, science, law, medicine, code, and casual conversation, the model must develop internal representations of all those domains. Knowledge is an emergent byproduct of accurate prediction.

The Loss Function: Cross-Entropy

At each step, the model produces a probability distribution over the vocabulary: each of the V tokens gets a probability, all positive, summing to 1. The loss function measures how well this distribution matches the truth. The correct token should have high probability. All other tokens should have low probability.

Cross-entropy loss measures this precisely. If the correct token has true probability 1 and the model assigns it probability p, the loss is -log(p). When p = 1.0 (perfect prediction), loss = 0. When p = 0.5, loss = 0.693. When p = 0.01, loss = 4.6. The loss penalizes overconfidence in wrong predictions and rewards high probability on correct ones.

Averaged across all token positions in a batch, the cross-entropy loss is a single number that tells you how well the model is predicting the training data. Models are often evaluated on perplexity, which is just exp(cross-entropy loss). A perplexity of 10 means the model is about as surprised by each token as if it were choosing uniformly among 10 equally likely options. Lower is better.

How the Model Learns: Gradient Descent at Scale

Training a language model is an optimization problem. The weights of the model are initialized randomly. The cross-entropy loss is computed on a batch of training text. Backpropagation computes the gradient of the loss with respect to every weight: the direction each weight should move to reduce the loss. The optimizer then takes a step in that direction.

The optimizer used in modern LLM training is usually AdamW, an adaptive learning rate optimizer that maintains per-parameter momentum and variance estimates. This is critical at the scale of billions of parameters: different weights need different learning rates, and AdamW adapts these automatically based on the history of gradients for each parameter.

Training happens over many steps. Each step processes one batch (often 512 to 2,048 sequences of ~2,048 tokens each), computes the loss, and updates the weights. Modern pretraining runs for hundreds of thousands of steps on hundreds of GPUs running in parallel. The total compute can be measured in FLOPs: a floating-point operation, the basic unit of neural network computation.

Scale matters GPT-2 was trained on 40 GB of text. GPT-3 was trained on roughly 570 GB of filtered text across ~300 billion tokens. Current frontier models train on trillions of tokens. The capability improvements at each scale have been consistently larger than researchers expected, a pattern that motivated the systematic study of scaling laws.

Data Curation: Where Training Data Comes From

Raw internet text is not suitable for training. It contains spam, duplicate content, low-quality text, personal information, and content that would actively harm model quality. Data curation is the process of filtering this raw material into a high-quality training corpus.

The key steps in a modern data pipeline are: URL and language filtering (remove non-target-language content and known low-quality domains); deduplication (remove near-duplicate documents, which otherwise overfit the model to repeated content); quality filtering (score documents by heuristics like average word length, symbol ratio, and classifier-based quality scores); and content filtering (remove harmful content, personally identifiable information, and copyright-sensitive material).

Deduplication deserves special attention. Web crawls contain massive amounts of near-duplicate content: the same news article republished on hundreds of sites, boilerplate legal text, forum threads with repeated signatures. Training on duplicate data wastes compute and biases the model toward overrepresented content. Lee et al. (arXiv:2107.06499) showed that deduplication significantly improves model quality for a given compute budget.

The Chinchilla Scaling Laws: How Much Data and How Large a Model?

Before 2022, the dominant practice was to train as large a model as the compute budget allowed, then train it until diminishing returns set in. A 2022 paper changed this calculus. Hoffmann et al. "Training Compute-Optimal Large Language Models" (arXiv:2203.06840), commonly called the Chinchilla paper, trained a suite of models of different sizes on different amounts of data and measured the resulting loss.

Their finding: for a given compute budget C (measured in FLOPs), the optimal strategy is to scale model parameters N and training tokens D roughly equally. The compute-optimal ratio is approximately N ~ D: one parameter per training token. A 70B parameter model, for example, should be trained on roughly 1.4 trillion tokens to be compute-optimal under this analysis.

This was a significant departure from the prior approach. GPT-3, by this analysis, was trained on far fewer tokens than its parameter count implied was optimal. Chinchilla itself (70B parameters, 1.4T tokens) matched or exceeded models several times larger that had been undertrained. The practical implication: for many organizations, the better investment is more data rather than a larger model, up to the compute-optimal point.

Fig 1 · Chinchilla Scaling: Compute-Optimal Model Size and Tokens (directional illustration based on Hoffmann et al. arXiv:2203.06840)
TRAINING TOKENS (D) MODEL PARAMS (N) COMPUTE-OPTIMAL FRONTIER: N ~ D Overtrained model (large N, small D) Compute-optimal (N and D balanced) FEW TOKENS MANY TOKENS SMALL LARGE C1 C2
Python · Cross-Entropy Loss for Next-Token Prediction
import numpy as np

def cross_entropy_loss(logits, target_token_id):
    """
    logits: raw scores from the model, shape (vocab_size,)
    target_token_id: the correct next token index
    Returns: scalar loss value
    """
    # Softmax: convert logits to probabilities
    # Subtract max for numerical stability
    shifted = logits - logits.max()
    exp_logits = np.exp(shifted)
    probs = exp_logits / exp_logits.sum()

    # Cross-entropy: -log(probability of correct token)
    loss = -np.log(probs[target_token_id] + 1e-10)
    return loss, probs[target_token_id]

# Simulate model output for vocabulary of 50,000 tokens
vocab_size = 50_000
np.random.seed(0)
logits = np.random.randn(vocab_size)

# Correct next token is "Paris" at index 12345
target = 12345

# Before training: random model, low probability on correct token
loss, prob = cross_entropy_loss(logits, target)
perplexity = np.exp(loss)
print(f"Random model — Loss: {loss:.3f}, P(correct): {prob:.6f}, Perplexity: {perplexity:.1f}")

# After training: model assigns higher logit to correct token
logits_trained = logits.copy()
logits_trained[target] += 8  # simulates learned preference for "Paris"
loss_t, prob_t = cross_entropy_loss(logits_trained, target)
print(f"Trained model — Loss: {loss_t:.3f}, P(correct): {prob_t:.4f}, Perplexity: {np.exp(loss_t):.1f}")
Try this

Run the code above and observe the perplexity values. A random model over 50,000 tokens has perplexity near 50,000 (it is equally uncertain about every token). The trained model with a boosted logit for the correct token has much lower perplexity. Now calculate: what logit boost would you need to achieve perplexity of 10? (Hint: set prob_t = 0.1 and work backward from the softmax formula. The answer will tell you how much the model needs to push the correct token above all others.)

NTP
Next-Token Prediction
The pretraining objective. Given the sequence so far, predict the next token. All world knowledge is instrumental: the model learns it because accurate prediction requires it.
CE
Cross-Entropy Loss
Measures the gap between the model's probability distribution and the correct answer. Minimizing this loss makes the model assign more probability to correct tokens.
CL
Chinchilla Laws
For a given compute budget, scale model parameters and training tokens roughly equally. Undertrained large models waste compute; smaller compute-optimal models can outperform them.
Interactive 3: Chinchilla Compute Budget Explorer Try it

Set a compute budget in GPU-days and see the compute-optimal model size and token count according to the Chinchilla scaling analysis (Hoffmann et al. arXiv:2203.06840). This is a directional illustration based on the paper's findings.

5,000
Optimal model size: --    Optimal token count: --
If pretraining just predicts text, why is the model so much better at some tasks than others? ▼
The quality of pretraining on a task is largely determined by how much relevant data exists in the training corpus and how well it represents the task's structure. Code generation is something LLMs are generally good at because code is abundant on the web, has predictable structure, and rewards consistent internal consistency (programs that run correctly appear as coherent training examples). Mathematical reasoning in natural language is harder because correct mathematical proofs are less abundant and the connection between text and formal reasoning is less direct. Tasks that appear rarely in the training distribution, or that require reasoning that text rarely exemplifies, will be weaker. This is why fine-tuning matters so much: it specializes a generally capable base model on a specific distribution.
Knowledge check
What is the key insight from the Chinchilla paper (Hoffmann et al. arXiv:2203.06840) about compute-optimal training?
Correct. Hoffmann et al. found that previous large models were undertrained: they were scaled up in parameters but not in data proportionally. The compute-optimal approach scales both equally, meaning a smaller, well-trained model can match a larger, undertrained one on the same compute budget.
Not quite. The Chinchilla paper's central finding is that compute-optimal training requires scaling both model size and data roughly equally. Many large models before Chinchilla were simply undertrained: they used far fewer tokens than their parameter count warranted.
Why does a model trained only to predict next tokens also learn factual knowledge about the world?
Correct. To predict that "Paris" follows "The capital of France is," the model must represent the relationship between France and Paris. Knowledge is an instrumental byproduct of accurate prediction, not a separately trained objective.
Not quite. The model learns factual knowledge because that knowledge is necessary for accurate prediction. It does not memorize sentence by sentence or receive factual labels. Knowledge emerges because the prediction task requires it.
Before you go
Reflection: A pretrained base model can complete code, write poetry, and summarize articles. But it will also complete harmful content if you start a harmful prompt. Why does pretraining produce capability without safety?
You might also like
Next Module
Fine-Tuning and RLHF
Blog
LLM Pretraining Explained
Related
Scaling Laws
Was this module helpful?

Advising on pretraining data strategy or compute budget allocation for your model development?

Schedule a call with Arjun

References

  1. Hoffmann, J. et al. "Training Compute-Optimal Large Language Models." arXiv:2203.06840. arxiv.org/abs/2203.06840
  2. Kaplan, J. et al. "Scaling Laws for Neural Language Models." arXiv:2001.08361. arxiv.org/abs/2001.08361
  3. Brown, T. et al. "Language Models are Few-Shot Learners." arXiv:2005.14165. arxiv.org/abs/2005.14165
  4. Lee, K. et al. "Deduplicating Training Data Makes Language Models Better." arXiv:2107.06499. arxiv.org/abs/2107.06499
  5. Loshchilov, I. and Hutter, F. "Decoupled Weight Decay Regularization (AdamW)." arXiv:1711.05101. arxiv.org/abs/1711.05101
  6. Gao, L. et al. "The Pile: An 800GB Dataset of Diverse Text for Language Modeling." arXiv:2101.00027. arxiv.org/abs/2101.00027
← Module 2: Tokenization and Embeddings Module 4: Fine-Tuning and RLHF →