LLM Pretraining Explained: Data, Loss, and Chinchilla

The Training Objective

The pretraining objective is autoregressive language modeling: given a context of t tokens, predict the probability of the token at position t+1. Across a training batch of sequences, the model minimizes the average cross-entropy loss between its predictions and the actual next tokens.

This is the negative log-likelihood of the training data under the model. Lower loss means the model is assigning higher probability to the tokens that actually appear in the text. Since the model is trained on a large and diverse corpus, it must assign high probability to tokens that are appropriate in many different contexts: scientific writing, fiction, code, dialogue, legal documents, and everything else that appears in the pretraining data.

Perplexity is a common way to report this loss: it is 2 raised to the power of the average bits-per-token loss (or equivalently, the exponential of the natural-log cross-entropy). A perplexity of 1 would mean the model perfectly predicts every next token. A perplexity equal to the vocabulary size would mean the model has no information at all and assigns equal probability to everything. Real models on general text typically achieve perplexities in the range of 5 to 15 depending on the domain and the model size.

Data: The Deciding Factor

Before 2022, the dominant intuition was that model size was the main driver of performance. The Chinchilla result (Hoffmann et al., 2022) changed that: for a fixed compute budget, the optimal strategy is to train a smaller model on more data rather than a larger model on less data.

But the data quality matters as much as the quantity. Web-crawled text contains spam, boilerplate, low-quality content, and near-duplicate documents. Training on this without filtering produces models that are less capable than models trained on smaller, higher-quality corpora. Standard preprocessing steps include:

• Quality filtering: removing documents that score below a threshold on a classifier trained to distinguish high-quality text from low-quality text.
• Deduplication: removing near-duplicate documents, which otherwise inflate the effective dataset size without adding information. Lee et al. (2021) showed that deduplication significantly improves model quality and reduces memorization.
• Domain mixing: controlling the fraction of data from different sources (web crawl, books, code, scientific papers) to ensure the model develops balanced capabilities.
• Safety filtering: removing documents containing personal information, explicit content, and other categories that should not be in the training distribution.

The Optimizer: AdamW

Almost all large-scale LLM pretraining uses the AdamW optimizer, which combines Adam (adaptive moment estimation) with decoupled weight decay. Adam maintains a running estimate of the first moment (mean) and second moment (variance) of the gradient for each parameter, and uses these to adapt the learning rate per parameter. This makes it much more robust to gradient scale differences across layers than SGD with a fixed learning rate.

Weight decay is added as a separate L2 regularization term on the model weights (decoupled from the gradient update), which helps prevent overfitting and keeps weight magnitudes in a reasonable range. A typical pretraining run uses a warmup of a few thousand steps where the learning rate increases linearly from 0, followed by a cosine decay schedule that reduces the learning rate to near zero by the end of training.

Data curation is where frontier model labs differentiate themselves. The architecture choices are largely converged. The optimizer is standardized. The data is the moat.

The Chinchilla Insight

Kaplan et al. (2020) found that model performance improves as a power law of compute, parameters, and data, with parameters scaling being relatively more important than data. This influenced model designs for the following two years: researchers scaled parameters aggressively while keeping training data fixed.

Hoffmann et al. (2022), training the Chinchilla model, ran a careful set of experiments to find the compute-optimal frontier: for each compute budget C, what is the optimal combination of parameters N and training tokens D? Their finding was that N and D should scale roughly equally: doubling the compute budget should roughly double both the parameter count and the number of training tokens.

Concretely, they found that a compute-optimal model should be trained on approximately 20 tokens per parameter. A 7-billion-parameter model should be trained on 140 billion tokens to be compute-optimal. A 70-billion-parameter model should be trained on 1.4 trillion tokens.

This result had an immediate practical consequence: most models deployed before 2022 were overtrained relative to their parameter count but undertrained relative to what would be compute-optimal. The implication for inference economics was significant: a smaller, more data-efficient model can match the quality of a larger undertrained model while costing less to deploy.

Why Pretraining Is Hard to Redo

Unlike fine-tuning, which can be run in hours on a single node, pretraining a frontier model requires thousands of GPUs running in parallel for weeks or months. The cost of a single pretraining run for a 70B-parameter model is in the range of tens of millions of dollars in compute. This makes every architectural and data decision consequential: a wrong choice discovered at 10% through a training run may not be economically justifiable to fix by restarting.

Infrastructure reliability becomes a training quality issue at this scale. A GPU failure or network partition that interrupts training for hours can cause a "loss spike," a sudden increase in training loss that may take many steps to recover from. Frontier labs invest heavily in fault-tolerant training infrastructure, checkpoint management, and gradient monitoring systems that would be unnecessary at smaller scale.

Parallelism Strategies at Scale

A 70-billion-parameter model trained in 16-bit precision requires roughly 140 GB of memory for the parameters alone. No single GPU holds that much memory. Distributing training across hundreds or thousands of GPUs requires careful parallelism strategies, and the wrong choice can waste most of the available compute budget on communication overhead rather than computation.

Data parallelism replicates the model on every GPU and splits the training batch across GPUs. Each GPU computes gradients for its share of the batch, and gradients are averaged across all GPUs before the optimizer step. This approach is simple and scales well when the model fits on a single GPU, but it does not help when the model is too large for one device.

Tensor parallelism splits individual weight matrices across multiple GPUs. The matrix multiplications in attention and FFN layers are partitioned so that each GPU computes a fraction of each operation, with a communication step (an all-reduce) to combine results. This allows models that are too large for a single GPU to be trained, but it requires high-bandwidth interconnects because the communication volume is proportional to batch size times model dimension, occurring at every layer in every forward and backward pass.

Pipeline parallelism assigns different layers to different GPUs and passes activations between them. GPUs earlier in the pipeline compute forward passes for one micro-batch while GPUs later in the pipeline compute backward passes for a previous micro-batch. When done well, this keeps most GPUs busy most of the time. When done poorly, "pipeline bubbles" leave GPUs idle while waiting for activations or gradients to arrive from adjacent stages.

The Pretraining Data Mix

What goes into the pretraining corpus is at least as important as how much. The composition of the corpus determines what the model knows, how it reasons, what languages it handles well, and what biases it amplifies. Researchers working on Llama 3 reported spending substantial effort on data mix design, adjusting the proportion of web crawl, books, code, and scientific text to improve specific downstream capabilities.

Code data deserves special attention. Including code in the pretraining corpus has been shown to improve not just code generation, but also logical reasoning and instruction-following on non-code tasks. The hypothesis is that code provides dense, structured examples of multi-step reasoning where every step has a verifiable consequence. Training on this data teaches the model patterns of systematic, step-by-step problem decomposition that transfer to other domains.

The ratio of different data sources involves genuine tradeoffs. Increasing the fraction of scientific and technical text improves performance on knowledge-intensive tasks but may reduce fluency on casual dialogue. Increasing the fraction of web text improves breadth but introduces more noise, informal writing, and content of uncertain quality. Increasing the code fraction helps reasoning but can lead to models that produce overly structured, code-like responses to natural language questions. Finding the mix that optimizes for a specific deployment use case requires systematic evaluation, not guesswork.

Curriculum Learning and Data Ordering

The order in which training data is presented to the model during pretraining is not entirely arbitrary. Curriculum learning refers to the strategy of presenting training examples in a purposeful order, typically starting with simpler or higher-quality data and introducing more complex or noisy data as training progresses.

The most commonly reported curriculum strategy in LLM pretraining is quality upsampling near the end of training. After the bulk of training on a broad web corpus, the model undergoes additional training with a higher proportion of high-quality text: books, curated web content, and domain-specific data. This annealing phase is reported to improve benchmark performance without additional compute on the full corpus. The intuition is that the model has already learned the statistical regularities of language from the large corpus, and the annealing phase shapes the final weight configuration to prioritize clean, factual, well-structured content.

Data ordering also interacts with deduplication. Near-duplicate documents in the corpus, if processed in adjacent batches, can cause the model to overfit to those patterns in a way that is hard to recover from. Shuffling the data thoroughly before training, and ensuring that near-duplicates are spread across the training horizon rather than clustered, is standard practice.

What Pretraining Does Not Teach

A pretrained model has absorbed an enormous range of factual associations, stylistic patterns, and reasoning structures from its training data. What it has not learned is how to apply those capabilities in response to instructions. Left to its own devices, a pretrained model will complete any text with statistically plausible continuations. If you give it a question, it may produce more questions (because questions are often followed by more questions in the training data). If you give it a news headline, it may produce the rest of the article.

This is why fine-tuning is necessary. The pretraining objective teaches the model what language looks like. Fine-tuning teaches it what a useful response to a specific type of input looks like. These are different things, and conflating them leads to poor system design. Practitioners who try to use a raw pretrained model as an instruction-following assistant, without fine-tuning, generally find that sophisticated prompt engineering can coax the model into following instructions some of the time, but the behavior is unstable and hard to rely upon.

The distinction between pretraining and fine-tuning also has implications for evaluation. Benchmarks that measure knowledge and language understanding test what was acquired during pretraining. Benchmarks that measure instruction following, refusal of harmful requests, and helpful conversation test what was shaped during fine-tuning. A model that scores well on knowledge benchmarks may still be difficult to use if its post-training alignment was done poorly. Evaluation must cover both dimensions to give a complete picture of a model's practical capabilities.

Continual Pretraining and Domain Adaptation

Pretraining does not have to start from scratch. Continual pretraining, sometimes called domain-adaptive pretraining, takes an existing pretrained model and continues training it on a domain-specific corpus. This allows an organization to inject specialized knowledge into a general-purpose model without bearing the cost of training from random initialization.

The tradeoff is catastrophic forgetting: when a model is trained on a narrow domain corpus, it can forget general capabilities that were encoded during the original pretraining. This happens because the new training signal updates weights that are also responsible for general language understanding, and the narrow data distribution pushes those weights away from the configurations that supported general capability. Mitigations include mixing domain-specific data with a fraction of general data during continual pretraining, using a lower learning rate than was used in the original pretraining run, and evaluating on a broad suite of general tasks throughout the process to detect degradation early.

Domain-specific pretraining has produced meaningful gains in specialized fields. Models pretrained on large corpora of scientific literature, legal documents, or biomedical text consistently outperform general models on domain-specific tasks, even when the general model is larger. For organizations with access to large proprietary text corpora, continual pretraining on that data is often a more cost-effective path to domain capability than prompt engineering or fine-tuning alone, because it shapes the model's internal representations rather than just its output format.

The decision to pursue continual pretraining versus instruction fine-tuning versus retrieval-augmented generation depends on the nature of the domain gap. If the domain requires specialized vocabulary and reasoning patterns that are absent from the base model's training distribution, continual pretraining closes that gap most effectively. If the domain requires applying general reasoning to proprietary documents, RAG is typically cheaper and more maintainable. If the domain requires a specific output format or interaction style, SFT is the right first step. Understanding pretraining as the foundation that determines what the model knows, separate from fine-tuning that shapes how it expresses that knowledge, clarifies which intervention addresses which problem.

Pretraining is ultimately a bet that the statistical regularities of human-generated text at scale encode the knowledge and reasoning patterns needed to be useful across a wide range of tasks. The evidence from the past several years suggests that bet has paid off beyond most early expectations. But it also means that the quality and diversity of the pretraining data remain the binding constraint on what any fine-tuned or prompted version of the model can do. No amount of fine-tuning teaches a model something it had no exposure to during pretraining; it can only shape and direct capabilities that pretraining established. This is the most important architectural fact about LLMs for anyone designing systems around them.