LLM Scaling Laws: What They Predict and Where They Break Down

The Kaplan Power Laws

Kaplan et al. (2020) trained a large number of transformer language models at different scales and measured how test loss changed as a function of three variables: the number of model parameters (N), the number of training tokens (D), and the total compute budget (C). Their central finding was that test loss follows a power law with each of these variables when the other two are held near optimal:

These exponents imply that doubling the number of parameters reduces loss by about 5%, doubling data reduces loss by about 6.4%, and doubling compute reduces loss by about 3.4%. The relationships are remarkably consistent across many orders of magnitude, spanning models from millions to tens of billions of parameters.

The practical implications are significant. A power law relationship between compute and performance means that you can extrapolate: if you measure loss at several small compute budgets, you can fit the curve and predict where loss will be at a 10x or 100x larger budget. This is exactly what labs do when planning expensive training runs: they run a series of small scaling experiments to fit the curve and validate predictions before committing to a full run.

The Chinchilla Correction

The Kaplan paper found that parameters scaled more steeply than data in the compute-optimal regime, suggesting that given a fixed compute budget, it was better to train a larger model on less data. This drove the model design decisions of 2020 and 2021.

Hoffmann et al. (2022) repeated the analysis more carefully, varying both model size and data quantity more systematically. They found that the Kaplan result was biased: the original experiments had not trained data-optimally across all model sizes. When corrected, parameters and tokens should scale roughly equally, meaning the compute-optimal token count is approximately 20 times the parameter count.

This is not a minor adjustment. It means that GPT-3 (175B parameters, trained on 300B tokens) was significantly undertrained relative to its parameter count. A model with a small fraction of GPT-3's parameters but trained on 10 to 20 times more data could match GPT-3's quality at a fraction of the inference cost. This is exactly what Chinchilla (70B parameters, 1.4T tokens) demonstrated.

Emergent Capabilities

Wei et al. (2022) documented a different kind of scaling behavior: tasks where small models score near zero and performance jumps sharply as model scale crosses some threshold. They called these "emergent capabilities" and identified them across 57 tasks covering arithmetic, symbolic reasoning, and language understanding. The appearance of in-context learning, multi-step reasoning, and chain-of-thought generation were cited as examples.

Emergence is striking because it suggests that scale creates qualitatively new capabilities, not just quantitative improvements. If true, it would imply that there are thresholds below which certain tasks are simply not achievable, and above which they suddenly become possible.

Whether emergence is a property of the model or a property of the evaluation metric is not a philosophical question. It has real consequences for how you plan deployments and interpret benchmark results.

The Mirage Argument

Schaeffer et al. (2023) challenged the emergence narrative. Their argument is that apparent emergent capabilities are often an artifact of the evaluation metric, not a genuine property of the underlying model.

Consider a task where the model must get every step of a multi-step problem correct to receive any credit (all-or-nothing accuracy). Small models get zero because they fail on at least one step. As scale increases, the probability of getting each individual step correct increases smoothly. The probability of getting all steps correct is the product of per-step probabilities, which grows rapidly once each individual probability crosses 50%. The result is a sharp increase in task accuracy that looks like emergence but is actually just smooth per-step improvement made invisible by the nonlinearity of the metric.

When Schaeffer et al. replaced discontinuous metrics (exact match, all-or-nothing accuracy) with continuous metrics (token-level probability or partial credit), the sharp transitions disappeared and performance improved smoothly with scale. This does not prove that all emergence is metric artifact, but it does establish that many of the most-cited examples can be explained this way.

What Scaling Laws Do Not Predict

Scaling laws predict how loss changes with scale under a fixed training recipe. They do not predict how fine-tuned behavior changes, how instruction-following capability changes, or how performance on specific downstream tasks changes. A model with lower pretraining perplexity will generally perform better on downstream tasks, but the relationship is not simple or universal: different tasks benefit differently from scale, and some tasks appear to saturate quickly while others continue improving.

Scaling laws also do not predict failure modes. A model with excellent aggregate perplexity can still produce factually wrong answers with high confidence, reason incorrectly about multi-step problems, and behave inconsistently across superficially similar prompts. These failures are not captured in the loss metric that scaling laws track.

Perhaps most importantly, scaling laws do not generalize across training distributions. A law fit on web-crawled English text will not accurately predict loss on code, on a new language, or on a specialized domain. The power-law structure is a property of the architecture and optimizer under a specific data distribution, not a universal law of deep learning.

Practical Use of Scaling Laws

Despite these limitations, scaling laws are genuinely useful for the decisions they were designed to inform: planning compute allocation, estimating the cost of a training run, and comparing architectural choices at small scale before committing to large runs. A team planning a training run on a specific data distribution should run small-scale experiments to fit their own scaling curve rather than relying on published exponents from a different data distribution.

The most reliable use of scaling laws is relative comparison: not "this model will achieve loss X," but "option A is expected to achieve lower loss than option B at the same compute budget, by this amount." The relative relationships are more stable than the absolute predictions.

Data-Constrained Scaling

The Chinchilla result implies a specific data requirement: compute-optimal training requires approximately 20 tokens per parameter. For a 70-billion-parameter model, that is 1.4 trillion tokens. As models grow larger, the data requirement grows proportionally. At some point, this requirement exceeds the available supply of high-quality text.

Muennighoff et al. (2023) studied what happens when the available data is exhausted before training reaches the compute-optimal point. Their finding was that repeating data, a practice generally assumed to be harmful, degrades performance less than might be expected when the data is repeated a small number of times, typically up to four repetitions. Beyond four repetitions, performance degrades more sharply. This result is practically important because many organizations training domain-specific models will face data constraints before hitting compute constraints.

The data constraint problem also motivates interest in synthetic data. If a model can generate training examples that are sufficiently diverse and accurate, those examples could supplement the finite supply of human-generated text. The risk is that models trained on their own outputs may amplify their own errors through a feedback loop, a problem sometimes called model collapse. Maintaining a substantial fraction of real human-generated data appears to be important for preventing this.

Inference-Time Scaling: A Different Axis

The scaling laws discussed so far describe how model quality improves with training compute, parameters, and data. A separate scaling relationship governs inference: how model quality changes as more compute is applied at inference time, through techniques like chain-of-thought generation, majority voting over multiple samples, and iterative self-refinement.

Repeated sampling followed by a verification or selection step can improve accuracy on well-defined tasks substantially. For mathematical reasoning problems with verifiable answers, generating 64 candidate solutions and selecting the most frequent correct answer outperforms single-sample generation by a large margin. The tradeoff is that inference cost scales linearly with the number of samples, so this approach is only practical when the output can be verified programmatically or with a reliable automated evaluator.

The existence of inference-time scaling creates an additional degree of freedom in system design. A smaller model run with many inference samples may match or exceed the quality of a larger model run with a single sample at lower total cost, depending on the task. Understanding when to scale training versus when to scale inference is an active research question, and the answer depends on the specific task, the available compute budget, and whether the task admits reliable automated verification.

Scaling Laws in Practice: How Labs Use Them

When a lab plans a large training run, they do not simply guess at the optimal configuration. They run a systematic set of scaling experiments at small compute budgets, fit a power-law curve to the results, and extrapolate to the target budget. This allows them to compare architectural variants, data mixes, and optimizer configurations before committing to the cost of a full run.

The fit is imperfect. Extrapolating across many orders of magnitude introduces errors, and unexpected phenomena (a sudden improvement from a new data source, a training instability that affects only large models) can cause the actual large-scale result to deviate from the prediction. Nonetheless, even an imperfect prediction is more informative than no prediction at all, and labs that fit scaling curves rigorously make more efficient use of their compute budget than those that rely on intuition.

The HELM evaluation framework (Liang et al., arXiv:2211.09110) provides an important complement to loss-based scaling analysis. By measuring performance across dozens of diverse tasks and scenarios, it reveals whether improvements in training loss translate into improvements on the tasks that actually matter for deployment. Models that improve rapidly on perplexity but slowly on diverse downstream tasks may be overfitting to distributional properties of the training data without generalizing the capabilities that practitioners need. Combining loss-based scaling analysis with comprehensive capability evaluation gives a more complete picture of where model development resources are best spent.

Scaling Laws and Enterprise AI Planning

For organizations that are consumers of frontier models rather than developers of them, scaling laws matter in a different way. They inform expectations about how model quality will change over time and help distinguish genuine capability improvements from benchmark optimization. When a new model release claims substantially better performance on key benchmarks, it is worth asking whether the improvement holds on tasks that represent your specific use case, not just on the metrics highlighted in the release announcement.

The power-law structure also informs build-versus-buy decisions for domain-specific AI capabilities. If a general frontier model scores substantially lower than a specialized model on a domain-specific evaluation, the question is whether that gap is attributable to scale, data, or fine-tuning. If it is primarily a data issue, the gap may close as frontier models scale and train on more diverse data. If it is a fine-tuning issue, the gap may persist regardless of scale, and a fine-tuned domain model retains a durable advantage.

Understanding the source of a capability gap is essential for making durable technology choices. Organizations that treat current capability gaps as permanent may over-invest in specialized solutions that become less necessary as frontier models improve. Organizations that assume all gaps will close through scaling may under-invest in domain-specific fine-tuning work for tasks where pretraining exposure alone cannot close the gap. Scaling laws are a tool for calibrating those expectations with empirical evidence rather than intuition, applied to the actual tasks and data distributions that matter for a specific deployment.

The Limits of Log-Linear Thinking

Scaling laws are expressed in log-log space, where the power-law relationship appears as a straight line. This representation makes the relationship look tidy and predictable. But plotting on log-log axes can create the impression of more regular behavior than actually exists. Deviations from the power-law trend that look small on a log scale correspond to large absolute errors in the predictions derived from those trends.

More fundamentally, scaling laws are fit on a narrow slice of the model design space. They assume a fixed architecture (dense transformer), a fixed training recipe (standard next-token prediction with AdamW), and a fixed data distribution. Changes to any of these break the extrapolation. A MoE architecture with the same parameter count as a dense model does not lie on the same scaling curve. A model trained with reinforcement learning from human feedback occupies a different capability frontier than one trained purely on next-token prediction at the same compute budget.

The responsible use of scaling laws is as one input among several, combined with empirical validation at intermediate scales before extrapolating to the full budget. Teams that use scaling law extrapolations as precise predictions rather than rough directional signals will systematically overestimate the reliability of their forecasts, especially for capabilities that go beyond the perplexity metric on which most scaling law research is based.

The appropriate attitude toward scaling laws is empirical rather than dogmatic. They are useful because they are approximately true across a wide range of conditions, not because they are exact. Treating them as engineering guides for compute allocation and architectural comparison, while maintaining awareness of their limitations for task-level prediction and cross-distribution generalization, captures most of their practical value without the overconfidence that comes from applying them outside their validated regime. For anyone building systems that depend on continued model improvement over time, scaling laws are the best available framework for thinking about that improvement systematically, even if they cannot predict it precisely.

The field is still learning where the boundaries of these laws lie. Whether they continue to hold through many more orders of magnitude of compute, whether architectural changes like MoE or inference-time scaling shift the frontier in ways that invalidate current extrapolations, and whether data constraints eventually impose a hard ceiling on scaling returns are open questions. What is clear is that the power-law relationship between compute and capability has held across enough experiments and enough orders of magnitude to be a reliable guide for near-term planning, even if it is not a complete theory of intelligence at scale. Practitioners who understand both the explanatory power and the limits of scaling laws are better equipped to interpret model releases, evaluate vendor claims, and make sound decisions about where to invest in AI capability development.