LLMs from Scratch
Home› AJ University› LLMs from Scratch› Module 6
Advanced 30 min Module 6 of 6
Module 6 of 6

Scaling Laws and Emergent Behavior

The claim that made AI a trillion-dollar industry was simple: make the model bigger, feed it more data, use more compute, and it gets better. Reliably. Predictably. In a way you can measure before you spend the money. This module covers the empirical laws behind that claim, what the evidence actually shows about "emergent" capabilities, and what the limits of scaling-based reasoning are.

By the end of this module you will be able to

The Basic Scaling Law

Kaplan et al. "Scaling Laws for Neural Language Models" (arXiv:2001.08361) established the foundational empirical relationship. Trained across a wide range of model sizes and data budgets, they found that language model loss L follows a power law with respect to three factors: the number of model parameters N, the number of training tokens D, and the total compute budget C.

The relationship takes the form: L(N) ~ N^(-0.076) when data is unlimited, L(D) ~ D^(-0.095) when model size is unlimited, and L(C) ~ C^(-0.050) for the compute-optimal frontier. These exponents are approximate and depend on architecture and data distribution, but the key point is that the relationships are smooth power laws, not step functions. Loss decreases predictably and continuously as you scale each factor.

This predictability is what makes scaling laws useful. You can train smaller models at lower compute, measure their loss, fit the power-law curve, and extrapolate to predict what loss a much larger model would achieve before you spend the compute to train it. Kaplan et al. validated this extrapolation across many orders of magnitude of compute, suggesting the relationship holds across a wide range.

The Chinchilla Correction

Hoffmann et al. (arXiv:2203.06840) identified a key limitation in the original Kaplan analysis: it was done in a data-limited regime where model size was varied more than data size. When Hoffmann et al. ran a more careful experiment varying both model size and data size jointly under a fixed compute budget, they found the optimal allocation was different from what Kaplan et al. implied.

The Kaplan analysis suggested approximately N ~ C^(0.73), meaning model size should scale faster than data for a given compute budget. The Chinchilla analysis found the optimal ratio is closer to equal scaling: N and D should both increase at roughly the same rate. Larger models trained on fewer tokens than implied by this ratio are simply undertrained. This is why the Chinchilla 70B model, trained on 1.4T tokens, was competitive with models many times larger trained on fewer tokens.

The Chinchilla finding had immediate practical consequences. If optimal training requires more data than earlier models received, then many large models already trained were operating suboptimally. For organizations planning new training runs, the prescription was clear: allocate more of your compute budget to data rather than model parameters, up to the compute-optimal ratio.

Emergent Capabilities: What the Evidence Actually Shows

The term "emergent capabilities" describes abilities that appear suddenly at scale: a model below some threshold cannot do the task at all, and a model above the threshold can. The apparent suddenness has been central to arguments both for and against continued scaling.

Wei et al. "Emergent Abilities of Large Language Models" (arXiv:2206.07682) documented numerous tasks where performance appeared to jump from near-random to well-above-random at specific model scales. Tasks like multi-step arithmetic, certain reasoning problems, and some language understanding benchmarks showed this pattern.

However, Schaeffer et al. "Are Emergent Abilities of Large Language Models a Mirage?" (arXiv:2304.15004) argued that many apparent emergent abilities are artifacts of the metrics used rather than genuine threshold phenomena. When benchmark accuracy is used as the metric (a discontinuous function that jumps when the model passes a threshold number of problems), capability appears to emerge suddenly. When continuous metrics are used instead (such as probability assigned to the correct answer rather than whether the answer is correct), the same models show smooth, predictable improvement with scale. The apparent emergence was in many cases a measurement artifact.

The key distinction Emergent in the scaling law sense means "unpredicted by extrapolation from smaller scales." It does not mean "impossible at smaller scales" or "suddenly acquired." Many tasks that appear emergent on accuracy metrics show smooth improvement on continuous metrics, suggesting capabilities develop gradually but only cross evaluation thresholds at larger scales.

What Scaling Laws Do Not Predict

Scaling laws predict loss on the pretraining distribution. They do not predict performance on specific downstream tasks, safety properties, alignment, reasoning ability in novel domains, or real-world usefulness. Loss is correlated with these properties, but the correlation is not deterministic.

Some capabilities are not well-predicted by next-token loss at all. A model can have excellent loss while being poor at formal mathematical reasoning, code debugging, or consistent long-horizon planning. These require specific structural properties of the training data and may require architectural innovations beyond raw scale. There is active research into whether current scaling trends will continue to improve these capabilities or whether a wall exists.

Scaling laws are also empirically established within a specific compute range. Kaplan et al. validated extrapolation across roughly 7 orders of magnitude of compute. Whether the same power laws continue at 10 or 12 orders of magnitude above the smallest models in their study is unknown. Historical evidence of sustained continuity is encouraging but does not guarantee the laws hold indefinitely.

Fig 1 · Power Law Scaling: Loss vs Compute (directional illustration, based on Kaplan et al. arXiv:2001.08361)
COMPUTE (LOG SCALE) LOSS (LOG SCALE) OPTIMAL FRONTIER LARGE N MED N SMALL N 10^20 10^22 10^24 10^26 3.0 2.5
Python · Power Law Scaling Prediction
import numpy as np

def predict_loss_from_params(N, alpha=-0.076, L_inf=1.0):
    """
    Predict test loss from model parameter count N.
    Based on Kaplan et al. (arXiv:2001.08361) power law approximation.
    L(N) ~ N^alpha + L_inf
    This is a simplified directional model, not exact.
    """
    return L_inf + N**alpha

def predict_loss_from_compute(C, beta=-0.050, L_inf=1.0):
    """Predict test loss from total compute FLOPs."""
    return L_inf + C**beta

# Compare three model sizes
model_sizes = [1e8, 1e9, 1e10, 1e11]  # 100M to 100B parameters
print("Model size vs predicted loss:")
for N in model_sizes:
    loss = predict_loss_from_params(N)
    print(f"  N={N:.0e}: predicted loss = {loss:.3f}")

# Chinchilla optimal: model size and tokens scale equally
# ~6N FLOPs per token, so for a budget C = 6 * N * D
# Optimal: N = D (approximately)
compute_budget = 1e23  # FLOPs
# C = 6 * N * D and N ~ D => N^2 ~ C/6 => N ~ sqrt(C/6)
optimal_N = np.sqrt(compute_budget / 6)
optimal_D = optimal_N  # equal scaling
print(f"\nFor budget C = {compute_budget:.0e} FLOPs:")
print(f"  Compute-optimal N: {optimal_N:.2e} parameters")
print(f"  Compute-optimal D: {optimal_D:.2e} tokens")
print(f"  (Hoffmann et al. arXiv:2203.06840)")
Try this

Find any publicly available LLM benchmark results table (BIG-Bench results, HELM leaderboard, or similar). For any benchmark where multiple model sizes from the same family are reported, plot accuracy on the y-axis and model size on the x-axis, first on a linear scale then on a log scale. Capabilities that appear to "emerge suddenly" on the linear scale often look like smooth improvement on the log scale. This exercise directly illustrates the Schaeffer et al. argument that apparent emergence may be a scale artifact.

PL
Power Law
Loss scales predictably with compute, parameters, and data. L ~ N^(-0.076) approximately. Enables reliable extrapolation before spending compute on large runs.
EM
Emergent Abilities
Capabilities that appear to jump from near-zero to above-chance at specific scales. Schaeffer et al. show many are metric artifacts: continuous metrics show smooth improvement.
LM
Law Limits
Scaling laws predict pretraining loss, not task performance, alignment, or reasoning. Validated empirically within specific compute ranges. Extrapolation beyond has uncertainty.
Interactive 6: Scaling Law Extrapolator Try it

Adjust model size and data tokens to see predicted loss under the Chinchilla scaling analysis (directional illustration based on Hoffmann et al. arXiv:2203.06840). Observe how undertrained models (high N, low D) have worse loss than compute-optimal allocations.

70B
300B
Predicted loss: --    vs compute-optimal: --
If scaling laws are reliable, why can't we predict exactly when new capabilities will appear? ▼
Scaling laws predict the next-token prediction loss, not downstream capability on specific tasks. The relationship between loss and task capability depends on the task structure, the evaluation metric, and how much the training distribution covers the task. A 1-point loss improvement might produce large gains on some tasks and no measurable change on others. Additionally, for tasks that require multi-step reasoning, the capability jump might require the model to simultaneously master several component skills. Until the model masters all of them, performance on the compound task is near-random. The final skill's acquisition looks like a sudden jump, but it is the final piece of a gradual process. This is why Schaeffer et al.'s argument matters: the apparent threshold crossing is often in the compound task metric, not in the underlying skills individually.
Knowledge check
What did Schaeffer et al. "Are Emergent Abilities of Large Language Models a Mirage?" (arXiv:2304.15004) argue about apparent emergent capabilities?
Correct. Schaeffer et al. showed that when accuracy (a discontinuous metric requiring a threshold number of correct answers) is used to measure capability, improvements appear sudden. When continuous metrics like probability assigned to correct answers are used, the same models show smooth improvement with scale. The "emergence" was often in the measurement, not the capability.
Not quite. Schaeffer et al. did not argue that capabilities are not real. They argued that the apparent discontinuity is a measurement artifact: binary accuracy metrics create sudden jumps when the continuous underlying capability crosses a threshold. Using continuous metrics often reveals smooth scaling behavior.
What did the Chinchilla paper (Hoffmann et al. arXiv:2203.06840) find about optimal compute allocation for a given budget?
Correct. Hoffmann et al. found that for a fixed compute budget, the compute-optimal strategy scales model size and training tokens roughly equally. Many earlier large models were undertrained: they were too large relative to their data budget, and a smaller model trained on proportionally more data would have matched them.
Not quite. The Chinchilla finding is that model size and training tokens should scale roughly equally. Neither dominates: more parameters without more data is wasteful, and more data without sufficient model capacity is also suboptimal.
Before you go
Reflection: If you were planning a training run on a fixed compute budget, what information would you need before deciding how to split that budget between model parameters and training tokens?
You might also like
Capstone
Complete the Course
Blog
LLM Scaling Laws Explained
Related
Evaluating AI Models
Was this module helpful?

Advising on AI scaling strategy, model selection, and compute budget planning?

Schedule a call with Arjun

References

  1. Kaplan, J. et al. "Scaling Laws for Neural Language Models." arXiv:2001.08361. arxiv.org/abs/2001.08361
  2. Hoffmann, J. et al. "Training Compute-Optimal Large Language Models." arXiv:2203.06840. arxiv.org/abs/2203.06840
  3. Wei, J. et al. "Emergent Abilities of Large Language Models." arXiv:2206.07682. arxiv.org/abs/2206.07682
  4. Schaeffer, R. et al. "Are Emergent Abilities of Large Language Models a Mirage?" arXiv:2304.15004. arxiv.org/abs/2304.15004
  5. Liang, P. et al. "Holistic Evaluation of Language Models." arXiv:2211.09110. arxiv.org/abs/2211.09110
  6. Anil, R. et al. "Palm 2 Technical Report." arXiv:2305.10403. arxiv.org/abs/2305.10403
← Module 5: Inference and Efficiency Capstone: Complete the Course →