LLMs from Scratch
Home› AJ University› LLMs from Scratch› Module 5
Advanced 30 min Module 5 of 6
Module 5 of 6

Inference and Efficiency

Training a model is expensive but you only do it once. Inference is expensive and you do it millions of times per day. The cost and latency of running an LLM in a real product are not fixed properties of the model: they are design variables. This module covers the four main levers: autoregressive decoding, the KV cache, quantization, and cost routing with FrugalGPT. By the end you will understand why a 70B model can run fast enough for interactive use and how to reduce inference cost without destroying quality.

By the end of this module you will be able to

Autoregressive Decoding: One Token at a Time

A transformer processes all input tokens in parallel during the forward pass. But generating output is different. Output is produced autoregressively: the model generates one token, appends that token to the input, and then runs the forward pass again to generate the next token. This continues until the model produces an end-of-sequence token or reaches the maximum length.

The consequence: generating a 100-token response requires 100 sequential forward passes. Unlike the pretraining forward pass which can process many documents in a batch simultaneously, generation is inherently sequential in the output dimension. This is the fundamental constraint on latency. You cannot parallelize across output tokens because each output token depends on all previous output tokens.

You can, however, batch multiple users' requests together. If 32 users each need a response and you run their inputs through the model simultaneously, you perform one forward pass that serves 32 requests in parallel. Batching is the primary tool for throughput optimization: at the cost of slightly higher latency per request, you dramatically increase the number of tokens generated per second per GPU.

The KV Cache: Avoiding Redundant Computation

In autoregressive decoding, the prompt tokens are identical across all decoding steps. When generating token 50, the model still needs the Key and Value representations of all the prompt tokens, and of output tokens 1 through 49. Without optimization, it would recompute these representations at every decoding step.

The KV cache solves this. The Key and Value matrices computed in each attention layer for each token are stored in GPU memory after the first time they are computed. On subsequent decoding steps, only the new token's K and V are computed and appended. The full attention computation can then use the cached K and V without recomputation.

The memory cost of the KV cache is significant. For a model with L layers, H attention heads, and d_k head dimension, the cache for one token uses 2 * L * H * d_k floating point numbers (factor of 2 for K and V). For a 70B parameter model with 80 layers and 64 heads of dimension 128, caching 2,000 tokens requires several GB of GPU memory. The maximum sequence length is therefore not just a model capability: it is also a GPU memory budget question.

The KV cache tradeoff The KV cache trades memory for compute. It eliminates redundant Key/Value recomputation, which reduces latency dramatically. But it uses GPU memory proportional to sequence length, which limits how long sequences can be and how many concurrent requests fit on a GPU. Managing this tradeoff is a core challenge in LLM serving infrastructure.

Quantization: Running Large Models in Less Memory

Neural network weights are typically stored as 32-bit or 16-bit floating point numbers. Quantization reduces the precision of these representations to lower bit widths: 8-bit integers (INT8) or 4-bit integers (INT4). This reduces memory usage by a factor of 2 or 4 respectively, which allows larger models to fit on a given set of GPUs and increases the number of model copies that can be served in parallel.

INT8 quantization maps the continuous distribution of weight values to 256 discrete levels. The key challenge is calibration: choosing the mapping from float to int carefully so that the distribution of weight values is well-covered by the 256 levels. With good calibration, INT8 quantization produces models with accuracy very close to the full-precision original, at half the memory footprint.

INT4 quantization (using only 16 discrete levels) is more aggressive. Accuracy degrades more noticeably, particularly for smaller models. Techniques like GPTQ (Frantar et al., arXiv:2210.17323) and AWQ (Lin et al., arXiv:2306.00978) address this with careful layer-wise quantization that minimizes accuracy loss. In practice, a 70B model quantized to INT4 can often match or exceed the accuracy of a 30B model in full precision, while fitting in significantly less GPU memory.

Speculative Decoding: Parallel Verification

Speculative decoding uses a smaller, faster draft model to propose multiple tokens at once, then uses the large target model to verify all proposed tokens in a single parallel forward pass. If the large model agrees with the draft tokens, all of them are accepted. If it disagrees at some position, the draft tokens from that position onward are rejected and generation continues normally.

The key insight: the large model's forward pass processes multiple tokens in parallel (as it does during training) rather than one at a time. If the draft model's proposals are accepted even 70-80% of the time, the effective token generation rate can increase by 2-3x without changing the output distribution of the large model. This is not an approximation: rejected draft tokens are discarded and the large model's own distribution is used, so the output is exactly equivalent to autoregressive decoding from the large model.

FrugalGPT: Routing Requests by Difficulty

Not every query requires the most capable model. Chen, Zaharia, and Zou, "FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance" (arXiv:2310.11409), showed that a routing strategy that sends easy queries to cheaper models and hard queries to expensive ones can reduce LLM API costs substantially while maintaining overall quality.

The FrugalGPT approach maintains a cascade of models ordered by cost. A router classifies each query by estimated difficulty and routes it to the cheapest model likely to answer it correctly. A quality verifier checks the response; if it fails a confidence threshold, the query is escalated to the next model in the cascade. The cascade continues until a satisfactory response is produced or the most capable model is reached.

This is not just about cost. Chen et al. found that the LLM cascade can also exceed the quality of the most expensive single model alone, because different models have different strengths. A routing strategy that directs queries to the model most likely to answer each query type correctly outperforms any single model on quality metrics. This is why FrugalGPT's framing includes both "reducing cost" and "improving performance" in its title.

KV
KV Cache
Caches Key and Value matrices for all processed tokens. Eliminates recomputation on each decoding step. Costs memory proportional to sequence length and model size.
Q4
INT4 Quantization
Reduces weight precision from 16 to 4 bits, halving memory vs INT8. Requires careful calibration techniques (GPTQ, AWQ). Enables large models on fewer GPUs.
SD
Speculative Decoding
Draft model proposes multiple tokens; target model verifies all in one parallel pass. Accepted tokens skip sequential decoding. Output is identical to autoregressive generation.
Interactive 5: Inference Cost and Latency Explorer Try it

Adjust model size, quantization, and batch size to see the relative memory footprint, throughput, and latency. Values are directional illustrations, not exact benchmarks.

13B
FP16
1
GPU memory: --    Relative throughput: --    Relative latency: --
Python · Greedy vs Temperature Sampling Decoding
import numpy as np

def greedy_decode(logits):
    """Always pick the highest-probability token. Deterministic."""
    return np.argmax(logits)

def temperature_sample(logits, temperature=1.0):
    """
    Sample from the distribution after scaling by temperature.
    temperature < 1: sharper distribution, more predictable
    temperature > 1: flatter distribution, more creative
    temperature = 0: equivalent to greedy
    """
    if temperature == 0:
        return greedy_decode(logits)
    scaled = logits / temperature
    exp_logits = np.exp(scaled - scaled.max())
    probs = exp_logits / exp_logits.sum()
    return np.random.choice(len(probs), p=probs)

def top_p_sample(logits, p=0.9, temperature=1.0):
    """
    Nucleus sampling: only sample from the top-p probability mass.
    Prevents sampling from the long tail of low-probability tokens.
    """
    scaled = logits / temperature
    exp_logits = np.exp(scaled - scaled.max())
    probs = exp_logits / exp_logits.sum()

    sorted_idx = np.argsort(-probs)
    cumulative = np.cumsum(probs[sorted_idx])
    cutoff = np.searchsorted(cumulative, p) + 1
    top_idx = sorted_idx[:cutoff]

    top_probs = probs[top_idx]
    top_probs /= top_probs.sum()
    return top_idx[np.random.choice(len(top_idx), p=top_probs)]

# Demo: vocabulary of 10, logits from a pretend model
np.random.seed(42)
logits = np.array([1.0, 3.0, 0.5, 2.0, 0.2, 1.5, 0.8, 2.5, 0.3, 1.2])

print("Greedy:", greedy_decode(logits))           # always picks argmax
print("Temp=1.0:", temperature_sample(logits, 1.0)) # samples proportionally
print("Temp=0.5:", temperature_sample(logits, 0.5)) # more concentrated
print("Top-p=0.9:", top_p_sample(logits, 0.9))    # nucleus sampling
Try this

Call the same model API twice with identical prompts but different temperature settings: once at temperature 0 (or 0.0001) and once at temperature 1.0. Run each 5 times. The temperature 0 responses will be identical every time. The temperature 1.0 responses will vary. Now try temperature 2.0 if the API allows it. You will observe the output becoming less coherent: high temperature flattens the probability distribution so that unlikely tokens are sampled as often as likely ones, producing creative but often incoherent output. This is the latency-quality-diversity tradeoff at work.

If speculative decoding produces the exact same output as autoregressive decoding, why isn't it always used? ▼
Speculative decoding requires a draft model that is both fast enough to propose tokens quickly and accurate enough to have a high acceptance rate. If the draft model is too slow, the overhead of running it offsets the speedup from parallel verification. If the draft model's proposals are rejected too often (low acceptance rate), the cascade overhead dominates and the net speedup disappears. The speedup also depends on hardware: on systems where memory bandwidth is the bottleneck (which is typical for inference), speculative decoding helps because verification uses less bandwidth than sequential decoding. On compute-bound systems, the benefit is smaller. Draft model selection and acceptance threshold tuning are active engineering problems in LLM serving systems.
Knowledge check
What does the KV cache store and why does it help inference speed?
Correct. At each decoding step, computing attention requires K and V for all previous tokens. Without caching, these would be recomputed from scratch at every step. The KV cache stores these representations so only the new token's K and V need to be computed, reducing the work per decoding step.
Not quite. The KV cache stores the Key and Value matrices computed for each token in each attention layer. Without the cache, attention at step N would require recomputing K and V for all N-1 previous tokens. The cache eliminates this redundancy.
According to FrugalGPT (Chen et al. arXiv:2310.11409), what approach can both reduce cost and improve quality compared to using a single large model?
Correct. FrugalGPT routes each query to the cheapest model likely to answer it correctly. Because different models excel at different query types, the cascade can exceed the quality of any single model while using far less total compute on average.
Not quite. FrugalGPT uses a cascade of models with a router that directs each query based on estimated difficulty. Easy queries go to cheap models; hard queries escalate to expensive ones. Because different models have different strengths, the cascade can exceed single-model quality while using less average compute.
Before you go
Reflection: If speculative decoding produces the same outputs as standard decoding, what metric would you use to decide whether it is worth the additional system complexity?
You might also like
Next Module
Scaling Laws
Related
Fine-Tuning Models
Prev Module
Fine-Tuning and RLHF
Was this module helpful?

Need help with LLM inference cost strategy or model selection for your product?

Schedule a call with Arjun

References

  1. Chen, L., Zaharia, M., Zou, J. "FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance." arXiv:2310.11409. arxiv.org/abs/2310.11409
  2. Leviathan, Y. et al. "Fast Inference from Transformers via Speculative Decoding." arXiv:2211.17192. arxiv.org/abs/2211.17192
  3. Frantar, E. et al. "GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers." arXiv:2210.17323. arxiv.org/abs/2210.17323
  4. Lin, J. et al. "AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration." arXiv:2306.00978. arxiv.org/abs/2306.00978
  5. Holtzman, A. et al. "The Curious Case of Neural Text Degeneration." arXiv:1904.09751. arxiv.org/abs/1904.09751 (nucleus sampling)
  6. Dettmers, T. et al. "LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale." arXiv:2208.07339. arxiv.org/abs/2208.07339
← Module 4: Fine-Tuning and RLHF Module 6: Scaling Laws →