You have worked through every layer of the stack: attention, tokenization, pretraining objectives, alignment pipelines, inference efficiency, and scaling laws. This capstone ties those layers into a single coherent picture and gives you a practical challenge to take further.
The Complete Forward Pass
Every LLM inference call runs the same sequence of operations. Below is the full pipeline from raw text to predicted token, with the key equations at each stage.
Step
Operation
Key Formula
What Happens
01
Tokenization
text → [id1, id2, ...]
BPE or WordPiece splits raw text into subword units and maps them to integer IDs from a fixed vocabulary.
02
Embedding Lookup
E[token_id]
Each token ID indexes into a learned embedding matrix of shape (V, d_model). The result is a dense vector of dimension d_model.
03
Positional Encoding
x = E[id] + PE[pos]
A positional signal (sinusoidal, learned, or RoPE) is added to each token embedding so the model can distinguish sequence order.
04
Self-Attention (per head)
Attn = softmax(QKᵀ / sqrt(d_k)) V
Queries, keys, and values are projected from the input. Each token attends to all preceding tokens. A causal mask zeroes out future positions.
05
Multi-Head Concat
MHA = concat(head_1, ..., head_h) W_O
h attention heads run in parallel across d_model/h dimensions each. Their outputs are concatenated and projected back to d_model.
06
Add and Norm
LayerNorm(x + MHA(x))
Residual connection adds the original input to the attention output. Layer normalization stabilizes the distribution before the next sub-layer.
07
Feed-Forward Network
FFN = max(0, xW_1 + b_1) W_2 + b_2
Two linear layers with a ReLU or GELU activation expand the representation to 4x d_model and back. Residual and LayerNorm follow again.
08
Repeat N Layers
x_L = Block_N(...Block_1(x_0)...)
Steps 4 through 7 repeat for each transformer layer. Depth (N) and width (d_model) are the two primary scaling axes.
09
Language Model Head
logits = x_L W_LM
The final hidden state is projected to vocabulary size V, producing a logit for every token in the vocabulary.
10
Sampling
p = softmax(logits / T)
Temperature scales the logits before softmax. The next token is drawn by greedy decoding (argmax), top-k, or nucleus sampling from the probability distribution.
Each of steps 4 through 7 uses a KV cache during autoregressive decoding so that only the new token's attention computation is performed at each step, rather than recomputing the full sequence. That cache is what makes large-context inference economically viable.
Transformer Forward Pass Visualizer
Adjust the model configuration and watch how the memory footprint and compute cost shift across the forward pass stages.
Module 1 (Transformers) showed that self-attention is a weighted average of value vectors, where the weights come from the compatibility between query and key vectors. Scaling by 1/sqrt(d_k) keeps the dot products from entering the flat region of softmax at large dimensions.
Module 2 (Tokenization) covered how BPE builds a vocabulary bottom-up by iteratively merging the most frequent byte pair. The choice of vocabulary size is a direct tradeoff: larger vocabularies shrink sequence length but increase the embedding matrix and the LM head, both of which scale with V.
Module 3 (Pretraining) established that next-token prediction with cross-entropy loss is the engine behind every foundation model. The Chinchilla result (Hoffmann et al., 2022) showed that most models before 2022 were undertrained relative to their parameter count: compute-optimal training scales parameters and data tokens roughly equally.
Module 4 (Fine-tuning and RLHF) traced the alignment pipeline from supervised fine-tuning through reward model training to PPO, and then showed how DPO (Rafailov et al., 2023) collapses that three-stage process into a single closed-form loss on preference pairs, eliminating the reward model entirely while achieving comparable alignment quality.
Module 5 (Inference) dissected the costs of autoregressive decoding: the KV cache trades memory for compute, quantization (INT8, INT4) trades precision for memory and throughput, and speculative decoding trades small-model compute for large-model latency. FrugalGPT adds a routing layer that directs easy queries to cheap models.
Module 6 (Scaling Laws) grounded the intuition that bigger is better in actual empirical relationships: Kaplan et al. found power-law improvements with parameters, data, and compute; Chinchilla corrected the parameter-heavy bias. Schaeffer et al. challenged the concept of sharp emergence by showing that many apparent phase transitions are artifacts of non-linear evaluation metrics rather than genuine discontinuities in model capability.
Practical Challenge
Your First Real Task
Build a Minimal Transformer in NumPy
The best way to solidify this course is to implement a one-layer, single-head transformer from scratch without using PyTorch or a deep learning framework. You should be able to do this now.
Write a BPE tokenizer that operates on a small text corpus (100 merges is enough).
Implement the scaled dot-product attention function: Q, K, V projections, the dot product, scaling by 1/sqrt(d_k), softmax, and the weighted sum of V.
Add a two-layer FFN with ReLU and residual connections around both attention and FFN.
Train it to predict the next character in a 10,000-character text using cross-entropy loss and gradient descent (you can use NumPy's autodiff or compute gradients by hand).
Measure perplexity on a held-out 1,000-character split and compare it to a unigram baseline.
This exercise will surface every place the theory above gets concrete: where matrix shapes need to align, why the causal mask is necessary, what perplexity actually measures.
Certificate of Completion
Enter your name below, set the date, and print your certificate.
I just completed "LLMs from Scratch" on Arjun Jaggi's Free AI University.
Six modules covering transformer architecture (Q/K/V attention, multi-head, residuals), tokenization (BPE, WordPiece, SentencePiece), pretraining objectives and the Chinchilla scaling laws, RLHF and DPO alignment, inference efficiency (KV cache, quantization, speculative decoding), and scaling law limits.
If you want to understand how the models you use every day actually work, the course is free: arjunjaggi.com/course/llms-from-scratch/
If you are evaluating a model change, scoping a fine-tuning project, or deciding between inference strategies, a 30-minute call can save weeks of trial and error.