How Transformers Work

The Problem Transformers Solve

Before 2017, the dominant approach to sequence modeling was the recurrent neural network. To predict the next word in a sentence, an RNN read the sentence left to right, updating a hidden state at each step. The problem: by the time it reached word 50, its memory of word 1 had been diluted through 49 successive updates. Long-range dependencies were hard to capture.

The transformer, introduced in Vaswani et al. "Attention Is All You Need" (arXiv:1706.03762), discarded the recurrence entirely. Instead it processes the entire sequence at once, letting every position attend directly to every other position. Long-range dependency becomes trivial: position 1 and position 50 can interact directly with equal computational cost as adjacent positions.

That architectural decision is what made modern large language models possible. Scale a transformer wide enough, train it on enough data, and it learns to model language with a fidelity that surprised most researchers. The rest of this module explains the exact mechanism behind that.

Step 1: Tokens to Vectors

A language model cannot operate on words directly. It needs numbers. The first step is tokenization (covered fully in Module 2) and embedding. Each token in the vocabulary is assigned a vector of fixed dimension, called the embedding dimension or model dimension d_model. For reference, GPT-2 small uses d_model = 768; larger models use 4096 or more.

The embedding is a learned lookup table. At the start of training it is random. After training, similar tokens end up with similar vectors. "King" and "queen" will sit close together in the embedding space. "King" and "carburetor" will sit far apart. The geometry of the embedding space encodes semantic relationships the model has learned from data.

Positional information is added at this stage too, because unlike an RNN, the transformer has no built-in notion of order. A positional encoding vector is added to each token embedding before any attention is computed. The combined vector now carries both "what this token means" and "where it sits in the sequence." Module 2 covers both embedding and positional encoding in detail.

Step 2: Self-Attention and the Q/K/V Mechanism

Self-attention is the heart of the transformer. Here is the intuition: every token in the sequence produces three different representations of itself, called the Query, the Key, and the Value. These come from multiplying the token's embedding by three separate learned weight matrices: W_Q, W_K, and W_V.

Think of it like a search engine. The Query is the question a token is asking: "what context do I need to understand myself?" The Keys are what every other token is broadcasting: "here is what I can tell you about." The Values are the actual information each token contributes if selected. Attention is the process of scoring each key against each query, then taking a weighted sum of the corresponding values.

The scoring step uses a dot product. For token i querying token j, the score is:

score(i, j) = Q_i · K_j

Dot products measure alignment. If Q_i and K_j point in similar directions, their dot product is large and positive. If they are orthogonal, the dot product is near zero. This is how relevance is computed: two vectors that "agree" produce a high score, meaning i will attend strongly to j.

The Scaling Factor

There is a problem with raw dot products. As d_model grows, the magnitude of dot products also grows, because you are summing more terms. Large dot products drive softmax into a regime where the gradients are vanishingly small, making training unstable. The fix, from Vaswani et al., is to divide each dot product by the square root of the key dimension d_k before applying softmax:

Attention(Q, K, V) = softmax(QK^T / sqrt(d_k)) V

Dividing by sqrt(d_k) keeps the variance of the dot products roughly constant regardless of dimensionality. This is not an arbitrary design choice: it directly addresses a numerical stability problem that would otherwise prevent training from converging.

After scaling, softmax converts the raw scores into a probability distribution: positive numbers that sum to 1. The token with the highest score gets the highest weight. The attention output is then the weighted sum of all Value vectors, with higher-scored tokens contributing more.

Multi-Head Attention: Running Many Attention Patterns in Parallel

A single attention head can learn one type of relationship between tokens. But language has many types of relationships simultaneously: grammatical agreement, coreference (which "it" refers to), semantic similarity, positional proximity. A single attention head must pick one focus.

Multi-head attention runs h separate attention operations in parallel, each with its own W_Q, W_K, W_V matrices projecting into a smaller dimension d_k = d_model / h. Each head specializes in a different type of relationship. The outputs of all h heads are concatenated and projected back to d_model through a final W_O matrix.

GPT-2 small uses 12 attention heads. Larger models use 32, 64, or more. Different heads in trained models have been shown empirically to specialize: some heads track grammatical structure, others track semantic content, others handle positional relationships. This specialization emerges from training, not from design.

The Transformer Block: Feed-Forward, Residuals, and LayerNorm

Attention is not the only component in a transformer block. After multi-head attention, the output passes through a position-wise feed-forward network (FFN): two linear transformations with a ReLU or GELU activation in between. The FFN is applied independently to each token position. Its purpose is to transform the attended representation nonlinearly, giving the model more expressive power than attention alone provides.

Two more elements complete the block: residual connections and layer normalization. Residual connections add the input of each sublayer (attention or FFN) directly to its output before normalization. This solves the vanishing gradient problem for deep networks: gradients can flow backward through the addition operation without diminishing. A transformer with 96 layers (like GPT-3) would not be trainable without residual connections.

Layer normalization stabilizes the activations at each layer, normalizing each token's representation to have zero mean and unit variance. The combined recipe: attention then add-and-norm, FFN then add-and-norm. This block is repeated N times. GPT-2 small repeats it 12 times. GPT-3 repeats it 96 times.

import numpy as np

def scaled_dot_product_attention(Q, K, V):
    """
    Q: (seq_len, d_k) — queries
    K: (seq_len, d_k) — keys
    V: (seq_len, d_v) — values
    Returns: (seq_len, d_v) weighted sum of values
    """
    d_k = Q.shape[-1]

    # Step 1: Compute dot products between every Q and every K
    scores = Q @ K.T          # shape: (seq_len, seq_len)

    # Step 2: Scale to prevent large dot products from saturating softmax
    scores = scores / np.sqrt(d_k)

    # Step 3: Softmax to get attention weights (each row sums to 1)
    exp_scores = np.exp(scores - scores.max(axis=-1, keepdims=True))
    weights = exp_scores / exp_scores.sum(axis=-1, keepdims=True)

    # Step 4: Weighted sum of values
    output = weights @ V
    return output, weights

# Example: 4 tokens, d_k = 8, d_v = 8
np.random.seed(42)
seq_len, d_k, d_v = 4, 8, 8
Q = np.random.randn(seq_len, d_k)
K = np.random.randn(seq_len, d_k)
V = np.random.randn(seq_len, d_v)

output, attn_weights = scaled_dot_product_attention(Q, K, V)
print("Attention weights (each row sums to 1):")
print(np.round(attn_weights, 3))
print("\nOutput shape:", output.shape)

Putting It All Together: One Forward Pass

Now you can trace a token through one complete transformer block. Start with a sequence of N tokens. Each becomes a d_model-dimensional vector through the embedding lookup plus positional encoding. These N vectors enter the multi-head attention sublayer.

In multi-head attention, each vector is projected h times into d_k dimensions (once per head) to produce queries, keys, and values. Each head computes scaled dot-product attention over all N positions, producing N output vectors of dimension d_k. The h heads' outputs are concatenated (giving N vectors of dimension h * d_k = d_model) and projected through W_O.

The result is added back to the original input via the residual connection, then layer-normalized. This goes to the FFN: a linear layer expanding to 4 * d_model dimensions, a GELU activation, and a linear layer contracting back to d_model. Another residual connection and layer normalization complete the block.

Repeat this N times. After the final block, a linear projection maps d_model dimensions to vocabulary size (say, 50,000 tokens). Softmax over this vocabulary distribution gives the probability of each possible next token. The token with the highest probability (or a sampled token, depending on the decoding strategy) becomes the next output. Module 5 covers decoding strategies in detail.