How the Transformer Architecture Works: Attention, Layers, and Context

The transformer architecture, introduced in a single 2017 paper, became the foundation for virtually every significant language model built in the following eight years. Understanding why it works, not just that it works, gives you the conceptual tools to reason about model capabilities, context limitations, and why certain tasks are easier or harder for language models than others. This post explains the architecture in plain English, starting from the problem it solves.

Before the transformer, language models were built on recurrent architectures: networks that processed text one token at a time, maintaining a hidden state that carried information about what had been seen so far. This sequential processing had a fundamental problem: it was slow to train and made it difficult for the model to connect information that was far apart in the sequence. By the time the model was processing token 500, the signal from token 1 had been diluted through 499 state updates. This limited how much context the model could effectively use.

The transformer solved this by processing all tokens in parallel and using a mechanism called attention to directly compute relevance relationships between every pair of tokens in the sequence. Instead of maintaining a sequential hidden state, the transformer builds a representation of each token that is explicitly conditioned on every other token, with learned weights that determine which other tokens are most relevant for interpreting each one. This is why transformers can be parallelized efficiently on modern hardware and why they generalize so well across tasks.

The Problem Attention Solves

To understand attention, start with the problem it solves. Consider the sentence: "The trophy didn't fit in the suitcase because it was too small." What does "it" refer to: the trophy or the suitcase? A human reader immediately understands that "it" refers to the suitcase, because "too small" makes more sense applied to the suitcase that cannot hold the trophy than to the trophy itself.

For a model processing this sentence token by token, making this inference requires connecting "it" (token 11 or so, depending on tokenization) with "suitcase" (token 7) and "trophy" (token 3) and "small" (token 13), computing which connection best resolves the ambiguity, and updating the representation of "it" accordingly. No sequential model could do this reliably because the relevant tokens are spread across the sequence and the relationship is bidirectional, requiring the model to look both backward and forward from "it."

Attention handles this directly. For each token, the attention mechanism computes a score representing how relevant every other token is for interpreting this one. These scores are used to compute a weighted sum of the other tokens' representations, producing a new representation of the current token that has been contextualized by everything in the sequence. The word "it" gets a representation that is partly composed of the representations of "trophy" and "suitcase," weighted by how relevant each is, and the model learns those relevance weights from training data.

Attention does not process tokens one at a time. Every token attends to every other token simultaneously, computing relevance relationships in parallel.

How Self-Attention Works

The self-attention mechanism works by transforming each token's representation into three vectors: a query, a key, and a value. These names come from an analogy to information retrieval systems, where a query is matched against keys to retrieve values, but the analogy is not essential to understanding the mechanism.

The query vector for a token represents the question "what am I looking for?" The key vectors for all tokens represent "what do I contain?" The attention score between a query token and a key token is computed as the dot product of the query vector and the key vector, scaled by the square root of the dimension of the key vectors to prevent the scores from becoming too large. These scores are then passed through a softmax function to produce attention weights that sum to 1 across all tokens. The output representation for the query token is the weighted sum of the value vectors of all tokens, where the weights are the attention scores.

The query, key, and value transformations are learned during training. This means the model learns to compute meaningful relevance relationships, not just arbitrary dot products. Through training on text, the model learns that pronouns attend to likely antecedents, that adjectives attend to the nouns they modify, that questions attend to the words most relevant for forming an answer, and countless other linguistic relationships that make contextual understanding possible.

Multi-head attention

The original transformer paper introduced multi-head attention: running the self-attention operation multiple times in parallel with different learned query/key/value transformations. Each "head" can learn to attend to different types of relationships: one head might specialize in syntactic dependencies, another in coreference, another in semantic similarity. The outputs of all heads are concatenated and passed through a linear transformation to produce the final representation.

The number of attention heads is a design choice. The original transformer had 8 heads. GPT-3's 175 billion parameter model used 96 attention heads in each of its 96 layers (Brown et al., arXiv:2005.14165). More heads give the model more capacity to learn diverse relationship types, at the cost of more parameters and compute.

The Full Transformer Layer

A transformer layer is composed of two main sub-components: the multi-head self-attention mechanism described above, and a feedforward network applied independently to each token's representation after the attention step.

The feedforward network is a simple two-layer neural network with a nonlinear activation function between the layers. Its role is to further transform each token's representation after it has been contextualized by the attention mechanism. The feedforward network does not involve attention and does not allow tokens to interact with each other; it applies the same transformation independently to every token position. This is where most of the model's factual knowledge is believed to be stored, based on analysis of what information is recoverable from different parts of the transformer (Geva et al., arXiv:2012.14913).

Both the attention sub-layer and the feedforward sub-layer use residual connections: the input to each sub-layer is added to its output, which makes it possible to train very deep networks without the gradient vanishing problem that affected earlier deep architectures. Layer normalization is applied before or after each sub-layer (the exact placement varies across architectures) to stabilize training.

Encoder-Only, Decoder-Only, and Encoder-Decoder Architectures

The original transformer paper described an encoder-decoder architecture for machine translation. Over the following years, two variants emerged that became dominant for different tasks.

Encoder-only transformers, like BERT and its descendants, use bidirectional attention: every token can attend to every other token in both directions. This makes them very good at tasks that require understanding the full context of a passage, like text classification, named entity recognition, and producing sentence embeddings for retrieval. The tradeoff is that they cannot generate text autoregressively; they produce a fixed representation of the input rather than a sequence of new tokens.

Decoder-only transformers, like the GPT family and Llama, use causal attention: each token can only attend to tokens that came before it in the sequence. This is necessary for text generation: when the model is predicting the next token, it must not be allowed to see the future tokens it is trying to predict. The causal masking implements this constraint. Decoder-only models dominated the scaling era because the next-token prediction training objective is simple, scalable, and produces models with surprisingly broad capabilities.

Encoder-decoder transformers, like T5 and BART, combine both variants with a cross-attention mechanism that allows the decoder to attend to the encoder's representations. These architectures are particularly well-suited for tasks where the input and output have different structures: translation, summarization, and tasks where the input must be fully processed before the output begins.

Positional Encoding: Teaching the Model About Order

Because self-attention processes all tokens in parallel with no inherent notion of order, transformers need an additional mechanism to represent the position of each token in the sequence. Without this, "the dog bit the man" and "the man bit the dog" would produce identical representations, since both contain the same tokens with the same attention weights between any pair.

The original transformer paper used sinusoidal positional encodings: fixed mathematical functions of token position added to each token's embedding. Later architectures moved to learned positional embeddings, and more recent large language models use rotary position embeddings (RoPE, Su et al., arXiv:2104.09864) or ALiBi (Press et al., arXiv:2108.12409), which have better extrapolation properties when processing sequences longer than those seen during training.

Why Context Window Length Matters

The context window is the maximum number of tokens a transformer can process in a single forward pass. Everything outside the context window is invisible to the model. Early transformer models had context windows of 512 to 2048 tokens. By 2024-2025, frontier models extended this to 128,000 tokens or more.

There are two reasons context window length matters practically. First, longer context allows the model to process longer documents, maintain coherence over longer conversations, and handle tasks that require reasoning over large amounts of information simultaneously. Second, and more subtly, the quadratic scaling of attention with sequence length means that doubling the context window roughly quadruples the compute and memory required for the attention operation. Techniques like FlashAttention (Dao et al., arXiv:2205.14135) address the memory efficiency problem; techniques like sliding window attention and sparse attention patterns reduce the compute cost at the expense of some cross-position connectivity.

How Depth and Width Scale Transformer Capability

A transformer model is not just a single attention layer. It is a stack of identical transformer layers applied in sequence. Each layer takes the representations from the previous layer and produces new representations that incorporate both the attention-derived context and the feedforward transformation. The depth of the stack (number of layers) and the width of each layer (dimensionality of the token representations, called the model dimension or hidden size) are the primary architectural parameters that determine model capacity.

Adding more layers allows the model to compute increasingly abstract representations. Early layers tend to capture local syntactic patterns: which words are adjacent, what grammatical roles they play. Middle layers capture more semantic relationships: word meaning, coreference, semantic role. Later layers capture the most abstract properties relevant to the prediction task. This progression has been characterized through interpretability research that probes what information is recoverable from each layer's representations (Tenney et al., arXiv:1905.05950).

The number of attention heads, the model dimension, and the number of layers all scale together in modern architectures. GPT-3 at 175 billion parameters uses 96 layers, a model dimension of 12288, and 96 attention heads per layer (Brown et al., arXiv:2005.14165). Smaller models like the 7 billion parameter Llama variants use 32 layers with correspondingly smaller dimensions. The compute required for a single forward pass scales roughly with the product of these dimensions, which is why larger models have substantially higher inference costs per token.

Understanding this scaling relationship matters for practical architecture and deployment decisions. A larger model is not always the right choice for a deployment constrained by latency or cost. A model that is much larger than the task requires may be slower and more expensive to run without providing any meaningful quality improvement on that specific task. Task-specific evaluation, covered in the next post in this series, is the only reliable way to determine whether a larger model justifies its higher inference cost per token in your application.

What Transformers Are Not Good At

Understanding the limitations of the transformer architecture is as important as understanding its strengths. Several failure modes follow directly from how the architecture works.

Transformers do not have persistent memory across independent inference calls. Each call starts from scratch with a fresh context window. This is why chatbots must maintain a conversation history by concatenating prior turns into the context: the model has no built-in mechanism for recalling previous conversations. Once the context window is full, earlier content must be compressed or discarded.

Transformers struggle with precise counting and arithmetic. The attention mechanism is designed to learn soft relevance relationships, not hard symbolic operations. A transformer that has learned that "2 + 2 = 4" from training text has learned a statistical pattern, not an arithmetic procedure. Errors on multi-step arithmetic, especially with numbers not frequently seen in training, are a known and persistent failure mode. Chain-of-thought prompting, which encourages the model to show its reasoning step by step, partially mitigates this by breaking multi-step problems into shorter steps (Wei et al., arXiv:2201.11903).

Transformers can also exhibit position bias: when the same information appears at different positions in a long context, the model may weigh it differently based on its distance from the beginning or end of the context window. Recent work has characterized this as the "lost in the middle" phenomenon, where information in the middle of a long context is attended to less reliably than information at the beginning or end (Liu et al., arXiv:2307.03172).

The Free Course Module on Transformer Architecture

Module 4 of the free Model Development Course covers the transformer architecture in full: self-attention, multi-head attention, the feedforward component, residual connections, layer normalization, positional encoding, and the differences between encoder-only, decoder-only, and encoder-decoder variants. The module builds each component from the problem it solves, in the same sequence used in this post, and includes interactive questions to check understanding at each stage.