The Transformer Architecture

The Problem with Reading One Word at a Time

Before the Transformer, the dominant way to build language models was a family of architectures called recurrent neural networks. The core idea was intuitive: process text word by word, left to right, carrying a "memory" of what you had seen so far as you moved forward. It felt like how humans read.

The problem was that this memory was limited. By the time the network processed the hundredth word in a sentence, the signal from the first word had been diluted through dozens of steps. Long-range connections in language, where a pronoun refers back to a noun many sentences earlier, or where a clause at the end of a paragraph depends on context from the beginning, were hard to preserve. The memory faded.

There was a second problem: because each word depended on the previous one, you could not process words in parallel. Training had to march through the text sequentially, one step at a time. This made training slow and expensive, which put a hard ceiling on how large these models could grow.

The Transformer solved both problems at once by abandoning sequential processing entirely. Instead of reading one word at a time, it looks at all the words simultaneously and decides, for each word, which other words are most relevant to understanding it. This mechanism is called attention.

What Attention Actually Means

Attention is best understood through an example. Consider the sentence:

"The trophy did not fit in the suitcase because it was too big."

What does "it" refer to? The trophy or the suitcase? You know immediately: the trophy, because if the trophy was too big, it would not fit. To reach that conclusion, your brain did not just look at the word "it" in isolation. It scanned back across the sentence, weighing the relevance of each word to figure out what "it" points to. You paid more attention to "trophy" and "fit" and "big" than to "because" or "the."

Transformers do exactly this, but mathematically. For every word (or token, as we called them in Module 3), the model computes a score representing how relevant every other word is to it. These scores are converted into weights that sum to one, and the word's representation is updated by pulling in information from the other words, weighted by how relevant they are. Words that score high contribute more; words that score low contribute less.

Self-Attention: The Mechanism in Detail

The specific form of attention that powers Transformers is called self-attention, because the words are attending to each other within the same sentence, rather than attending to some separate source of information.

Here is how it works at a conceptual level. For each token, the model learns three different representations:

• A Query: what this token is looking for ("I am a pronoun, what noun am I referring to?")
• A Key: what this token has to offer ("I am a noun, available to be referred to")
• A Value: the information this token actually contributes when selected ("I carry the meaning 'trophy'")

The attention score between two tokens is computed by comparing the query of one token with the key of another. A high score means those two tokens are highly relevant to each other. The scores are then used to weight the values, so the output for each token is a mixture of information from all other tokens, blended according to relevance.

The model learns the query, key, and value transformations during training. It discovers which kinds of relationships matter: subject-verb agreement, pronoun resolution, adjective-noun pairing, and many more. No one programs these rules in. They emerge from the data.

Multi-Head Attention: Watching Multiple Things at Once

A sentence contains many kinds of relationships simultaneously. The word "bank" might attend to "river" for its literal meaning, or to "loan" for its financial meaning. A single pass of attention can only capture one kind of relationship at a time.

Transformers solve this with multi-head attention: running several attention computations in parallel, each with its own learned query, key, and value transformations. Each "head" learns to notice a different type of relationship in the text. Their outputs are then concatenated and combined into a single representation.

Think of it as multiple specialists attending the same committee meeting. One specialist tracks grammatical roles, another tracks semantic similarity, another tracks long-range references. Together they produce a richer analysis than any one of them could alone.

Position Matters: But Attention Does Not Know That

Here is a subtle problem. Attention looks at all words simultaneously, without any inherent sense of order. The sentence "the dog bit the man" and "the man bit the dog" would produce the same attention scores if order were ignored, even though they mean opposite things.

To fix this, Transformers add positional encoding to each token's representation before the attention layers see it. Positional encoding is a signal added to each token that encodes where in the sequence that token appears. The first word gets one pattern, the second word gets a slightly different pattern, and so on.

The Feed-Forward Layer

After the attention layer has let each token gather information from its neighbours, the Transformer passes each token through a small, independent neural network called a feed-forward layer. This layer processes each token on its own, applying a nonlinear transformation to its representation.

If attention is the step where tokens talk to each other and exchange information, the feed-forward layer is where each token privately processes what it has learned and updates its own internal representation. The two steps alternate: gather from others, then think for yourself.

Stacking Layers: Depth Gives Understanding

A single attention layer followed by a feed-forward layer is called a Transformer block. In practice, dozens of these blocks are stacked on top of each other. The output of one block becomes the input to the next.

Early layers tend to capture simple, local patterns: nearby words, punctuation, basic syntax. Later layers capture more abstract relationships: topics, argument structure, factual associations. Depth allows the model to build meaning up from basic patterns to complex understanding, layer by layer.

Large language models today stack anywhere from 12 to 96 or more Transformer blocks. Each additional layer adds representational capacity, which is part of why scaling models larger can make them dramatically more capable.

Why This Architecture Won

The Transformer became the dominant architecture for three interconnected reasons:

Parallelism. Because attention looks at all tokens simultaneously rather than one at a time, the entire computation can run in parallel on modern graphics processors. This dramatically cuts training time and makes it feasible to train on enormous datasets.

Long-range context. Every token can directly attend to every other token in the sequence, regardless of distance. A word at the beginning of a document can directly influence the representation of a word at the end. Earlier architectures had to pass signals through many sequential steps to achieve this, losing fidelity along the way.

Scalability. More data and more compute consistently produce better Transformers, with no signs of plateauing at the scales that have been tried. This scaling property is what made training models on trillions of tokens worthwhile: the architecture rewarded the investment.