The Science Behind the Headlines: Circuits, Superposition, and Sparse Autoencoders

Mechanistic interpretability is not a philosophy about AI. It is an experimental research programme producing verifiable, peer-reviewed findings about what happens inside neural networks. If you have been following interpretability coverage without reading the primary sources, you are missing most of what is actually interesting about it.

This post covers the core scientific concepts and key findings at a level of precision that is useful for technical leaders, researchers in adjacent fields, and scientists who want to understand what the fuss is actually about. I will not simplify to the point of inaccuracy, but I will not assume you have a background in deep learning either. The goal is a precise conceptual map, not a tutorial.

Starting with Transformers

Almost all modern large language models are transformer architectures. A transformer processes text by converting tokens (roughly, words or word fragments) into high-dimensional vectors, then passing those vectors through a series of layers. Each layer contains two main components: attention heads, which allow tokens to communicate information with each other across the sequence, and multilayer perceptron (MLP) blocks, which apply learned transformations to each token position independently.

The key question for interpretability is: what are these components actually computing? Not in a mathematical sense, as the matrix multiplications are fully specified, but in a conceptual sense. When an attention head attends to certain tokens, what information is it routing and why? When an MLP block applies its transformation, what concept or operation is it implementing?

For most of the history of deep learning, these questions were considered unanswerable in any satisfying way. Transformers were treated as black boxes whose outputs could be evaluated but whose mechanisms could not be understood. The mechanistic interpretability programme challenges that assumption directly.

The Circuits Framework

The core intellectual contribution of the mechanistic interpretability programme is the "circuits" framework, introduced in "A Mathematical Framework for Transformer Circuits" (Transformer Circuits Thread, 2021). The central claim of this framework is that neural networks implement specific, identifiable computational algorithms, and those algorithms can be discovered through careful analysis of the weights and activations.

A circuit, in this framework, is a subgraph of the neural network that performs a specific computation. Circuits can be composed: a larger computation is built from smaller circuits, which are themselves built from individual neurons and attention heads. The programme's goal is to enumerate these circuits, verify the algorithms they implement, and eventually produce a complete mechanistic account of model behavior.

The first major empirical finding in this programme was the identification of induction heads in transformer models. Induction heads are pairs of attention heads that together implement a specific algorithm: given a sequence that contains a repeated pattern, induction heads enable the model to continue the pattern by attending to the context that followed the previous occurrence of the current token.

This discovery is significant not just for what it found but for how it was verified. The researchers did not merely observe that certain attention heads seemed to correlate with pattern completion. They formed a specific hypothesis about the algorithm the heads implement, designed experiments to test that hypothesis, and confirmed it through ablation studies (removing the heads and observing the effect on the task), activation patching (replacing activations from one forward pass with those from another to trace information flow), and mathematical analysis of the weight matrices. This is the methodological standard of the programme: claims about mechanisms must be experimentally verifiable, not just plausible.

The Superposition Hypothesis

One of the most important and counterintuitive findings in mechanistic interpretability is the superposition hypothesis, laid out in "Toy Models of Superposition" (arXiv:2209.11895, 2022). Understanding superposition is essential to understanding why interpreting neural networks is hard and what the most promising current tools are trying to solve.

The intuitive model of how a neural network stores knowledge is that each neuron represents a distinct concept or feature. A neuron fires when the model encounters a specific concept; its activation value encodes the presence and strength of that concept. In this model, a network with N neurons can represent N features.

The superposition hypothesis says this model is wrong in a specific and important way. Neural networks can represent far more features than they have neurons, by encoding multiple features as overlapping directions in the high-dimensional activation space. Think of it geometrically: in a high-dimensional space, you can pack a very large number of nearly-orthogonal vectors. If features are encoded as these nearly-orthogonal directions rather than as individual neuron activations, the network can represent exponentially more features than it has neurons, at the cost of some interference between features that share similar representations.

The research in "Toy Models of Superposition" demonstrates this phenomenon in a controlled setting, showing that networks learn to use superposition when the number of features to be represented exceeds the number of available neurons, and that the degree of superposition is governed by the frequency and importance of features in the training data. Frequent, important features tend to get dedicated neuron-level representations. Rare features tend to be packed into superposition with other features.

Superposition means the model may know far more than a direct inspection of individual neurons can reveal. It also means that seemingly unrelated concepts may interfere with each other in the model's internal representations.

Polysemanticity as a Consequence

Superposition produces polysemanticity: individual neurons respond to multiple unrelated concepts. A single neuron might activate strongly for both a specific named entity, a grammatical construction, and a domain-specific concept, not because these are related but because they happen to share a representational direction in the model's weight space.

Polysemanticity is a significant obstacle to interpretability because it means you cannot reliably interpret a model by examining what individual neurons respond to. If a neuron activates for multiple unrelated concepts, observing its activation does not tell you which concept is active in any specific context, and suppressing the neuron would affect all the concepts it represents rather than just the one you are targeting.

This has practical consequences for enterprise AI governance. Probing studies that test whether a model "knows" a specific concept by training a linear classifier on neuron activations are measuring a noisy proxy. Ablation studies that remove specific neurons to test whether a capability depends on them may produce misleading results if those neurons implement multiple capabilities in superposition. The interpretability tools most commonly applied in enterprise contexts are not designed for polysemantic representations.

Sparse Autoencoders: The Current Best Tool

Sparse autoencoders (SAEs) are the current state-of-the-art tool for addressing polysemanticity and making model internals more inspectable. The key paper is "Towards Monosemanticity: Decomposing Language Models With Dictionary Learning" (Anthropic, 2023), which demonstrates the approach on a one-layer transformer model.

The core idea is to train a sparse autoencoder to reconstruct the activations of a specific layer, with the constraint that the reconstruction must use a sparse combination of a larger dictionary of features. The autoencoder is not trained on any task-specific objective; it is trained only to reconstruct activations sparsely. When this works, the dictionary entries learned by the autoencoder tend to correspond to monosemantic features: distinct, identifiable concepts that activate cleanly for a single concept rather than mixing multiple concepts in superposition.

The 2023 paper found that applying this approach to a small transformer model produces a large dictionary of highly interpretable features, including features corresponding to specific named entities, thematic topics, syntactic structures, and more abstract properties. Many of these features could be identified by inspection: looking at the inputs that maximally activate a given dictionary entry, a researcher could assign a coherent label to the feature in most cases.

Subsequent research, including "Scaling and Evaluating Sparse Autoencoders" (arXiv:2406.04093, 2024), has extended this work to larger models, demonstrating that the approach scales with model size and that features remain interpretable at larger scales. This is an important result because it suggests that SAEs may eventually be applicable to frontier-scale models used in enterprise deployments, not just the small models used in initial research.

Activation Patching and Causal Tracing

Beyond SAEs, mechanistic interpretability uses a family of techniques collectively called activation patching or causal tracing to test hypotheses about where specific computations happen in a model. The key paper demonstrating causal tracing at scale is "Locating and Editing Factual Associations in GPT" (arXiv:2202.05262, 2022), which showed that factual associations in a language model can be localised to specific layers and positions in the model.

Activation patching works by running two forward passes: one with a "clean" input and one with a "corrupted" input that differs in a specific way. You then systematically replace the activations at different positions and layers in the corrupted pass with the corresponding activations from the clean pass, and observe the effect on the output. Positions and layers where swapping activations restores the clean output are identified as causally important for the specific capability being tested.

This technique has produced some of the most striking results in mechanistic interpretability. It can identify, with layer-level and sometimes head-level precision, where specific types of information are encoded and processed in a model. The findings are verifiable: another researcher with the same model can run the same patching experiments and observe the same effects.

The Scale Question

The honest limitation of the current mechanistic interpretability programme is that most of its most precise findings come from small models. The circuits framework has been applied most rigorously to transformer models with millions of parameters, not the hundreds of billions of parameters found in frontier models. The question of whether the circuits found in small models correspond to circuits in large models, or whether large models implement qualitatively different algorithms, is an active research question.

Research on whether circuit analysis scales, including work by Lieberum et al. examining sparse autoencoders in language models at larger scale, examines this question directly. The findings are cautiously optimistic: circuits identified in smaller models tend to have analogues in larger models, and the circuits framework continues to produce verifiable findings at larger scales. But the research also notes that larger models implement more complex circuits that are harder to fully characterise, and that the combinatorial explosion of possible circuits makes exhaustive enumeration increasingly impractical.

For enterprise leaders, the practical implication is that full mechanistic interpretability of the frontier models currently in enterprise deployment is not yet achievable. What is achievable is a partial but meaningful analysis: identifying circuits responsible for specific capabilities, localising factual associations, and using SAEs to examine the feature representations in specific layers. This partial analysis is significantly more informative than post-hoc attribution methods, even if it falls short of a complete mechanistic account.

What the Science Means for Practice

The scientific developments described here have direct practical implications that most enterprise AI teams have not yet incorporated into their governance frameworks.

First, polysemanticity means that the model's representation of a concept you care about, whether for governance, compliance, or safety reasons, is very likely entangled with representations of other concepts. You cannot safely excise a concept from a model by identifying and removing neurons associated with that concept. The entanglement means that targeted interventions are much harder than simple neuron-level analysis suggests.

Second, SAEs provide a genuinely better basis for concept auditing than direct neuron inspection. If you need to establish whether a model has encoded a specific concept or type of information in its representations, running an SAE on the relevant layers and examining the feature dictionary is more likely to surface that encoding than probing individual neurons.

Third, activation patching provides a method for testing mechanistic hypotheses rather than just observing correlations. If you want to know whether a model's behavior on a specific type of input is causally driven by a specific layer or component, activation patching gives you an experimental tool rather than a correlation-based inference.

These tools do not make full mechanistic interpretability of large models easy or cheap. But they raise the standard of what "we understand this model" means, and they give technically capable teams a more principled basis for making claims about model behavior than post-hoc attribution alone provides.

The Residual Stream and Why It Matters

One concept from the circuits framework that has significant practical implications is the residual stream. Transformers are built around a residual connection architecture: each layer writes to and reads from a shared residual stream, adding its contribution rather than replacing the representation of the previous layer. This means that information computed by early layers is available to all subsequent layers, and that late layers can read directly from early representations without having to reconstruct them.

For interpretability, the residual stream creates a specific and testable architecture for how information flows through the model. Activation patching exploits this architecture directly: by patching the residual stream at a specific position and layer, you can precisely control what information is available to subsequent computation. The clean results achieved by causal tracing (arXiv:2202.05262) in localising factual associations depend on this architectural property. If transformers used a different information flow pattern, the same technique would produce noisier results.

For enterprise governance teams, the residual stream matters because it gives you a principled framework for thinking about which parts of the model are relevant to a specific behavior. Rather than treating the model as a monolithic black box, you can reason about layers and positions as sites of specific computations, and target your interpretability analysis accordingly. This is more efficient than brute-force feature attribution across all model components.

Universality: Do Models Learn the Same Circuits?

One of the most striking findings in mechanistic interpretability is the evidence for universality: the hypothesis that different neural networks, trained independently on different datasets and architectures, implement similar circuits for similar tasks. Early evidence for universality came from computer vision, where similar curve and feature detectors appeared independently in different convolutional networks. More recent evidence suggests the pattern extends to language models.

The Transformer Circuits Thread documented universality for induction heads, a two-component circuit responsible for pattern completion, across many different transformer models. Induction heads appear consistently in transformers trained from different random seeds and on different data distributions. This is not a trivial result: it suggests that the circuits framework is identifying genuine algorithmic solutions to learning problems, not artifacts of specific training runs.

The practical implication for enterprise governance is significant. If circuits are universal, then interpretability findings from one model provide meaningful prior expectations about what to look for in other models. An enterprise team that has characterised the relevant circuits in one version of a model has a head start on characterising the same circuits in the next version, because the circuits are expected to be present and similar. This makes mechanistic interpretability investment more durable than behavioral characterisation that must be repeated from scratch with each model update.