What Interpretability Reveals That Surprises Everyone

The empirical findings from mechanistic interpretability research are not what most people in enterprise AI expect. They are stranger, more structured, and more practically consequential than the coverage suggests. This post covers the findings that have changed how researchers think about what language models are and how they work.

Each finding described here is drawn from peer-reviewed or preprint research with specific experimental methodology. None of these are theoretical conjectures. They are observed phenomena in specific models, replicated by independent research groups in some cases, and verifiable by anyone with access to the relevant model weights and the computational resources to run the experiments.

Finding One: Feature Geometry Is Structured and Meaningful

One of the most surprising findings in interpretability research is that the geometric relationships between feature representations in a model's activation space carry meaningful semantic content. Features that represent semantically related concepts tend to be geometrically close in the activation space. But the structure goes beyond simple clustering.

Research in this area has found that certain concepts are represented using regular geometric structures that correspond to semantic relationships. The relationship between a country and its capital, for example, is encoded as a consistent direction in the activation space: adding the vector corresponding to "capital of" to the representation of France moves you toward the representation of Paris. This is the arithmetic property that made word2vec famous, but it appears in transformer models too, and mechanistic interpretability research has begun to identify the specific circuits that implement these geometric relationships.

More striking is the finding that features representing opposite concepts tend to be encoded as roughly antipodal directions, and features representing categorical distinctions tend to form regular geometric patterns such as pentagons and hexagons in two-dimensional projections of the relevant activation subspace. These geometric regularities are not imposed by the architecture. They emerge from training on natural language data.

The practical implication for enterprise AI is that models have internal representations of relational structure that go far beyond surface pattern matching. When a model reasons about the relationship between a supplier and its contractual obligations, or between a patient's diagnosis and appropriate treatments, it is operating on geometric relationships in a high-dimensional feature space that are meaningfully structured, not just statistically correlated. This is good news for reliability in-distribution and an important caution about how the model might behave when relationships in the input deviate from the patterns represented in its training data.

Finding Two: Universality Across Models and Architectures

A finding that has surprised many researchers is the degree to which specific circuits and features appear consistently across different models trained on different data with different architectures. This "universality" hypothesis, explored in "Zoom In: An Introduction to Circuits" (Distill, 2020), proposes that models trained on similar data tend to learn similar features and circuits, even if trained independently.

The evidence for universality is strongest for low-level features. Curve detectors, frequency detectors, and certain types of syntactic processing circuits appear across multiple models and architectures. For language models specifically, induction heads, the pattern-completion circuit described in Post 2 of this series, have been found across a range of transformer sizes and training configurations, suggesting they are a convergent solution to a common statistical structure in language data.

Universality has significant practical implications. If models trained on similar data tend to learn similar circuits, then interpretability findings from one model may transfer to other models trained in similar conditions. A vulnerability or bias discovered through mechanistic analysis of one model may be present in models trained by different organisations on similar data. And conversely, a safety mechanism verified in one model may or may not be present in a retrained or fine-tuned version, depending on whether the training conditions preserve the relevant circuits.

Universality means that a mechanistic finding in one model is not just a curiosity about that model. It may be a finding about a class of models trained on similar data.

Finding Three: Models Implement Lookup Tables, Not Just Pattern Matching

"Locating and Editing Factual Associations in GPT" (arXiv:2202.05262, 2022) produced a finding that challenged assumptions about where and how factual knowledge is stored in language models. Using causal tracing, the research team found that factual associations, the model's ability to complete "The Eiffel Tower is in" with "Paris," are stored primarily in the weights of specific MLP layers, and that these storage locations can be identified with layer and position precision.

This finding has two important implications. First, factual knowledge in language models is localised, not uniformly distributed across all parameters. The model is not doing uniform pattern matching across all its weights for every query; it has something closer to a lookup mechanism implemented in specific layers, with the attention heads doing the work of routing queries to the correct lookup location.

Second, and more practically relevant for enterprise AI teams, the localisation of factual associations means that targeted editing of factual knowledge is at least theoretically feasible. The same paper demonstrates "ROME" (Rank-One Model Editing), a technique that modifies a single factual association in a model by directly editing the identified weight matrix, without retraining. This is not currently a mature enterprise tool, but it points toward a future where model knowledge can be audited and selectively corrected at the factual level, not just through fine-tuning.

Finding Four: Attention Heads Implement Specific Named Functions

Mechanistic analysis has identified specific classes of attention heads that implement well-defined, nameable functions across many models. Beyond induction heads (described in Post 2), research has identified:

The existence of named, function-specific attention head types is significant because it means the model is not doing uniform, undifferentiated computation at every layer. It has specialised computational modules with identifiable functions, more like a program with subroutines than a uniform statistical interpolation engine.

Finding Five: Safety-Relevant Features Are Entangled with Capability Features

One of the most practically consequential findings from interpretability research is that features relevant to safety and alignment are not cleanly separable from features relevant to capability. The circuits that implement a model's ability to reason about harmful content are often closely related to the circuits that implement its general reasoning capabilities.

Research on "representation engineering" (arXiv:2310.01405, 2023) identified that properties such as honesty, emotional valence, and content harmfulness are represented as identifiable directions in the model's activation space, and that steering along these directions can produce targeted changes in model behavior. This finding suggests that safety-relevant properties are encoded as genuine representational features, not just surface-level output patterns.

But the same research shows that these representations are entangled with capability-relevant representations in complex ways. Interventions that shift a model's honesty representation can produce side effects on other behaviors. Safety training that modifies the model's outputs may or may not modify the underlying representations in the intended direction. Post-hoc analysis of model behavior using output evaluations alone does not tell you whether the underlying representations are aligned with the intended change.

For enterprise AI governance, this finding has direct implications. A model that passes output-level safety evaluations may still have internal representations that would produce unsafe behavior on inputs outside the evaluation distribution. Conversely, a model that occasionally produces problematic outputs may have safety-relevant representations that are functioning correctly but being overridden by other features in specific contexts. Distinguishing these cases requires looking at internal representations, not just outputs.

Finding Six: Emergent Capabilities Have Mechanistic Precursors

One of the most debated phenomena in large language model research is "emergence": the apparently sudden appearance of capabilities as model scale increases, which seems discontinuous with the gradual improvement expected from smooth scaling laws. Interpretability research has begun to provide a mechanistic account of why emergence appears discontinuous even when the underlying learning process is continuous.

Research on in-context learning and on the development of induction heads during training provides an early example. Induction heads develop suddenly during training at a specific point in the training process, corresponding to a phase transition in the model's ability to perform in-context learning. Before this transition, the model's performance on tasks requiring pattern completion across a context window is substantially worse. After the transition, it improves sharply. The mechanistic account is that the circuits required for in-context learning are not gradually built up; they are either present and functional, or absent. The transition point is when all the required components are in place and the circuit becomes operational.

This finding has a general implication: emergent capabilities are not mysterious discontinuities in an otherwise smooth process. They are the behavioural signatures of circuits becoming operational. If we can identify the circuits required for a capability, we can in principle predict when that capability will emerge during training, and monitor for its emergence by watching for the relevant circuits rather than waiting to observe the behavioral effect.

Finding Seven: The Geometry of Features Encodes Relationships

One of the more subtle findings in interpretability research is that the geometric relationships between feature representations encode semantic and logical relationships between the concepts they represent. This goes beyond the observation that features have directions in activation space; it shows that the directions are systematically organised in ways that mirror the structure of the concepts themselves.

The classic example from word embedding research is directional arithmetic: the difference vector between the representation of a capital city and its country is approximately constant across many countries. This is not a trivial observation: it means the model has discovered, without explicit supervision, that the relationship "capital city of" is a consistent directional transformation in its representation space. The same principle extends to transformer models: the difference between feature representations encodes relational information that the model uses to perform analogical reasoning and multi-hop inference.

For enterprise governance, this finding matters because it means that a model's internal representations are not just a lookup table of memorised facts. They are a structured, relational knowledge representation that the model uses to compose knowledge for novel queries. Understanding the structure of this representation, rather than just cataloguing what specific facts the model knows, is what gives interpretability tools their predictive power for novel inputs.

What These Findings Change

Taken together, these findings change the conceptual framework that should govern enterprise AI decision-making in several important ways.

The model is not a statistical interpolation engine. It has internal structure: identifiable circuits implementing specific functions, geometric feature representations with meaningful semantic content, localised factual storage mechanisms, and emergent computational modules. This structure is more legible than "black box" framing implies, but it requires specific tools and methods to access.

Output evaluation is necessary but not sufficient for governance. A model that produces correct outputs on your evaluation set may have internal representations that are misaligned with your intended design in ways that will surface only on out-of-distribution inputs. The findings on feature entanglement and superposition mean that the gap between output behavior and internal representation is potentially large and consequential.

Interpretability is not primarily about explaining past decisions. The enterprise framing of explainability has focused on producing after-the-fact justifications for individual decisions. Mechanistic interpretability is about understanding the mechanism well enough to predict behavior on future inputs, identify vulnerabilities before they surface, and verify that governance interventions have the intended effect on model internals rather than just on output distributions.

The next post covers how these scientific developments intersect with the regulatory requirements your organisation is already subject to or will be subject to within the current compliance cycle.

The Practical Significance for Enterprise AI Teams

Each of the findings described in this post has a direct operational implication that should affect how enterprise AI teams approach model governance. These are not abstract observations about neural network architecture. They are evidence about how specific, high-stakes behaviors in deployed models are implemented, and that evidence should inform what questions your governance process asks and what tools it uses to answer them.

The finding that factual associations are localised to specific MLP layers means that if your model produces a factual error that is consequential in your deployment, there is a specific location in the model where that incorrect association is stored. This is actionable: it means targeted analysis can determine whether a factual error is a storage problem (the model has encoded incorrect information) or a retrieval problem (the model has correct information encoded but fails to retrieve it appropriately in context). These are different problems with different remediation strategies. A post-hoc attribution approach cannot make this distinction. Causal tracing can.

The finding that named attention head types exist with specific functions means that the attention patterns your model uses when processing a specific category of input are not arbitrary. If you can identify which attention head types are active on the inputs that produce your highest-risk outputs, you have a structural account of what the model is doing at the computational level. This is substantially more informative than knowing which input tokens the model attended to, which is what standard attention visualisation provides.

The finding that safety-relevant features are entangled with capability features means that safety evaluation at the output level is insufficient for high-stakes deployments. A model that produces safe outputs on your evaluation set may have safety-relevant representations that are functional but close to boundaries that in-distribution evaluation does not probe. The only way to assess this is to look at the representations directly, which requires access to model internals and tools capable of interpreting those internals.

The finding that emergent capabilities have mechanistic precursors means that monitoring model behavior purely through output evaluation gives you a lagging indicator of capability changes. If you are tracking a deployed model for capability drift, watching for the activation patterns associated with circuit formation gives you an earlier signal than waiting for output-level changes to become visible. This is particularly relevant for models that are fine-tuned periodically on new data, where unexpected capability gains or losses may occur without being reflected immediately in your evaluation metrics.

What Has Not Yet Been Found

A responsible account of interpretability research findings must also acknowledge what has not yet been established. Full mechanistic characterisation of frontier-scale models remains out of reach. The circuits identified in GPT-2 Small and similar small models have partial analogues in larger models, but the combinatorial complexity of larger models means that no complete circuit-level account of any complex capability exists for models at the scale of current frontier deployments.

The universality hypothesis, while supported by substantial evidence, is not established as a universal law. There are capabilities for which different models appear to implement different circuits, and the conditions under which universality holds versus breaks down are not fully understood. This limits how much you can generalise from interpretability findings on one model to predictions about another model, particularly when the models differ substantially in architecture, training data, or scale.

The relationship between mechanistic structure and behavioral alignment is still being established. The finding that safety-relevant features have identifiable representations in activation space does not yet translate into a reliable method for verifying that a model's safety representations are functioning correctly in all relevant contexts. The research direction is promising, but the tools for using it in enterprise governance are still maturing.

These limitations are not reasons to discount the findings that have been established. They are reasons to be precise about what the evidence does and does not support, which is exactly the kind of epistemic precision that good governance requires.