Neural networks represent far more concepts than they have neurons. This is superposition: a known, studied phenomenon that makes mechanistic interpretability hard and enterprise AI auditing harder. Here is what it is, what the research shows, and what it means for teams deploying AI at scale.
When an enterprise AI system gives an unexpected output, the first question is always the same: why did it do that? The answer is almost never available. Not because the engineers did not try to find it, but because the internal structure of the model does not map cleanly onto human-readable concepts. Neurons do not correspond to ideas. Activations do not correspond to reasoning steps. The gap between what the model computes and what humans can read is fundamental, not incidental.
This is not a new observation. The "black box" critique of neural networks has been common since deep learning entered enterprise deployment. But the mechanistic interpretability research of the last four years has done something more useful than naming the problem: it has explained the mechanism. The reason models are hard to interpret is not just that they are complex. It is that they are doing something specific called superposition, and superposition is now well enough understood that we can reason about what it implies for deployment, auditing, and regulatory compliance.
The research comes primarily from two Anthropic papers: "Toy Models of Superposition" (Elhage et al., arXiv:2209.11895, 2022) and "Towards Monosemanticity: Decomposing Language Models With Dictionary Learning" (Bricken et al., arXiv:2309.08600, 2023). Together they establish what superposition is, why networks develop it, and what a technical path toward more interpretable representations looks like. Neither paper is aimed at enterprise practitioners. This post is.
Start with a simple observation about how neural networks work. A network layer with N neurons has N dimensions of activation space. In the most straightforward interpretation, you might expect that network to represent at most N distinct concepts or features, one per neuron. A neuron that reliably fires for "legal language" and only legal language would be monosemantic: one neuron, one concept.
What Elhage et al. found is that this is not what happens in practice. Real networks are polysemantic: single neurons respond to multiple unrelated concepts simultaneously. A neuron might activate for "legal language" and also for "references to body parts" and also for "mentions of currencies." These concepts have no obvious relationship. The neuron is not computing a meaningful generalization across them. It is storing multiple unrelated features in a single unit via overlapping linear directions in activation space.
Superposition is the phenomenon where a neural network represents more features than it has neurons by storing features as overlapping, nearly-orthogonal directions in activation space. Features can coexist because they rarely activate at the same time: the interference between them stays tolerable because the features are sparse.
The key insight is why this is possible. If features are sparse, meaning any given feature is active only rarely in the training data, then multiple features can share the same neurons without creating too much interference with each other. When feature A and feature B rarely co-occur, the network can store both of them in the same neurons without the interference between them degrading performance beyond an acceptable threshold. The network is making an implicit trade: it sacrifices a small amount of precision for a large increase in the number of concepts it can represent.
In their toy model experiments, Elhage et al. showed that a network with N neurons can represent significantly more than N features when feature sparsity is high. The relationship between sparsity and representable features is superlinear: as features become sparser, the packing density increases substantially. This is not a theoretical curiosity. It is what large language models are doing across billions of parameters: packing far more semantic content into their activation space than a naive count of neurons would suggest.
The reason superposition creates an interpretability problem is direct. If you want to understand what a model is "thinking" at a given layer, you look at which neurons are activating and how strongly. But if neurons are polysemantic, that information is ambiguous at the level of individual neurons. Neuron 47 activating does not tell you which of the several concepts associated with neuron 47 is relevant to the current input. The model knows, in the sense that the downstream computation uses it correctly, but the activation value alone does not encode which concept triggered it in human-readable form.
This matters enormously for enterprise applications. Interpretability tools built on individual neuron analysis are fundamentally limited. Attribution methods that try to explain an output by tracing it back through layer activations hit the same wall: the features the model is using are not stored cleanly in individual neurons. They are distributed across multiple neurons in overlapping directions that require decomposition to read.
The practical implication: most activation-based interpretability methods in enterprise AI systems today are reading at the wrong granularity. They are looking at neurons when the meaningful unit of analysis is a feature direction in activation space, and these two things do not align cleanly in models that use superposition. You can observe that certain neurons activate on certain inputs. You cannot straightforwardly read off which feature the model is using, because polysemantic neurons carry multiple features simultaneously.
The "Towards Monosemanticity" paper (Bricken et al., arXiv:2309.08600) is the practical follow-on to the theoretical foundation. Its approach: train a sparse autoencoder (SAE) on the internal activations of a language model, specifically the MLP layer of a small one-layer transformer. The SAE learns a dictionary of features, each represented as a linear direction in the model's activation space, where the dictionary is overcomplete: it has far more feature directions than the model has neurons.
The result is striking. The SAE recovers thousands of distinct, interpretable features from a small model with limited neurons. Many of these features correspond to recognizable semantic concepts: features that activate for base64-encoded content, for DNA sequences in biological text, for legal language, for mathematical notation, for names in different languages. These are not categories the researchers imposed. They emerged from the SAE decomposition and were verified by examining which inputs caused each feature to activate most strongly.
Crucially, these SAE-recovered features behave much more monosemantically than the underlying model neurons. A given feature direction activates for a coherent, identifiable concept. The polysemanticity is still in the original model, but the SAE provides a lens that decomposes the superposed representation into a more readable form. This is mechanistic interpretability's most concrete result to date: a method that partially bridges the gap between what the model computes and what humans can read.
A single neuron activates for legal text, currency mentions, and geographic references. The activation value tells you something activated the neuron, not which concept. Standard attribution tools read ambiguous signal.
A SAE-recovered feature direction activates specifically for legal language. The feature is verifiable: examine the highest-activating inputs and they form a coherent semantic category. Attribution is now tractable.
The SAE approach is not yet production-ready for most enterprise teams. Training an SAE adds computational overhead. The features it recovers are not guaranteed to be complete: there may be concepts the model uses that the SAE does not cleanly recover. And the research so far has focused on small models and specific layers, not the full architecture of frontier models in enterprise deployment. But the direction is clear and the research is moving quickly. Within two to three years, SAE-based analysis tools are likely to be a standard part of the interpretability toolchain for serious enterprise AI deployment.
The research has several direct implications for teams running AI at scale. These are not speculative. They follow from the mechanism.
Most enterprise interpretability and explainability tools operate at the level of input attribution: which tokens or input features most influenced a given output? These methods can be useful for debugging individual outputs, but they do not expose the internal representations the model built on the way to that output. Because of superposition, the internal representations are distributed across overlapping directions in activation space. Attribution at the input level cannot fully explain behavior that arises from these internal representations. Teams that need to audit model internals need to understand this gap and plan for it.
Article 13 of the EU AI Act (Regulation 2024/1689) requires that high-risk AI systems be sufficiently transparent to allow users to interpret the system's output and use it appropriately. Article 17 requires technical documentation of the system's design logic. These requirements are not satisfiable with black-box outputs alone. For AI systems in regulated categories including credit scoring, employment screening, and medical device decision support, interpretability is now a compliance question, not just an engineering preference. The superposition problem is directly relevant: if your model's internal features cannot be read, meeting these documentation requirements at a meaningful level of depth becomes substantially harder.
In a single-turn language model deployment, an unexpected output is visible and reviewable before action is taken. In an agentic system, the model makes intermediate decisions that are not always surfaced to a human reviewer: which tool to call, what parameters to pass, whether the result of one step is sufficient to proceed to the next. Each of these intermediate decisions is shaped by the model's internal representations. Because those representations use superposition, the features the model is acting on at each step are not directly readable. As agentic deployment becomes standard in enterprise AI, the interpretability gap becomes a deployment risk question, not just a research question.
Interpretability is moving from a research topic to an operational requirement. The superposition hypothesis tells you why your current tools are limited. The monosemanticity work tells you what the technical path forward looks like. Enterprise teams that start building familiarity with SAE-based analysis now will be ahead of the compliance and audit requirements that are coming.
The two papers are worth reading directly if you have technical staff who need to engage with interpretability tooling decisions. Here is a practical guide to what each one establishes and what to take from it.
This paper establishes the phenomenon. It uses small, synthetic networks where the ground truth is known and shows that these networks develop superposition when the number of features in the training data exceeds the number of neurons. It derives the conditions under which superposition emerges: feature sparsity and the network's tolerance for interference between features. The key takeaway is that superposition is not a bug or an artifact of scale. It is what networks rationally do when they need to represent more features than they have neurons, which is the typical condition in real language models.
What practitioners should take from it: the interpretability problem has a specific mechanism, not just a general complexity argument. That mechanism is superposition. Any team building interpretability tooling needs to grapple with this mechanism rather than treating the black box as opaque by assumption.
This paper provides the technical path toward a solution. It shows that sparse autoencoders can decompose the polysemantic representations in a real language model into a larger set of more monosemantic features. The features recovered are verifiable and interpretable. The paper also shows that these features have meaningful causal properties: artificially activating a feature direction in the model's activation space produces the behavior you would predict from knowing what the feature represents.
What practitioners should take from it: SAE-based analysis is a real method with demonstrated results, not a speculative direction. It is not yet plug-and-play for production deployment, but it is far enough along that teams doing serious interpretability work should be tracking it. The fact that features have causal properties is important: it means the representations the SAE recovers are not just labels, they are the actual computational units the model uses to produce output.
Several active research directions extend from this foundation. Scaling SAEs to larger models and to more layers of the network is the immediate frontier. Circuit analysis, which maps how features interact across layers to produce a specific behavior, is a parallel track that connects feature-level interpretability to behavior-level explanations. Automated interpretability methods, which use language models to label the features that SAEs recover, address the bottleneck of human labeling at scale. Each of these directions is moving toward the same goal: making the internal computation of large language models readable at the level of features, not just neurons.
Before deploying AI in a high-stakes or regulated context, evaluate your interpretability posture against these questions:
Superposition and polysemanticity are not just technical puzzles. They are the mechanism behind most of the deployment failures that enterprise AI teams struggle to explain. A model that refuses a reasonable request, or helps with an unreasonable one, or produces confidently wrong output is doing so because of how it has organized its internal representations. Understanding that those representations use superposition does not immediately tell you how to fix the behavior, but it does tell you that looking at individual neurons or surface-level attributions will not get you to the root cause.
The mechanistic interpretability research program is trying to build the tools that would let engineers actually read what a model is doing internally. That research is still early relative to the complexity of frontier models. But it is far enough along to tell enterprise practitioners that the gap between "the model computed something" and "we can read what it computed" has a known mechanism and a technical path forward. The path is sparse autoencoders, feature-level analysis, and circuit tracing. None of these are ready for casual adoption, but all of them are advancing rapidly.
The organizations that will be positioned to audit, explain, and govern their AI systems in 2028 are the ones building literacy with this research direction now. The compliance window for high-risk AI interpretability is closing. The technical tooling is advancing to meet it. Understanding what superposition is and why it matters is the prerequisite for navigating both.