The Architecture Nobody Reads

When an enterprise team evaluates a language model, they run benchmarks, test prompts, measure latency. What almost no team does is ask: how does this model actually represent the concepts it reasons about? That question sounds academic. The answer is not.

Research from Anthropic's interpretability team, published in 2022, established that neural networks routinely encode far more features than they have neurons by using the same neurons for multiple overlapping purposes [1]. The researchers called this superposition. For enterprise AI, it is not a curiosity. It is a structural explanation for a category of reliability failures that no monitoring dashboard currently captures.

This post names that gap, gives it a term, and gives you a framework for governing it.

Coined Term: Superposition Blindness

Superposition Blindness is the organizational condition in which an enterprise AI governance program lacks any mechanism to detect, measure, or respond to failures caused by a model encoding multiple features in the same representational space. A team experiencing Superposition Blindness can identify that an output is wrong, but cannot determine whether the root cause is a data issue, a prompt issue, or a representational collision inside the model itself.

What Superposition Actually Means

In a traditional symbolic system, each concept gets its own variable. In a neural network, there is no such separation. A neuron does not represent one thing. Research by Elhage and colleagues [1] showed that a network with N neurons can represent far more than N features by encoding them as quasi-orthogonal directions in a high-dimensional space. This works because most features are only rarely active simultaneously, so the interference between overlapping representations stays small most of the time.

The word "most of the time" is doing enormous work in that sentence. When two features that share representational space are simultaneously relevant, the model's output degrades in ways that look like random errors but are structurally determined by the geometry of the representation.

Research Context

Elhage et al. [1] demonstrated this in toy models and found evidence it scales. Bricken et al. [2] subsequently showed that sparse autoencoders can be used to decompose model activations into more monosemantic (single-feature) representations, recovering thousands of interpretable features from a single layer. This does not fix superposition; it makes it legible.

For enterprise teams, the practical translation is this: your model may have learned to compress a concept like "legal obligation" and a concept like "optional recommendation" into overlapping representational space. In most queries, only one is active, and the output is correct. In an edge case where both are simultaneously relevant, you get an output that confidently misclassifies one as the other. Your monitoring system sees an error. It cannot tell you why.

Coined Term: Feature Collapse

Feature Collapse is the specific failure mode in which two semantically distinct concepts that share representational space in a language model produce an incorrect output when both are simultaneously relevant to a query. Feature Collapse is distinct from hallucination caused by training data gaps: the model has learned both concepts and represents them, but the representation geometry causes interference at inference time.

The Three Failure Modes Superposition Causes

Superposition does not cause all LLM failures. It causes a specific subset. Governance programs that treat all failures as equivalent will apply the wrong mitigations to each.

Failure Mode 1: Confidence Inversion

The model produces a high-confidence output that is wrong precisely because the overlapping features reinforce each other in a locally consistent but globally incorrect direction. The confidence calibration is misleading because the interference looks like strong signal.

Failure Mode 2: Context Sensitivity Collapse

A query that differs from a prior query by a single contextual detail (jurisdiction, time period, document type) triggers a different feature in the same representational space, producing a drastically different output despite apparent similarity. Regression testing fails to catch this because the triggering delta is subtle.

Failure Mode 3: Adversarial Amplification

An attacker who understands the approximate geometry of a model's representational space can craft prompts that deliberately activate overlapping features, causing reliable misbehavior. This is a harder attack than simple prompt injection because it exploits the model's learned structure, not its instruction-following surface.

What This Is Not

Superposition-driven failures are not training data hallucination, not tokenization artifacts, and not context window overflow. Applying grounding or retrieval mitigations to Feature Collapse does not fix it. Misidentifying the failure class is itself an organizational failure.

Architecture: Where Superposition Lives in Your Stack

Representational Risk Stack
Input Layer: Tokenization and Embedding Representation Layer: Superposition Risk Zone Features compressed into quasi-orthogonal directions Attention and MLP Layers: Feature Interaction Output Layer: Token Probability Distribution Governance Gap: No enterprise tool currently measures feature collision in this zone Output monitoring catches errors; it cannot diagnose representational root cause
The representation layer sits between tokenization and output. Enterprise monitoring instruments the output layer; the source of Feature Collapse is the representation layer. This gap is Superposition Blindness.

What Mechanistic Interpretability Can Tell You

Mechanistic interpretability is the research program that attempts to reverse-engineer the learned computations inside neural networks. Olah and colleagues [3] showed that specific circuits, meaning recurring subgraphs of attention heads and MLP neurons, implement identifiable algorithms. Conmy and colleagues [4] automated the discovery of these circuits for specific model behaviors.

The practical value for enterprise teams is not that you will run circuit analysis on your production models. Most teams will not. The value is what it reveals about failure classes: superposition-driven failures have different signatures than data-gap failures, and governance programs that can distinguish them will apply better mitigations.

Failure Mode Distinguishability by Monitoring Approach
Illustrative comparison of how well four monitoring approaches detect each failure class. Superposition-driven failures (Feature Collapse, Confidence Inversion) are poorly detected by output-layer monitoring alone. Directional illustration based on practitioner observation.

The Governance Gap

Current enterprise AI governance frameworks, including NIST AI RMF [5] and ISO/IEC 42001, define risk categories at the application layer: bias, fairness, accuracy, robustness. These are appropriate categories. They do not reach the representational layer.

This creates a structural gap: an organization can be fully compliant with every current governance standard and still have no mechanism to detect or respond to Feature Collapse. Compliance and safety are not the same thing when the framework does not yet address the relevant failure class.

Governance Framework Coverage vs. Failure Class
Coverage matrix: which governance frameworks address which failure classes. Green indicates substantial coverage; the superposition row reveals the gap across all current frameworks. Directional illustration; not derived from systematic audit data.

Decision Framework: Do You Have Superposition Exposure?

Not every enterprise AI deployment has equal superposition exposure. Four variables determine your risk level.

  1. Domain precision requirement. Applications where the distinction between two similar concepts (obligatory vs. permissive, liable vs. not liable, effective vs. ineffective) determines a consequential outcome have high superposition exposure. Applications where approximate conceptual matching is sufficient have low exposure.
  2. Query distribution breadth. Narrow, well-specified query types are less likely to trigger feature interference than broad, open-ended queries. A model that only answers questions about one product's specifications is less exposed than a general enterprise assistant.
  3. Confidence reliance. Downstream decisions that treat model confidence scores as meaningful signals are more exposed, because Confidence Inversion specifically corrupts confidence calibration while leaving output structure intact.
  4. Adversarial surface area. Applications where external users can freely craft inputs have higher Adversarial Amplification exposure than internal-only tools with structured input schemas.

Three Enterprise Scenarios

Scenario 1 · Legal Technology

Contract Obligation Classifier

A CLO deploys a language model to classify contract clauses as obligatory, permissive, or aspirational. In testing on standard commercial contracts, accuracy exceeds 94%. In production, a subset of claims settlement clauses that combine obligatory language with conditional permissive carve-outs begins producing misclassifications at 23% rate. The failure is not a data gap: the model has seen both clause types. The failure is Feature Collapse between two representational directions that overlap in the model's learned geometry. The fix is not retrieval-augmented generation; it is output verification against a deterministic rule engine for the specific clause subtype, and flagging the query for human review when both feature directions are likely active. Cross-link: see the hallucination taxonomy post for how Instruction Confabulation compounds this failure in agentic workflows.

Scenario 2 · Financial Services

Credit Memo Summarization

A CFO team deploys a summarization model for credit memos. The model performs well on standard corporate credit but begins producing subtle errors on leveraged buyout memos where the same company appears as both borrower and guarantor in overlapping roles. The model's representation of "borrower" and "guarantor" appear to share representational space, and the dual-role trigger causes confidence inversion: the model produces high-confidence summaries that misattribute obligations. Output monitoring detects errors only after a downstream credit decision has already been flagged. The governance intervention: add a structured pre-check that identifies dual-role entities and routes those memos to a narrower, specialized model fine-tuned on LBO structure. Cross-link: AI infrastructure strategy covers model routing architectures.

Scenario 3 · Healthcare Operations

Clinical Protocol Classification

A COO deploys a model to route clinical queries to the correct protocol. The model handles standard acute and chronic care routing well. A class of queries involving patients with concurrent acute and chronic presentations (comorbid acute exacerbation) begins producing systematic routing errors. The failure signature, including high confidence paired with incorrect routing, matches Feature Collapse. The architecture decision: implement a comorbidity detector as a pre-step that identifies concurrent presentations and forces a human-in-the-loop review before model output determines routing. The model is not replaced; its superposition exposure for this query class is compensated by an upstream guard. Cross-link: prompt debt covers how workaround accumulation around failure modes creates its own governance burden.

Executive Checklist: Governing Superposition Exposure

01
Have you mapped your high-precision concept pairs?

Good: a list of domain concepts where misclassification is consequential and the concepts are semantically adjacent. Red flag: "the model was evaluated on accuracy overall" without concept-pair analysis.

02
Does your failure taxonomy distinguish representational failures from data-gap failures?

Good: different escalation paths for high-confidence wrong outputs vs. low-confidence wrong outputs. Red flag: all errors treated as equivalent in post-incident review.

03
Do you have output verification for high-precision concept pairs?

Good: deterministic rule engine or specialized model as a second-pass verifier for the highest-risk concept pairs. Red flag: model confidence score used as the sole quality signal.

04
Is your regression test suite structured to surface context-sensitivity failures?

Good: test cases that vary a single contextual variable across semantically adjacent concepts. Red flag: regression suite tests only known-good examples, not near-boundary cases.

05
Is your adversarial testing scoped to representational attacks, not just prompt injection?

Good: red team exercises that specifically attempt to activate overlapping features through crafted inputs. Red flag: adversarial testing limited to jailbreak and injection patterns.

06
Do your governance reports distinguish Superposition Blindness as a risk category?

Good: board-level AI risk reporting includes a representational risk category with specific monitoring metrics. Red flag: governance reporting covers accuracy, fairness, and compliance, with no representational risk category.

Build, Buy, or Configure

Build

  • High-precision concept-pair test suites for your domain
  • Output verification rules for your highest-risk classification pairs
  • Failure taxonomy documentation distinguishing representational failures
  • Human-review triggers for high-confidence wrong-direction signals

Buy (Vendor Category)

  • Model evaluation platforms with confidence calibration analysis
  • Sparse autoencoder tooling for feature-level model inspection
  • Adversarial testing platforms that go beyond prompt injection
  • Interpretability-as-a-service (emerging category)

Configure

  • Existing monitoring to flag high-confidence errors separately
  • Routing logic to specialized models for high-precision concept pairs
  • Existing governance templates to add representational risk category

Three-Phase Roadmap

Phase 1 · Weeks 1-6

Exposure Mapping

  • Identify high-precision concept pairs in your domain
  • Build targeted test suites for each pair
  • Run existing models against near-boundary cases
  • Gate: failure rate map with concept-pair breakdown
Phase 2 · Weeks 7-14

Mitigation Architecture

  • Build output verification layer for highest-risk pairs
  • Implement confidence-inversion detection heuristics
  • Add human-review routing for flagged outputs
  • Gate: 80% reduction in undetected high-confidence errors
Phase 3 · Weeks 15+

Governance Integration

  • Add representational risk to governance reporting
  • Run adversarial testing scoped to representational attacks
  • Evaluate interpretability tooling for ongoing monitoring
  • Gate: representational risk category in board AI risk report

Cost of Inaction

High-Confidence Wrong Outputs

Confidence Inversion specifically corrupts the signal that downstream review processes use to triage. Organizations that trust confidence scores will systematically miss the highest-severity errors from superposition-driven failures.

Governance Compliance Gap

Current AI governance frameworks do not require representational risk assessment. Organizations that wait for regulatory mandates will find themselves building the capability reactively, after an incident that becomes the mandate's origin story.

Adversarial Surface Exposure

Adversarial Amplification via representational geometry is a harder attack to detect than prompt injection. Organizations that only test for injection attacks have an unmonitored attack surface in models deployed to external users.

Misdiagnosed Root Cause

Post-incident reviews that misclassify Feature Collapse as a data quality problem apply retrieval mitigations to a representational failure. The root cause persists. The mitigation investment does not address it. Both costs compound.

Excited about AI, innovation, and growth?

Start a conversation

References

  1. Elhage, N., et al. "Toy Models of Superposition." Anthropic, arXiv:2209.11895, 2022.
  2. Bricken, T., et al. "Towards Monosemanticity: Decomposing Language Models With Dictionary Learning." Anthropic Transformer Circuits Thread, 2023.
  3. Olah, C., et al. "Zoom In: An Introduction to Circuits." Distill, 2020. DOI:10.23915/distill.00024.001.
  4. Conmy, A., et al. "Towards Automated Circuit Discovery for Mechanistic Interpretability." arXiv:2304.14997, 2023.
  5. National Institute of Standards and Technology. "Artificial Intelligence Risk Management Framework (AI RMF 1.0)." NIST AI 100-1, 2023.