What Is Actually Usable Today: An Honest Assessment

The vendor claims have arrived. Interpretability is now a marketing term, which means the gap between what vendors promise and what their tools actually deliver requires active attention. This post is a practical audit: what interpretability methods are ready for enterprise use, what remains primarily a research instrument, and what questions to ask any vendor or internal team claiming to offer interpretability capability.

Posts 2 and 3 of this series described the science of interpretability research: circuits, superposition, sparse autoencoders, activation patching, and causal tracing. Post 4 described the regulatory requirements that create demand for this science in enterprise contexts. This post answers the question that enterprise leaders and architects actually need to answer when evaluating interpretability investments: what can I deploy in a real system today, and what should I wait on?

The answer is more nuanced than either enthusiasts or skeptics suggest. There are genuine capabilities available now. There are also genuine limitations that are being elided in vendor pitches and in some of the breathless coverage of interpretability research. Navigating the gap between the two requires a clear framework for evaluation.

The Readiness Spectrum

Interpretability methods exist on a spectrum from well-validated and deployable to theoretically promising but not yet enterprise-ready. The spectrum is not purely determined by the sophistication of the method: some technically simpler methods are more reliably useful than more sophisticated ones, because reliability depends on how well-validated a method is across the kinds of inputs and model behaviors you care about, not just on how impressive the research results look.

What Activation Analysis Can Actually Do for You

Activation analysis is the most practically deployable interpretability method available today. The approach is conceptually straightforward: extract the internal activations of a model at one or more layers for a set of inputs, then analyse those activations to detect patterns. The analysis can be supervised (train a linear probe on labeled examples to detect whether a specific concept is represented) or unsupervised (cluster activations to identify natural groupings in the representation space).

The practical value for enterprise governance is in three areas. First, behavioral consistency monitoring: by tracking the distribution of internal activations on a held-out validation set, you can detect when a model's internal representations drift relative to baseline, which is an early signal of behavioral change that precedes output-level degradation. This is significantly more sensitive than monitoring output quality metrics alone, because activation drift can occur before it is visible in aggregate outputs.

Second, failure mode characterisation: by training linear probes on activations from examples where the model fails, you can often identify internal features that are predictive of failure. This produces interpretable descriptions of the conditions under which the model is likely to fail, rather than just a list of failure examples. The EU AI Act documentation obligations described in Post 4 benefit directly from this capability.

Third, distribution shift detection: when inputs arrive that are substantially outside the training distribution, activation patterns often reflect this before outputs become visibly degraded. Activation-based anomaly detection can serve as a monitoring layer that flags inputs for human review before the model has produced an output that may be used in a consequential decision.

Limits of Activation Analysis

Activation analysis has well-characterised limitations that should be understood before deployment. The method is correlational, not causal: a linear probe that predicts failure is evidence of association, not a mechanistic explanation of why failure occurs. The probe may be detecting a correlate of failure rather than the cause. Acting on probe outputs without understanding the causal structure can lead to interventions that target symptoms rather than root causes.

Additionally, linear probes assume that the concept being detected is linearly separable in the representation space. For simple concepts in lower-dimensional projections, this assumption often holds well. For complex, compositional concepts in frontier models where superposition means that individual neurons represent multiple features, linear probes may understate the true richness of the representation. They are useful and reliable, but they are not a complete picture.

Attention: The Most Misused Interpretability Tool

Attention visualisation became popular as an interpretability tool shortly after the transformer architecture was introduced, partly because attention weights are easily visualised and intuitively appealing as explanations. The narrative is seductive: here are the tokens the model attended to when generating this output, which means these are the inputs the model "considered" most relevant.

This narrative is substantially misleading. Research published in 2019 and subsequently replicated across multiple model families showed that attention weights are not faithful explanations of model predictions. The model's outputs can be reproduced even when attention weights are permuted or set to uniform distributions in ways that would completely change the tokens being "attended to." Attention is a routing mechanism for information flow in the residual stream, not a direct measure of which tokens caused an output. Visualising attention and presenting those visualisations as explanations of model behavior is a form of interpretability theater that does not survive scrutiny.

The appropriate use of attention visualisation is narrower: as a diagnostic tool to identify anomalous attention patterns that may indicate unusual model behavior, and as an input to more rigorous causal analysis rather than as a standalone explanation. Vendors who offer attention heatmaps as their primary interpretability product are offering a tool that has known validity problems for the explanatory use case.

Attention heatmaps are a diagnostic starting point. They are not an explanation. Building compliance documentation on attention visualisation alone is not defensible.

Sparse Autoencoders: Genuine Progress, Genuine Constraints

Sparse autoencoders (SAEs) represent the most significant recent advance in practical interpretability tooling. As described in Post 2, SAEs are trained to decompose the internal activations of a language model into a larger set of sparse, interpretable features. The monosemanticity work published by Anthropic in 2023 (Bricken et al., arXiv:2309.08600) demonstrated that SAE-derived features are often human-interpretable and causally relevant to model behavior, validating the approach at scale.

The subsequent scaling work (arXiv:2406.04093) extended this to frontier models, demonstrating that SAE features at very large scale include identifiable features corresponding to specific concepts, people, locations, and behaviors. This is a substantial capability: it means you can, in principle, examine the feature space that a specific model is using to represent concepts in a domain relevant to your application, and identify features that correspond to concepts you care about, including potentially problematic ones.

The constraints are significant, however. Training a high-quality SAE on a frontier model requires access to the model's internal activations at scale, which means access to model weights or to an API that exposes activations (rare in current commercial offerings). Running the SAE inference to decompose activations at serving time adds latency and computational cost. And interpreting SAE outputs requires both domain expertise and interpretability expertise, because the features the SAE discovers are not automatically labeled or organised. A domain expert needs to review the features relevant to their application and determine which are well-interpreted and which require further investigation.

For teams with the expertise to deploy SAEs well, they represent genuine capability for characterising what a model represents. For teams that do not have both the technical depth and the domain expertise to validate SAE outputs, they can produce false confidence: a feature that looks interpretable may be conflating multiple underlying concepts in ways that only become visible when the feature is tested causally.

Activation Patching: The Causal Evidence Standard

Activation patching is the most rigorous tool available for answering specific causal questions about model behavior: does this component of the model causally contribute to this output, or is it merely correlated with the inputs that produce that output? The method works by running the model twice, once on a clean input and once on a corrupted input, and then restoring the activations from the clean run at specific locations in the model graph. If restoring activations at a location restores the clean output, that location is causally responsible for the output difference.

The causal tracing work of Meng et al. (arXiv:2202.05262) used activation patching to demonstrate that factual associations in GPT-2 XL are causally stored in specific middle-layer MLP blocks, with a precision that prior correlation-based methods could not achieve. This validated activation patching as a methodology for identifying causal structure in transformer models.

For enterprise governance purposes, activation patching is best suited to targeted investigations of specific high-risk behaviors. If you have identified a class of outputs that is concerning from a governance perspective, and you want to understand which components of the model are causally responsible for those outputs, activation patching provides the strongest available evidence. It does not require training any auxiliary models, unlike SAEs. It does require careful experimental design: the choice of clean and corrupted inputs, the choice of which locations to patch, and the interpretation of results all require expertise.

Off-the-shelf tooling for activation patching is improving. The TransformerLens library (Nanda and Bloom, 2022) provides accessible Python tooling for mechanistic interpretability on transformer models, including activation patching. It is a research-oriented library, not an enterprise monitoring product, but it is actively maintained and increasingly used for applied interpretability work beyond the academic context.

The Vendor Claim Audit

The growth of interpretability as a regulatory and governance concern has produced a wave of vendor claims that merit careful evaluation. Below is an assessment of common claim patterns, calibrated against what the research actually supports.

What a Serious Enterprise Interpretability Stack Looks Like

Across the teams that have invested seriously in interpretability for governance rather than for marketing purposes, a pattern has emerged. The stack is not a single product. It is a set of complementary methods applied at different levels of investigation depth, with clear protocols for when each method is invoked.

The first layer is behavioral monitoring: systematic tracking of model outputs against expected distributions, with automated alerts when distributional drift is detected. This is not interpretability in the mechanistic sense, but it is the prerequisite for knowing when deeper investigation is warranted.

The second layer is activation monitoring: tracking internal activation distributions at one or more model layers, with anomaly detection tuned to the specific failure modes that have been characterised during development. This layer provides earlier warning than output monitoring and a richer signal for triage.

The third layer is targeted causal investigation: when a behavioral concern is identified that warrants mechanistic investigation, applying activation patching or SAE analysis to characterise the internal structure responsible for the concerning behavior. This layer is not continuous; it is invoked for specific high-priority investigations. It requires the most expertise and produces the most governance-defensible evidence.

The fourth layer is documentation: maintaining a structured record of behavioral characterisations, failure modes, and the causal investigations that have been conducted, with links to the specific method outputs and the interpretation provided by domain and technical reviewers. This documentation is the artifact that satisfies regulatory obligations and that enables consistent human oversight.

The Closed-Weights Problem

A significant practical constraint for enterprises using frontier commercial models is that full mechanistic interpretability requires access to model weights and internal activations. For models accessed via API, activation-level analysis is only possible if the provider exposes internal activations as part of their API offering, which most commercial providers do not currently do.

This is a genuine strategic consideration. Enterprises that are serious about mechanistic interpretability for their highest-risk applications should factor model access into their vendor evaluation. Open-weight models deployed on enterprise infrastructure allow full activation access. API-only access to closed-weight models limits interpretability to behavioral monitoring and whatever post-hoc analysis can be conducted on inputs and outputs without access to internals.

FrugalGPT (Chen, Zaharia, Zou, arXiv:2310.11409) presents a complementary consideration: cost-optimised inference routing can reduce the total cost of operating large models by routing queries to less expensive models when confidence in the output is sufficient. For interpretability purposes, a routing architecture that uses simpler, open-weight models for the highest-stakes queries, where full activation access is required for governance, while using closed-weight frontier models for lower-stakes queries, may represent the most defensible architecture for regulated high-risk applications.

What "Interpretable Enough" Actually Means

The final question this post addresses is practical and often avoided in vendor conversations: what level of interpretability is actually adequate for your specific deployment? The answer is not "as much as possible." Interpretability investments have costs in development time, operational complexity, and latency. The right level of investment is determined by the risk profile of the deployment.

For applications where errors are recoverable, consequences are limited, and human oversight is high-bandwidth, behavioral monitoring and activation anomaly detection may be sufficient. For applications where errors are consequential, consequences include regulatory exposure or harm to individuals, and human oversight is limited in bandwidth, targeted mechanistic investigation of the highest-risk output categories is justified and, under the EU AI Act, arguably required to satisfy Article 9's risk management obligations.

The practical implication is that interpretability investment should be concentrated where risk is concentrated. A blanket "we use interpretability" claim is less meaningful than a specific account of which behaviors have been mechanistically investigated, what the investigation found, and how the findings informed the governance and oversight design for the deployment.

The next and final post in this series translates this assessment into a concrete 18-month roadmap: what to build, what to buy, what to wait on, and how to position interpretability capability as a competitive differentiator in regulated markets.