What Interpretability Actually Means, and Why Leaders Need to Care Now

Something important is happening in AI research, and it is being misread by most enterprise leaders. The word gaining traction is "interpretability." Most executives hear it and file it next to "explainability," a checkbox they already have covered. That filing error is going to be expensive.

Explainability and interpretability are not synonyms. They are different answers to different questions about AI systems, and conflating them produces a false sense of governance coverage that regulators, auditors, and your own risk function will not accept as AI deployments scale.

This series is for two audiences simultaneously: technical leaders and researchers who want the strategic and regulatory framing, and business leaders who need enough scientific grounding to make real decisions. I will not condescend to either group. You can follow the technical thread or the strategic thread; ideally you follow both, because the question of what is actually happening inside a language model is no longer separable from the question of whether your organisation can defend its AI decisions under scrutiny.

The Explainability Trap

Over the past several years, enterprises deploying machine learning in regulated contexts adopted explainability frameworks as their answer to the "why did the model do that" problem. LIME (Local Interpretable Model-agnostic Explanations) and SHAP (SHapley Additive exPlanations) became standard components of model governance stacks. The pitch was straightforward: even if you cannot understand the full model, you can approximate a locally linear explanation for any individual prediction. That approximation can be presented to a regulator, a customer, or an internal audit committee as evidence that the decision is understood.

The problem is that these explanations describe what features the model weighted in a specific prediction, not what the model actually knows or does. They are post-hoc approximations of a different, simpler model fitted to match the original model's outputs in a local region. When the two models diverge, the explanation diverges from the truth. Research has shown that LIME and SHAP explanations can be manipulated to produce misleading outputs while the underlying model's behavior remains unchanged. They are useful tools for debugging and communication, but they do not constitute a mechanistic account of model behavior.

For low-stakes applications, this limitation is manageable. For high-stakes applications in finance, healthcare, legal reasoning, or security, it is not. A SHAP value cannot tell you whether a model has learned a genuine causal relationship or a spurious correlation. It cannot tell you how the model will behave on inputs that are slightly outside its training distribution. It cannot tell you whether a model is using a concept you thought was excluded from its decision process.

Explainability tells you what features a model weighted. Interpretability asks what the model actually knows and how that knowledge is structured.

What Interpretability Research Actually Does

Mechanistic interpretability is a research programme with a specific and ambitious goal: to reverse-engineer the algorithms that neural networks implement. Not to approximate their behavior from the outside, but to open them up and describe what is actually computed at each layer, in each set of neurons, through each attention head.

The foundational framing comes from researchers working on what they call the "circuits" approach to neural network analysis. The premise is that neural networks implement recognisable computational subroutines, and those subroutines can be identified, described, and verified. A group of neurons that collectively represents a specific concept or performs a specific operation is a "circuit." The goal of mechanistic interpretability is to catalogue these circuits and understand how they compose to produce the model's overall behavior.

This is not an aspirational research direction. Work in this area has produced concrete, verifiable findings about specific algorithms implemented in specific models. Induction heads, a class of attention pattern identified in transformer models, implement a specific sequence-completion algorithm that can be described precisely. The discovery was verified by identifying the heads, hypothesising the algorithm they implement, and confirming the hypothesis through ablation and activation patching experiments.

The research programme is documented across a series of papers including "A Mathematical Framework for Transformer Circuits" (Transformer Circuits Thread, 2021), "Toy Models of Superposition" (arXiv:2209.11895, 2022), and "Towards Monosemanticity: Decomposing Language Models With Dictionary Learning" (Anthropic, 2023). These are not theoretical position papers. They are empirical findings with verifiable experimental methodology.

Three Concepts Every Leader Needs to Understand

You do not need to read the full technical literature to develop an informed view of interpretability. Three concepts are enough to ask the right questions of your AI teams and vendors.

Why This Is Now a Strategic and Regulatory Issue

For most of the history of enterprise AI, interpretability was a research concern with no immediate business relevance. That changed with two developments arriving in close proximity: the scaling of large language models into consequential enterprise applications, and the passage of binding AI regulation with explicit transparency and auditability requirements.

The EU AI Act (Regulation 2024/1689), which took effect in 2024 with staged compliance timelines, establishes transparency obligations for high-risk AI systems under Article 13. High-risk systems must be designed and developed such that their operation is sufficiently transparent to enable deployers to interpret the system's output and use it appropriately. For high-risk systems in sectors including credit, employment, education, essential services, and law enforcement, this is not optional guidance. It is a legal requirement.

The EU AI Act also establishes a two-tier penalty structure: prohibited AI practices can result in fines of up to 7% of global annual turnover, while violations of provider and deployer obligations (including transparency requirements) can result in fines of up to 3% of global annual turnover. These are not per-incident fines. They are calibrated to global revenue, which means a serious compliance failure at a large enterprise is a material financial event.

In the United States, the NIST AI Risk Management Framework 1.0 (2023) establishes explainability and interpretability as distinct dimensions of trustworthy AI, and recommends that organisations maintain the ability to understand and explain AI system behavior to the degree necessary to manage risks. While the NIST AI RMF is voluntary guidance rather than binding regulation, it is increasingly referenced in procurement requirements, financial services guidance, and pending federal AI legislation.

The Gap Between What You Think You Have and What You Actually Have

Most enterprises that have deployed AI in regulated contexts believe they have addressed explainability and consider interpretability a future concern. The following questions reveal where the gap typically sits.

Can you explain not just what features your model weighted in a specific prediction, but what concepts and relationships the model has encoded in its parameters? For a simple gradient-boosted tree or logistic regression model, the answer is likely yes. For a fine-tuned large language model handling the same task, the answer is almost certainly no.

Can you verify that a concept you intended to exclude from your model's decision process is actually absent? For example, if your credit model was trained without demographic attributes, can you confirm that the model has not learned proxy representations of those attributes from other features? Post-hoc attribution methods can tell you what features were most predictive of a specific output but cannot confirm the absence of a latent representation that is not directly expressed in the feature space.

Can you explain why your model behaves consistently on in-distribution inputs but differently on inputs slightly outside its training distribution? SHAP and LIME both rely on locally linear approximations and do not transfer well to distribution shift scenarios. A mechanistic account of what the model has learned would allow you to reason about out-of-distribution behavior from first principles. A post-hoc attribution approach does not.

Can you audit the model's internal representations after a fine-tuning or instruction-tuning step to confirm that the intended changes were made and no unintended changes occurred? Current practice for most enterprises is to evaluate outputs on held-out test sets. This tests behavior but does not test mechanism. A model can produce acceptable outputs on a test set while having encoded representations that will produce unacceptable outputs on inputs not covered by the test set.

What Good Looks Like at Different Maturity Levels

Interpretability is not a binary property. Organisations operate at different levels of interpretability maturity, and the realistic goal for most enterprises in the near term is not full mechanistic transparency but a meaningful upgrade from the current baseline.

At the foundational level, good practice means having a clear taxonomy of your AI systems by risk tier, with explicit documentation of what interpretability evidence is required at each tier. Low-risk systems may require only standard output logging and basic attribution. High-risk systems should require a documented audit trail of model behavior on representative inputs, evidence that the model's decision-relevant features are consistent with the intended design, and a clear process for investigating unexpected outputs.

At the intermediate level, good practice means integrating mechanistic interpretability tools, particularly sparse autoencoders for language models, into your model evaluation pipeline. This does not require building interpretability research capabilities in-house. It requires understanding what the available tools can and cannot tell you, and building the organisational capability to interpret their outputs and incorporate them into governance decisions.

At the advanced level, good practice means the ability to provide a mechanistic account of model behavior for any high-stakes decision, to detect changes in internal representations across model versions, and to verify that safety-relevant properties are present at the level of model internals rather than just at the level of output behavior.

The Competitive Dimension

Most enterprise AI governance conversations frame interpretability as a compliance cost. That framing misses the competitive dimension. Organisations that develop genuine interpretability capabilities will be able to do things their competitors cannot.

They can deploy AI in higher-stakes contexts because they can provide the level of assurance those contexts require. They can adapt models more confidently because they understand what changes when a model is retrained or fine-tuned. They can detect model degradation earlier because they are monitoring mechanisms, not just outputs. They can defend their AI decisions under regulatory scrutiny because they have evidence that goes beyond post-hoc attribution.

The organisations that will benefit most from the current wave of interpretability research are not the AI labs doing the research. They are the enterprises that invest early in understanding and applying the findings. The research is becoming usable faster than most enterprise AI teams realise, and the window for building a genuine capability advantage is open now.

The next post in this series covers the science: what mechanistic interpretability researchers have actually found inside language models, explained for a technical but non-specialist audience. The findings are more surprising, and more practically relevant, than most enterprise AI coverage suggests.

The Three Questions Every Governance Leader Should Be Able to Answer

The practical test for interpretability readiness is not whether you have a stated policy. It is whether you can answer three specific questions about each high-risk AI system you operate, with evidence rather than assertion.

First: what are the conditions under which this system will produce outputs that are wrong, biased, or harmful, and how do you know? This is the failure mode question. Most enterprises can answer it at the level of "we tested on a held-out validation set and accuracy was X." That is not a failure mode characterisation. A failure mode characterisation identifies specific input types, contexts, and distributions that are associated with output degradation, and provides evidence for why those failure modes exist, not just that they exist.

Second: if this system's behavior changes after a model update, fine-tuning, or prompt change, how will you know what changed and why? Output monitoring detects that something changed. Mechanistic evidence tells you what changed. The governance difference is significant: knowing that outputs shifted on a particular category of input is the beginning of an investigation. Understanding what changed in the model's internal representations tells you whether the change is contained, whether it is likely to produce other downstream effects, and whether it is the intended result of the intervention.

Third: if a regulator, an auditor, or an adverse party in litigation asks you to explain why this system produced a specific output in a specific case, what will you say? Post-hoc attribution tools can provide a partial answer. A mechanistic account provides a qualitatively stronger one. For enterprises operating in regulated industries, the difference between these two answers is worth careful consideration before deployment, not after a harm event.

The organisations that can answer these three questions with genuine evidence rather than approximations are the ones that interpretability capability is built for. The next post in this series covers the underlying science, making the research findings accessible to the technical leaders who need to understand what is actually being studied and why it matters for their governance programs.