Interpretability and Regulation: What Compliance Actually Requires

The EU AI Act does not require you to open the neural network. What it requires is something that many organisations find harder: documented, defensible evidence that you understand how your system behaves, where it fails, and how those failures are detected. The science of interpretability is now directly relevant to whether you can produce that evidence.

Post 4 of this series focuses on the regulatory dimension. Posts 2 and 3 established what interpretability research actually finds inside neural networks: circuits, superposition, polysemanticity, and the structural organisation of model knowledge. This post asks: when regulators and auditors arrive at your door, what do they actually want to see? And how does the emerging science of model internals change what is possible to show them?

Two Frameworks, One Underlying Question

Two frameworks dominate enterprise AI governance: the EU AI Act (Regulation 2024/1689), which applies to AI systems deployed or affecting persons in the European Union, and the NIST AI Risk Management Framework 1.0 (NIST AI RMF), which is voluntary in the United States but has become a de facto reference standard for enterprise AI governance programs. They approach the same underlying question from different directions.

The EU AI Act is prescriptive. It establishes risk tiers, assigns obligations to providers and deployers, and attaches penalties to non-compliance. The NIST AI RMF is structured around organisational practice: Map, Measure, Manage, Govern. It does not specify what you must do so much as it structures the thinking required to decide what you should do. Together, they represent the current outer boundary of what enterprise AI governance requires.

Both frameworks, when read carefully, are asking a version of the same question: can you demonstrate that you understand what your system does, to whom, under what conditions, and with what confidence in that understanding?

EU AI Act Article 13: Transparency and Traceability

Article 13 of the EU AI Act establishes transparency obligations for high-risk AI systems. The article requires that high-risk AI systems be designed and developed in such a way as to ensure that their operation is sufficiently transparent to enable deployers to interpret the system's output and use it appropriately. It mandates that providers supply deployers with an instructions-for-use document that covers the system's purpose, intended use, performance characteristics, known limitations, and the human oversight measures required for appropriate use.

The operative word in Article 13 is "sufficiently." The regulation does not define a universal technical threshold. It requires that the level of transparency be adequate for the specific deployment context, assessed against what a reasonably informed deployer needs to make appropriate use of the system's outputs. This is a judgment standard, not a technical checklist. What counts as sufficient transparency for a document summarisation tool differs substantially from what is required for a tool that recommends credit limits or triage priority.

Article 13 transparency obligations apply to the deployer, not just the original model developer. If your organisation is deploying a third-party foundation model in a high-risk application, you are responsible for understanding and documenting its behavior in your specific context. The provider's general technical documentation does not discharge your obligation to characterise performance in your deployment conditions.

What "Sufficient Transparency" Means Operationally

In practice, satisfying Article 13 requires answers to a concrete set of questions about your deployed system. The questions are not primarily about the model architecture. They are about documented knowledge of behavior:

• What are the accuracy characteristics of this system on the specific input distribution it will encounter in deployment, not just on the benchmark used to evaluate it during development?
• What are the failure modes, and how have they been characterised? Where does the system produce outputs with high confidence that are incorrect?
• Are there identifiable subpopulations or input types on which performance degrades relative to the overall distribution?
• What human oversight is required, and what is the human oversight process designed to catch? Does the oversight process assume competence in identifying the failure modes that have been characterised?
• How will anomalous behavior in deployment be detected? What monitoring is in place?

Interpretability tools are relevant to several of these questions in ways that post-hoc explainability is not. If you have used activation patching or causal tracing (as described in Post 3) to identify which internal circuits are responsible for a class of outputs, you have a mechanistic account of that behavior that is substantially more defensible than a SHAP value generated after the fact. If you have used sparse autoencoder analysis to characterise the feature space the model is representing in a domain relevant to your application, you have structural evidence about what the model knows. These are answers, not approximations.

Article 9: Risk Management as an Ongoing Process

Article 9 of the EU AI Act establishes risk management obligations that run throughout the lifecycle of a high-risk AI system. The risk management system must be a continuous iterative process run throughout the entire lifecycle of a high-risk AI system. It must include evaluation of known and reasonably foreseeable risks, testing to verify that risks are addressed, and post-market monitoring once the system is deployed.

The post-market monitoring obligation is significant for interpretability. It requires that providers establish and document systems for collecting and reviewing data about the performance of high-risk AI systems in the real world. This is not a one-time certification. It is an ongoing obligation to know how your system is behaving and to respond when behavior diverges from what was characterised during the conformity assessment.

This ongoing monitoring obligation creates a direct use case for interpretability-adjacent tooling: not full mechanistic reverse-engineering of the model, but systematic monitoring of internal representations and activation patterns to detect drift in model behavior before it manifests as downstream harm. Several enterprise AI teams are beginning to build this capability, and it represents a genuine competitive differentiation in regulated industries.

Penalty Structure: What Is Actually at Stake

The EU AI Act establishes two tiers of maximum penalties. For violations involving prohibited AI practices under Article 5 (systems that manipulate persons through subliminal techniques, exploit vulnerabilities, enable social scoring by public authorities, and specific biometric categorisation uses), the maximum fine is the higher of thirty-five million euros or seven percent of total worldwide annual turnover.

For violations involving obligations on providers and deployers of high-risk systems, including transparency and traceability obligations under Articles 9 through 15, the maximum fine is the higher of fifteen million euros or three percent of total worldwide annual turnover. This three-percent tier is where most enterprise AI deployments live in terms of compliance risk. It is not a theoretical ceiling. It is the exposure associated with documented failures of governance practice in high-risk applications.

The practical implication is that compliance risk under the EU AI Act is not concentrated in exotic edge cases. The obligations under Articles 9 through 15 apply to any organisation deploying high-risk AI in EU-affecting contexts, and the three-percent penalty tier applies to failures of documentation, transparency, and ongoing monitoring. These are governance failures, not malicious-use failures. They are the kind of failure that happens when an organisation deploys AI faster than its governance processes mature.

NIST AI RMF: The Governance Architecture

The NIST AI Risk Management Framework, released in January 2023, does not impose legal obligations. It has nonetheless become the reference architecture for enterprise AI governance in organisations that take AI risk seriously. It organises AI risk management into four functions: Map (identify context and risks), Measure (analyse and assess risk), Manage (prioritise and implement responses), and Govern (establish policies, processes, and accountability). The Govern function is explicitly cross-cutting: it describes the organisational and cultural conditions required for the other three functions to work.

The NIST AI RMF's treatment of explainability and interpretability is instructive. The framework distinguishes between explainability (the degree to which a process, system, or its outcomes can be communicated in human-understandable terms) and interpretability (the degree to which humans can consistently predict a model's outputs). These are defined as complementary, not equivalent. The framework recognises that different deployment contexts require different levels and types of each.

More importantly, the NIST AI RMF treats transparency not as a binary property but as a risk-calibrated spectrum. The Measure function includes explicit guidance on evaluating AI system trustworthiness across dimensions that include explainability, interpretability, reliability, safety, and security. Each dimension is to be assessed in proportion to the risk profile of the specific deployment, not according to a universal standard.

What the NIST Framework Requires in Practice

For organisations using the NIST AI RMF as their governance foundation, the interpretability question surfaces most acutely in the Measure function. The framework expects organisations to assess AI risks using quantitative, qualitative, and mixed methods, and to document the basis for their risk assessments. When the risk being assessed is behavioral, the quality of your interpretability evidence directly affects the quality of your risk assessment.

Consider a concrete example: an organisation deploying a language model for document review in a legal context. The NIST Measure function asks the organisation to characterise the reliability of the system, its failure modes, and the conditions under which outputs should not be trusted. A governance team that can point to causal tracing results showing which internal circuits drive the system's handling of specific document types has a mechanistic basis for its characterisation. A governance team that can only point to aggregate accuracy benchmarks on held-out test sets has a weaker, less defensible characterisation, particularly when the test distribution diverges from deployment conditions.

Interpretability tools do not replace governance. They provide the evidentiary basis that makes governance defensible.

The Documentation Gap

The single most common compliance failure in enterprise AI deployments is not technical. It is documentary. Organisations deploy systems with genuine understanding of their behavior during development, and then fail to produce the documentation that allows that understanding to be reviewed, audited, and updated as the system operates in the real world.

Article 11 of the EU AI Act mandates technical documentation for high-risk AI systems. The documentation must include a general description of the system, a description of its design and development process, information about training data, validation and testing procedures, the monitoring and logging design, a description of the human oversight measures, and the conformity assessment procedures. This is substantial documentation, and much of it does not exist in most current enterprise AI deployments.

Interpretability methods are relevant to documentation in a specific way: they provide durable, mechanistic accounts of model behavior that survive model version changes in a way that behavioral benchmarks do not. If you have characterised the internal circuits responsible for a class of outputs, that characterisation remains meaningful even after fine-tuning, because you can re-run the same analysis on the updated model and compare. Behavioral benchmarks, by contrast, tell you outputs have changed, but not why, which makes them harder to use as the basis for update documentation.

What Regulators Will Actually Ask

Enforcement under the EU AI Act is still maturing, and guidance from national supervisory authorities continues to develop. Based on the text of the regulation and analogous enforcement experience in financial services AI regulation (particularly under the European Banking Authority's guidelines on internal governance for model risk), there are several categories of question that compliance teams should be prepared to answer.

Question Category	What Is Being Assessed	Interpretability Relevance
Scope of intended use	Whether the system is being used within the boundaries for which it was characterised	Activation analysis can identify input types outside the characterised distribution
Failure mode documentation	Whether known failure modes were identified, documented, and disclosed to deployers	Causal tracing identifies which internal processes drive problematic output categories
Monitoring evidence	Whether post-deployment behavior is monitored and whether anomalies trigger review	Internal representation monitoring detects behavioral drift before it becomes output-level
Human oversight adequacy	Whether human reviewers are positioned to catch the failure modes that have been characterised	Circuit-level understanding informs what human reviewers need to be trained to recognise
Update governance	Whether changes to the model trigger re-characterisation and updated documentation	Mechanistic analysis enables comparison across model versions, not just output benchmarks

The Healthcare Context

Healthcare AI deployments face overlapping regulatory obligations that make interpretability evidence particularly valuable. The EU AI Act classifies AI systems used in the management of critical infrastructure and systems used for safety components of other regulated products as high-risk under Annex III. AI systems embedded in medical devices are subject to both the EU AI Act and the Medical Device Regulation (MDR), creating layered transparency obligations.

Research in the healthcare AI domain has specifically examined the question of what constitutes adequate characterisation of model behavior for clinical deployment. The MEDFIT-LLM evaluation framework (Rao, Jaggi, Naidu, IEEE RMKMATE 2025, DOI: 10.1109/RMKMATE64574.2025.11042816) addresses the challenge of evaluating large language model performance in medical contexts, identifying dimensions of evaluation that go beyond accuracy to include robustness, consistency, and behavior under distribution shift. These dimensions map directly to the documentation obligations under Article 13.

Three Governance Postures

Organisations approaching this terrain tend to adopt one of three postures, which have meaningfully different compliance and risk profiles.

The first posture is documentation compliance: satisfying the letter of transparency and documentation obligations using behavioral benchmarks, test set performance, and post-hoc explainability tools. This is the minimum viable compliance approach. It satisfies the current interpretation of Article 13 for lower-risk high-risk applications, but it creates a brittle documentation record that cannot answer deeper questions about model behavior if those questions are raised by an auditor or a harm event.

The second posture is behavioral characterisation: investing in systematic behavioral testing that goes beyond held-out benchmarks to include adversarial inputs, out-of-distribution examples, and subpopulation analysis. This is substantially better than documentation compliance alone. It produces evidence about failure modes that is genuinely useful for human oversight design and post-market monitoring. It still does not provide mechanistic accounts of why the failure modes exist.

The third posture is mechanistic evidence: integrating interpretability methods into the development and governance process to produce mechanistic accounts of specific high-risk behaviors. This is not full reverse-engineering of the model. It is targeted: identify the specific output categories that carry the most governance risk, apply causal tracing and activation analysis to those categories, document the results. This produces governance evidence that is qualitatively different from what behavioral testing can produce, and it is more defensible under regulatory scrutiny because it answers the question of what the model is doing, not just what outputs it produces.

Positioning for 2026 and Beyond

The EU AI Act's high-risk system obligations are fully applicable from August 2026 under the regulation's phased implementation timeline. Organisations that have not yet begun conformity assessment for systems that fall under Annex III are already behind. Organisations that are beginning now and are relying on documentation compliance as their posture are taking on risk that is manageable but not negligible, particularly in sectors where national supervisory authorities have signalled active enforcement interest.

The more important strategic point is that regulatory requirements in this area will tighten over time, not loosen. The EU AI Act establishes a floor, not a ceiling. The interpretability methods described in Posts 2 and 3 of this series are moving from research into tooling rapidly. Organisations that invest now in the capability to apply these methods to their deployed systems will be positioned to satisfy transparency obligations that are currently aspirational before they become mandatory.

The next post in this series turns from what regulation requires to what is actually available today: an honest audit of which interpretability tools are ready for enterprise use, which are still primarily research instruments, and what distinguishes genuine progress from vendor claims.