The 18-Month Interpretability Roadmap: What to Build, Buy, or Wait On
Five posts in this series have established the science, the findings, the regulatory requirements, and the honest assessment of what is usable today. This final post turns to the question that matters most for people with decisions to make: what do you actually do about this, in what order, over the next 18 months?
The answer is not to become an interpretability research team. It is to develop targeted interpretability capabilities matched to the specific governance obligations and competitive pressures of your industry, built in a sequence that produces defensible evidence at each stage and positions your organisation to absorb the advances that are coming in the next two years.
This roadmap is structured around three phases. The first phase builds the foundation that every serious enterprise AI program needs regardless of interpretability investment: behavioral monitoring, documentation architecture, and the ability to identify which behaviors in your deployed systems carry the most governance risk. The second phase adds activation-level capability and the tooling to conduct targeted mechanistic investigations when the governance risk justifies it. The third phase positions your organisation to adopt emerging capabilities as they mature, and to use interpretability capability as a differentiator in regulated markets.
The Strategic Frame
Before the roadmap, the frame. Interpretability is not primarily a research capability or a compliance checkbox. In regulated industries, it is becoming a competitive variable. The organisations that can demonstrate the highest quality of behavioral understanding for their AI systems will be able to deploy more aggressively in high-stakes contexts, because they have the evidence base to satisfy regulators, reassure counterparties, and govern their systems with precision rather than with blunt behavioral constraints.
The inverse is also true. Organisations that are caught in the gap between the EU AI Act's documentation obligations and their actual ability to characterise their systems' behavior will face a choice between constraint and exposure. Constraint means limiting AI deployment to contexts where behavioral characterisation is easy but value is modest. Exposure means deploying in high-value contexts without adequate governance evidence, which is viable until it is not.
The organisations building interpretability capability now are not primarily making compliance investments. They are making competitive ones. They are acquiring the ability to deploy where others cannot, and to demonstrate to the customers, regulators, and counterparties who increasingly ask hard questions about AI systems that their answers are grounded in evidence rather than in confidence.
Interpretability capability is not the ability to explain everything. It is the ability to answer the specific questions that your highest-stakes deployments will face from regulators, auditors, and counterparties.
Phase One: Foundations (Months 1 to 6)
The first phase is not about interpretability tooling. It is about establishing the preconditions that make interpretability investments useful. You cannot build a monitoring layer on top of a system whose baseline behavior is not characterised. You cannot document failure modes you have not identified. You cannot conduct mechanistic investigations of behaviors you have not prioritised.
The concrete deliverables for this phase are:
- An inventory of every AI system in your organisation that falls under the EU AI Act's high-risk classification, with an assigned risk tier and a named owner responsible for governance documentation.
- For each high-risk system: a behavioral characterisation document that covers the input distribution the system is intended to handle, the performance characteristics on that distribution, and the failure modes that have been identified through systematic testing.
- A documentation architecture that can hold and version this characterisation over time, linked to the system versions that produced each characterisation and the testing evidence that supports it.
- A baseline output monitoring pipeline for each high-risk system, tracking distributional properties of outputs against a characterised baseline, with alert thresholds set based on the consequence severity of distributional drift for that specific application.
This phase will surface gaps quickly. Most organisations that undertake this inventory discover that their behavioral characterisation is thinner than they believed. Documentation that seemed adequate when it was produced looks incomplete when measured against the specific questions that Article 13 asks. The gaps are normal; identifying them is the point of the phase.
The talent required for this phase is governance and product talent, not research talent. The primary technical skill is systematic behavioral testing, which is available in most mature software engineering teams with the right prioritisation. The primary organisational skill is connecting AI governance obligations to the people who operate the relevant systems, which requires executive sponsorship and clear accountability structures.
Phase Two: Activation Capability (Months 6 to 12)
Phase two adds activation-level capability to the behavioral monitoring established in phase one. The goal is twofold: richer ongoing monitoring that detects behavioral drift earlier and more specifically than output monitoring alone, and the ability to conduct targeted mechanistic investigations when a behavioral concern is identified that warrants deeper analysis.
The concrete deliverables for this phase are:
- Activation extraction infrastructure for your highest-risk deployed systems. For systems using open-weight models deployed on enterprise infrastructure, this means integration with a library such as TransformerLens or equivalent tooling. For systems using commercial API models, this means establishing which providers expose activation-level information and whether activation-based monitoring is feasible for your architecture.
- Linear probe training on labeled examples of high-risk behaviors for each system in scope. The probes should cover the failure modes characterised in phase one, enabling real-time scoring of inputs against the likelihood of failure before outputs are generated.
- Activation anomaly detection tuned to each system's baseline activation distribution, with alert thresholds calibrated to the governance risk of the system. Anomalous activations should trigger queuing for human review, not automatic output suppression.
- A protocol for targeted mechanistic investigation when a behavioral concern is elevated by the monitoring layer. The protocol specifies the conditions that justify a mechanistic investigation, the methods to be applied (activation patching for causal questions, SAE analysis for feature characterisation), the expertise required, and the documentation format for investigation findings.
This phase requires a different talent profile than phase one. At least one member of the team needs sufficient interpretability expertise to design activation monitoring correctly, validate probe outputs, and conduct or oversee mechanistic investigations. This person does not need to be a mechanistic interpretability researcher, but they need to understand the difference between correlational and causal evidence, the known limitations of attention-based approaches, and the experimental design requirements for activation patching to be informative.
For most enterprise teams, the fastest path to this capability is not hiring: it is a targeted training investment for an existing technical team member, combined with engagement with the open-source interpretability community (EleutherAI, Anthropic's interpretability research output, the TransformerLens community) and with specialist external advisors for the highest-stakes investigations.
Phase Three: Differentiation (Months 12 to 18)
By month twelve, organisations that have executed phases one and two have something genuinely uncommon in the market: behavioral characterisation that is documented, versioned, and linked to mechanistic evidence for the behaviors that matter most. This is a governance asset. The question in phase three is whether to treat it purely as a governance asset or to leverage it as a competitive one.
The competitive leverage comes from the ability to enter markets and relationships that require evidence of understood AI behavior. These include:
- Regulated financial services deployments where counterparties and regulators are asking for behavioral characterisation that goes beyond aggregate performance benchmarks.
- Healthcare AI deployments where procurement requires evidence of failure mode characterisation and monitoring at a level that most competitors cannot currently provide.
- Enterprise sales motions where CISOs and Chief AI Officers are asking vendors about model governance as a condition of procurement, and where the ability to provide a governance evidence package rather than a generic compliance statement differentiates the sale.
- Partnership and licensing contexts where the ability to document model behavior at the mechanistic level supports IP protection and contract negotiations about model behavior guarantees.
The scaling investment in phase three is in tooling and process, not primarily in additional expertise. The investigation protocols established in phase two should be streamlined into repeatable workflows. The documentation architecture should be extended to support external disclosure in the formats that specific regulatory contexts and counterparties require. The monitoring infrastructure should be audited for coverage and extended to newly deployed systems as the AI portfolio grows.
Phase three is also when to track and evaluate the emerging capabilities that are not yet enterprise-ready. Automated circuit discovery, large-scale SAE deployment across full model families, and tooling for interpretation of multimodal models are all areas where the research is advancing faster than the enterprise tooling. Organisations with phase two capability in place are positioned to absorb these advances as they mature, rather than starting from the foundation when they become practically relevant.
Build, Buy, or Wait: The Decision Framework
- Behavioral characterisation documentation for your specific applications
- Activation monitoring pipelines tuned to your deployment context
- Linear probes for failure modes specific to your use cases
- Internal investigation protocols and documentation templates
- Training for existing technical team on interpretability methods
- Specialist interpretability expertise for highest-stakes investigations
- Behavioral testing platforms with strong distributional analysis
- Model governance platforms with documentation and versioning (once mature)
- EU AI Act conformity assessment support for high-risk system documentation
- Automated circuit discovery tools (false positive rates too high)
- Full SAE deployment for closed-weight API models (access constraints)
- Vendor claims of "full model transparency" for frontier models
- Interpretability tooling for multimodal models (research stage)
Industry-Specific Priorities
The roadmap above is the general case. Industry context shapes which phases to accelerate and which capabilities are most valuable first.
Financial services organisations subject to EBA model risk guidance and EU AI Act obligations for credit and risk management systems should accelerate phase one documentation to align with existing model validation obligations, and should prioritise failure mode characterisation for models used in credit decisions, fraud detection, and risk scoring. The regulatory pressure in financial services is the most mature, and the documentation standards expected by supervisory authorities are correspondingly high.
Healthcare organisations deploying AI in clinical workflows face overlapping EU AI Act and Medical Device Regulation obligations. The MEDFIT-LLM evaluation framework (Rao, Jaggi, Naidu, IEEE RMKMATE 2025, DOI: 10.1109/RMKMATE64574.2025.11042816) addresses evaluation standards for large language models in medical contexts specifically. Healthcare organisations should prioritise the behavioral characterisation in phase one around the specific clinical decision types supported, with particular attention to performance across patient subpopulations and to the failure modes that carry clinical consequence.
Legal and professional services organisations using AI for document review, contract analysis, or legal reasoning face governance questions about consistency and reliability that make activation-level monitoring particularly valuable. The ability to detect when a model is operating on inputs that are substantially outside its characterised distribution is directly relevant to the duty of care obligations that professional services firms have to their clients.
The Talent Equation
One of the most common mistakes in enterprise interpretability investment is treating it as a research hiring problem. The right question is not how to hire mechanistic interpretability researchers from frontier AI labs. It is how to develop interpretability literacy in the AI engineering and governance team you already have, and how to access specialist expertise for the specific high-stakes investigations that warrant it.
Interpretability literacy, the ability to understand what activation analysis, linear probes, and causal tracing can and cannot show, is acquirable for most technical practitioners through targeted study and practice. The open-source tooling and published research are accessible. The gap is not intelligence or technical background; it is prioritisation and structured learning time. Most enterprise AI teams that have invested in this have found that it pays back quickly in the quality of behavioral characterisation they can produce, well before they reach the mechanistic investigation capabilities that require deeper specialist expertise.
Specialist expertise for targeted mechanistic investigations is a different matter. For the highest-stakes governance questions in regulated deployments, the quality of the mechanistic evidence matters, and producing that evidence well requires experience that goes beyond interpretability literacy. The practical model here is not full-time hire but engagement: external advisors, academic collaborators, or specialist consulting relationships that can be drawn on for specific investigations without carrying the overhead of permanent research headcount.
What to Tell Your Board
The framing that lands best with boards and audit committees in regulated industries is not about research or technology. It is about risk and capability.
The risk framing: under the EU AI Act and its analogues, your organisation's ability to document and defend the behavior of high-risk AI systems is a governance obligation with material financial consequences. The three-percent-of-turnover penalty tier for provider and deployer obligation failures is not a theoretical maximum; it is the exposure associated with documented governance failures in high-risk deployments. Building the capability to satisfy those obligations is not discretionary; the question is only when and how.
The capability framing: interpretability capability is becoming a differentiator in regulated markets. Organisations that can demonstrate high-quality behavioral understanding of their AI systems can deploy in contexts where competitors cannot, because they have the evidence to satisfy the questions that regulators, counterparties, and sophisticated buyers are increasingly asking. This is a strategic asset that compounds over time: the governance evidence built on one deployment reduces the cost of characterising the next, and the expertise developed for one investigation accelerates the next.
Together, these framings make the investment case: the downside of not investing is regulatory exposure and constrained deployment scope; the upside of investing is the ability to deploy more aggressively in the markets that matter most.
Closing the Series
This series began with a simple observation: interpretability is not explainability, and conflating the two produces false governance confidence that regulators, auditors, and sophisticated counterparties will not accept as AI deployment scales. It has traced the science of mechanistic interpretability from first principles through to specific findings about what is inside language models, mapped those findings to specific regulatory obligations, audited the honest state of enterprise-usable tooling, and proposed a concrete sequence for building the capabilities that matter.
The field is moving fast. The findings described in Posts 2 and 3 represent the research frontier as of mid-2026; the next 18 months will extend them, challenge some of them, and produce tooling that makes today's research instruments into tomorrow's standard practice. The organisations that are investing now are not trying to solve interpretability completely. They are building the foundation and the expertise to absorb advances as they arrive, rather than starting from zero when those advances become regulatory requirements.
The question is not whether your organisation needs interpretability capability. The question is whether you build it on your terms, with your timeline, or on the regulator's timeline, at the regulator's pace. The answer to that question is a choice you can make in the next six months.
Directional roadmap. Timelines vary by organisation size, existing governance maturity, and deployment risk profile.
- Post 1: What Interpretability Actually Means, and Why Leaders Need to Care Now
- Post 2: The Science of Interpretability, Explained Without Jargon
- Post 3: What Interpretability Research Has Actually Found Inside Neural Networks
- Post 4: Interpretability and Regulation: What Compliance Actually Requires
- Post 5: What Is Actually Usable Today: An Honest Assessment
- Post 6: The 18-Month Roadmap: What to Build, Buy, or Wait On
Ready to build interpretability capability in your organisation?
I work with enterprise leadership and AI teams to design interpretability programs matched to their risk profile, regulatory obligations, and competitive priorities. If you are ready to move from awareness to action, let us talk.
Schedule a conversation →References
- EU AI Act, Regulation (EU) 2024/1689. Article 9 (Risk management), Article 11 (Technical documentation), Article 13 (Transparency), Article 17 (Quality management). eur-lex.europa.eu
- NIST AI Risk Management Framework 1.0. National Institute of Standards and Technology, January 2023. doi.org/10.6028/NIST.AI.100-1
- Nanda, N., and Bloom, J. TransformerLens. 2022. github.com/TransformerLensOrg/TransformerLens
- Bricken, T., et al. Towards Monosemanticity: Decomposing Language Models With Dictionary Learning. Anthropic, 2023. arXiv:2309.08600
- Meng, K., et al. Locating and Editing Factual Associations in GPT. 2022. arXiv:2202.05262
- Chen, L., Zaharia, M., and Zou, J. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. 2023. arXiv:2310.11409
- Rao, A. K. G., Jaggi, A., and Naidu, S. MEDFIT-LLM: A Comprehensive Evaluation Framework for Large Language Models in Medical Fitness Assessment. IEEE RMKMATE 2025. DOI: 10.1109/RMKMATE64574.2025.11042816
- European Banking Authority. Guidelines on internal governance (EBA/GL/2017/11). eba.europa.eu
- Conmy, A., et al. Towards Automated Circuit Discovery for Mechanistic Interpretability. NeurIPS 2023. arXiv:2304.14997