The most common mistake enterprise leadership makes when beginning an AI program is assuming the infrastructure question comes first. It does not. But if you are running mainframes, COBOL batch systems, or on-premises ERP from 2005, the infrastructure question is never far away. This post is a decision framework for what you actually do.
The question is not whether your legacy infrastructure is "AI-ready." It almost certainly is not, in the narrow sense. The question is: which AI use cases can you unlock on existing infrastructure right now, which require selective modernization, and which genuinely require a platform rebuild before they become viable? Those are three different answers requiring three different investment theses.
When a CTO presents the AI infrastructure question to the board, it typically gets framed as a binary: "We need to modernize our infrastructure before we can do AI." This framing is wrong in two directions simultaneously. It overstates the barrier to starting and understates the complexity of modernization itself.
The result is a governance deadlock. AI initiatives stall waiting for infrastructure projects that take three to five years. Infrastructure projects expand in scope to justify their budgets. And the organization falls further behind competitors who moved on use cases that did not require replatforming at all.
IBM's z16 mainframe, announced April 2022, includes the Telum processor with an on-chip AI accelerator capable of processing 300 billion inference requests per day across the system (IBM Newsroom, April 5, 2022). The z16 can run AI inference co-located with transaction processing, at the data source, without data movement. Many "legacy" systems already have more AI-native capability than their operators realize.
The better framing: which AI value can we extract from existing infrastructure in the next 90 days, and what infrastructure investments unlock the next tier of value? That is a roadmap question, not a binary modernization question.
Legacy is not a synonym for old. It means infrastructure that was not designed with API-first data access, containerized workloads, or real-time data streaming in mind. Legacy infrastructure typically shares four characteristics:
These four constraints map directly to four questions in the decision framework. Not all AI use cases require all four to be resolved.
Every organization running legacy infrastructure has three viable paths to AI value. The right path depends on the use case, the data accessibility level, and the risk tolerance for infrastructure change. These paths are not mutually exclusive: most enterprises run all three simultaneously, applied to different use case categories.
Add AI inference capabilities via API above existing systems. No infrastructure changes required. Works when existing data exports or scheduled extracts are sufficient for the use case.
Time to first value: 6 to 12 weeks
Infrastructure change: None required
Ceiling: Limited by data freshness and accessibility
Expose specific data assets or subsystems via APIs or streaming pipelines, without replatforming the core. Adds a data access layer rather than replacing the system of record.
Time to first value: 3 to 9 months
Infrastructure change: Data layer only
Ceiling: Near-real-time, bounded by subsystem scope
Migrate core workloads to modern infrastructure. Required for use cases demanding real-time inference co-located with transactions, or full data model control. Highest investment, highest ceiling.
Time to first value: 18 to 36+ months
Infrastructure change: Core systems
Ceiling: Full AI-native architecture
The fastest way to demonstrate AI value in a legacy environment is to treat existing systems as a read-only data source and build inference capabilities above them. This is not a compromise: it is the correct architecture for a large category of enterprise AI use cases.
Document intelligence is the canonical Path A use case. If your accounts payable team processes invoices, contracts, or compliance documents, an AI layer can extract structured data, flag anomalies, and route decisions without touching the ERP system at all. The AI reads the document. The result is written back into the existing workflow. The mainframe never knows there was an AI in the loop.
The same logic applies to employee-facing tools. A natural language interface that allows employees to query HR policy, retrieve procedural guidance, or navigate internal knowledge does not require the underlying systems to change. Retrieval-augmented generation (RAG) architectures can index existing documentation repositories and serve AI-generated answers. The system of record stays unchanged.
Before committing to Path A for a use case, confirm: (1) the data the AI needs is available in existing exports or readable file systems; (2) the latency requirement is greater than your current extract cycle (batch analytics, document processing, and periodic reporting all qualify); (3) the result of AI inference does not need to update the source system in real time. If all three are true, Path A is viable without infrastructure change.
The constraint on Path A is data freshness. If a use case requires AI decisions based on data that changes faster than your extract cycle, Path A breaks. Real-time fraud detection, live customer personalization, and dynamic pricing all require data fresher than a nightly batch. Those use cases belong to Path B or C.
Selective modernization does not mean rewriting systems. It means building a data access layer that exposes specific subsystems to modern tooling without changing the system of record. The mainframe still runs payroll. The ERP still manages inventory. But a streaming extract from the transaction log, or a change data capture feed from a specific table, makes that data available to AI inference in near-real time.
The IBM z16 supports this architecture natively. The Telum processor includes an on-chip AI accelerator specifically designed to run fraud scoring inference co-located with transaction processing. Banks using z16 can score transactions for fraud at the point of processing, without data movement, at the sub-millisecond latency that card authorization requires. This is not future capability: it is in production at multiple institutions today.
For organizations not on z16, change data capture tools can create a streaming data layer above legacy databases. This approach exposes the data without migrating it. The AI inference environment is modern; the system of record is not. The boundary between them is the streaming pipeline.
Change data capture (CDC) reads the transaction log of a database and streams row-level changes in real time to a target system. CDC does not require changes to the source database schema or application code. It works with most legacy relational databases. The result is a real-time data feed from a system that was not designed to produce one.
The Path B investment is primarily in the data layer: CDC tooling, streaming infrastructure (Apache Kafka or equivalent), and schema mapping from the legacy format to the format the AI system expects. This is a bounded engineering project, typically three to nine months for a production-quality implementation covering a specific data domain.
There are AI use cases that genuinely require replatforming. The decision criteria are narrow but real. Replatforming is the correct answer when three conditions are simultaneously true:
If all three conditions are not met, Path B is almost always the better choice. Replatforming that is not driven by a specific use case requirement tends to expand in scope, exceed budget, and delay value delivery.
Large-scale infrastructure migrations routinely extend beyond original timelines and budgets. The primary cost driver is not the technology: it is the institutional knowledge embedded in legacy systems that must be reverse-engineered before it can be rebuilt. Every month a migration extends is a month where AI value is deferred and the organization remains exposed to the risks the migration was supposed to eliminate. Model the downside before committing.
Most organizations that have attempted enterprise AI programs report the same finding: the infrastructure was not the binding constraint. The data was. Specifically: the data exists in the legacy system, but it is not accessible in a form the AI can use.
Three data problems account for the majority of AI delays in legacy environments:
Legacy systems often have schemas that were designed for the application's internal logic, not for external consumption. Field names are abbreviated codes. Relationships are enforced by application code, not foreign keys. Understanding what the data means requires the same institutional knowledge as understanding the business process. Before an AI system can use the data, someone must document it.
Legacy data accumulates quality debt over decades. Duplicate records, inconsistent formats, orphaned rows, and fields that meant something different ten years ago are common. AI models trained or fine-tuned on uncleaned legacy data inherit that noise and amplify it in their outputs.
In many legacy systems, personally identifiable information (PII) is not isolated. It is embedded throughout the data model because the original designers did not anticipate a regulatory environment requiring data minimization. Exposing legacy data to an AI system without a PII review creates compliance risk.
The 90-day plan is not about deploying AI. It is about generating the evidence base that drives the right infrastructure investment decisions while demonstrating early value on Path A use cases.
The ROI framing that resonates with boards and CEOs is not cost reduction. It is competitive parity and risk. The question is not "what does this AI project save us?" The question is "what does our competitor do with AI that we currently cannot, and what is the cost of that gap compounding for another 18 months?"
For CIOs, the infrastructure investment case is most compelling when it is framed as a capability unlock rather than a modernization project. "This data access layer allows us to run fraud scoring at authorization time, reducing our false negative rate" is a business case. "We need to modernize our data infrastructure" is not.
For CTOs, the key metric is time-to-value for AI pilots. Every pilot that stalls on a data or infrastructure constraint is a measurement of infrastructure debt, expressed in weeks. Aggregate those delays across the portfolio and you have a quantified cost of the infrastructure problem that is defensible in a board discussion.
For organizations that do move to Path B or C architectures, inference cost optimization becomes material at scale. Research by Chen, Zaharia, and Zou (arXiv:2310.11409) demonstrated that intelligently routing queries across models of different capability and cost can reduce inference cost substantially without sacrificing output quality on most query types. This is relevant to infrastructure design: a hybrid architecture that routes simple queries to smaller, cheaper models and complex queries to larger ones requires an inference routing layer, which is a Path B concern, not a Path C one.
Three patterns consistently stall AI programs in legacy environments:
Waiting for a "data platform" before starting AI: The data platform project expands to cover everything the organization thinks it might need someday. AI pilots wait. The platform misses its deadline. The pilots are already a year late when the platform delivers. Start Path A pilots now, in parallel with any data platform work.
Treating AI infrastructure as an IT project: The business use case drives the infrastructure requirement. Infrastructure projects that are not attached to a specific AI use case with a defined ROI tend to drift in scope and lose executive support. Every infrastructure investment should be traceable to a use case and a value calculation.
Conflating cloud migration with AI readiness: Moving to the cloud does not automatically create AI-ready data access. Organizations that migrate legacy systems to cloud infrastructure (lift-and-shift) frequently discover they have the same data accessibility problems in a new location. Cloud migration and data modernization are different projects and should be scoped and funded separately.