What You're Actually Measuring (and Why It's Wrong)

Prompts sent. Tokens generated. Seats licensed. Pilot completions. These are the metrics most organizations track for AI. None of them are ROI metrics.

The measurement problem in enterprise AI is not that data is unavailable. Vendor dashboards produce large volumes of data. The problem is that the available data answers the wrong question. It answers "are people using this?" rather than "is this producing a return?"

Usage and return are related but not equivalent. High usage can coexist with zero return if the time savings are not redeployed, if errors introduced by AI offset errors prevented, or if the processes being accelerated were not limiting factors in the first place. CFOs who push back on AI ROI claims are often doing so because they recognize this distinction and see it not being made.

The Vanity Metric Trap

Vanity metrics are measurements that feel meaningful because they go up and to the right but do not connect to business outcomes. In enterprise AI, the vanity metric trap is almost universal: adoption metrics got embedded in AI dashboards, AI leadership teams report them upward, and boards receive numbers that look like progress but carry no information about return.

The trap is partly a vendor artifact. AI vendors are incentivized to show usage because usage correlates with license renewal. They build dashboards that highlight prompts sent, active users, and features accessed. Organizations that adopted vendor reporting as their AI measurement approach ended up measuring vendor-defined success, which is adoption, not return.

The trap is also a speed artifact. Building real measurement infrastructure requires upfront work: defining the use case, establishing baselines, specifying the value hypothesis, designing attribution. In a deployment environment where speed was the priority, that upfront work was skipped. The vanity metrics that were already available from vendors filled the reporting vacuum.

McKinsey's State of AI in 2024 found that organizations identifying as AI high performers were significantly more likely to have formal measurement processes that connected AI activity to business outcomes. The measurement discipline that distinguishes high performers is not about tracking more metrics. It is about tracking different ones, specifically those that trace from AI activity through mechanism to business outcome.

Adoption is not return. An organization where every employee uses AI daily but where no one can specify what outcome that usage produces has not demonstrated ROI. It has demonstrated adoption.

What CFOs Actually Want to See

CFOs are not hostile to AI. They are hostile to claims that cannot be audited, traced to specific outcomes, or compared against alternatives. The specific objections that recur in CFO conversations about AI ROI share a common structure: there is a claim, there is no mechanism specified for how the claim was generated, and there is no way to verify it independently.

Four categories of measurement satisfy CFO scrutiny when done with appropriate rigor:

The Attribution Problem

Even organizations that track the right categories of metrics run into attribution problems. AI rarely changes only one thing. An AI deployment typically coincides with: process redesign to accommodate the AI, training programs that change how workers approach tasks, personnel changes as teams restructure, and in some cases concurrent technology changes in adjacent systems.

When outcomes improve after an AI deployment, attributing the improvement specifically to AI requires eliminating or accounting for these confounding factors. The gold standard is a randomized controlled trial: deploy AI with a randomly selected group, withhold it from a comparable control group, and compare outcomes. This is feasible in some enterprise contexts and challenging in others.

Where controlled trials are not feasible, the alternatives are less rigorous but still useful: documenting what else changed during the measurement period, using pre/post measurement within the same team or process, and being explicit about the upper and lower bounds of attributable impact. A CFO who sees this kind of careful reasoning about attribution is far more likely to credit the claimed value than one who sees an unqualified productivity claim.

The Anthropic Economic Index (2025) documents significant within-occupation variation in AI use even among occupations with high average AI adoption. This finding reinforces the attribution challenge: even in organizations with high AI adoption, individual workers vary substantially in how much they use AI and how effectively. Attributing outcomes to AI at the team or organization level requires controlling for this variation, which requires individual-level measurement that most organizations have not implemented.

The Baseline Problem

The most common reason AI ROI cannot be measured is not the absence of a measurement framework. It is the absence of a baseline. Organizations that deployed AI without documenting the pre-deployment state of the processes AI touches cannot calculate the delta. The pre-deployment state must be inferred from memory, from proxy metrics, or from industry benchmarks, none of which produce defensible numbers.

The practical implication is that any AI use case being considered for future deployment should trigger a mandatory baseline measurement phase. Before a single prompt is sent in a production context, the current state of the relevant process should be documented: labor hours, error rates, cycle times, cost per unit, and any other metric the value hypothesis depends on. This documentation is the foundation of every credible ROI claim that follows.

McKinsey's Superagency in the Workplace (2025) emphasizes that AI value realization patterns differ significantly between organizations that invested in measurement infrastructure before deployment and those that did not. The implication is clear: the measurement infrastructure is not a reporting convenience. It is the mechanism by which organizations identify which AI investments are working, iterate on those that are not, and build the institutional knowledge that makes future investments more productive.

How to Build Toward Real Metrics

The Productivity Research and Its Limits

The Noy and Zhang controlled experiment (Science, 2023) is the most rigorously designed study of generative AI's effect on knowledge worker productivity. The finding that workers with access to generative AI completed a specific writing task 37% faster is real, peer-reviewed, and provides a useful anchor for productivity discussions. It is also frequently misapplied in enterprise contexts.

The study examined a specific type of writing task performed under controlled conditions. The productivity gain was largest for workers who were less skilled at the task without AI assistance. The implication is not that any knowledge work will speed up by 37% when AI is introduced. It is that AI can meaningfully accelerate specific tasks, particularly where the current constraint is skill or experience rather than effort or information access.

Translating the research finding into an enterprise ROI claim requires three additional steps that most organizations skip. First, identify whether the specific tasks in your use case are sufficiently similar to the studied task to expect a comparable effect. Second, estimate the skill distribution of workers performing those tasks, since the productivity benefit varied substantially by worker skill level in the research. Third, specify how the time saved will be captured as productive output rather than dissipating into less-visible forms of slack. None of these steps are technically difficult. All of them require more discipline than simply citing the 37% figure in a business case.

Why the Measurement Transition Is Hard

Moving from vanity metrics to ROI metrics is harder than it sounds, and not because the concepts are difficult to understand. The difficulty is organizational. Vanity metrics are widely available, easy to produce, and easy to present. ROI metrics require upfront investment in baselines and value hypotheses, ongoing investment in measurement infrastructure, and the willingness to present numbers that may show the AI program is underperforming expectations.

That last point deserves emphasis. One reason measurement infrastructure is absent in many enterprise AI programs is that it creates accountability for outcomes, not just activity. A measurement system that shows whether value hypotheses are materializing will, in some use cases, show that they are not. That information is genuinely valuable for portfolio management, but it is uncomfortable to produce and present. The absence of measurement is not always an oversight. Sometimes it reflects an implicit choice to avoid accountability for outcomes.

The organizations that make the transition to outcome-based measurement typically do so as a result of external pressure: a CFO who will not approve the next budget cycle without credible ROI numbers, a board that explicitly asks for evidence of return, or an audit process that requires financial accountability. The external pressure creates the internal motivation to invest in the measurement infrastructure that should have existed from the beginning.

Building that infrastructure under pressure is harder than building it in advance. The baselines that should have been established before deployment are gone. The value hypotheses that should have been written before deployment must be reverse-engineered from imperfect data. The attribution mechanisms that should have been designed in advance must now contend with months of unmeasured concurrent changes. Retroactive measurement infrastructure is possible but weaker than prospective measurement infrastructure by design.

The Anthropic Economic Index (2025) found that even in occupations with high AI adoption rates, the within-occupation variation in AI use intensity was substantial. This finding is relevant to measurement transition: any credible ROI measurement framework must account for variation in how individual workers use AI, not just whether tools are deployed. Aggregate adoption metrics mask this variation. Use-case-level measurement with individual tracking, where feasible and appropriate, produces a more accurate picture of actual return and a stronger foundation for the portfolio management decisions that follow.