Skip to content
Book a consult

Insights  /  Measurement

Measuring AI: KPIs tied to business outcomes

How to avoid vanity metrics and connect AI activity to performance.

Sopra Labs AI  ·   ·  8 min read
SopraLabs Hero Look like

Key takeaways

  • Licenses assigned, prompts run and self-reported hours saved describe activity. They cannot tell leadership whether revenue, cost, risk or customer experience has changed.
  • The gap between use and results is wide. In McKinsey’s 2026 survey of 1,719 participants, 37% of respondents said AI has contributed positively to their organization’s EBIT, essentially unchanged from 2025 (McKinsey).
  • Self-reported time savings are unreliable. In a 2025 randomized trial, experienced developers took 19% longer with AI tools and still believed they had been 20% faster (METR).
  • A workable KPI model is small. Choose the business outcome first, record a baseline before launch, and pair an adoption measure with an outcome measure and a quality or risk measure.

Nine thousand hours, and nothing moved

Quarterly business review, the AI slide. The numbers are large and green. Twelve hundred licenses assigned. Eighty-four percent of staff active in the last month. Nine thousand hours saved this quarter.

The CFO asks where the hours went.

It is a fair question, and she is not being difficult. Headcount is flat. The monthly close still takes nine days. The support backlog is where it was in the spring. Margin has not moved in any way she can trace to the program.

The head of IT explains that the hours come from a survey. Employees were asked how much time the tools save them each week, and the answers were multiplied out. The COO adds that people like the tools, which is true.

Nobody in the room is misleading anyone. Every number on the slide is accurate. The slide just cannot answer the question the CFO is paid to ask: is the business better off?

The slide counts activity, not results

This room is typical. In McKinsey’s 2026 global survey of 1,719 participants, 37% of respondents said AI has contributed positively to their organization’s EBIT (earnings before interest and taxes), essentially unchanged from 2025 (McKinsey). Use keeps spreading. Enterprise-level results, by respondents’ own account, have not followed.

A second survey points the same way. A February 2026 NBER working paper drew on nearly 6,000 senior executives at US, UK, German and Australian firms. It found that 69% of firms actively use AI, with nine in ten executives reporting no impact on employment or productivity over the last three years (NBER).

The reason the slide looks the way it does is practical. Activity numbers are the ones the tools hand over: licenses, active users, prompts. Outcome numbers live in finance and operations systems that belong to other people. A dashboard gets built from what is at hand.

Hours saved looks like the exception, a real outcome. It is the weakest number on the slide, because it is self-reported. In 2025 the research group METR ran a randomized controlled trial with 16 experienced open-source developers and 246 real tasks from their own projects. With AI tools allowed, they took 19% longer. Afterward, “they still believed AI had sped them up by 20%” (METR).

METR has since said developers are likely faster with AI now than in that study (METR). The lasting lesson is not about speed. Skilled people could not tell from the inside whether the tool was helping.

The UK Department for Business and Trade found a similar gap. It made 1,000 Microsoft 365 Copilot licenses available to staff for three months in late 2024. Of survey respondents, 72% were satisfied or very satisfied. The evaluators still “did not find robust evidence to suggest that time savings are leading to improved productivity,” though they note this was not a key aim. In observed tasks with only three users and three non-users, Copilot users completed Excel data analysis “more slowly and to a worse quality and accuracy than non-users” (Department for Business and Trade).

Even an accurate hour saved is not yet a result. It becomes one only when it turns into more output, lower cost or fewer errors. Until someone decides where the time goes, the week absorbs it.

Build a small KPI model

A KPI model for AI can be small. Three measures per use case, agreed before launch, will do more than a wall of charts.

Pick the outcome first. Before choosing a metric, decide which of four things the use case is meant to move: revenue, cost, risk or customer experience. Then name the one operating number that would show it, such as days to close, cost per ticket or proposal turnaround, and the executive who already owns that number. If nobody can name it, call the work an experiment and fund it as one.

Set the baseline before launch. Record the number for a full cycle before the tool arrives. Where you can, start one team later than another so there is something to compare. One early study of generative AI at work did exactly this. It followed the staggered introduction of an AI assistant, using data from 5,179 customer support agents, and measured issues resolved per hour, which rose 14% on average (NBER). That figure came from operating data, not a survey. Stanford’s 2026 AI Index notes that productivity gains are largest “in structured, measurable work where outputs are easy to monitor” (Stanford HAI).

Pair three measures. An adoption measure shows whether the intended people use the tool in the intended workflow, which is not the same as licenses assigned. An outcome measure is the operating number. A quality or risk measure, such as error rate, rework or escalations, shows whether speed is being bought with mistakes. The UK evaluation is the reminder of why the third one matters.

Keep surveys in their place. Ask employees where the tool helps and where it gets in the way, and use the answers to fix the workflow. Do not multiply them into dollars.

Set the review date and the decision. Agree in advance when the three numbers will be read and what each result means: scale, fix or stop. In McKinsey’s 2026 survey, the 6% of respondents it calls AI high performers were twice as likely as others to report “defined processes to measure the impact of those initiatives” (McKinsey). An earlier McKinsey survey, fielded in July 2024, asked about 12 practices, and each correlated positively with EBIT impact. The practice most associated with bottom-line impact was tracking well-defined KPIs for gen AI solutions, and less than one in five respondents said their organizations did it (McKinsey).

The usual objection is that AI’s contribution cannot be isolated, because too much else changes in a quarter. That is true, and precision to the decimal is not the goal. A baseline, a later-starting team and a number finance already trusts will give a defensible range. Other investments are held to that standard. Some use cases will show nothing, and that is a result too. It frees the budget for the ones that do.

The same slide, with fewer numbers

Next quarter the AI slide has fewer numbers. Three use cases, each with three lines: who is using it, what moved, and what happened to quality. Days to close is down by one. Support cost per ticket is flat, and that pilot is under review. The hours-saved figure is gone.

The CFO asks where the gain went, and this time someone points to a line she already tracks. The slide is less impressive than the last one. It is also the first one she can use.

The opening scenario is a composite of common patterns, not a client engagement. SopraLabs builds the KPI map, dashboard spec and review cadence with leadership teams as the KPI Model & Measurement module of our services, which defines success measures tied to business outcomes, not vanity metrics.

Frequently asked questions

What KPIs should we use to measure AI?

Use three per use case: an adoption measure, an outcome measure and a quality or risk measure. The outcome measure is an operating number the business already tracks, such as cost per ticket or days to close, tied to revenue, cost, risk or customer experience. Company-wide counts of licenses and prompts help manage the tools. They do not show value.

How do you measure the ROI of generative AI?

Compare an operating number before and after launch, against a baseline recorded in advance and, where possible, against a team that started later. Convert the change to money only after it shows up in that number. Self-reported time savings are not a sound basis for an ROI figure.

Are “hours saved” a reliable AI metric?

Not on their own. In METR’s 2025 randomized trial, experienced developers took 19% longer with AI tools and afterward believed they had been 20% faster (METR). An hour saved becomes value only when it shows up as more output, lower cost or fewer errors.

What is a vanity metric in AI?

A vanity metric is a number that rises with activity whether or not the business improves. Licenses assigned, active users, prompts run and survey-based hours saved are the common ones. They are worth tracking to manage licenses and adoption, but they should not be presented as results.

Sources

Keep reading

Browse all insights