Everyone Knows AI Works. Nobody Can Prove It.
Hi all,
I keep getting the same question from clients lately: "Where's the ROI on all this AI spend?"
It's a fair question. Two summers ago, Sequoia published AI's $600B Question, pointing out the industry had committed something like $600 billion to AI infrastructure with nowhere near that much revenue to show for it. A week later, Goldman Sachs put out a report bluntly titled Gen AI: Too Much Spend, Too Little Benefit? and quoted MIT's Daron Acemoglu estimating gen AI would lift U.S. productivity by just 0.53% over the next decade.
Two years on, the spending hasn't slowed. And it's not just tokens and tools anymore. Organizations are reskilling entire workforces, re-architecting core processes, and rewriting strategy around this technology. When the check is that big, "trust me, it's working" doesn't cut it with the CFO.
Why This Is So Hard to Measure
Conor Grennan has a thought experiment recently that I love. Imagine a pill that makes everyone in your company 30% smarter and 30% faster. Everyone takes it on day one. Six months later, the CFO asks whether it paid off.
Easy, right? Look at output.
Except what's the output of the analyst whose job is to be right about the market? Or the strategist whose most valuable move last quarter was talking you out of a bad project?
Every previous technology came with its own meter in the box. New machine on the line: units per hour. Salesforce: cycle time. AI doesn't. It makes people faster and makes their decisions better, and the second one has never shown up cleanly in a spreadsheet.
Every Number You've Seen Is Probably Wrong
My friend and fellow economist Ben Falk wrote a piece earlier this year with a title I wish I'd thought of: Every AI ROI number you've seen is wrong.
Economists have a very high bar for causal claims, and almost nobody in enterprise AI is clearing it. "We saved X hours, an hour costs Y" is a manufacturing formula applied to knowledge work. No control group means no causal claim. Measuring the person using the tool while ignoring everyone upstream and downstream inflates the headline. And a 30-day pilot tells you almost nothing, because people are still learning the tool.
His closing line is one I've been quoting all year (verbatim): "Everything else is post-hoc rationalisation dressed up as analysis."
Between vendors with a commercial interest in big numbers and consultants with an interest in validating their clients' decisions, there's a lot of confident arithmetic floating around that wouldn't survive one reply from a skeptical economist.
We've Seen This Movie Before
Back in grad school, I studied under Lorin Hitt, who studied under Erik Brynjolfsson at MIT. Together they spent years chasing a famous riddle called the IT Productivity Paradox. Nobel laureate Robert Solow put it best in 1987: "You can see the computer age everywhere but in the productivity statistics."
It was maddening. Everyone knew computers were transforming work. The data refused to show it. Some studies would triumphantly declare "we found it!" only for others to replicate the method and discover that pencils were apparently productivity game-changers too.
By the late 1990s, the paradox unraveled. The gains were real, but they showed up only in industries that reorganized around IT (finance, retail, logistics), not the ones that bolted computers onto old processes. The payoff required training, redesign, and time. Traditional metrics couldn't capture it. And the benefits concentrated in the sectors that reinvented themselves.
My Take
I suspect the ROI on AI will look obvious in hindsight, the same way IT's eventually did. It'll show up when three things happen:
- We measure it properly. Full production chain, real counterfactuals, quality and not just hours.
- We account for the lag. These things take time to implement and to create an impact.
- AI gets baked into processes natively instead of bolted on. The organizations that reorganize around it will pull away. The ones layering a chatbot onto 2019 workflows will keep asking where the ROI is.
How We Approach It at AIPL
As an economist, I take this seriously, and so does my team at the AI Performance Lab. When we engage with a company, we measure before and after, with metrics tied to the actual work rather than seat counts and hours saved. The result is an ROI you can defend to a board. In our experience, it typically comes in at many multiples of the investment.
If you're wrestling with how to prove (or disprove) the return on your own AI investments, hit reply. I'd love to have that conversation.
Best,
Dr. Michael "House" Housman