Reading an AI-ROI Report Like a CPA

I just got back from a week of fishing in Canada, my first real week off in a long time. Coming back, the first thing in my inbox was a new AI research report, and it happened to be from Alteryx, a tool I know really well. I was a heavy Alteryx user for years at PwC, from my first year as an intern all the way through when I left, helping teams deploy it across the firm. So when Alteryx put out a report on AI ownership, agents, and ROI, I read it with two hats on: someone who’s used the product, and someone who reads a lot of these vendor reports for this newsletter.

And that’s actually what I want to talk about today…less the specific findings, and more how to read reports like this one, because you’re going to keep getting sent them.

The two numbers everyone will quote

The report surveyed 1,400 people (roughly 1,000 IT leaders and 400 people in AI and workflow roles) and the two numbers that’ll travel are that 69% of respondents report moderate or significant ROI from AI investments, and 93% say they’re confident agentic AI will show ROI within two years. Both are self-reported. And the report doesn’t say much about how those 1,400 people were selected, or what the response rate was. Did they survey 10,000 people and hear back from 1,400, or something else? We don’t know.

Why I dug past the headline page

Most people flip straight to the page with the big numbers on it. I did too, at first. But once I started digging into the details, I found things that didn’t add up. On page 6, the body text says one AI investment category delivered 22% clearer ROI than others. The chart right next to it says 27% for that same category. Later in the report, a chart on confidence in agentic AI has four bars that are supposed to cover the full range of answers. Add them up and they come to 107%.

I don’t think that’s malicious. My guess is that more of these reports are being put together quickly, maybe with AI doing some of the drafting, and errors are slipping through. But it’s a real signal about how much scrutiny went into the report before it was published, and it’s worth catching before you repeat a number to a client.

The playbook behind these reports

These reports aren’t independent studies, and that’s not a knock on Alteryx specifically…it’s the format. A vendor funds a survey, a research firm (in this case, Coleman Parkes) fields it, and the report comes out with a last page that quietly explains how the vendor’s product solves the exact problems the survey uncovered. I spent years in go-to-market roles, and I can tell you: these reports are built to be a warm reason for a sales rep or customer success manager to reach back out to you. That doesn’t make them worthless. It just means you read them differently than you’d read an academic study.

The 60-second check I run now

• Who funded it, and who actually fielded the research? The last page usually tells you the real agenda.
• How much do they disclose about methodology? The more detail on how they ran the survey, the more I trust it.
• Do the numbers hold up? Check any chart against the text next to it — and this is a genuinely good use case for AI: paste the report in and ask it to flag inconsistencies.
• Who’s actually in the sample? A report of IT leaders is telling you what IT leaders think, not what finance leaders think — and a lot of these skew geographically in ways that aren’t always obvious.

Where the real signal is

None of this means you should throw the report out. Once you filter past the marketing, this one lines up with what I’ve seen elsewhere: workflow automation is where companies are actually seeing ROI right now, more than copilots or predictive analytics. Workflow automation is a mature field with structure and governance already built in, so it’s a lot easier to measure than something like “are people using the copilot more.” And on the agentic side, only 7% of respondents describe their AI as fully autonomous with no human involved. The other 93% keep a human in the loop or use AI to work through predefined steps. Given how much governance matters in finance specifically, that’s exactly where I think things should be right now.

Key Takeaways

• Check who funded and who fielded the report — the last page usually reveals the real agenda.
• The size of the methodology section is a decent proxy for how much you should trust the findings.
• When a chart and the surrounding text disagree, don’t repeat either number until you’ve resolved it.
• Know your sample — a survey of IT leaders tells you what IT leaders think, not your whole client base.
• The findings are still useful. You just have to read past the top line to get there.

Want the CPE credit? Take the full lesson on EverydayCPE and earn 0.2 CPE credits: Reading an AI-ROI Report Like a CPA