The spreadsheet that embarrassed a CFO and changed how we measure AI

0:00Welcome to the Leaders Insights podcast.

0:07Today's episode, the spreadsheet that embarrassed a CFO and changed how we measure AI.

0:12A CFO stood up in front of the board, projected millions in AI savings, and then watched the

0:18number melt to almost nothing.

0:20What actually happened?

0:21The article walks through a major retailer that ran a customer service AI pilot and

0:26celebrated something like $11 million in projected annual savings.

0:32Then someone finally added up the full-cost, ORS-1, the licensing, the cloud compute, the

0:37humans reviewing bad answers, and the net dropped to a rounding error.

0:42The CFO, the chief financial officer, learned in public that the headline number was fiction.

0:48But projections are always optimistic.

0:51Why is this the story that supposedly changed how we measure AI?

0:54Because it's not one bad forecast, it's a pattern that repeated across industries through

0:58the early 2020s.

1:01Companies counted the gross benefit and ignored what the article calls the full cost.

1:07The savings were real on one side of the ledger, and the spending was invisible on

1:11the other.

1:12That asymmetry is the whole disease.

1:19Four things.

1:20Inference.

1:21The cost of running the model every single time a customer types a question, which never

1:26stops.

1:27Then the review layer.

1:29Humans checking outputs because the model is confidently wrong, maybe 5-10% of the time.

1:36Then integration engineering to wire it into your existing systems.

1:40And retraining because the model drifts.

1:43Its accuracy decays as the world changes around it.

1:47The pilot budget covers none of that.

1:49So the demo looks brilliant, and the operating bill quietly eats the gain.

1:53Exactly!

1:54A demo processes 10 tickets, production processes 2 million, and now that per-query inference

1:59cost is a monthly invoice with your name on it.

2:02I've seen internal decks, I won't name them, where the pilot showed a 60% cost reduction,

2:08and the 12-month reality was closer to 8% once support and error handling were counted

2:13honestly.

2:14Here's the uncomfortable one.

2:16Aren't the vendors selling these tools the same people supplying the savings math?

2:20Yes, and that's the tell.

2:22When Microsoft or Salesforce publishes a study saying their AI assistant makes agents 30

2:27or 40% more productive, read the footnote.

2:31That's a vendor measuring its own product, often in a controlled task, not your messy

2:35environment.

2:37Treat those numbers as marketing until you've reproduced them on your own data.

2:41The good ones will actually let you.

2:43Fine.

2:44How does a normal manager measure this without a data science team?

2:47Start with one number, nobody games.

2:50Fully loaded cost per completed task.

2:52Not per query, per finished piece of work, including the human who fixed it.

2:57Take your total spend, divide by tasks that actually shipped correctly.

3:02Compare that to what the same task cost before AI.

3:05If the AI number isn't lower, you don't have savings, you have a science project.

3:10What if the task quality is worse but cheaper?

3:13People hide behind that.

3:14Then you price the failure.

3:16If the AI resolves a support ticket for 40 cents but creates a second contact because

3:21the answer was wrong, that second contact costs you the full human handling, say $6.

3:27Your real cost per resolution is now higher than doing it right the first time.

3:32Rework is the line item that turns a win into an embarrassment.

3:36Give me a case where someone got the measurement right.

3:39The pattern that works, and Klarna, the payments company, talked about this publicly,

3:44was treating the AI assistant like a new hire on probation.

3:48They tracked resolution rate, repeat contacts, and customer satisfaction side by side,

3:54monthly, and were willing to pull it back.

3:57The discipline wasn't the technology,

3:59it was refusing to declare victory from a single quarter.

4:02So the skill isn't building AI, it's the accounting around it.

4:06The article's real point.

4:08Measuring AI return is the most underrated skill in enterprise technology,

4:13and it's just honest arithmetic wearing a lab coat.

4:16Give me the one thing a listener does Monday morning.

4:19Take your flagship AI project and calculate cost per completed correct task,

4:25with the human cleanup included.

4:27If nobody in the building can produce that number by Friday,

4:30that's your answer.

4:31You're guessing, not saving.

4:33That's a wrap.

4:34Fresh AI briefings drop daily at MBA-training.com.