Data literacy: measure decision quality, not training

September 16, 2026 · 5 min · Free, with full transcript

Why do data literacy programs teach medians, filters and chart types and still leave the business running on gut feeling? The position here is blunt: knowledge and behavior are different things, and almost nobody measures the second one. The fix is to build training around real recurring decisions like pricing, hiring and inventory, and to require anyone proposing a decision to state in advance what evidence would change their mind. You come away with a way to score decision quality across ten recurring decisions before and after a program, a read on Booking.com's experimentation default, and evidence from MIT Sloan Management Review, O'Reilly Radar and DBT Labs on why culture and trust, not skills, are the bottleneck.

Listen to the podcast

4 min

Chapters

Key takeaways

  • Pick ten recurring decisions, record whether each was made on gut, politics or evidence, then measure again six months later.
  • Require the person proposing a decision to write down what evidence would change their mind before the meeting.
  • Start the literacy program with the twelve executives who actually make the calls, not with five thousand employees.
  • Tie learning to the actual workflow, like a plant manager changing one reorder decision on a forecast, rather than a training calendar.
  • Treat vendor data such as DBT Labs' claims with care and cross-check against independent work like O'Reilly's annual surveys.
Read the full transcript

Host:Welcome to the Leaders Insights podcast. Today's episode, Data Literacy Programs that Change Behavior. The concept CDOs get wrong. Most data literacy programs are a waste of money. You've said that out loud in a room full of CDOs. Defend it.

Expert:Happy to. Walk into 90% of these programs and what are they doing? Teaching people what a median is, how to filter a dashboard, the difference between a bar chart and a line chart. Then everyone gets a certificate and goes back to running the business on gut feeling exactly like they did before.

Host:But surely people need to understand the tools before they can use them?

Expert:Understanding the tool isn't the goal. Changing the decision is the goal. I can teach you what a confidence interval is, the range where the true number probably sits. And you'll still approve a marketing budget because someone in the room sounded certain. Knowledge and behavior are different animals. Nobody measures the second one.

Host:So what does a program that actually changes behavior look like?

Expert:It's built around decisions, not concepts. You take a real recurring decision—pricing, hiring, inventory—and you rewire how it gets made. You require the person proposing it to show what evidence would change their mind before they walk in. That one question does more than a semester of Excel training.

Host:Give me a company that did it right.

Expert:Booking.com, comma, years back. They didn't run a literacy course. They made experimentation the default. Every product change had to go through a controlled test, and thousands of people were empowered to run those tests. The literacy came from doing, not from a slide deck. People learned what a P-value was because they needed it to ship, not because HR mandated it.

Host:That's a tech company, though. Easy for them. What about a bank? A manufacturer?

Expert:Fair jab. But the principal travels. MIT Sloan Management Review has been tracking this. The organizations that see returns tie learning to the actual workflow, not to a training calendar. The manufacturer doesn't need everyone to understand regression, fitting a line through messy data. They need the plant manager to change one reorder decision based on a forecast and see it pay off. Small, concrete, tied to their day.

Host:You're describing something slow and unglamorous. Teachers want a number. What number do you give them?

Expert:You give them a decision quality number, and yes, it's harder to fake. Pick 10 recurring decisions before the program. Track how they were made. Ciao. Gut. Politics. Or evidence. Six months later, track again. DBT Labs published data suggesting teams with mature data practices ship decisions faster, though they sell data tooling. So I'd cross-check that against O'Reilly's annual survey work, which is independent and

Host:less flattering.

Expert:Less flattering how? O'Reilly's radar research keeps finding the same thing. The bottleneck isn't skills. It's culture and trust in the data. You can train every analyst in the building and it changes nothing if the CFO, the finance chief, still overrides the model whenever it disagrees with him.

Host:The failure is at the top, and no literacy program touches the top.

Expert:So the CDO's own boss is the problem. Constantly. I've watched chief data officers roll out gorgeous training to 5,000 employees, while the executive committee makes every real call in a hallway conversation with zero data. You want to change the culture. Start the program with the 12 people who actually decide things.

Host:Uncomfortable, but that's where the ROI, the return, the money back, lives.

Expert:Isn't that just politics dressed up as data? All of it is politics. Anyone who tells you data culture is a technical problem is selling you a platform. The technology's been fine for a decade. What's broken is that showing evidence and being told you're wrong is socially expensive, and people avoid it.

Host:Fix the social cost, and the literacy follows for free.

Expert:One thing a listener can do Monday morning, go. Pick a decision you make every month. Before you make the next one, write down what evidence would change your mind. If the honest answer is, nothing would. You've just learned you don't need data. You need to admit you've already decided.

Host:That's more literacy than most programs deliver in a year. Sources for today's episode. The new stack, DBT Labs, vendor, data tooling, KD Nuggets, O'Reilly Radar, MIT Sloan Management Review. That's your five. The full CDO track, structure, no fluff, is at MBA-training.com.

Read and go deeper