Natural-language BI and the analyst's new role: why the "democratisation" story is only half true

August 22, 2026 · 5 min · Free, with full transcript

Natural-language query tools promise to put business intelligence in everyone's hands, removing the analyst bottleneck. The reality is more complicated, and CDOs who act on the simple version of this story will make costly structural mistakes.

Listen to the podcast

4 min

Read the full transcript

Welcome to the Leaders Insights podcast.

Today's episode, Natural Language BI and the Analyst New Role, why the democratization

story is only half true.

Every vendor pitch I've heard this year says the same thing.

Type a question in plain English, get a chart back, and you can finally fire your analytics

team.

You're telling me that's wrong.

I'm telling you it's the most expensive half-truth in the industry right now.

The typing a question part works.

The firing the team part is where CDOs, E's, chief data officers, the people who own a company's

data strategy, light money on fire.

Strong start, but be concrete.

What actually breaks?

The question isn't the hard part.

The definition behind it is, ask what was revenue last quarter?

And the tool has to know, recognized revenue or booked?

Does it include refunds?

Which quarter?

Calendar or fiscal?

But one retailer I worked with, the same question gave finance and marketing numbers

that differed by 11 percent.

Because the Natural Language tool guessed the join between two tables and guessed wrong.

But the vendors say the semantic layer fixes exactly that.

The dictionary that tells the tool what revenue means.

Right, and I love the semantic layer.

The shared dictionary of business definitions the machine reads before answering.

Here's the catch, nobody prints on the box.

Someone has to build and maintain that dictionary.

That someone is an analyst.

So the tool that supposedly kills the analyst runs entirely on the analyst's work.

So you've just moved the bottleneck, not removed it?

Exactly.

You've moved it upstream, where it's less visible and more dangerous.

When a human analyst gave you a wrong number, they hedged.

This looks off, let me check.

The tool answers in confident prose with a nice bar chart.

It has the tone of certainty and none of the doubt.

That's how bad decisions get made fast.

Give me a number.

How often are these things actually right?

The honest benchmarks are sobering.

There's public research.

The spider benchmark, an academic test of turning English into database queries, where

even the best 2025 models landed around 80 to 90 percent accuracy on clean, well-labeled

data.

Sounds great until you realize real company data is filthy and a 15 percent error rate

on financial reporting is not a rounding issue.

It's a resignation letter.

Vendors would say those numbers are old and their tool is better.

Of course they would.

And their internal benchmarks, which they ran themselves so great accordingly, always show

95 plus.

I've watched those same tools confidently sum a column of customer IDs and report

it as a metric.

It's like hiring an intern who never says, I don't know and never sleeps.

So who's actually winning with these tools, if anyone?

The companies that treat it as an amplifier, not a replacement.

Spotify and a few others have talked openly about internal query assistance.

But the point is they sit on top of a governed data model built by a real team.

The tool handles the boring 70 percent.

How many active users in Germany last week?

The analyst is freed for the 30 percent that pays their salary.

The why did they drop question no natural language box can answer.

Because that requires knowing the marketing calendar and the app outage on Tuesday.

So the analyst's job changes.

Into what exactly?

Two things.

First, a definer.

The person who decides and documents what every metric means so the tool can't improvise.

Second, an investigator.

The one who takes the tool's answer and asks whether it's actually true.

The pure report runner, the person whose whole day was building the same dashboard

for the fourth time.

Yes, that role shrinks.

But that was never analysis.

That was clerical work wearing a data badge.

So what's the structural mistake a CDO makes if they believe the simple story?

They cut headcount first and build the semantic layer never.

Then the tool spreads confident wrong answers across the whole company.

And now everyone's a self-served analyst producing garbage at scale.

You didn't democratize insight.

You democratized error.

Give me the one thing to do, Monday.

Before you buy any natural language tool, ask your team to write down the definition

of your top five metrics.

If they can't agree in a room, the tool will just automate the disagreement, faster,

and with prettier charts.

Done.

Want to know where you actually stand?

Take the CDO self-assessment at MBA-training.com.

Read and go deeper