Marketing mix modeling: why holdout tests beat dashboards

September 15, 2026 · 5 min · Free, with full transcript

Can marketing mix modeling alone win a budget argument with the CFO? The position here is no. Modeling estimates where spend worked, but it can be tuned to flatter whoever signs off on the budget, and it says nothing about why or what happens if spend doubles. Controlled experiments, starting with geo holdouts, settle the question. The Airbnb paid search holdout around 2019 is the worked example, alongside HubSpot and MIT Sloan Management Review findings on testing habits, with vendor claims flagged as vendor claims. You walk away with a loop you can run: let the model raise a hypothesis, test it in half your markets, feed the result back. Plus a Monday starting point that needs no data science team, only the nerve to switch a channel off.

Listen to the podcast

4 min

Chapters

Key takeaways

  • Treat marketing mix modeling as a hypothesis generator, not proof, and check who benefits from the model's conclusions.
  • Run a geo holdout: switch one channel off in a handful of markets for four weeks and leave everything else untouched.
  • Feed experiment results back into the model so next quarter's estimates rest on tested numbers rather than assumptions.
  • Discount vendor benchmarks from analytics and marketing software sellers and cross-check them against independent research.
  • Stop using dashboards as evidence of causation; they report what happened, not what caused it.
Read the full transcript

Welcome to the Leaders Insights podcast.

Today's episode, Marketing Mix Modeling and Experimentation Culture, how CMOs build the

evidence machine.

Most CMOs I talk to spend more time defending their budget than growing it.

Why is that?

Because they walk into the finance meeting with a story instead of a machine.

They say the brand campaign lifted awareness, and the CFO, the chief financial officer,

the person holding the purse, nods politely and cuts 15% anyway.

Story loses to spreadsheet every time.

So the fix is marketing mix modeling.

But isn't that just an expensive way to confirm what you already believe?

That's the trap, and it's a real one.

Marketing mix modeling, gwer, the statistical technique that estimates how each channel

contributes to sales, is only as honest as the person feeding it.

I've seen teams tune the model until TV comes out as the hero because the head of TV signs

off on the budget.

That's not evidence.

That's astrology with a regression line.

Strong words.

So what separates the real thing from the astrology?

Experiments.

This is what everyone gets wrong.

They treat modeling as the answer.

Modeling tells you where the money worked, roughly.

It cannot tell you why, and it definitely can't tell you what happens if you double the

spend tomorrow.

For that, you run controlled tests.

Oh, duh-oh-co-weed.

You hold out a region.

You turn a channel off in 10 markets.

You watch what breaks.

Give me a real example.

Not a hypothetical.

Airbnb.

Around 2019, their model said search advertising drove a mountain of bookings, so they ran

a geo holdout, switched off paid search in a chunk of markets, and measured the gap.

Find out a big share of those bookings would have happened anyway.

People typing Airbnb were already sold.

The model gave paid search credit for a sale it didn't cause.

They redirected serious money to brand marketing off the back of that.

No experiment, no discovery.

But most companies aren't Airbnb.

They don't have a data science army.

You don't need one to run a holdout.

You need the nerve to turn something off.

That's the actual bottleneck.

Not tooling, courage.

Every marketer is terrified that if they pause a channel and numbers hold steady, they've

just proven their own job was optional.

There's the discomfort.

Is that fear justified?

Occasionally, yes, and that's the funny part.

The experiment that saves the budget is also the one that might quietly fire you.

But the CMOs who grow budgets are the ones who'd rather know.

HubSpot puts marketers running regular experiments at meaningfully higher return on ad spend,

though worth noting they sell marketing software.

So cross-check that against independent work.

And MIT Sloan Management Review has been blunt for years that firms with a genuine testing

habit outperform the ones drowning in dashboards they never act on.

Dashboards get a lot of love.

You're saying they're mostly theater?

Mostly.

A dashboard tells you what happened.

It has no opinion on causation.

On to you on what actually caused what.

You can stare at a beautiful chart of website traffic climbing and have no idea whether

your campaign did it or a competitor imploded.

SEMrush will happily show you traffic curves all day.

Again, they sell the analytics tools, so treat their benchmarks as marketing, not gospel.

The number on the screen isn't proof.

The test is proof.

So modeling plus experiments.

How do they actually talk to each other?

The model generates hypotheses.

Podcasts look underpriced.

We're getting more than we pay for.

The experiment settles the bet.

You buy more podcast slots in half your markets and see if sales follow.

Then you feed the result back into the model.

So next quarter's estimate is grounded in something you tested, not something you

assumed.

It's a loop.

Each side covers the other's blind spot.

If someone's building this from zero on Monday, where do they start?

Make one channel you're convinced is working.

Then turn it off.

Bowl you in a handful of markets for four weeks, nowhere else.

If sales don't move, you just found money you were setting on fire.

If they drop, you've got the cleanest budget defense evidence you'll ever own.

One channel, one holdout.

That's the whole starter kit.

Turn something off to prove it matters.

I'll take it.

Sources for today's episode.

DigiDay SEMrush Vendor, SEO Analytics Tools, MIT Sloan Management Review, HubSpot Vendor,

CRM Marketing Automation.

That's it.

The CMO Decision Tools are live at MBA-training.com.

Read and go deeper