ML in production: why models die after the notebook

September 17, 2026 · 5 min · Free, with full transcript

Why do most machine learning projects never leave the lab? The position here is blunt: the model is about ten percent of the work and the other ninety percent is pipelines, monitoring and ownership. The episode walks through the three failure points at the notebook to production handoff, data drift, unowned pipelines after launch, and systems never designed for retraining, and why a drifting model costs money quietly instead of paging you at midnight. You come away able to name the roles that fix this, the ML engineer and the data engineer, read vendor claims from DBT Labs against O'Reilly Radar and MIT Sloan Management Review, apply data as a product, and ask the accountability question before funding an AI project.

Listen to the podcast

4 min

Chapters

Key takeaways

  • Before approving any model, ask who by name is on the hook to keep it running six months after launch.
  • Budget for an ML engineer who sits between the data scientist and the software team, not just modellers.
  • Treat each dataset as a product with an owner, a quality standard, documentation and a promise it will not silently change.
  • Monitor for data drift, because a degrading model keeps returning confident answers instead of crashing.
  • Discount vendor productivity claims such as DBT Labs surveys and cross check them against O'Reilly Radar or MIT Sloan research.
Read the full transcript

Host:Welcome to the Leaders Insights Podcast. Today's episode, the ML to Production Field Guide, who shapes what breaks and why.

Expert:Here's a stat that should terrify anyone funding an AI budget. Most machine learning projects never make it out of the lab. They die somewhere between a data scientist notebook and an actual working product. Why? Because a notebook is a sandbox and production is a war zone. In the notebook, that's the interactive coding environment where data scientists prototype. Everything is clean, static, and forgiving. In production, the data is late. The users are weird. And nobody warned you the upstream system changed a column name at 2 a.m. So, it's not that the models are bad, it's everything around them. The model is maybe 10% of the work. The other 90 is plumbing. Pipelines that move data, monitoring that tells you when it rots, and the boring governance of who owns what. O'Reilly's survey work has been saying this for years. The top barriers to ML adoption aren't algorithms, they're skills gaps and lack of data infrastructure.

Host:Give me the concrete failure. Where does the notebook to production handoff actually snap?

Expert:Three places. One, the data drifts. The real world inputs slowly stop looking like the training data, so a fraud model trained on 20-24 spending patterns quietly goes senile. Two, nobody owns the pipeline after launch, so it breaks and sits broken. Three, the thing was never designed to be retrained, so fixing it means rebuilding it.

Host:Go senile. That's the drift problem. How fast does that happen?

Expert:Faster than people expect. A recommendation model can degrade meaningfully in weeks if behavior shifts. And here's the trap. It doesn't crash. It keeps returning confident answers that are increasingly wrong. A broken server pages you at midnight. A drifting model just costs you money silently for a quarter.

Host:So who are the people who actually fix this?

Expert:The field guide is about roles. The key hire nobody budgets for is the ML engineer. The person who lives between the data scientist and the software team, whose whole job is making a model survive contact with reality. Then data engineers who build the pipes. Use those two and you've hired a Formula One driver and forgotten to build a track.

Host:Let's talk numbers. I've seen vendors claim huge productivity gains from better tooling. How much do you trust that?

Expert:Carefully. DPD labs. And note. They sell data transformation software, so they've got skin in the game. Publishes analytics engineering surveys showing data teams spend enormous time just cleaning and reshaping data before any modeling happens. Directionally true, but cross check it against something independent. MIT Sloan Management Review has done more sober work here and their line is consistent. The winners treat data as a managed product, not a byproduct.

Host:Data as a product. Unpack that because it sounds like a slogan.

Expert:It means someone owns a dataset the way a product manager owns an app. With a quality standard, documentation, and a promise it won't silently change. Most companies treat data like leftovers in the office fridge. Technically available, ownership unclear, increasingly dangerous.

Host:So an exec hears all this and thinks, I need to buy a platform. Are they right?

Expert:That's the reflex, and it's usually wrong. Tools don't fix an org that hasn't decided who's accountable when the model breaks at 3 a.m. MIT Sloan's research keeps landing on culture and clear ownership over any specific tool. You can buy the best monitoring in the world and still ignore the alerts.

Host:Give me the uncomfortable question then. What's the one thing companies fake?

Expert:They fake done. A demo works, everyone claps, the press release goes out. And then it never actually runs reliably for real users. Working in the demo and working in production are as different as a wedding and a marriage.

Host:So for the exec listening who's about to greenlight an AI project, what do they do Monday morning?

Expert:Before you approve the model, ask one question. Who is on the hook to keep this running six months after launch? And what's their name? If there's no name, you're not funding a product. You're funding a very expensive demo.

Host:A name, not a platform. That's the takeaway. Thanks for this.

Expert:Anytime.

Host:Sources for today's episode. Towards Data Science, KD Nuggets, The New Stack, DBT Labs, Vendor, Data Tooling, O'Reilly Radar, MIT Sloan Management Review. That's a wrap. Watch CDO briefings drop daily at MBA-training.com.

Read and go deeper