# Closing the POC-to-Production Gap
In 2019, a global consumer bank we'll keep anonymous ran 47 AI pilots across risk, marketing, and operations. Each one "worked": the notebooks ran, the accuracy metrics cleared the bar, the demos wowed the steering committee. Eighteen months later, exactly three were in production. The other 44 died in what the team's own retrospective called "the valley of proven-but-unusable." The models were fine. Everything around them was missing.
This is the defining failure mode of enterprise AI, and it is not a modeling problem. Gartner's oft-cited figure—that roughly half of AI projects never reachreachThe number of unique people exposed to your message in a given period. Unlike impressions, reach counts each person once, no matter how often they see it.View full definition → production—understates the pain, because it counts survival, not value. The real question for a CDO is not "can we build a model that predicts churn?" You already can. It's "why does a model that predicts churn with 84% AUC sit in a data scientist's environment for nine months while churn keeps happening?"
The gap between a proof-of-concept and a production system is not a gap of intelligence. It's a gap of industrialization. A POC optimizes for *proof*; production optimizes for *repeatability under constraint*. These are different disciplines with different owners, and the CDO's job is to close four specific gaps that a demo will always hide.
A POC is a controlled experiment. It uses a frozen, hand-cleaned dataset, a data scientist who understands every column, no latency requirement, no downstream consumer, and no auditor. Production removes every one of those luxuries. MapMapUsing software to automate repetitive marketing tasks and campaigns, enabling personalisation at scale across channels like email, web, and social.View full definition → the difference and you get your industrialization backlog.
The POC ran on a CSV someone pulled and cleaned over three days. Production needs that same data delivered fresh, at volume, on a schedule, with the same distribution the model was trained on—forever. This is where most pilots quietly die.
The concrete failure is training-serving skew: the model was trained on a feature (say, customer_tenure_days) computed one way in a batch SQLSQLSales Qualified Lead: a prospect the sales team has validated as ready for direct outreach and a proposal, having passed clear qualification criteria.View full definition → job, but at inference time that feature is computed a different way, or from a different source, or is simply unavailable in real time. The model isn't wrong; it's being fed a different reality than the one it learned.
Your Monday-morning test for any pilot claiming production-readiness: for every feature the model consumes, can you name the production pipeline that will produce it, its refresh cadence, and its SLA? If the answer for even one feature is "we'll figure that out," the pilot is not production-ready regardless of its accuracy. This is precisely what a feature storefeature storeA centralised repository managing ML features, ensuring consistency between training and serving environments.View full definition → solves—not as a technology fetish, but as the contract that guarantees the feature computed in training is the identical feature served in production.
In the POC, one talented person owns everything: data, model, evaluation, and the demo. In production, no single person can own a system that must run at 2 a.m. on a holiday. The most common organizational cause of stalled pilots is that the pilot has a builder but no operator.
Ask this brutal question in your next review: *when this model degrades in production—not if—whose pager goes off, and what runbook do they open?* If the data scientist who built it is the answer, you have a hero-dependency, not a product. Heroes don't scale, take vacations, and eventually quit.
A prediction that lands in a dashboard nobody's job requires them to open is not a product; it's a fact. Value is created only when the prediction changes a decision inside an existing workflow—the CRMCRMCustomer Relationship Management: software and strategy to manage and analyse customer interactions throughout their lifecycle.View full definition → screen the rep already uses, the underwriting queue, the replenishment order. Integrating into a living system with its own release cycle, security review, and product owner is often harder than the model itself.
In the demo, trust is granted by the room's enthusiasm. In production, trust must be *earned continuously* and defended to a regulator, a customer, or a business owner whose bonus depends on the outcome. This means monitoring, explainability at the individual-decision level, a documented approval trail, and a kill switch. A model the business doesn't trust won't be used, and an unused model in production is just a more expensive POC.
Closing four gaps sounds like four projects. In practice, you industrialize by refusing to let a pilot start without a Production Contract—a lightweight artifact the sponsoring business unit and the data team sign *before* any modeling. It converts the four gaps into explicit commitments and kills doomed pilots early, which is the single highest-ROIROIReturn on Investment: the ratio of net profit to the cost of an investment. A 300% ROI means each dollar invested returns $3.View full definition → move a CDO can make. Killing a bad pilot in week two saves the six months it would otherwise take to die in the valley.
The contract answers, in one page:
Notice the framing: the contract is business-led, but the CDO owns the *template and the veto*. A pilot that can't fill in the "consumer" and "operator" lines is not ready for funding, no matter how excited engineering is.
You cannot apply the same rigor to every use case without grinding to a halt. Tier your governance to the decision's blast radius:
The CDO's leverage is in setting these tiers once, so that 200 use cases don't each negotiate their own governance from scratch. This is how you scale judgement.
MLOpsMLOpsMachine Learning Operations: combining ML and DevOps practices to industrialise, deploy, monitor, and retrain models reliably in production.View full definition → is not a tool you buy; it's the set of automated guarantees that let you run many models without many heroes. You already know the basics of pipelines and CI/CD from the architecture module—here's what specifically closes the production gap.
The load-bearing capabilities, in priority order:
1. A feature store to kill training-serving skew (Gap 1).
2. A model registry so every deployed model is versioned, traceable to the data and code that produced it, and rollback-able (Gaps 2 and 4).
3. Automated monitoring for both operational health (latency, errors) *and* statistical health—data drift and model performance decay (Gap 4).
Drift monitoring is the one most teams skip and most regret. A model doesn't fail with an error message; it fails silently as the world drifts away from its training data. Your monitoring config should treat a distribution shift as a first-class alert, not a quarterly analysis:
monitor:
model: churn_scorer_v3
drift_detection:
feature: customer_tenure_days
method: population_stability_index
threshold: 0.25 # PSI > 0.25 = significant shift
window: 7d
performance:
metric: auc
baseline: 0.84
alert_below: 0.78 # trigger retrain review
action_on_breach: page_operator + open_retrain_ticketThe point of this snippet isn't the YAML—it's the discipline it encodes: a defined baseline, a defined breach threshold, and a defined human action. A model in production without these three things is unmonitored, which means it's untrusted, which means it will be switched off the first time it embarrasses someone.
The strategic mistake is treating each production deployment as a custom engineering project. The winning pattern—used by every company that has scaled AI past a handful of models—is the paved road: a standardized, opinionated path from registered model to deployed, monitored service that a data scientist can travel *without* filing a ticket to a platform team.
If deploying model number 12 is as bespoke as model number 1, you have no platform; you have a series of heroics. The CDO's investment case for an MLOpsMLOpsMachine Learning Operations: combining ML and DevOps practices to industrialise, deploy, monitor, and retrain models reliably in production.View full definition → platform is exactly this: it converts the marginal cost of the *next* production model from months to days. That is the number to put in front of the CFO—not "AI maturity," but deployment velocity and the operator headcount you *don't* have to hire.
Knowledge check
1. According to the lesson, why does the CDO's central question shift from 'can we build a model that predicts churn?' to 'why does a proven model sit unused for months?'
2. The lesson argues that a POC and a production system 'optimize for different things.' What is the core distinction being drawn?
3. Why does the lesson caution that Gartner's figure about AI projects failing to reach production actually 'understates the pain'?
4. Select ALL correct answers about the conditions that make a POC a 'controlled experiment' whose luxuries production removes.
Select all the correct answers.
5. Select ALL correct answers that correctly characterize the 'Data Gap' (from snapshot to supply chain) described in the lesson.
Select all the correct answers.
Knowing the four gaps and the backbone is not a plan. Sequencing is where CDOs go wrong—they either build a gold-plated platform before there's a single production use case (the "field of dreams" trap) or they ship a dozen bespoke pilots and drown in operational debt.
The right sequence is deliberately unglamorous:
First, ship one Tier 2 use case end-to-end—badly. Pick a use case with a real, willing business owner and push it all the way to production using whatever duct tape it takes. The goal is not the model; it's to surface every gap in *your specific organization*—which teams block, which data sources lie, which security review takes eleven weeks. You are mapping the terrain, not winning the war.
Second, pave the road you just walked. Now build platform capability against the friction you actually encountered, not the friction you imagined. If the security review was the bottleneck, standardize a pre-approved deployment pattern. If features were the problem, stand up the feature storefeature storeA centralised repository managing ML features, ensuring consistency between training and serving environments.View full definition →. This keeps platform investment ruthlessly tied to observed pain.
Third, industrialize the operating model. This is the part CDOs under-invest in. Stand up the operator function—whether you call it ML platform engineering, MLOpsMLOpsMachine Learning Operations: combining ML and DevOps practices to industrialise, deploy, monitor, and retrain models reliably in production.View full definition →, or a product-aligned pod—and make the Production Contract mandatory for funding. The organizational design is the deliverable here: the pilot has a builder, but production needs a builder *and* an operator, and someone has to own the boundary between them.
The anti-pattern to name explicitly: the innovation-lab-to-nowhere. A shiny AI lab that runs pilots but has no contract with, and no obligation to, the operating business will generate demos indefinitely and production value never. If your lab's success metric is "pilots launched" rather than "decisions changed in production," you are funding the valley of proven-but-unusable. Change the metric on Monday.
One more judgement call: retire aggressively. A production model has a maintenance cost forever. If a deployed model isn't clearing its kill criteria or has been quietly bypassed by its business owner, shut it down. A portfolio of 40 production models where 15 are unmonitored and unused is not a sign of maturity; it's a liability with a monitoring bill.
1. Treat the POC-to-production gap as four specific gaps—data supply, ownership, integration, and trust—not as a modeling problem. A demo hides all four by design; your review process must surface each one explicitly.
2. Institute a one-page Production Contract that every pilot signs before funding. If it can't name the decision it changes, the consumer system, the feature sources with SLAs, and the operator who holds the pager, it doesn't get built. Killing doomed pilots early is your highest-ROIROIReturn on Investment: the ratio of net profit to the cost of an investment. A 300% ROI means each dollar invested returns $3.View full definition → move.
3. Tier governance by decision blast radius, not by hype, so that low-risk use cases move fast and regulated decisions get full audit and bias controls—without every team renegotiating governance from scratch.
4. Justify the MLOps platform in the language of deployment velocity and avoided operator headcount, not "maturity." The paved road exists to make model number 12 take days, not months, and to eliminate hero-dependency.
5. Sequence deliberately: ship one use case end-to-end to map your real friction, then pave that specific road, then mandate the operating model—and retire unused models aggressively before they become a silent liability.