MLOps: monitoring, retraining & drift

# MLOpsMLOpsMachine Learning Operations: combining ML and DevOps practices to industrialise, deploy, monitor, and retrain models reliably in production.View full definition →: monitoring, retraining, and drift

In 2020, Zillow's iBuying algorithm was quietly buying homes at prices its models believed were fair. The models had been trained on a housing market that no longer existed. By late 2021, Zillow Offers had overpaid on thousands of properties, wrote down more than $500 million, laid off a quarter of its workforce, and shut the business down. The models did not crash. They did not throw errors. They kept producing confident predictions right up until the company killed the unit. That is the defining property of the failure mode you now own: a model in production does not fail loudly. It rots silently, and it takes the P&L with it.

Your fundamentals track taught you how a model gets built and deployed. This lesson is about the eighteen months *after* deployment, the operating discipline that keeps a model honest as the world underneath it moves. Most organizations invest 90% of their ML effort in getting to the first deployment and almost nothing in what happens next. That asymmetry is where value dies.

The two kinds of rot, and why the distinction drives your response

The word "drift" gets used loosely in status meetings. As a CDO you need to hold the line on three distinct phenomena, because each demands a different intervention and a different owner.

Data drift is a change in the distribution of the model's *inputs*. Your fraud model was trained when 20% of transactions were card-present; now, post-pandemic, 70% are online. The relationship between features and outcome may be intact, but the model is now seeing inputs it rarely trained on. This is often benign or self-correcting, but it is your earliest warning signal.

Concept drift is a change in the relationship between inputs and the *target itself*. The same customer profile that predicted low churn in 2019 now predicts high churn because a competitor entered the market. Your inputs may look statistically identical while the underlying reality has inverted. This is the dangerous one, because input-monitoring alone will not catch it. Zillow suffered concept drift: the mapping from home features to future sale price shifted underneath a model that still saw familiar-looking houses.

Label drift / delayed ground truth is the operational problem that makes concept drift so insidious. In many high-value use cases, credit default, insurance claims, long-cycle churn, you don't *know* the true outcome for months. You cannot measure prediction accuracy in real time because the answer hasn't arrived yet. This forces you to monitor proxies rather than truth.

The practical judgment here: the longer your ground-truth latency, the more you must invest in input-side and proxy monitoring, because your accuracy metrics are a rear-view mirror pointed at a road you drove months ago.

A monitoring stack that maps to those three phenomena

Build monitoring in three layers, and insist every production model has all three before it ships:

1. Input monitoring (catches data drift, available immediately). Track feature distributions against a training reference. Population Stability Index (PSI) and Kolmogorov, Smirnov statistics are the workhorses. PSI above 0.2 on a material feature is a "look now" signal; above 0.25 is an alert.

2. Prediction monitoring (catches drift you can see before labels arrive). Track the distribution of the model's *outputs*. If your approval-rate model suddenly approves 40% instead of 25%, something moved even before you know if those approvals were good.

3. Performance monitoring (catches concept drift, delayed). Track accuracy, precision/recall, calibration, and business KPIs as ground truth arrives. This is truth, but it is late truth.

# Population Stability Index — the one metric to put on the wall
def psi(expected, actual, buckets=10):
    breakpoints = np.quantile(expected, np.linspace(0, 1, buckets + 1))
    e = np.histogram(expected, breakpoints)[0] / len(expected)
    a = np.histogram(actual, breakpoints)[0] / len(actual)
    e, a = np.clip(e, 1e-4, None), np.clip(a, 1e-4, None)
    return np.sum((a - e) * np.log(a / e))
# <0.1 stable | 0.1–0.25 investigate | >0.25 significant shift

The mistake I see most often is teams that monitor only layer 3 because it's the "real" metric. By the time layer 3 moves on a slow-label model, you've been shipping degraded decisions for a quarter. Layers 1 and 2 are your smoke detectors; layer 3 is the fire department arriving.

Deciding when to retrain: from reflex to policy

There are three retraining philosophies, and choosing among them is a genuine executive decision, not a data-science default.

Scheduled retraining, retrain on a fixed cadence (weekly, monthly). Simple, auditable, budgetable. It is also either wasteful (retraining a stable model) or too slow (a fast-moving domain drifts between cadences). Use it as a floor, not a strategy.

Triggered retraining, retrain when a monitoring signal breaches a threshold. This is where mature organizations live. The model retrains when PSI crosses a line, when accuracy drops below an SLA, or when a business metric degrades. It is efficient but requires disciplined thresholds and a governance layer so the model can't retrain itself into a worse state.

Continuous / online learning, the model updates constantly from streaming data. Powerful for recommendation and ad-ranking systems, dangerous almost everywhere else. It removes the human checkpoint, and it lets a poisoned or anomalous data stream corrupt the model in near-real-time. Reserve it for high-frequency, low-blast-radius, self-correcting problems.

The retraining decision framework

Don't let "the metric dropped" automatically mean "retrain." Retraining is a *cost and a risk*, not a free reset. Walk the decision through four questions:

1. Is the degradation real or noise? A single bad day is not drift. Require the signal to persist across a defined window before acting. Set the sensitivity to your decision cadence, not to the data's minute-by-minute jitter.

2. Will retraining actually fix it? This is the question teams skip. If the world genuinely changed (concept drift), retraining on fresh labeled data helps. But if the problem is a broken upstream data pipelinedata pipelineETL (Extract, Transform, Load) is a data integration process that pulls data from sources, reshapes it into a consistent format, and writes it into a target system.View full definition →, a feature that started arriving null, retraining will *bake the corruption into the new model*. Diagnose the source before you retrain. A shocking share of "drift" is actually a silently broken ETLETLETL (Extract, Transform, Load) is a data integration process that pulls data from sources, reshapes it into a consistent format, and writes it into a target system.View full definition → job.

3. Do you have enough fresh labeled data to retrain on? In delayed-label domains, you may detect drift long before you have enough new ground truth to train a better model. Sometimes the right move is to fall back to a simpler, more robust model or human review, *not* to retrain on thin data.

4. What is the blast radius of retraining wrong? A new model version is a new deployment with its own risk. It must clear the same validation gates as the original. Retraining under pressure, skipping validation, is how teams replace a slowly-degrading model with a rapidly-failing one.

Netflix and other high-maturity shops resolve this by treating every retrained model as a candidate, not a replacement. The candidate runs in shadow mode, scoring live traffic without affecting decisions, and is compared against the incumbent on live data before promotion. The champion/challenger discipline you know from experimentation applies directly: the incumbent is champion until a challenger *demonstrably* beats it on the metrics that matter, on current data.

Reference data: the quietly critical choice

Every drift metric compares "now" against a "reference." What is your reference window? If you compare against the original training set forever, seasonal businesses will scream "drift!" every December. If you compare against a rolling recent window, you may slowly normalize a genuine degradation, the boiling-frog problem, where the reference drifts along with the data and never triggers. The right answer is usually *both*: a fixed training-reference comparison for absolute drift, and a rolling comparison for velocity. Make this an explicit, documented choice, not a default someone picked in a notebook.

The MLOpsMLOpsMachine Learning Operations: combining ML and DevOps practices to industrialise, deploy, monitor, and retrain models reliably in production.View full definition → a CDO must fund, own, and insist on

You are not writing the monitoring code. Your job is to ensure the *operating system* around models exists, is funded, and has clear owners. Four capabilities separate organizations that operate models from organizations that merely deploy them.

A model registry and lineage system. You must be able to answer, for any prediction that caused a business or regulatory problem: which model version produced it, what data it was trained on, who approved it, and when. If your team cannot reconstruct that in minutes, you have an audit and liability exposure, not just a technical gap. This is the operational spine of the AI governance you already own.

A feature store with training-serving consistency. The single most common production bug is *training-serving skew*: a feature computed one way in the training pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.View full definition → and a subtly different way in the live serving path. The model looks great in validation and underperforms in production, and everyone blames drift. A feature storefeature storeA centralised repository managing ML features, ensuring consistency between training and serving environments.View full definition → that serves the *same computation* to both training and inference eliminates an entire category of silent failure. When a team reports "unexplained drift," skew is the first suspect.

Automated validation gates in the deployment pipeline. No model version, including a retrained one, reaches production without passing tests: performance thresholds, fairness checks across protected segmentssegmentsDividing a market into distinct groups of customers who share similar needs, characteristics or behaviours, so each group can be served with a tailored approach.View full definition →, calibration, and behavioral tests on known edge cases. This is what makes triggered retraining safe. The gate is what stands between "the model degraded" and "we panicked and shipped something worse."

A rollback path. Every deployed model needs a one-click return to the previous version and, ideally, a fallback to a simple, robust baseline (even a business-rules heuristic). When the sophisticated model misbehaves at 2 a.m., the on-call engineer should not be retraining anything. They should be rolling back. Ask your team the direct question: *"If our top revenue model started producing garbage right now, how long until we're safely on the previous version?"* If the answer is more than minutes, that's your Monday priority.

The org design question underneath all of this

Here is the failure pattern that no tool fixes. Data scientists build the model, hand it to engineering, and move to the next project. Six months later the model is degrading and *no one owns it*. The scientists are gone, the engineers don't understand the model logic, and the business only notices when the KPIKPIKey Performance Indicator, a measurable value that shows how effectively you're achieving a specific objective, tracked over time against a target.View full definition → craters.

The fix is organizational, and it's yours to make. Assign every production model a named owner accountable for its ongoing performance, the ML equivalent of a product manager. Establish a model performance review on a regular cadence where owners report drift status, retraining decisions, and business impact, the same way you'd review a product line. And define a model retirement policy: models, like products, reachreachThe number of unique people exposed to your message in a given period. Unlike impressions, reach counts each person once, no matter how often they see it.View full definition → end-of-life. A model no one can justify keeping should be decommissioned, not left running because deleting it feels risky. An inventory of un-owned, un-monitored models is not an asset; it's a set of unexploded liabilities on your balance sheet.

The maturity signal to aim for: your organization detects and responds to a degrading model through instrumentation and process *before* the business unit complains. If the business tells you the model is broken before your monitoring does, your MLOpsMLOpsMachine Learning Operations: combining ML and DevOps practices to industrialise, deploy, monitor, and retrain models reliably in production.View full definition → is decorative.

Key Takeaways

• Monitor in three layers, and never rely on accuracy alone. Input drift (PSI, KS) and prediction drift are your early-warning system; performance metrics arrive too late in any delayed-label domain to be your only defense. The longer your ground-truth latency, the more you must invest upstream.

• Treat retraining as a governed decision, not a reflex. Before retraining, confirm the degradation is real, verify the cause isn't a broken pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.View full definition → (which retraining would bake in), check you have enough fresh labeled data, and route every retrained model through the same validation gates and shadow-mode comparison as a new deployment.

• Eliminate training-serving skew before you blame drift. A feature storefeature storeA centralised repository managing ML features, ensuring consistency between training and serving environments.View full definition → with consistent computation across training and inference removes the most common source of "unexplained" production degradation. Skew should be the first suspect, not drift.

• Fund the four MLOps non-negotiables: a model registry with lineage, a feature storefeature storeA centralised repository managing ML features, ensuring consistency between training and serving environments.View full definition →, automated validation gates, and a fast rollback path to a robust baseline. Ask your team how many minutes it takes to roll back your top revenue model, the answer reveals your true maturity.

• Assign a named owner to every production model and run a model performance review. Un-owned models rot silently. Your target maturity: instrumentation flags a degrading model before the business unit ever complains.