Models in production: drift, monitoring & MLOps
Predictive models in production are not a one-time deployment, they're a living system that requires ongoing care. A model that performs well at launch will degrade over time. The world changes. Customer behavior shifts. Product features evolve. The data distribution the model was trained on diverges from the data it sees in production.
This is called model drift. It is inevitable. Managing it is an operational discipline.
Types of model drift
Data drift (covariate shift): The statistical distribution of input features changes. A fraud detection model trained on 2020 purchase patterns encounters 2024 behavior. The model still "works" mechanically, but its predictions are based on patterns that no longer hold.
Concept drift: The relationship between features and the target changes. A pricing model trained when interest rates were 2% faces a world where rates are 5%. The model's learned relationships are no longer correct.
Label drift: The definition of the target variable changes. If "conversion" was redefined to exclude trial conversions that don't activate within 48 hours, all historical labels are now inconsistent with the new definition.
Each requires different detection and remediation approaches.
Machine Learning Model Monitoring in Production
Knowledge check
1. A fraud detection model was trained on 2020 purchase patterns and now processes 2024 transactions. The input feature distributions have shifted, but the underlying relationship between features and fraud remains the same. What type of drift is this?
2. Why does the lesson describe a production model as a 'living system' rather than a one-time deployment?
3. A pricing model trained when interest rates were 2% now operates in a 5% environment, and its learned relationships between features and price are no longer correct. This best illustrates:
4. Select ALL correct statements about performance monitoring of a production model.
Select all the correct answers.
5. Select ALL statements that correctly describe the difference between output monitoring and business impact monitoring.
Select all the correct answers.
Monitoring production models
A production model should be monitored continuously on multiple dimensions:
Input monitoring: Are the input feature distributions stable? Alert when features drift beyond historical bounds (using statistical tests like KS-test or PSI, Population Stability Index).
Output monitoring: Is the distribution of predictions stable? A churn model that suddenly predicts 80% churn for all customers when it previously predicted 15% is signaling a problem.
Performance monitoring: Is the model still accurate against known outcomes? Requires labeled data, you need to know what actually happened (did the customer churncustomer churnChurn rate is the percentage of customers or revenue lost over a period. It measures how fast a business loses its existing customer base.View full definition →?) to evaluate the prediction (was the model right?). This creates a labeling lag, you can't evaluate fraud predictions until you know which transactions were actually fraudulent.
Business impact monitoring: Are the downstream business metrics that the model is supposed to improve still moving in the right direction? Model accuracy is a proxy, business impact is the actual goal.
The retraining strategy
When monitoring detects significant drift, the model needs retraining. But retraining strategy matters:
Scheduled retraining: Retrain on a fixed schedule (weekly, monthly) regardless of drift signals. Simple, predictable, but potentially wasteful (retraining when not needed) or slow (waiting for the schedule when drift is significant).
Triggered retraining: Retrain when monitoring detects drift above a threshold. More efficient, but requires robust drift detection and automated retraining pipelines.
Online learning: Continuously update the model as new labeled data arrives. Highest complexity, but appropriate for high-velocity applications (ad bid optimization, real-time personalization).
Most production ML systems use scheduled retraining for simplicity, with manual drift monitoring. Triggered retraining requires MLOps maturity.
MLOpsMLOpsMachine Learning Operations: combining ML and DevOps practices to industrialise, deploy, monitor, and retrain models reliably in production.View full definition →: the operational layer
MLOps (ML Operations) is the set of practices that make ML models reliable and scalable in production.
Core MLOps capabilities:
- Model registry: Versioned storage of trained models with metadata (training data, hyperparameters, performance metrics)
- CI/CD for models: Automated testing and deployment pipelines for model updates
- Feature store: Centralized repository of features used across multiple models, ensuring consistency between training and serving
- Model serving: Infrastructure to serve model predictions at scale with low latency
- Monitoring and alerting: Continuous performance monitoring with automated alerts on drift
Tools: MLflow (open-source, model registry + experiment tracking), Weights & Biases (experiment tracking), Kubeflow (Kubernetes-native MLOps), Vertex AI (Google managed MLOps), SageMaker (AWS managed MLOps).
Quiz Questions
- Qu'est-ce que le "concept drift" dans un modèle ML en production ?
A) Les données d'entrée changent de distribution
B) La relation entre les features et la variable cible change (ex: un modèle de pricing formé à 2% de taux d'intérêt face à un monde à 5%)
C) Le modèle consomme trop de mémoire
D) Les prédictions du modèle deviennent plus lentes
Réponse: B
- Qu'est-ce qu'un "Feature StoreFeature StoreA centralised repository managing ML features, ensuring consistency between training and serving environments.View full definition →" dans l'architecture MLOps ?
A) Un outil de monitoring de modèles
B) Un repository centralisé de features utilisées par plusieurs modèles, garantissant la cohérence entre l'entraînement et le serving
C) Une base de données de stockage de modèles entraînés
D) Un outil d'A/B testingA/B testingA/B testing is a controlled experiment that compares two versions of something (A and B) by splitting traffic randomly to learn which performs better on a chosen metric.View full definition → pour les modèles ML
Réponse: B
- Quelle stratégie de réentraînement est la plus complexe mais appropriée pour les applications à haute vélocité ?
A) Réentraînement planifié (hebdomadaire)
B) Réentraînement déclenché par détection de drift
C) Online learning (mise à jour continue)
D) Réentraînement manuel
Réponse: C
Frequently asked questions
What is model drift and why does it happen?
Model drift is the degradation of a predictive model's performance once it runs in production, because the data it sees diverges from the data it was trained on. Customer behavior shifts, product features change, market conditions move. A fraud model trained on 2020 purchase patterns faces 2024 behavior that no longer matches. Drift is inevitable, so managing it is an operational discipline rather than a bug to fix once.
What is the difference between data drift and concept drift?
Data drift (covariate shift) means the statistical distribution of the input features changes while the underlying relationships stay valid. Concept drift means the relationship between features and the target itself changes, as with a pricing model trained when interest rates were 2% now operating at 5%. The distinction matters because detection and remediation differ: data drift shows up in input monitoring, concept drift only surfaces when you compare predictions to actual outcomes.
What should you monitor on a machine learning model in production?
Four dimensions: input feature distributions, prediction distributions, accuracy against known outcomes, and downstream business metrics. A churn model that suddenly predicts 80% churn when it used to predict 15% signals a problem on output monitoring alone. Business impact monitoring matters most, since model accuracy is only a proxy for the result you actually want.
How do you detect data drift statistically?
Compare the current distribution of each input feature against its historical reference and alert when the gap exceeds a threshold. The two usual tests are the Kolmogorov-Smirnov test and the Population Stability Index (PSI). The same logic applies to the distribution of model outputs, which is often the fastest early warning.
Scheduled retraining or triggered retraining: which should you choose?
Scheduled retraining (weekly, monthly) is simple and predictable but can be wasteful when there is no drift, or too slow when drift is severe. Triggered retraining fires only when monitoring crosses a drift threshold, which is more efficient but demands reliable drift detection and automated pipelines. Most production systems start with scheduled retraining and manual drift monitoring; triggered retraining requires real MLOps maturity. Online learning, which updates the model continuously, fits high-velocity cases like ad bid optimization or real-time personalization.
What does a feature store do that a model registry does not?
A model registry stores versioned trained models with their metadata: training data, hyperparameters, performance metrics. A feature store centralizes the features consumed by several models and guarantees that the same computation is used at training time and at serving time, which removes a classic source of silent production errors. Both are standard MLOps components, alongside CI/CD for models, model serving and monitoring. MLflow covers registry and experiment tracking; Kubeflow, Vertex AI and SageMaker cover broader MLOps platforms.
What to do, from this lesson
These actions are compiled in the role's Playbook.
- Monitor production models across input, prediction, and performance layers with named owners
Related articles
Recent articles from the blog that build on this lesson.
- DataHow did Telefónica build a churn model that actually moved retention numbers?Telefónica's data teams spent years accumulating subscriber signals before their churn models started producing revenue-grade predictions. The mechanics of what they built, and where other telcos consistently fall short, carry direct lessons for any CDO running a retention program in 2026.
- DataThe ML-to-production field guide: who shapes what breaks and whyMost ML projects die somewhere between a promising notebook and a live system. This field guide identifies the people, projects, and organisations whose work reveals exactly where the gaps are and what serious practitioners do about them.
- DataFeature stores and the ML data supply chain: what CDOs actually need to understandFeature stores sit at the intersection of data engineering and machine learning operations, yet most organizations treat them as a tooling decision rather than a strategic one. This article explains how they work, why the architectural choice matters at the CDO level, and where the tradeoff between standardization and flexibility bites hardest.