Predictive models in production are not a one-time deployment, they're a living system that requires ongoing care. A model that performs well at launch will degrade over time. The world changes. Customer behavior shifts. Product features evolve. The data distribution the model was trained on diverges from the data it sees in production.
This is called model drift. It is inevitable. Managing it is an operational discipline.
Data drift (covariate shift): The statistical distribution of input features changes. A fraud detection model trained on 2020 purchase patterns encounters 2024 behavior. The model still "works" mechanically, but its predictions are based on patterns that no longer hold.
Concept drift: The relationship between features and the target changes. A pricing model trained when interest rates were 2% faces a world where rates are 5%. The model's learned relationships are no longer correct.
Label drift: The definition of the target variable changes. If "conversion" was redefined to exclude trial conversions that don't activate within 48 hours, all historical labels are now inconsistent with the new definition.
Each requires different detection and remediation approaches.
Machine Learning Model Monitoring in Production
Knowledge check
1. According to the lesson, what is 'concept drift'?
2. Why does performance monitoring create a 'labeling lag' in fraud detection?
3. The lesson mentions PSI (Population Stability Index) as a tool for input monitoring. What does PSI primarily measure?
4. Select ALL dimensions on which a production model should be monitored according to the lesson.
Sélectionnez toutes les réponses correctes.
5. Select ALL statements that are TRUE about model drift as described in the lesson.
Sélectionnez toutes les réponses correctes.
A production model should be monitored continuously on multiple dimensions:
Input monitoring: Are the input feature distributions stable? Alert when features drift beyond historical bounds (using statistical tests like KS-test or PSI, Population Stability Index).
Output monitoring: Is the distribution of predictions stable? A churn model that suddenly predicts 80% churn for all customers when it previously predicted 15% is signaling a problem.
Performance monitoring: Is the model still accurate against known outcomes? Requires labeled data, you need to know what actually happened (did the customer churncustomer churnChurn rate is the percentage of customers or revenue lost over a period. It measures how fast a business loses its existing customer base.View full definition →?) to evaluate the prediction (was the model right?). This creates a labeling lag, you can't evaluate fraud predictions until you know which transactions were actually fraudulent.
Business impact monitoring: Are the downstream business metrics that the model is supposed to improve still moving in the right direction? Model accuracy is a proxy, business impact is the actual goal.
When monitoring detects significant drift, the model needs retraining. But retraining strategy matters:
Scheduled retraining: Retrain on a fixed schedule (weekly, monthly) regardless of drift signals. Simple, predictable, but potentially wasteful (retraining when not needed) or slow (waiting for the schedule when drift is significant).
Triggered retraining: Retrain when monitoring detects drift above a threshold. More efficient, but requires robust drift detection and automated retraining pipelines.
Online learning: Continuously update the model as new labeled data arrives. Highest complexity, but appropriate for high-velocity applications (ad bid optimization, real-time personalization).
Most production ML systems use scheduled retraining for simplicity, with manual drift monitoring. Triggered retraining requires MLOpsMLOpsMachine Learning Operations: combining ML and DevOps practices to industrialise, deploy, monitor, and retrain models reliably in production.View full definition → maturity.
MLOpsMLOpsMachine Learning Operations: combining ML and DevOps practices to industrialise, deploy, monitor, and retrain models reliably in production.View full definition → (ML Operations) is the set of practices that make ML models reliable and scalable in production.
Core MLOpsMLOpsMachine Learning Operations: combining ML and DevOps practices to industrialise, deploy, monitor, and retrain models reliably in production.View full definition → capabilities:
Tools: MLflow (open-source, model registry + experiment tracking), Weights & Biases (experiment tracking), Kubeflow (Kubernetes-native MLOpsMLOpsMachine Learning Operations: combining ML and DevOps practices to industrialise, deploy, monitor, and retrain models reliably in production.View full definition →), Vertex AI (Google managed MLOpsMLOpsMachine Learning Operations: combining ML and DevOps practices to industrialise, deploy, monitor, and retrain models reliably in production.View full definition →), SageMaker (AWS managed MLOpsMLOpsMachine Learning Operations: combining ML and DevOps practices to industrialise, deploy, monitor, and retrain models reliably in production.View full definition →).
1. Qu'est-ce que le "concept drift" dans un modèle ML en production ?
A) Les données d'entrée changent de distribution
B) La relation entre les features et la variable cible change (ex: un modèle de pricing formé à 2% de taux d'intérêt face à un monde à 5%)
C) Le modèle consomme trop de mémoire
D) Les prédictions du modèle deviennent plus lentes
Réponse: B
2. Qu'est-ce qu'un "Feature StoreFeature StoreA centralised repository managing ML features, ensuring consistency between training and serving environments.View full definition →" dans l'architecture MLOpsMLOpsMachine Learning Operations: combining ML and DevOps practices to industrialise, deploy, monitor, and retrain models reliably in production.View full definition → ?
A) Un outil de monitoring de modèles
B) Un repository centralisé de features utilisées par plusieurs modèles, garantissant la cohérence entre l'entraînement et le serving
C) Une base de données de stockage de modèles entraînés
D) Un outil d'A/B testingA/B testingA/B testing is a controlled experiment that compares two versions of something (A and B) by splitting traffic randomly to learn which performs better on a chosen metric.View full definition → pour les modèles ML
Réponse: B
3. Quelle stratégie de réentraînement est la plus complexe mais appropriée pour les applications à haute vélocité ?
A) Réentraînement planifié (hebdomadaire)
B) Réentraînement déclenché par détection de drift
C) Online learning (mise à jour continue)
D) Réentraînement manuel
Réponse: C