Models in production: drift, monitoring & MLOps

Monitoring production models

A production model should be monitored continuously on multiple dimensions:

Input monitoring: Are the input feature distributions stable? Alert when features drift beyond historical bounds (using statistical tests like KS-test or PSI, Population Stability Index).

Output monitoring: Is the distribution of predictions stable? A churn model that suddenly predicts 80% churn for all customers when it previously predicted 15% is signaling a problem.

Performance monitoring: Is the model still accurate against known outcomes? Requires labeled data, you need to know what actually happened (did the customer churncustomer churnChurn rate is the percentage of customers or revenue lost over a period. It measures how fast a business loses its existing customer base.View full definition →?) to evaluate the prediction (was the model right?). This creates a labeling lag, you can't evaluate fraud predictions until you know which transactions were actually fraudulent.

Business impact monitoring: Are the downstream business metrics that the model is supposed to improve still moving in the right direction? Model accuracy is a proxy, business impact is the actual goal.

The retraining strategy

When monitoring detects significant drift, the model needs retraining. But retraining strategy matters:

Scheduled retraining: Retrain on a fixed schedule (weekly, monthly) regardless of drift signals. Simple, predictable, but potentially wasteful (retraining when not needed) or slow (waiting for the schedule when drift is significant).

Triggered retraining: Retrain when monitoring detects drift above a threshold. More efficient, but requires robust drift detection and automated retraining pipelines.

Online learning: Continuously update the model as new labeled data arrives. Highest complexity, but appropriate for high-velocity applications (ad bid optimization, real-time personalization).

Most production ML systems use scheduled retraining for simplicity, with manual drift monitoring. Triggered retraining requires MLOpsMLOpsMachine Learning Operations: combining ML and DevOps practices to industrialise, deploy, monitor, and retrain models reliably in production.View full definition → maturity.

MLOpsMLOpsMachine Learning Operations: combining ML and DevOps practices to industrialise, deploy, monitor, and retrain models reliably in production.View full definition →: the operational layer

MLOpsMLOpsMachine Learning Operations: combining ML and DevOps practices to industrialise, deploy, monitor, and retrain models reliably in production.View full definition → (ML Operations) is the set of practices that make ML models reliable and scalable in production.

Core MLOpsMLOpsMachine Learning Operations: combining ML and DevOps practices to industrialise, deploy, monitor, and retrain models reliably in production.View full definition → capabilities:

• Model registry: Versioned storage of trained models with metadata (training data, hyperparameters, performance metrics)
• CI/CD for models: Automated testing and deployment pipelines for model updates
• Feature store: Centralized repository of features used across multiple models, ensuring consistency between training and serving
• Model serving: Infrastructure to serve model predictions at scale with low latency
• Monitoring and alerting: Continuous performance monitoring with automated alerts on drift

Tools: MLflow (open-source, model registry + experiment tracking), Weights & Biases (experiment tracking), Kubeflow (Kubernetes-native MLOpsMLOpsMachine Learning Operations: combining ML and DevOps practices to industrialise, deploy, monitor, and retrain models reliably in production.View full definition →), Vertex AI (Google managed MLOpsMLOpsMachine Learning Operations: combining ML and DevOps practices to industrialise, deploy, monitor, and retrain models reliably in production.View full definition →), SageMaker (AWS managed MLOpsMLOpsMachine Learning Operations: combining ML and DevOps practices to industrialise, deploy, monitor, and retrain models reliably in production.View full definition →).

Quiz Questions

1. Qu'est-ce que le "concept drift" dans un modèle ML en production ?

A) Les données d'entrée changent de distribution

B) La relation entre les features et la variable cible change (ex: un modèle de pricing formé à 2% de taux d'intérêt face à un monde à 5%)

C) Le modèle consomme trop de mémoire

D) Les prédictions du modèle deviennent plus lentes

Réponse: B

2. Qu'est-ce qu'un "Feature StoreFeature StoreA centralised repository managing ML features, ensuring consistency between training and serving environments.View full definition →" dans l'architecture MLOpsMLOpsMachine Learning Operations: combining ML and DevOps practices to industrialise, deploy, monitor, and retrain models reliably in production.View full definition → ?

A) Un outil de monitoring de modèles

B) Un repository centralisé de features utilisées par plusieurs modèles, garantissant la cohérence entre l'entraînement et le serving

C) Une base de données de stockage de modèles entraînés

D) Un outil d'A/B testingA/B testingA/B testing is a controlled experiment that compares two versions of something (A and B) by splitting traffic randomly to learn which performs better on a chosen metric.View full definition → pour les modèles ML

Réponse: B

3. Quelle stratégie de réentraînement est la plus complexe mais appropriée pour les applications à haute vélocité ?

A) Réentraînement planifié (hebdomadaire)

B) Réentraînement déclenché par détection de drift

C) Online learning (mise à jour continue)

D) Réentraînement manuel

Réponse: C