+65 XP

Models in production: drift, monitoring & MLOps

Predictive models in production are not a one-time deployment, they're a living system that requires ongoing care. A model that performs well at launch will degrade over time. The world changes. Customer behavior shifts. Product features evolve. The data distribution the model was trained on diverges from the data it sees in production.

This is called model drift. It is inevitable. Managing it is an operational discipline.

Types of model drift

Data drift (covariate shift): The statistical distribution of input features changes. A fraud detection model trained on 2020 purchase patterns encounters 2024 behavior. The model still "works" mechanically, but its predictions are based on patterns that no longer hold.

Concept drift: The relationship between features and the target changes. A pricing model trained when interest rates were 2% faces a world where rates are 5%. The model's learned relationships are no longer correct.

Label drift: The definition of the target variable changes. If "conversion" was redefined to exclude trial conversions that don't activate within 48 hours, all historical labels are now inconsistent with the new definition.

Each requires different detection and remediation approaches.

Machine Learning Model Monitoring in Production

Watch on YouTube

Knowledge check

1. A fraud detection model was trained on 2020 purchase patterns and now processes 2024 transactions. The input feature distributions have shifted, but the underlying relationship between features and fraud remains the same. What type of drift is this?

2. Why does the lesson describe a production model as a 'living system' rather than a one-time deployment?

3. A pricing model trained when interest rates were 2% now operates in a 5% environment, and its learned relationships between features and price are no longer correct. This best illustrates:

MULTIPLE CHOICE

4. Select ALL correct statements about performance monitoring of a production model.

Select all the correct answers.

MULTIPLE CHOICE

5. Select ALL statements that correctly describe the difference between output monitoring and business impact monitoring.

Select all the correct answers.

Monitoring production models

A production model should be monitored continuously on multiple dimensions:

Input monitoring: Are the input feature distributions stable? Alert when features drift beyond historical bounds (using statistical tests like KS-test or PSI, Population Stability Index).

Output monitoring: Is the distribution of predictions stable? A churn model that suddenly predicts 80% churn for all customers when it previously predicted 15% is signaling a problem.

Performance monitoring: Is the model still accurate against known outcomes? Requires labeled data, you need to know what actually happened (did the customer churn?) to evaluate the prediction (was the model right?). This creates a labeling lag, you can't evaluate fraud predictions until you know which transactions were actually fraudulent.

Business impact monitoring: Are the downstream business metrics that the model is supposed to improve still moving in the right direction? Model accuracy is a proxy, business impact is the actual goal.

The retraining strategy

When monitoring detects significant drift, the model needs retraining. But retraining strategy matters:

Scheduled retraining: Retrain on a fixed schedule (weekly, monthly) regardless of drift signals. Simple, predictable, but potentially wasteful (retraining when not needed) or slow (waiting for the schedule when drift is significant).

Triggered retraining: Retrain when monitoring detects drift above a threshold. More efficient, but requires robust drift detection and automated retraining pipelines.

Online learning: Continuously update the model as new labeled data arrives. Highest complexity, but appropriate for high-velocity applications (ad bid optimization, real-time personalization).

Most production ML systems use scheduled retraining for simplicity, with manual drift monitoring. Triggered retraining requires MLOps maturity.

MLOps: the operational layer

MLOps (ML Operations) is the set of practices that make ML models reliable and scalable in production.

Core MLOps capabilities:

  • Model registry: Versioned storage of trained models with metadata (training data, hyperparameters, performance metrics)
  • CI/CD for models: Automated testing and deployment pipelines for model updates
  • Feature store: Centralized repository of features used across multiple models, ensuring consistency between training and serving
  • Model serving: Infrastructure to serve model predictions at scale with low latency
  • Monitoring and alerting: Continuous performance monitoring with automated alerts on drift

Tools: MLflow (open-source, model registry + experiment tracking), Weights & Biases (experiment tracking), Kubeflow (Kubernetes-native MLOps), Vertex AI (Google managed MLOps), SageMaker (AWS managed MLOps).

Quiz Questions

  1. Qu'est-ce que le "concept drift" dans un modèle ML en production ?

A) Les données d'entrée changent de distribution

B) La relation entre les features et la variable cible change (ex: un modèle de pricing formé à 2% de taux d'intérêt face à un monde à 5%)

C) Le modèle consomme trop de mémoire

D) Les prédictions du modèle deviennent plus lentes

Réponse: B

  1. Qu'est-ce qu'un "Feature Store" dans l'architecture MLOps ?

A) Un outil de monitoring de modèles

B) Un repository centralisé de features utilisées par plusieurs modèles, garantissant la cohérence entre l'entraînement et le serving

C) Une base de données de stockage de modèles entraînés

D) Un outil d'A/B testing pour les modèles ML

Réponse: B

  1. Quelle stratégie de réentraînement est la plus complexe mais appropriée pour les applications à haute vélocité ?

A) Réentraînement planifié (hebdomadaire)

B) Réentraînement déclenché par détection de drift

C) Online learning (mise à jour continue)

D) Réentraînement manuel

Réponse: C

Frequently asked questions

What is model drift and why does it happen?

Model drift is the degradation of a predictive model's performance once it runs in production, because the data it sees diverges from the data it was trained on. Customer behavior shifts, product features change, market conditions move. A fraud model trained on 2020 purchase patterns faces 2024 behavior that no longer matches. Drift is inevitable, so managing it is an operational discipline rather than a bug to fix once.

What is the difference between data drift and concept drift?

Data drift (covariate shift) means the statistical distribution of the input features changes while the underlying relationships stay valid. Concept drift means the relationship between features and the target itself changes, as with a pricing model trained when interest rates were 2% now operating at 5%. The distinction matters because detection and remediation differ: data drift shows up in input monitoring, concept drift only surfaces when you compare predictions to actual outcomes.

What should you monitor on a machine learning model in production?

Four dimensions: input feature distributions, prediction distributions, accuracy against known outcomes, and downstream business metrics. A churn model that suddenly predicts 80% churn when it used to predict 15% signals a problem on output monitoring alone. Business impact monitoring matters most, since model accuracy is only a proxy for the result you actually want.

How do you detect data drift statistically?

Compare the current distribution of each input feature against its historical reference and alert when the gap exceeds a threshold. The two usual tests are the Kolmogorov-Smirnov test and the Population Stability Index (PSI). The same logic applies to the distribution of model outputs, which is often the fastest early warning.

Scheduled retraining or triggered retraining: which should you choose?

Scheduled retraining (weekly, monthly) is simple and predictable but can be wasteful when there is no drift, or too slow when drift is severe. Triggered retraining fires only when monitoring crosses a drift threshold, which is more efficient but demands reliable drift detection and automated pipelines. Most production systems start with scheduled retraining and manual drift monitoring; triggered retraining requires real MLOps maturity. Online learning, which updates the model continuously, fits high-velocity cases like ad bid optimization or real-time personalization.

What does a feature store do that a model registry does not?

A model registry stores versioned trained models with their metadata: training data, hyperparameters, performance metrics. A feature store centralizes the features consumed by several models and guarantees that the same computation is used at training time and at serving time, which removes a classic source of silent production errors. Both are standard MLOps components, alongside CI/CD for models, model serving and monitoring. MLflow covers registry and experiment tracking; Kubeflow, Vertex AI and SageMaker cover broader MLOps platforms.

What to do, from this lesson

These actions are compiled in the role's Playbook.

  • Monitor production models across input, prediction, and performance layers with named owners
See the full action playbook →

Related articles

Recent articles from the blog that build on this lesson.