# Model risk on the line: when AI drifts, breaks, or misleads
At 2 a.m. on a Tuesday, a defect-detection camera system on an automotive stamping line stops flagging cracked brackets. Not because it broke. Because a new steel supplier changed the surface sheen just enough that the model's training data no longer matches reality. Six hours and 4,000 parts later, someone notices the reject rate dropped to zero, which is the giveaway: a healthy line rejects some parts. Nobody wanted this. Nobody coded it in. It happened because the model quietly drifted, and no one was watching.
This is model risk: the chance that an AI system produces wrong, biased, or unreliable outputs that cause financial, safety, or reputational harm. In manufacturing, model risk does not stay in a dashboard. It becomes scrap, recalls, missed shipments, or injured workers.
Manufacturing AI risk differs from, say, a retail recommendation engine, in three ways:
1. Defect-detection drift. A computer vision model trained on images from one lighting rig and one steel supplier starts missing defects when the plant switches suppliers or the camera lens gets a light film of oil. The model's accuracy on paper (from validation testing) says 98%. Its accuracy on the floor, six months later, might be materially lower, and nobody re-measured it.
2. Demand-forecast blindness to shocks. A forecasting model trained on five years of stable demand data has never seen a semiconductor shortage, a port closure, or a geopolitical export ban. When the 2021 to 2022 chip shortage hit automakers, models trained on pre-shock patterns underpredicted lead times and overpredicted supply availability, because the shock was outside the training distribution. This is a tail risk problem: models are typically strong on the "middle" of the distribution and weak on outliers they've never seen.
Borrow from the US Federal Reserve and OCC's SR 11-7 guidance on model risk management (originally written for banks but widely adapted across industries), which frames model risk along two axes:
1. Likelihood of model error (how often does it happen, how easily is it triggered by real-world change)
2. Severity of consequence (safety, cost, compliance, reputation)
| Risk tier | Example | Action |
|---|---|---|
| Low likelihood, low severity | Minor SKU forecast miss on a slow-moving part | Monitor quarterly |
| High likelihood, low severity | Seasonal demand model needs retraining every quarter | Scheduled retraining |
| Low likelihood, high severity | Vision model misses a structural crack in an aerospace part | Human-in-the-loop mandatory, redundant checks |
| High likelihood, high severity | Forecasting model exposed to frequent supply shocks with no fallback | Do not deploy without a rules-based override |
Anything in the top-right quadrant (high severity) needs documented human oversight before it ever touches production, regardless of accuracy scores.
You do not need exotic math to catch drift early. A common, accessible metric is Population Stability Index (PSI), which measures how much a variable's distribution has shifted between two time periods.
PSI = Σ (Actual% - Expected%) × ln(Actual% / Expected%)Rule of thumb interpretation (widely used in credit risk and adapted for ops monitoring):
Worked example: Say your defect model was trained when 5% of parts had surface scratches (the "expected" distribution). This month, incoming QA samples show 18% scratch rate because of the new steel supplier.
A single-bucket PSI of 0.166 already crosses into "moderate shift, investigate" territory. Run this across all key input features monthly, and you catch drift before it becomes a recall.
Manufacturing AI does not sit in a regulatory vacuum, even though there's no single "manufacturing AI law."
The practical takeaway for a plant or ops leader: even absent a specific mandate, document your model risk process. Regulators and auditors increasingly ask "can you show me you tested for drift" rather than "do you have an AI license."
Vérification des acquis
1. In the defect-detection example, why did a zero percent reject rate signal a problem rather than a success?
2. What is the key distinction between data drift and concept drift?
3. Why are AI models in manufacturing structurally weak at predicting rare, catastrophic failures like fires or structural defects?
4. Select ALL correct answers describing why manufacturing is a distinct AI risk environment compared to something like a retail recommendation engine.
Sélectionnez toutes les réponses correctes.
5. Select ALL correct answers about model risk in a manufacturing context.
Sélectionnez toutes les réponses correctes.
Before any model touches a live line, run these checks:
1. Out-of-distribution testing. Feed the model edge cases it wasn't trained on (new supplier batches, extreme lighting, discontinued SKUs) and confirm it flags low confidence rather than guessing silently.
2. Human-in-the-loop thresholds. Define confidence cutoffs below which a human, not the model, makes the call. For high-severity contexts (aerospace, automotive safety parts), this should be non-negotiable.
3. Shadow deployment. Run the new model in parallel with the existing process (or human inspectors) for a defined period before it takes over decisions. Compare outputs before trusting outcomes.
4. Monitoring cadence. Set PSI or equivalent drift metrics on a schedule (weekly for high-velocity lines, monthly for stable ones), not just at initial validation.
5. Rollback plan. Know, in advance, how to revert to the prior model or manual process if drift crosses your threshold. This should be a documented runbook, not an improvised Slack thread during a crisis.