The pre-deployment checklist: guardrails that catch failures early, MBA Training, MBA Training
3/4+150 XP
The pre-deployment checklist: guardrails that catch failures early
# The pre-deployment checklist: guardrails that catch failures early
A vision system at a European automotive stamping plant once passed every offline accuracy test, 99.2% detection of surface defects, then missed a batch of micro-cracks on the line because the plant's lighting rig ran at a different color temperature than the lab. The defective panels shipped for three shifts before a line worker caught the pattern by eye. Nothing was wrong with the model's math. Everything was wrong with what got checked before go-live.
This is the core lesson of this module: most AI failures in manufacturing are not algorithm failures, they are governance failures. A pre-deployment checklist is how you catch them before they touch the line.
Why checklists, not just model metrics
Model accuracy on a validation set tells you how the system performed on data it already knew about. It says nothing about:
Drift between lab conditions and shop-floor conditions (lighting, vibration, sensor calibration).
What happens when the model is confident and wrong.
Who can override it, and how fast.
What the plant does when the system goes down entirely.
Regulators are converging on this same view. The EU AI Act (in force since August 2024, with phased obligations through 2026-2027) classifies many industrial AI uses, like safety components in machinery, as "high-risk," requiring documented risk management, human oversight, and logging before deployment. In the US, there is no single federal AI law, but OSHA (Occupational Safety and Health Administration) still holds employers liable for a "recognized hazard" if an AI-driven process injures a worker, and NIST's AI Risk Management Framework (free, from NIST
) is becoming the de facto audit standard cited in insurance and procurement contracts.
The four-part checklist
1. Data validation: is the input trustworthy on day one?
Before any model touches production, confirm:
Sensor and label provenance: where did the training data come from, and does it match the target line's equipment generation, camera angle, and environment?
Distribution check: run the incoming production data stream through a statistical comparison (a simple population stability index or PSI works) against training data. A PSI above roughly 0.25 is a common industry rule of thumb for "material drift, investigate before trusting predictions."
Edge case coverage: does the dataset include rare but real conditions, night shift lighting, seasonal humidity, a new supplier's raw material batch?
A simple drift check in Python, the kind a plant data team can run weekly:
2. Human-in-the-loop overrides: who can stop the machine, and how fast?
"Human-in-the-loop" (HITL) means a person reviews or can veto the AI's decision before or during action, rather than after the fact.
Checklist items:
A named override authority on every shift, not just during business hours.
Override latency target: how many seconds from "operator sees a problem" to "system halts"? Siemens and other automation vendors typically design safety-critical override loops for sub-second response; if your AI layer adds delay to that, it needs its own budget.
No silent overrides: every override should be logged with a reason code, feeding back into retraining and audit trails (this logging requirement is explicit under the EU AI Act's Article 12 record-keeping obligations for high-risk systems).
3. Fallback-to-manual: what happens when the AI goes dark?
Every plant rollout needs a documented, rehearsed manual fallback. Ask:
Can the line run at reduced speed or in manual inspection mode without the AI, and for how long, before it must stop?
Is there a trained human backup for the specific task (defect sorting, predictive maintenance alerts, robotic path planning)?
Has the fallback actually been tested, not just written down? A fallback procedure nobody has rehearsed is a paper guardrail.
A useful benchmark from lean manufacturing risk practice: treat AI system downtime like any other critical equipment failure mode in your FMEA (Failure Mode and Effects Analysis), with a defined Mean Time to Fallback (MTTF) target, commonly under 5 minutes for inline quality systems in high-throughput automotive or electronics lines (this is a practical estimate, not a universal regulatory figure).
4. Performance thresholds: define "good enough" before launch, not after
Set numeric thresholds in advance, written into the deployment sign-off:
Minimum precision and recall for the specific defect class, not just overall accuracy (a model can be 99% "accurate" while missing the 1% of defects that matter most).
False negative ceiling: for safety-relevant detection, false negatives (missed defects, missed hazards) usually matter more than false positives. Set a hard ceiling and a rollback trigger if it's breached in production monitoring.
Confidence-based routing: predictions below a set confidence score (for example, under 85%, calibrated per use case) route automatically to human review rather than autonomous action.
Vérification des acquis
1. The vision system in the automotive stamping plant example had 99.2% detection accuracy in offline testing but failed on the line. What does this scenario primarily illustrate?
2. Why does the lesson argue that most AI failures in manufacturing are 'governance failures' rather than 'algorithm failures'?
3. A plant manager wants to deploy a defect-detection model that scored well on a validation dataset. According to the lesson, what additional step is essential before go-live?
CHOIX MULTIPLES
4. Select ALL correct answers about what model accuracy on a validation set fails to reveal, according to the lesson.
Sélectionnez toutes les réponses correctes.
CHOIX MULTIPLES
5. Select ALL correct answers about the regulatory landscape described in the lesson.
Sélectionnez toutes les réponses correctes.
Putting it together: a rollout gate, not a launch date
Leading manufacturers (Siemens, Bosch, and Toyota's production system teams have all published elements of this approach in various technical and sustainability disclosures) increasingly treat AI deployment like a Process FMEA gate: the system doesn't go live on a calendar date, it goes live when it clears a checklist, sequentially:
1. Data validation passed and documented.
2. HITL override tested with real operators, not just engineers.
3. Fallback-to-manual rehearsed under a simulated outage.
4. Performance thresholds set, monitored, and tied to an automatic rollback trigger.
This sequencing matters because skipping step 2 or 3 is the most common real-world failure. Teams pilot models extensively (step 1 and step 4) but treat the human backup as an afterthought, then discover during a real outage that nobody remembers the manual procedure.
🎬 [VIDEO: "Human-in-the-Loop AI: Why It Matters in Manufacturing" - youtube.com - search for recent manufacturing-AI conference talks covering HITL design patterns and override architecture on production lines]
A note on governance ownership
None of this works without a named owner. Under the EU AI Act, high-risk AI system providers and deployers must designate responsible parties for risk management and post-market monitoring. Even outside the EU, plants report better outcomes when one named quality or safety engineer, not a vendor, not a data science team alone, owns the pre-deployment checklist and has authority to delay a launch.
Key Takeaways
Model accuracy metrics are necessary but not sufficient; most manufacturing AI failures come from environment mismatch, unclear override authority, or untested fallback procedures, not bad algorithms.
Run a four-part checklist before any production deployment: data validation (including drift checks like PSI), human-in-the-loop override design with latency targets, rehearsed fallback-to-manual procedures, and pre-agreed performance thresholds with automatic rollback triggers.
Regulatory frameworks now expect this in writing: the EU AI Act requires documented risk management and logging for high-risk industrial AI, and NIST's AI RMF is becoming the reference standard even for US plants without a direct legal mandate.
Treat deployment as a gate, not a date: sequence the checklist so the system launches only when every guardrail has been tested, not just designed.
Assign a single named owner with authority to delay launch. Checklists without ownership become paperwork; checklists with ownership become guardrails.