# Evaluating AI vendors and pilots before you scale
Three vendors just pitched the same use case: computer vision to catch mislabeled packages on a bottling line. Vendor A demoed 99.7% accuracy. Vendor B quoted a two-week install. Vendor C offered the lowest price per camera. Six months later, only one of these deployments would still be running. This lesson gives you the scorecard to know which one, before you sign.
Why demos lie (a little)
Every vendor demo works. That is the point of a demo. The model was trained and tuned on curated footage, often shot in the vendor's own lab or on a client's best-case production day.
The real question is not "does it work on stage." It is "does it work on your line, with your lighting, your label stock, your changeover frequency, three months from now, at 2 a.m. on the night shift."
This gap between demo performance and production performance is sometimes called "demo-ware": a system that looks production-ready but was never stress-tested against the messiness of a real plant. Manufacturing is especially exposed to this because conditions vary line to line: vibration, dust, glare, SKU (stock keeping unit, a distinct product/packaging variant) changeovers, and operator behavior all differ from the vendor's reference site.
The three-part scorecard
Score every vendor pitch on three dimensions before running a pilot.
1. Data readiness
Ask: what data does this model need, and do you actually have it, in the right format, at the right volume?
For computer vision (defect detection, label verification): How many labeled images of defects does the vendor need? Rare defects (a scratched cap that occurs 1 in 20,000 units) are hard to train on. Ask how the vendor handles class imbalance (when one outcome, like "good part," vastly outnumbers the other, like "defect," in the training data).
For predictive maintenance: Do you have historical sensor data (vibration, temperature, current draw) tagged with actual failure events? Without failure labels, the model has nothing to learn from. Many plants have sensor logs but not clean maintenance records linking them to breakdowns.
For quality/process optimization: Is your historian data (time-series records from your SCADA or MES system, i.e. Supervisory Control and Data Acquisition / Manufacturing Execution System) clean, timestamped consistently, and free of sensor drift?
A vendor who says "we'll figure out the data during the pilot" is telling you the pilot is really a data audit in disguise. That is fine, but price and scope it as one.
2. Integration effort
This is where budgets and timelines usually break.
Ask concretely:
Does this require a new camera, PLC (programmable logic controller, the industrial computer that controls machinery) connection, or edge device, or does it read from existing infrastructure?
Who owns the OT (operational technology, the industrial control systems) network, and has IT/OT security signed off? Manufacturing plants increasingly separate OT and IT networks for cybersecurity reasons, and any AI system touching both is a review, not a plug-in.
What is the actual cycle time impact? A vision system adding 200 milliseconds per unit can be invisible or catastrophic depending on your line speed.
Does it require stopping the line to install, and for how long?
A useful gut check: if the vendor cannot answer these questions in specifics (not "typically," but "on your line, this means X"), they have not scoped your plant, only their product.
3. Proof-of-concept (PoC) design
A PoC (proof of concept, a small-scale test to validate feasibility before wider investment) is only useful if it is designed to fail informatively. Weak PoCs are designed to succeed.
A credible PoC should specify, in writing, before it starts:
Success metric with a number. Not "improved defect detection" but "reduce false negative rate on cap defects from current baseline to below 0.5%, measured over 10,000 units."
Duration long enough to hit real variability. A one-week pilot during a stable production run tells you little about performance during a changeover week or a supplier material switch.
A defined baseline. What is the current human inspector or existing system's actual error rate? Many plants don't know this precisely, which makes "AI improves accuracy" unfalsifiable.
An exit clause. What happens, contractually, if the PoC fails the metric? Good vendors agree to this. Vendors who resist a hard failure threshold are signaling low confidence.
For a useful primer on structuring evaluation metrics for classification systems (the kind of logic under most defect-detection tools), see this accessible explainer from Google's own Machine Learning Crash Course: Classification: Accuracy, recall, precision.
A worked mini-example
Suppose Vendor A's demo claims 99.7% accuracy on defect detection. Your line currently ships 500,000 units a month with a human-inspection defect escape rate (defects that get through) estimated internally at 0.3%, i.e. roughly 1,500 defective units reachreachThe number of unique people exposed to your message in a given period. Unlike impressions, reach counts each person once, no matter how often they see it.Voir la définition complète → customers monthly.
Accuracy alone is misleading here because most units are "good," so a lazy model that always predicts "good" could still score high on accuracy while catching zero defects. What matters is recall on the defect class specifically: of all actual defects, what percentage did the model catch?
If Vendor A's 99.7% figure is overall accuracy but their disclosed defect recall is only 80%, that means 20% of true defects still slip through, in this case around 300 units a month still escaping. That may or may not beat your current human baseline, but you cannot know without asking the vendor to disclose recall and precision (of everything flagged as defective, what percentage really was) separately, not just blended accuracy.
Red flags checklist
Accuracy quoted without a stated baseline or class breakdown
No mention of how the model handles edge cases (new SKU, new label design, lighting change)
Pricing based on number of cameras/sensors with no mention of data engineering or integration labor
PoC proposal with no hard failure criteria
Vendor cannot name a comparable production deployment (not a demo, an actual running plant) you can reference-check
🎬 [VIDEO: "How to Evaluate AI Vendors" - youtube.com/results?search_query=how+to+evaluate+ai+vendors+manufacturing - search for recent manufacturing-focused vendor evaluation talks from industry conferences like Hannover Messe or MODEX, which regularly post session recordings]
Vérification des acquis
1. Why can a vendor demo showing very high accuracy still fail to predict real-world performance on your line?
2. A vendor's computer vision model needs to detect a defect that occurs in only 1 out of 20,000 units. What is the core data challenge this illustrates?
3. What is the most useful reframing of the question 'does this AI system work' before scaling a pilot?
CHOIX MULTIPLES
4. Select ALL correct answers about factors that create a gap between vendor demo performance and real production performance in manufacturing.
Sélectionnez toutes les réponses correctes.
CHOIX MULTIPLES
5. Select ALL correct answers about what 'data readiness' assessment should involve when evaluating an AI vendor pilot.
Sélectionnez toutes les réponses correctes.
Adoption realities after the pilot passes
Passing a PoC is not the same as being ready to scale. Two more checks matter before rollout across multiple lines or plants:
Does it generalize? A model tuned on Line 3's lighting and camera angle may need retraining, not just redeployment, for Line 7. Ask the vendor what "scaling" actually costs: is it copy-paste, or a new mini-project per line?
Who maintains it? Models drift (performance degrades as real-world conditions shift from training conditions). Ask who monitors model performance in production, how often retraining happens, and what that costs annually. This ongoing cost is frequently left out of the initial pitch and can rival the original implementation cost over two to three years (this is a general pattern reported across enterprise AI deployments, not a fixed industry figure, so validate it against your specific contract).
Key Takeaways
Score every vendor pitch on three dimensions: data readiness (do you have the labeled data the model needs), integration effort (what it takes to connect to your actual OT/IT environment), and PoC design (whether success/failure is defined in advance with numbers).
Demand class-specific metrics (recall, precision) for imbalanced problems like defect detection, not blended accuracy, which can hide poor real-world performance.
A credible PoC has a numeric success threshold, a realistic duration that captures production variability, a stated baseline, and a contractual exit if it fails.
Passing a pilot on one line does not guarantee it scales cheaply across others; ask about per-line retraining costs and ongoing model maintenance before committing budget.
Reference-check actual production deployments, not demos, and be skeptical of vendors unwilling to commit to hard failure criteria.