Evaluating vendor claims and proof of concept design

# Evaluating vendor claims and proof of concept design

A loss prevention director at a mid-size grocery chain once told a vendor's sales team: "Your demo caught 94% of theft. Show me on my cameras, in my stores, with my shoplifters." The vendor's accuracy dropped to 61%. That gap between demo and reality is the entire reason this lesson exists.

Vision-AI shrink detection (computer vision systems that flag suspected theft or scan avoidance at checkout) is one of the fastest-growing AI purchases in retail loss prevention. It is also one of the easiest categories to get burned in, because vendor demos are built on curated footage, not your store's lighting, camera angles, or customer mix.

Why vendor demos mislead by default

Vendor demos are optimized to sell, not to inform. Three structural reasons:

• Cherry-picked footage. Demo reels typically show clean, well-lit, high-confidence detections. The vendor selects clips where the model performs best.
• Different base rates. A model trained on big-box retail theft patterns may perform poorly on a convenience store's razor-thin aisles or a pharmacy's basket sizes.
• No exposure to your false positive drivers. Reusable bags, employee restocking, parents grabbing items for kids, these trigger false alarms in ways a generic demo never surfaces.

This is why an RFP (Request for Proposal, the formal document retailers issue to solicit vendor bids) needs a scorecard that goes beyond "does it work" to "does it work on my data, at my scale, under my constraints."

Anatomy of a shrink-detection RFP scorecard

A realistic scorecard for a vision-AI shrink vendor typically weights five categories. Here is a simplified version modeled on what large retailers actually use:

| Category | Weight | What you're testing |

|---|---|---|

| Detection accuracy on your footage | 30% | Precision and recall on a labeled sample from your own stores |

| False positive rate in production conditions | 25% | Alerts per store per day that staff must review |

| Integration effort | 15% | Compatibility with existing camera hardware, POS (point of sale) systems, and staff workflows |

| Data privacy and compliance | 15% | Handling of biometric data, retention policy, state and EU rules |

| Total cost of ownership over 3 years | 15% | Licensing, hardware upgrades, staffing to review alerts |

Notice: "accuracy" is only 30%. Vendors love to lead with a single accuracy number. A serious buyer treats that number as one input among five.

Precision and recall, defined for non-technical buyers

Two terms you must nail down before any negotiation:

• Precision: of all the alerts the system raised, what percentage were real theft events? Low precision means staff waste time chasing false alarms.
• Recall: of all the actual theft events that happened, what percentage did the system catch? Low recall means shrink keeps leaking through undetected.

There is almost always a tradeoff. A vendor claiming both 95% precision and 95% recall on a single demo should raise your skepticism, not your confidence. Ask for the confusion matrix (the table breaking down true positives, false positives, true negatives, false negatives), not just a headline percentage.

Designing the proof of concept (POC)

A POC (proof of concept, a limited-scale test before full purchase) should be structured like a controlled experiment, not an extended sales pitch.

Step 1: Pick representative stores, not flagship stores.

Choose 3 to 5 locations that reflect your actual footprint: different layouts, lighting, camera age, urban and suburban mix. Vendors will push for your best-equipped store. Resist that.

Step 2: Build your own labeled ground truth.

Before the POC starts, your loss prevention team (or a sampled review process) should tag a set of known incidents from historical footage, both theft events and non-events that look suspicious (the reusable-bag problem, restocking staff, etc.). This becomes the answer key the vendor's system is graded against, independent of the vendor's own reporting.

Step 3: Run a fixed time window, minimum 60 to 90 days.

Shoplifting patterns vary by season, day of week, and staffing level. A two-week POC during a slow period tells you almost nothing. Retail loss prevention associations, including the National Retail Federation's Loss Prevention Research Council, publish annual shrink survey data that can help you benchmark what "normal" incident rates look like for your format.

Step 4: Score against the RFP scorecard, not vendor dashboards.

Vendors will offer their own performance dashboard. Cross-check a sample manually. If the vendor reports 88% precision and your manual audit of 100 flagged alerts finds 60 were real, that discrepancy is the single most important data point in the whole evaluation.

Step 5: Stress-test the false positive cost.

Calculate the labor cost of alert review. A simple worked example:

• System generates 40 alerts per store per day
• Average review time: 3 minutes per alert
• That's 120 minutes (2 hours) of staff time per store per day, every day
• Across 50 pilot stores, that is 100 staff-hours per day just reviewing alerts

If the fully loaded cost of that review time exceeds the shrink dollars recovered, the system is a net negative even if it "works."

Data privacy and compliance checks

Vision-AI systems that track individuals raise real regulatory exposure. In the US, states including Illinois (Biometric Information Privacy Act, BIPA) and Texas have specific biometric data statutes with private right of action or state attorney general enforcement. In the EU and UK, the GDPR (General Data Protection Regulation) and UK GDPR govern any processing of biometric identifiers, and facial recognition in retail settings has drawn scrutiny from data protection authorities including the UK's Information Commissioner's Office. Confirm with your legal team, before the POC even begins, whether the vendor's system performs facial recognition (matching identity) versus behavior detection (flagging suspicious movement patterns without identifying who). These carry very different compliance burdens.

A simple vendor evaluation script

For technically inclined teams, even a basic script comparing vendor alerts to your labeled ground truth clarifies the picture fast:

# Compare vendor alerts to your ground truth labels
import pandas as pd

vendor_alerts = pd.read_csv("vendor_alerts.csv")   # columns: incident_id, flagged
ground_truth = pd.read_csv("ground_truth.csv")     # columns: incident_id, actual_theft

merged = ground_truth.merge(vendor_alerts, on="incident_id", how="left")
merged["flagged"] = merged["flagged"].fillna(False)

true_positives = ((merged.flagged) & (merged.actual_theft)).sum()
false_positives = ((merged.flagged) & (~merged.actual_theft)).sum()
false_negatives = ((~merged.flagged) & (merged.actual_theft)).sum()

precision = true_positives / (true_positives + false_positives)
recall = true_positives / (true_positives + false_negatives)

print(f"Precision: {precision:.2%}, Recall: {recall:.2%}")

This is not production code, it is a sanity-check tool any analyst can run in an afternoon to validate a vendor's self-reported numbers.

Negotiating based on POC results

Once you have real numbers, negotiate contract terms around them:

• Performance-linked pricing: tie a portion of fees to measured precision and recall on your data, not the vendor's benchmark.
• Right to re-audit: build in quarterly re-testing rights, since model performance can drift as store layouts and merchandise change (a phenomenon called model drift).
• Exit clauses: since vision-AI vendors often require hardware installation (cameras, edge computing units), ensure you are not locked into proprietary hardware that limits switching later.

🎬 [VIDEO: "How Retailers Are Using AI to Fight Shoplifting" - youtube.com - search for recent segmentssegmentsDividing a market into distinct groups of customers who share similar needs, characteristics or behaviours, so each group can be served with a tailored approach.Voir la définition complète → from business news outlets covering retail loss prevention AI deployments and their real-world limitations]

Key Takeaways

• Never evaluate a shrink-detection vendor on their demo alone. Insist on a POC using your own store footage, your own lighting conditions, and your own historical incidents as ground truth.
• Build an RFP scorecard that weights accuracy alongside false positive cost, integration effort, and compliance, accuracy alone is an incomplete picture.