# Underwriting AI vendor claims before you buy
A fraud-detection vendor tells your risk committee: "Our model cuts false positives by 40%." The slide has a clean bar chart. The sales rep has a case study logo wall. Nobody in the room asks: 40% compared to what, measured how, on whose data?
Six months later your fraud ops team is drowning in a queue that looks nothing like the pitch. This lesson gives you the checklist to prevent that outcome, before a pilot ever touches production data.
A false positive (a legitimate transaction wrongly flagged as fraud) has a real cost: annoyed customers, abandoned transactions, call center volume. Vendors know reducing false positives is a top pain point, so it's the headline metric of choice.
But "40% fewer" is a ratio, not a fact. You need the denominator. Ask:
If the vendor cannot produce the confusion matrix (the 2x2 table of true/false positives and negatives) behind the headline number, treat the claim as marketing, not evidence.
Ask for a data sheet describing: time period, geography, transaction types (card-present vs. card-not-present, wire, ACH), fraud label definition, and class balance (what % of the dataset was actually fraudulent). Fraud is rare, often well under 1% of transactions, so a model tested on an artificially balanced 50/50 dataset will look far better than it performs live.
"Accuracy" is almost useless for rare-event detection like fraud or anti-money laundering (AML, the regulatory regime requiring banks to detect and report suspicious financial activity) because a model that predicts "not fraud" every time can be 99%+ "accurate" while catching zero fraud. Insist on:
Ask explicitly: was the test set completely separate in time from the training set (out-of-time validation), or just a random split? Random splits on transaction data can leak information (a customer's later transactions inform predictions about their earlier ones), inflating results artificially. Out-of-time testing, training on 2023 data and testing on 2024 data, is the realistic standard.
A model benchmarked on large-bank card data may degrade badly on a community bank's smaller, different transaction mix, or on cross-border wires if it was trained mostly on domestic ACH. Ask for a champion-challenger pilot: run the vendor model in shadow mode (scoring live transactions without acting on them) against your existing system for a defined period, then compare results on your actual population.
Fraud patterns shift constantly, and so does normal customer behavior. Ask how often the model is retrained, what triggers retraining, and what happens to performance between retraining cycles (this is called model drift or performance decay). A vendor with no answer to "how do you monitor drift" is asking you to buy a snapshot, not a system.
Suppose your fraud rate is realistically 0.3% of transactions (a plausible order-of-magnitude estimate for card transactions; actual rates vary by issuer and channel). Out of 1,000,000 transactions, that's roughly 3,000 fraudulent ones.
Vendor claims: 90% recall (catches 90% of fraud) and a false positive rate of 2%.
So your fraud team reviews roughly 22,640 flagged transactions to find 2,700 real fraud cases. That's a precision of about 12%. The vendor's "40% fewer false positives than our old model" claim might be true and still leave you with an unworkable review queue, because the base rate of fraud is so low that even small false-positive percentages generate huge absolute volumes.
This is the base rate problem, and it's the single most common reason vendor pilots disappoint in production. Always convert percentage claims into absolute counts against your actual transaction volume before signing anything.
In the US, the Federal Reserve's SR 11-7 guidance on model risk management sets the supervisory expectation that banks independently validate any model, including vendor-supplied ones, before deployment, and monitor it on an ongoing basis. You cannot outsource validation responsibility to the vendor's own benchmark report.
In the EU, the AI Act (entered into force 2024, with phased compliance obligations through 2026 and beyond) classifies most creditworthiness and fraud-scoring systems used by banks as "high-risk AI systems," which triggers requirements for documentation, human oversight, and accuracy/robustness testing that examiners can inspect. European banks should mapmapUsing software to automate repetitive marketing tasks and campaigns, enabling personalisation at scale across channels like email, web, and social.View full definition → any vendor claim directly to these documentation requirements as part of procurement, not after.
Either way, the regulatory expectation and the commercial due diligence point the same direction: independent validation on your own dataown dataData collected directly from your own customers and prospects through your own channels: your most reliable and privacy-compliant source.View full definition →, not vendor marketing collateral, is the standard of proof.
Knowledge check
1. A vendor claims their model reduces false positives by 40%. Why is this claim alone almost meaningless?
2. Why can a vendor always reduce false positives simply by loosening the detection threshold?
3. What should a buyer conclude if a vendor cannot produce the confusion matrix behind their headline performance claim?
4. Select ALL correct answers: which questions should a buyer ask to properly interpret a vendor's 'X% fewer false positives' claim?
Select all the correct answers.
5. Select ALL correct answers: what information should be included in a benchmark dataset description requested from an AI vendor before a pilot?
Select all the correct answers.
A practical procurement checklist for AI vendor claims in banking should score each vendor on:
1. Transparency of methodology: will they share the confusion matrix, not just the headline ratio?
2. Data relevance: was the benchmark built on data resembling your portfolio (geography, channel, customer segment)?
3. Shadow-mode pilot availability: will they run live, non-production scoring on your data before go-live?
4. Explainability: can the model's flagged decisions be explained to a customer or examiner (relevant under fair lending laws like the US Equal Credit Opportunity Act, and under GDPR's automated-decision provisions in the EU)?
5. Drift monitoring and retraining SLA: what's contractually guaranteed for ongoing performance, not just launch-day performance?
A simple snippet for tracking this internally during vendor evaluation:
# minimal vendor claim sanity-check
fraud_rate = 0.003 # estimate: adjust to your actual portfolio
total_txns = 1_000_000
recall = 0.90 # vendor-claimed true positive rate
fpr = 0.02 # vendor-claimed false positive rate
fraud_txns = total_txns * fraud_rate
non_fraud_txns = total_txns - fraud_txns
true_positives = fraud_txns * recall
false_positives = non_fraud_txns * fpr
precision = true_positives / (true_positives + false_positives)
print(f"Flagged for review: {true_positives + false_positives:,.0f}")
print(f"Precision: {precision:.1%}")Run this with the vendor's actual claimed numbers and your real transaction volume before any contract discussion.
🎬 [VIDEO: "Model Risk Management Explained" - https://www.youtube.com/results?search_query=model+risk+management+banking+explained - search results for accessible explainer videos on model validation and governance frameworks like SR 11-7, useful background before vendor evaluation meetings]