Evaluating vendor AI claims in RFPs and demos

3. "Up to 40%": "Up to" is the tell. It describes a best case, possibly from one pilot site, one quarter, cherry-picked. Ask for the distribution, not the ceiling.

4. "Across the RAN": Was this validated on 4G, 5G, both? Urban macro cells, rural, indoor small cells? A model trained on dense urban RAN traffic patterns often degrades on rural cells with different noise profiles.

Five questions that separate proof from polish

1. What is the baseline?

"40% reduction in MTTR" versus what? Manual triage? A legacy rules-based system? Ask for the specific comparator and the time period measured.

2. What data was the model trained and tested on?

Ask directly: how many cell sites, how many months, which vendor's equipment (Ericsson, Nokia, Huawei RAN hardware behaves differently), and was the test set from a different time period than the training set (out-of-time validation) to rule out overfitting.

3. What are precision and recall, not just accuracy?

For fault prediction or anomaly detection, accuracy is often meaningless because faults are rare events. If only 1% of time windows contain a real fault, a model predicting "no fault" always scores 99% accuracy while being useless. Insist on precision (of flagged faults, how many were real) and recall (of real faults, how many were caught).

4. Was this validated by a third party or only internally?

Independent validation, a customer reference willing to speak, or a published case study with methodology, carries more weight than an internal white paper. Ask for the analogous public resource: TM Forum publishes vendor-neutral maturity frameworks for AI in telecom operations, useful as an outside benchmark.

5. Does the demo reflect production conditions?

Demos run on curated data. Ask: can we run this on our own held-out data before signing? A vendor confident in its claims should agree to a proof of concept (PoC) with your data, your KPIs, and a pre-agreed success threshold.

A simple framework: the four-layer claim test

| Layer | Question | Red flag |

|---|---|---|

| Definition | What exact metric, over what window? | "Improves efficiency" with no metric |

| Baseline | Compared to what prior state? | No comparator mentioned |

| Evidence | Internal, customer-cited, or peer-reviewed? | Only "internal studies show" |

| Transferability | Tested on data like ours? | Single pilot, different market/geography |

If a claim fails two or more layers, treat it as marketing, not a technical result, and price the deal accordingly (shorter contract, PoC gate, penalty clauses tied to real KPIs).

Worked example: sizing a claim before you believe it

A vendor claims their AI-based predictive maintenance cuts unplanned outages by 30%, saving an estimated $2M annually (vendor's estimate, unverified) for a mid-size operator with 10,000 cell sites.

Sanity-check the arithmetic yourself:

This roughly matches the vendor's $2M figure, which is a good sign the claim is internally consistent. But consistency is not proof. You still need your own outage rate and cost-per-outage numbers, not the vendor's assumed ones, and a PoC to test whether 30% holds on your network.

Contract and evaluation clauses worth insisting on

• PoC before commitment: 60 to 90 days, on your data, against your defined KPIKPIKey Performance Indicator, a measurable value that shows how effectively you're achieving a specific objective, tracked over time against a target.View full definition →, with an explicit "no purchase" exit if the threshold is missed.
• Explainability requirement: for any model influencing customer-facing decisions (e.g., churn prediction affecting retention offers), ask how the vendor addresses the EU AI Act's transparency obligations for relevant AI systems, and whether they can produce feature-level explanations, not just a score.
• Data provenanceData provenanceData lineage maps how data moves and transforms across systems, from origin to consumption, showing where it came from, what changed it, and where it goes.View full definition →: where was training data sourced, does it include your competitors' anonymized data (common in multi-tenant network analytics platforms), and does that raise antitrust or confidentiality concerns.
• Performance decay monitoring: does the vendor commit to retraining cadence and to reporting model drift (performance degrading as network conditions change)?

🎬 [VIDEO: "How to Read an AI Vendor's Whitepaper Critically" - youtube.com - search for talks from MLOpsMLOpsMachine Learning Operations: combining ML and DevOps practices to industrialise, deploy, monitor, and retrain models reliably in production.View full definition → or applied ML conferences on evaluating vendor ML claims; look for ones that walk through precision/recall traps and baseline comparisons in enterprise AI sales material]

What genuinely good vendor evidence looks like

• Named customer references, ideally at operators comparable in scale and geography to you.
• A confusion matrix or precision/recall table, not just a single "accuracy" number.
• Disclosure of the model type (even at a high level: gradient boosting, LSTM for time-series, transformertransformerA Transformer is a neural network architecture that uses self-attention to process sequences in parallel, powering most modern language and generative AI models.View full definition →-based anomaly detection) and why it fits the use case.
• A published or third-party audited benchmark, or willingness to be benchmarked against a competitor on shared data.
• Clear statement of limitations: where the model underperforms (e.g., "accuracy drops on newly deployed 5G small cells with under 3 months of history").

Vendors who volunteer limitations are, counterintuitively, more credible. Overclaiming with no caveats is itself a signal.

Key Takeaways

• Treat every AI performance number in an RFP as a hypothesis, not a fact, until you see the baseline, the dataset, and precision/recall detail behind it.
• "Up to X%" language signals a best-case, likely non-representative result; ask for the distribution across sites and time periods.
• Demand a proof of concept on your own dataown dataData collected directly from your own customers and prospects through your own channels: your most reliable and privacy-compliant source.View full definition → with a pre-agreed KPIKPIKey Performance Indicator, a measurable value that shows how effectively you're achieving a specific objective, tracked over time against a target.View full definition → threshold before signing; demos on vendor-curated data prove little.
• Use a simple four-layer test (definition, baseline, evidence, transferability) to triage claims quickly during RFP review.
• Vendors who disclose model limitations and offer independent validation are generally more trustworthy than those with polished claims and no caveats.