Vetting an AI vendor's claims in energy, MBA Training, MBA Training
2/5+150 XP
Vetting an AI vendor's claims in energy
# Vetting an AI vendor's claims in energy
The pitch that sounded too good
"Our AI predicts outages 48 hours before they happen, with 94% accuracy."
That line, or something close to it, has been pitched to nearly every utility grid operations team in the US and Europe since 2023. The vendor deck shows a slick dashboard, a case study logo wall, and a promise: fewer truck rolls, faster restoration, happier regulators.
Here's the problem. That claim, on its own, tells you almost nothing. What counts as "an outage"? Accuracy measured how, on what data, over what time window? Would a simple weather-based heuristic get close to 94% anyway, given that most outages cluster around storms?
Utilities sign multi-year SCADA (Supervisory Control and Data Acquisition) integrations and outage management system contracts worth millions based on decks like this. This lesson gives you six questions to ask before you sign, using the outage detection pitch as the running example.
Why energy is an easy sector to fool
Energy AI vendors fall into two camps:
1. Purpose-built: trained on grid topology, sensor telemetry (SCADA, AMI smart meter data, weather feeds), and utility-specific failure modes (transformertransformerA Transformer is a neural network architecture that uses self-attention to process sequences in parallel, powering most modern language and generative AI models. thermal aging, vegetation contact, animal contact outages).
2. Repackaged horizontal AI: a general anomaly-detection or forecasting engine, originally built for retail demand or manufacturing, with an energy-flavored UI layered on top.
Both can look identical in a sales demo. The difference shows up only when you ask about data, validation, and failure modes. That's the gap this lesson closes.
Question 1: What exact outcome was measured, and against what baseline?
"94% accuracy" is not a number, it's a fragment. Ask:
Accuracy at predicting *any* outage, or specifically unplanned equipment failures?
Measured against what comparison? A naive baseline (like "storm warning plus historical outage rates") often gets 70 to 80% directional accuracy for free, especially in regions with strong seasonal weather correlation (Gulf Coast hurricanes, Nor'easters, European windstorms like Storm Eunice in 2022).
Precision versus recall matters enormously here. A model that flags every feeder as "at risk" during a storm gets high recall (catches real outages) but terrible precision (floods crews with false alarms). Ask for both numbers, not one blended figure.
Good vendor answer: "We beat a weather-only baseline by X percentage points on precision, holding recall constant, across 3 years of PJM Interconnection outage records." (PJM is the regional transmission organization covering mid-Atlantic and Midwest US grid operations.)
Red flag answer: "94% accuracy" with no baseline mentioned.
Question 2: Was the model validated on your grid, or someone else's?
Grid topology is not portable the way, say, a customer churncustomer churnChurn rate is the percentage of customers or revenue lost over a period. It measures how fast a business loses its existing customer base.View full definition → model is. A model trained on underground urban distribution in a European city (dense, less weather-exposed) will not transfer cleanly to overhead rural distribution in Texas (sparse, storm- and vegetation-exposed).
Ask the vendor for:
Which utilities' data trained the model.
Whether your grid's topology (voltage classes, overhead versus underground mix, tree canopy density) resembles the training set.
Whether they re-train or fine-tune per client, or ship one frozen model to everyone.
This is the single clearest tell of sector-agnostic AI in a utility costume: a vendor who can't describe grid-specific training data usually doesn't have it.
Question 3: What data do you actually need to feed it, and do you have it?
Outage prediction tools typically need some combination of:
AMI (Advanced Metering Infrastructure) ping data: smart meters reporting "last gasp" signals when power drops.
SCADA telemetry: real-time voltage, current, breaker status from substations.
Weather feeds: often licensed from providers like NOAA (National Oceanic and Atmospheric Administration, free and public in the US) or commercial services.
Asset age and maintenance records: often in legacy systems, sometimes still partly paper-based.
Vegetation management data: LiDAR or satellite imagery of canopy proximity to lines.
A quick gap check: if your utility has AMI penetration below 60% (still common in parts of Europe and rural US as of 2025 estimates), any model leaning heavily on meter-ping data will have blind spots exactly where you need coverage most, low-density rural feeders.
Question 4: How does it fail, and who notices first?
Every predictive model has failure modes. The honest vendor names them upfront:
Cold start problem: new substations or newly instrumented feeders have no history, so predictions default to population averages.
Concept drift: grid conditions change (new solar interconnections, EV charging load) faster than the model retrains, degrading accuracy silently.
Rare event blindness: catastrophic but rare failures (like the equipment failure linked to California's 2018 Camp Fire) are underrepresented in training data by definition, since they're rare.
Ask: "Walk me through a case where your model missed an outage. What did you learn?" A vendor with no answer, or a rehearsed non-answer, hasn't stress-tested their own product.
Question 5: What's the human workflow around the alert?
An AI prediction that never changes a dispatcher's decision has zero ROIROIReturn on Investment: the ratio of net profit to the cost of an investment. A 300% ROI means each dollar invested returns $3.View full definition → (return on investmentreturn on investmentReturn on Investment: the ratio of net profit to the cost of an investment. A 300% ROI means each dollar invested returns $3.View full definition →), regardless of model accuracy. Ask:
Does the alert integrate into the existing Outage Management System (OMS), or is it a separate dashboard nobody checks during a storm?
What's the false-positive tolerance before crews stop trusting alerts? (This is the "cry wolf" problem: utilities have reported crew fatigue and alert dismissal when false-positive rates run high, an operational risk vendors rarely volunteer.)
Who is accountable if the model misses a major outage that later triggers a regulatory inquiry, for instance under US state Public Utility Commission reliability reporting rules, or Europe's equivalent national regulators?
Question 6: What does the ROIROIReturn on Investment: the ratio of net profit to the cost of an investment. A 300% ROI means each dollar invested returns $3.View full definition → calculation actually include?
Vendors often quote savings from reduced SAIDI/SAIFI (System Average Interruption Duration/Frequency Index, standard US reliability metrics reported to regulators). A believable ROIROIReturn on Investment: the ratio of net profit to the cost of an investment. A 300% ROI means each dollar invested returns $3.View full definition → case shows the arithmetic, not just the conclusion.
Simple worked example (illustrative, not a vendor-verified figure):
Utility has 10,000 truck rolls per year for unplanned outages.
Vendor claims 15% of rolls could be prevented or better-routed via earlier prediction (a plausible, moderate estimate for mature deployments, per industry discussion at events like DistribuTECH).
Average truck roll cost: roughly $300 to $500 per dispatch (industry-cited estimate, varies by utility and region).
Savings: 10,000 × 0.15 × $400 = $600,000/year.
Compare against software licensing, integration engineering, and change-management cost. If the vendor's annual contract exceeds this figure without counting reliability or regulatory benefits, the payback period stretches past typical 2 to 3 year utility technology evaluation horizons.
Always ask the vendor to show their assumed prevention rate and cost-per-truck-roll inputs separately, not bundled into one "$X million saved" headline.
Knowledge check
1. A vendor claims '94% accuracy' for outage prediction. Why is this figure alone insufficient to evaluate the claim?
2. Why might a naive baseline (e.g., storm warnings plus historical outage rates) already achieve 70-80% directional accuracy?
3. What is the key practical difference between a 'purpose-built' energy AI vendor and a 'repackaged horizontal AI' vendor?
MULTIPLE CHOICE
4. Select ALL correct answers about why energy is described as an 'easy sector to fool' when evaluating AI vendor claims.
Select all the correct answers.
MULTIPLE CHOICE
5. Select ALL correct answers regarding what a rigorous evaluation of an outage-prediction accuracy claim should include.
Select all the correct answers.
Putting it together: a vetting checklist
Before the next AI vendor meeting, bring this list:
1. Baseline comparison for every accuracy claim
2. Training data source and topology match to your grid
3. Required input data and your actual data readiness
4. Named failure modes and a real example of a miss
5. Workflow integration and alert fatigue plan
6. ROIROIReturn on Investment: the ratio of net profit to the cost of an investment. A 300% ROI means each dollar invested returns $3.View full definition → math with assumptions shown, not just the total
🎬 [VIDEO: "How AI Is Predicting Power Outages Before They Happen" - youtube.com - search for utility industry or DistribuTECH conference talks on AI-driven outage prediction for real deployment case studies with named utilities]
Key Takeaways
A single accuracy number is meaningless without a stated baseline and precision/recall breakdown; demand both.
Grid AI does not transfer cleanly between utilities; ask what topology and region trained the model.
MapMapUsing software to automate repetitive marketing tasks and campaigns, enabling personalisation at scale across channels like email, web, and social.View full definition → the vendor's required data inputs against your actual AMI, SCADA, and asset data maturity before assuming the model will work as demoed.
Insist on hearing about failure modes and a real miss; a vendor with no failure story hasn't tested seriously.
Rebuild the ROIROIReturn on Investment: the ratio of net profit to the cost of an investment. A 300% ROI means each dollar invested returns $3.View full definition → calculation yourself from stated assumptions (prevention rate, cost per truck roll, contract cost) rather than trusting the headline savings figure.