# Data qualityData qualityThe degree to which data is fit for purpose: accurate, complete, consistent, timely, valid and unique. Poor quality data undermines analytics, reporting and AI.View full definition → dimensions for energy datasets: accuracy, completeness, timeliness
At 3 a.m. during a February ice storm in Texas, a distribution utility's SCADA (Supervisory Control and Data Acquisition) system stops receiving telemetry from 40,000 smart meters. Grid operators are flying blind on exactly the feeders most likely to fail. When the data comes back six hours later, someone has to decide: do we trust it, is it complete, and did it arrive in time to matter. That is data qualitydata qualityThe degree to which data is fit for purpose: accurate, complete, consistent, timely, valid and unique. Poor quality data undermines analytics, reporting and AI.View full definition →, and in energy it is not an abstraction. It decides whether outage restoration crews go to the right pole.
This lesson works through the three dimensions that matter most for energy datasets, with real scenarios and the metrics you'd actually calculate.
Energy datasets combine three hard properties that make quality control harder than in, say, retail analytics:
Accuracy means the recorded value reflects reality. A meter reading of 4.2 kWh (kilowatt-hours) for an interval is accurate if the customer actually consumed 4.2 kWh.
Real case: after a meter swap (replacing an old analog meter with a smart meter), technicians sometimes enter the wrong multiplier or CT (current transformertransformerA Transformer is a neural network architecture that uses self-attention to process sequences in parallel, powering most modern language and generative AI models.View full definition →) ratio for large commercial accounts. The result: readings that are internally consistent but wrong by a fixed factor, sometimes 10x or 100x off. This is called a "meter multiplier error" and it is one of the most common billing-accuracy incidents in the industry.
How you catch it: validation rules comparing new readings against historical consumption patterns for the same account. A commercial customer whose load suddenly appears to be 100x higher should trigger an automatic flag, not a bill.
Completeness means you have all the data points you're supposed to have, for the time period and asset set you're measuring.
Real case: a meter swap creates a data gap. The old meter stops reporting at 2:00 p.m.; the new smart meter doesn't start transmitting until it's provisioned in the head-end system (the central system that collects meter data), sometimes hours or days later. Every 15-minute interval in between is missing.
Completeness rate formula:
Completeness rate = (Actual intervals received / Expected intervals) x 100Worked example: A meter should report 96 intervals per day (24 hours x 4 fifteen-minute intervals). After a swap, it reports only 72 intervals due to a 6-hour provisioning gap.
Completeness rate = (72 / 96) x 100 = 75%Utilities commonly target completeness rates above 98 to 99% for billing-grade meter data (this threshold is a common industry practice, not a universal legal standard). Below that, utilities often fall back to "estimated" billing for the gap, using historical load profiles, which itself needs disclosure to the customer under most US state tariff rules.
Timeliness means data arrives fast enough to be useful for its purpose. Note this is distinct from completeness: data can be complete but late.
Real case: during a storm, telemetry from Advanced Metering Infrastructure (AMI, the network of smart meters and communication systems) can be delayed by network congestion or backhaul outages, even though the meters themselves keep recording locally. Readings eventually arrive, complete and accurate, but 8 hours late. For grid operations trying to detect outages in real time, 8-hour-late data is close to useless. For monthly billing, it's fine.
This is why utilities define timeliness thresholds by use case:
| Use case | Typical timeliness requirement (industry estimate) |
|---|---|
| Real-time outage detection | Under 5 minutes |
| Grid balancing / demand response | Under 15 to 60 minutes |
| Monthly billing | Within 24 to 48 hours |
| Regulatory reporting (e.g., NERC reliability metrics) | Per filing deadline, often days to weeks |
Utilities often run automated validation, editing, and estimation (VEE) processes on incoming meter data. This is standard practice referenced in AMI system documentation from vendors like Itron and Landis+Gyr. VEE checks each reading against rules: is it within a plausible range, does it match the meter's historical pattern, is it flagged by the meter itself as a fault code.
Reading-validation pass rate formula:
Pass rate = (Readings passing all VEE checks / Total readings received) x 100Worked example: A distribution utility receives 10 million interval readings in a day. VEE flags 150,000 as failing at least one check (out of range, negative usage on a non-generating account, missing timestamp).
Pass rate = ((10,000,000 - 150,000) / 10,000,000) x 100 = 98.5%A pass rate below roughly 95% (a commonly cited operational benchmark, not a regulatory one) usually triggers a data qualitydata qualityThe degree to which data is fit for purpose: accurate, complete, consistent, timely, valid and unique. Poor quality data undermines analytics, reporting and AI.View full definition → investigation: is it a systemic meter firmware issue, a communication network problem, or a one-off event like the storm above.
Knowledge check
1. A distribution utility loses SCADA telemetry from thousands of smart meters during a storm and regains it six hours later. Which data quality question is most distinct from asking whether the returned values are accurate?
2. After a meter swap, a technician enters the wrong multiplier, causing recorded consumption to be off by a consistent factor even though a reading was captured for every interval. Which dimension is most directly compromised?
3. Why is data quality control generally harder for energy datasets than for typical retail analytics datasets, according to the lesson's framing?
4. Select ALL correct answers about why missing telemetry during a grid emergency (like the ice storm scenario) is especially risky.
Select all the correct answers.
5. Select ALL correct answers about the regulatory dimension of energy data quality mentioned in the lesson.
Select all the correct answers.
To ground these dimensions, know the core datasets in play:
Each dataset has different accuracy, completeness, and timeliness requirements. Settlement data can tolerate a multi-day lag but demands near-perfect accuracy because it moves money between market participants. Outage telemetry demands near-instant timeliness but can tolerate occasional gaps if redundant signals exist (e.g., customer calls backing up meter "last gasp" data).
For a deeper technical reference on smart meter data standards, the US Department of Energy's Green Button initiative documents standardized data formats used across US utilities.
None of these metrics matter without ownership. Leading utilities assign data stewards to specific datasets (meter data, outage data, asset data) with explicit accountability for the three dimensions above. This is increasingly formalized as data governance frameworks mature across the sector, often modeled on general frameworks like DAMA-DMBOK (Data Management Body of Knowledge), adapted to energy-specific assets and regulatory reporting deadlines.