# From smart meters to grid telemetry: the energy data stack
A single home with a smart meter generates 96 readings a day. At 15-minute intervals, that is one reading every 900 seconds, capturing kilowatt-hours consumed. Multiply that by the roughly 130 million smart meters installed across US homes and businesses (a figure that has grown steadily for over a decade), and you get a firehose of data measured in terabytes per day.
Here is the uncomfortable truth: most utilities store almost all of it and use almost none of it. Let us follow one reading and find out why.
Meet the Nguyen household. At 6:15 PM, they turn on the oven, the AC kicks in, and an EV starts charging in the garage. Their meter records the cumulative energy used in that 15-minute window.
The device on the wall is an AMI meter (Advanced Metering Infrastructure). Unlike the old spinning-dial analog meters that a human read once a month, AMI meters record interval data and communicate it back to the utility automatically, usually over a wireless mesh network or cellular link.
The Nguyen reading is now a tiny packet: a meter ID, a timestamp, a kWh value, and often a quality flag (was the read clean or estimated?).
The packet travels to the head-end system (HES), the software that talks directly to the meter fleet. Think of it as the switchboard. Its job is narrow but critical:
The HES speaks the meter's protocol. It is not where analysis happens. It is plumbing.
Next stop: the MDMMDMMaster Data Management (MDM) is the discipline of creating and maintaining a single, consistent, trusted version of an organization's core business entities like customers, products, and suppliers.View full definition → (Meter Data Management system). This is the system of record for consumption data. The MDMMDMMaster Data Management (MDM) is the discipline of creating and maintaining a single, consistent, trusted version of an organization's core business entities like customers, products, and suppliers.View full definition → does the unglamorous but essential work:
By the time the Nguyen reading reaches the MDMMDMMaster Data Management (MDM) is the discipline of creating and maintaining a single, consistent, trusted version of an organization's core business entities like customers, products, and suppliers.View full definition →, it has been checked, gap-filled, and standardized. This is the data that ends up on their bill.
Consumption is only half the picture. The grid itself is instrumented with its own telemetry:
So the energy data stack has two rivers: customer-side (meters, MDMMDMMaster Data Management (MDM) is the discipline of creating and maintaining a single, consistent, trusted version of an organization's core business entities like customers, products, and suppliers.View full definition →) and grid-side (SCADA, PMUs). They rarely meet.
Now the core problem. The Nguyen reading was useful for one thing: the bill. After that, it mostly sits in a database.
The billing team owns the MDMMDMMaster Data Management (MDM) is the discipline of creating and maintaining a single, consistent, trusted version of an organization's core business entities like customers, products, and suppliers.View full definition →. The operations team owns SCADA. The planning team owns yet another system. Each was procured separately, often years apart, from different vendors. They do not share a common data model.
A concrete example: knowing that the Nguyens' EV charges at 6:15 PM could help the planning team decide whether the local transformertransformerA Transformer is a neural network architecture that uses self-attention to process sequences in parallel, powering most modern language and generative AI models.View full definition → needs an upgrade. But the meter data lives in one silo and the asset data in another. Nobody connects them.
Consider the raw math for a mid-sized utility:
1,000,000 meters
× 96 reads/day (15-min intervals)
= 96,000,000 reads/day
× 365 days
≈ 35 billion reads/yearAdd multiple channels per meter (kWh delivered, kWh received from rooftop solar, voltage) and the count multiplies. Grid telemetry from PMUs adds far more, because sampling 30 times per second dwarfs a 15-minute meter read. Storing this is cheap. Making it queryable, clean, and cross-referenced is expensive.
Utilities are regulated monopolies in most of their service territory. Their revenue is set through rate cases (proceedings where a regulator approves what the utility can charge). Historically, regulators rewarded building physical assets, not extracting value from data. So the data was collected to satisfy a billing mandate, and the business case to do more with it was weak.
That is shifting as regulators push for grid modernization and demand-side programs, but the legacy mindset lingers.
When utilities actually connect the rivers, the Nguyen reading becomes valuable in ways that go well beyond a bill.
Compare energy delivered on a feeder against energy billed to every meter downstream. A persistent gap can signal meter tampering, faulty meters, or theft. Interval data makes this detectable.
Aggregate the interval reads of every home on a single transformertransformerA Transformer is a neural network architecture that uses self-attention to process sequences in parallel, powering most modern language and generative AI models.View full definition →. If the Nguyens plus three neighbors all charge EVs at 6:15 PM, that transformertransformerA Transformer is a neural network architecture that uses self-attention to process sequences in parallel, powering most modern language and generative AI models.View full definition → may overload. Utilities can spot this before failure instead of after.
Interval data enables time-of-use rates and demand response (paying or nudging customers to shift usage away from peak hours). Without granular meter data, none of this is measurable or fair.
A cluster of meters that suddenly stop reporting ("last gasp" messages) tells operators exactly where an outage is, often before customers call.
For a solid grounding in how this data infrastructure is meant to evolve, the US Department of Energy's Grid Modernization Initiative is a free, credible starting point.
Modern utilities are trying to break the silos with a shared data lakedata lakeA data lake is a centralized repository that stores large volumes of raw data in its native format, from structured tables to unstructured files, until needed.View full definition → or lakehouselakehouseA hybrid architecture combining the flexibility of a data lake with the analytical capabilities of a data warehouse, on a single storage layer.View full definition →: a central store where MDMMDMMaster Data Management (MDM) is the discipline of creating and maintaining a single, consistent, trusted version of an organization's core business entities like customers, products, and suppliers.View full definition → data, SCADA feeds, weather data, and asset records all land in a common format.
On top sits an analytics layer feeding forecasting models, a digital twin (a virtual model of the physical grid updated with live telemetry), and increasingly, machine learning for load prediction.
The blocker is rarely technology now. It is governance: agreeing on data definitions, ownership, access rights, and privacy. Interval data is sensitive. It can reveal when a household is home, asleep, or on vacation, which is why data privacy rules (varying by jurisdiction) constrain how it is shared and monetized.
Knowledge check
1. The lesson notes that utilities store almost all of their smart meter data but use almost none of it. What does this best illustrate about the energy data stack?
2. Why is the head-end system (HES) described as 'plumbing' rather than where analysis happens?
3. A single reading arrives with a quality flag indicating it was 'estimated' rather than a clean read. Which system is responsible for resolving such data quality issues before it becomes part of the system of record?
4. Select ALL correct answers about what fundamentally distinguishes an AMI meter from an old analog spinning-dial meter.
Select all the correct answers.
5. Select ALL correct answers that correctly describe the flow and division of responsibilities in the energy data stack.
Select all the correct answers.
Notice the pattern. The technical journey from meter to MDMMDMMaster Data Management (MDM) is the discipline of creating and maintaining a single, consistent, trusted version of an organization's core business entities like customers, products, and suppliers.View full definition → is largely a solved problem. The hard part is organizational and regulatory.
A useful mental model: the energy data stack has three layers of maturity.
1. Collect: get the reads reliably. Most utilities are here or past it.
2. Clean and store: VEE, aggregation, a common data model. Many are working on this.
3. Act: cross-reference customer and grid data to drive decisions. Few do this well.
The terabytes are not the problem. The problem is that the value sits in layer three, and most of the spending and the regulatory reward has historically stopped at layer one.
For an MBA reading this: the opportunity in energy data is rarely about building a fancier model. It is about the boring, high-leverage work of connecting systems that were never designed to talk, and building the business case that makes a regulator say yes.