From smart meters to grid telemetry: the energy data stack

# From smart meters to grid telemetry: the energy data stack

A single home with a smart meter generates 96 readings a day. At 15-minute intervals, that is one reading every 900 seconds, capturing kilowatt-hours consumed. Multiply that by the roughly 130 million smart meters installed across US homes and businesses (a figure that has grown steadily for over a decade), and you get a firehose of data measured in terabytes per day.

Here is the uncomfortable truth: most utilities store almost all of it and use almost none of it. Let us follow one reading and find out why.

The journey of one reading

Meet the Nguyen household. At 6:15 PM, they turn on the oven, the AC kicks in, and an EV starts charging in the garage. Their meter records the cumulative energy used in that 15-minute window.

Step 1: The AMI meter

The device on the wall is an AMI meter (Advanced Metering Infrastructure). Unlike the old spinning-dial analog meters that a human read once a month, AMI meters record interval data and communicate it back to the utility automatically, usually over a wireless mesh network or cellular link.

The Nguyen reading is now a tiny packet: a meter ID, a timestamp, a kWh value, and often a quality flag (was the read clean or estimated?).

Step 2: The head-end system

The packet travels to the head-end system (HES), the software that talks directly to the meter fleet. Think of it as the switchboard. Its job is narrow but critical:

• Collect readings from millions of meters.
• Send commands back (a remote disconnect, a firmware update, an on-demand read).
• Handle communication failures and retries.

The HES speaks the meter's protocol. It is not where analysis happens. It is plumbing.

Step 3: The meter data management system (MDMMDMMaster Data Management (MDM) is the discipline of creating and maintaining a single, consistent, trusted version of an organization's core business entities like customers, products, and suppliers.View full definition →)

Next stop: the MDMMDMMaster Data Management (MDM) is the discipline of creating and maintaining a single, consistent, trusted version of an organization's core business entities like customers, products, and suppliers.View full definition → (Meter Data Management system). This is the system of record for consumption data. The MDMMDMMaster Data Management (MDM) is the discipline of creating and maintaining a single, consistent, trusted version of an organization's core business entities like customers, products, and suppliers.View full definition → does the unglamorous but essential work:

• Validation, Estimation, and Editing (VEE): if the Nguyen meter missed a read, VEE rules fill the gap using historical patterns, then flag it as estimated.
• Aggregation: rolling 15-minute reads into hourly, daily, and monthly totals.
• Billing determinants: producing the clean numbers the billing system needs.

By the time the Nguyen reading reaches the MDMMDMMaster Data Management (MDM) is the discipline of creating and maintaining a single, consistent, trusted version of an organization's core business entities like customers, products, and suppliers.View full definition →, it has been checked, gap-filled, and standardized. This is the data that ends up on their bill.

Step 4: The grid side

Consumption is only half the picture. The grid itself is instrumented with its own telemetry:

• SCADA (Supervisory Control and Data Acquisition): the traditional backbone that monitors substations, breakers, and feeders, typically polling every few seconds.
• PMUs (Phasor Measurement Units): high-speed sensors that sample voltage and current phase 30 to 60 times per second, giving operators a near real-time view of grid stability.

So the energy data stack has two rivers: customer-side (meters, MDMMDMMaster Data Management (MDM) is the discipline of creating and maintaining a single, consistent, trusted version of an organization's core business entities like customers, products, and suppliers.View full definition →) and grid-side (SCADA, PMUs). They rarely meet.

Why utilities drown

Now the core problem. The Nguyen reading was useful for one thing: the bill. After that, it mostly sits in a database.

The data is siloed by design

The billing team owns the MDMMDMMaster Data Management (MDM) is the discipline of creating and maintaining a single, consistent, trusted version of an organization's core business entities like customers, products, and suppliers.View full definition →. The operations team owns SCADA. The planning team owns yet another system. Each was procured separately, often years apart, from different vendors. They do not share a common data model.

A concrete example: knowing that the Nguyens' EV charges at 6:15 PM could help the planning team decide whether the local transformertransformerA Transformer is a neural network architecture that uses self-attention to process sequences in parallel, powering most modern language and generative AI models.View full definition → needs an upgrade. But the meter data lives in one silo and the asset data in another. Nobody connects them.

The volume is genuinely hard

Consider the raw math for a mid-sized utility:

1,000,000 meters
× 96 reads/day (15-min intervals)
= 96,000,000 reads/day
× 365 days
≈ 35 billion reads/year

Add multiple channels per meter (kWh delivered, kWh received from rooftop solar, voltage) and the count multiplies. Grid telemetry from PMUs adds far more, because sampling 30 times per second dwarfs a 15-minute meter read. Storing this is cheap. Making it queryable, clean, and cross-referenced is expensive.

Regulation shapes what gets used

Utilities are regulated monopolies in most of their service territory. Their revenue is set through rate cases (proceedings where a regulator approves what the utility can charge). Historically, regulators rewarded building physical assets, not extracting value from data. So the data was collected to satisfy a billing mandate, and the business case to do more with it was weak.

That is shifting as regulators push for grid modernization and demand-side programs, but the legacy mindset lingers.

What the data could unlock

When utilities actually connect the rivers, the Nguyen reading becomes valuable in ways that go well beyond a bill.

Non-technical loss detection

Compare energy delivered on a feeder against energy billed to every meter downstream. A persistent gap can signal meter tampering, faulty meters, or theft. Interval data makes this detectable.

TransformerTransformerA Transformer is a neural network architecture that uses self-attention to process sequences in parallel, powering most modern language and generative AI models.View full definition → load management

Aggregate the interval reads of every home on a single transformertransformerA Transformer is a neural network architecture that uses self-attention to process sequences in parallel, powering most modern language and generative AI models.View full definition →. If the Nguyens plus three neighbors all charge EVs at 6:15 PM, that transformertransformerA Transformer is a neural network architecture that uses self-attention to process sequences in parallel, powering most modern language and generative AI models.View full definition → may overload. Utilities can spot this before failure instead of after.

Demand response and dynamic pricingdynamic pricingAutomatically adjusting prices in real time based on demand, competition or user behaviour to optimise revenue, margin or conversion.View full definition →

Interval data enables time-of-use rates and demand response (paying or nudging customers to shift usage away from peak hours). Without granular meter data, none of this is measurable or fair.

Outage detection

A cluster of meters that suddenly stop reporting ("last gasp" messages) tells operators exactly where an outage is, often before customers call.

For a solid grounding in how this data infrastructure is meant to evolve, the US Department of Energy's Grid Modernization Initiative is a free, credible starting point.

The emerging stack

Modern utilities are trying to break the silos with a shared data lakedata lakeA data lake is a centralized repository that stores large volumes of raw data in its native format, from structured tables to unstructured files, until needed.View full definition → or lakehouselakehouseA hybrid architecture combining the flexibility of a data lake with the analytical capabilities of a data warehouse, on a single storage layer.View full definition →: a central store where MDMMDMMaster Data Management (MDM) is the discipline of creating and maintaining a single, consistent, trusted version of an organization's core business entities like customers, products, and suppliers.View full definition → data, SCADA feeds, weather data, and asset records all land in a common format.

On top sits an analytics layer feeding forecasting models, a digital twin (a virtual model of the physical grid updated with live telemetry), and increasingly, machine learning for load prediction.

The blocker is rarely technology now. It is governance: agreeing on data definitions, ownership, access rights, and privacy. Interval data is sensitive. It can reveal when a household is home, asleep, or on vacation, which is why data privacy rules (varying by jurisdiction) constrain how it is shared and monetized.

Where the value actually lives

Notice the pattern. The technical journey from meter to MDMMDMMaster Data Management (MDM) is the discipline of creating and maintaining a single, consistent, trusted version of an organization's core business entities like customers, products, and suppliers.View full definition → is largely a solved problem. The hard part is organizational and regulatory.

A useful mental model: the energy data stack has three layers of maturity.

1. Collect: get the reads reliably. Most utilities are here or past it.

2. Clean and store: VEE, aggregation, a common data model. Many are working on this.

3. Act: cross-reference customer and grid data to drive decisions. Few do this well.

The terabytes are not the problem. The problem is that the value sits in layer three, and most of the spending and the regulatory reward has historically stopped at layer one.

For an MBA reading this: the opportunity in energy data is rarely about building a fancier model. It is about the boring, high-leverage work of connecting systems that were never designed to talk, and building the business case that makes a regulator say yes.

Key Takeaways

• One meter, 96 reads a day. Across a national fleet, that is terabytes daily, but most of it is used only once, for billing.
• Learn the four stops: AMI meter to head-end system to MDMMDMMaster Data Management (MDM) is the discipline of creating and maintaining a single, consistent, trusted version of an organization's core business entities like customers, products, and suppliers.View full definition → for customer data, plus SCADA and PMUs for grid telemetry. These are two separate rivers that rarely merge.
• The bottleneck is organizational, not technical. Data lives in vendor silos owned by different teams with no shared model. Connecting them is where value hides.
• Interval data unlocks real money: loss detection, transformertransformerA Transformer is a neural network architecture that uses self-attention to process sequences in parallel, powering most modern language and generative AI models.View full definition → load management, demand response, and faster outage response, but only when customer and grid data are joined.
• Regulation drives behavior. Utilities historically got rewarded for building assets, not exploiting data. As grid modernization mandates grow, the business case for layer-three analytics is finally strengthening.