Mapping the energy data landscape: sources, owners, and formats

SCADA (Supervisory Control and Data Acquisition) systems are the nervous system of physical assets: they poll sensors on generators, substations, and pipelines in near real time, often every few seconds. This data is proprietary, vendor-specific (Siemens, GE, Schneider Electric are major suppliers), and historically kept air-gapped from corporate IT for cybersecurity reasons.

Meter Data Management (MDM) systems sit downstream of smart meters. In the US, over 100 million smart meters were deployed as of 2023 (estimate, U.S. Energy Information Administration, EIA), each reporting interval consumption every 15 minutes to hourly. MDMMDMMaster Data Management (MDM) is the discipline of creating and maintaining a single, consistent, trusted version of an organization's core business entities like customers, products, and suppliers.Voir la définition complète → systems clean, validate, and estimate (VEE process) this data before it reaches billing.

ISO/RTO market data comes from Independent System Operators and Regional Transmission Organizations, the entities that run wholesale electricity markets. PJM Interconnection, ERCOT, CAISO, and MISO in the US, or ENTSO-E in Europe, publish locational marginal prices, load forecasts, and generation mix data, typically on 5-minute to hourly cadences, through public APIs.

Weather APIs feed renewable forecasting and demand models. Providers like NOAA (National Oceanic and Atmospheric Administration, free) or commercial vendors like Tomorrow.io deliver temperature, wind speed, and irradiance data at varying spatial resolutions.

Regulatory filings are the slowest-moving layer: annual FERC Form 1 reports, EIA-923 generation and fuel data, or EU emissions filings under the Monitoring, Reporting and Verification (MRV) regulation. These arrive quarterly or annually, often as PDFs or structured but delayed datasets.

The cadence and format mismatch

| Source | Typical cadence | Common format | Owner |

|---|---|---|---|

| SCADA | 1-4 seconds | OPC-UA, proprietary binary | Asset operator / OT team |

| Smart meter (MDMMDMMaster Data Management (MDM) is the discipline of creating and maintaining a single, consistent, trusted version of an organization's core business entities like customers, products, and suppliers.Voir la définition complète →) | 15 min to 1 hour | CSV, XML, MultiSpeak | Utility distribution ops |

| ISO/RTO market data | 5 min to 1 hour | JSON, CSV via REST APIREST APIApplication Programming Interface: a standardised interface that lets applications communicate and exchange data without knowing each other's internal workings.Voir la définition complète → | ISO/RTO (e.g., PJM, ERCOT) |

| Weather feeds | 15 min to 6 hours | JSON, GRIB2 | NOAA, commercial vendors |

| Regulatory filings | Monthly to annual | PDF, XBRL, flat files | FERC, EIA, national regulators |

This table is the seed of what you should build first: a data source inventory. It's a living document (a spreadsheet or a lightweight database) that catalogs every data source your organization touches, who owns it, its refresh rate, its format, and its known quality issues.

Building a data source inventory

A good inventory answers five questions for every source:

1. What is it? (name, description, business purpose)

2. Who owns it? (internal team or external party; include a contact)

3. How often does it update? (cadence, and whether that cadence is guaranteed or best-effort)

4. What format and access method? (APIAPIApplication Programming Interface: a standardised interface that lets applications communicate and exchange data without knowing each other's internal workings.Voir la définition complète →, SFTP, flat file, manual upload)

5. What are the known gaps? (missing history, unreliable timestamps, unit inconsistencies)

Here is a minimal schemaschemaA schema is the formal blueprint that defines how data is structured, named, typed, and related within a database, file, or message.Voir la définition complète → you could use in a database or as a config file:

Doing this exercise across even 10 to 15 core sources typically surfaces two things fast: duplicate data being pulled by different teams in incompatible formats, and critical gaps (for example, no owner assigned to a legacy SCADA historian nobody has touched in three years).

Where the gaps usually hide

Three patterns show up repeatedly across utilities and generators:

• Orphaned systems: A SCADA historian or an old MDMMDMMaster Data Management (MDM) is the discipline of creating and maintaining a single, consistent, trusted version of an organization's core business entities like customers, products, and suppliers.Voir la définition complète → instance where the original vendor contract ended and no internal team formally owns it anymore.
• Format drift: An ISO changes its APIAPIApplication Programming Interface: a standardised interface that lets applications communicate and exchange data without knowing each other's internal workings.Voir la définition complète → schemaschemaA schema is the formal blueprint that defines how data is structured, named, typed, and related within a database, file, or message.Voir la définition complète → (this happens periodically, PJM and CAISO have both revised data formats in recent years) and downstream pipelines silently break or misparse fields.
• Cadence mismatch: Combining 5-minute market prices with hourly weather data and monthly regulatory filings without resampling correctly, producing analyses that quietly conflate different time resolutions.

A practical starting point for exploring real public data: the EIA Open Data API gives free access to US generation, consumption, and pricing datasets, a good sandbox for practicing inventory and format-mapping skills before touching proprietary utility systems.

From inventory to governance

Once your inventory exists, it becomes the foundation for basic data governancedata governanceData governance is the set of policies, roles, and processes that ensure data is accurate, secure, well-defined, and used responsibly across an organization.Voir la définition complète → questions: Which sources are "system of record" versus derived copies? Which sources need a data qualitydata qualityThe degree to which data is fit for purpose: accurate, complete, consistent, timely, valid and unique. Poor quality data undermines analytics, reporting and AI.Voir la définition complète → SLA (Service Level Agreement) with an internal or external owner? Which gaps are acceptable risk versus which block a regulatory report?

For instance, if your emissions reporting under the EU's MRV regulation depends on a fuel consumption feed that has a known 3-day lag and occasional missing days, that's not just a technical footnote. It's a compliance risk that needs an escalation path, not a silent workaround in a spreadsheet.

Regulators increasingly expect this kind of traceability. FERC's data reporting requirements and the EU's Agency for the Cooperation of Energy Regulators (ACER) transaction reporting under REMIT (Regulation on Energy Market Integrity and Transparency) both assume organizations can show data lineagedata lineageData lineage maps how data moves and transforms across systems, from origin to consumption, showing where it came from, what changed it, and where it goes.Voir la définition complète →, that is, where a reported number came from and what transformations touched it along the way.

Key Takeaways

• Energy data comes from five structurally different layers (SCADA, MDMMDMMaster Data Management (MDM) is the discipline of creating and maintaining a single, consistent, trusted version of an organization's core business entities like customers, products, and suppliers.Voir la définition complète →, ISO/RTO markets, weather, regulatory filings), each with its own cadence, format, and owner. Treat them as separate systems to mapmapUsing software to automate repetitive marketing tasks and campaigns, enabling personalisation at scale across channels like email, web, and social.Voir la définition complète →, not one unified pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing. to assume.