Mapping the energy data landscape: sources, owners, and formats
A single wind farm in Texas generates SCADA telemetry every 4 seconds, sends settlement data to ERCOT every 5 minutes, reports availability to NERC monthly, and gets weather forecasts refreshed hourly from three different vendors. None of these systems were designed to talk to each other. Welcome to energy data in 2026.
This fragmentation is not an accident. It is the byproduct of a sector built over a century by different actors (utilities, regulators, market operators, equipment vendors) who each optimized for their own function. Your job as a data-literate professional is not to fix this. It is to mapmapUsing software to automate repetitive marketing tasks and campaigns, enabling personalisation at scale across channels like email, web, and social.View full definition → it, understand its gaps, and know where to look before you build anything on top of it.
Why the landscape is so fragmented
Energy systems mix physical infrastructure (turbines, transformers, pipelines) with financial markets (wholesale power, capacity, carbon credits) and regulatory oversight (safety, reliability, environmental compliance). Each layer generates its own dataown dataData collected directly from your own customers and prospects through your own channels: your most reliable and privacy-compliant source.View full definition →, on its own schedule, in its own format.
SCADA (Supervisory Control and Data Acquisition) systems are the nervous system of physical assets: they poll sensors on generators, substations, and pipelines in near real time, often every few seconds. This data is proprietary, vendor-specific (Siemens, GE, Schneider Electric are major suppliers), and historically kept air-gapped from corporate IT for cybersecurity reasons.
Meter Data Management (MDM) systems sit downstream of smart meters. In the US, over 100 million smart meters were deployed as of 2023 (estimate, U.S. Energy Information Administration, EIA), each reporting interval consumption every 15 minutes to hourly. MDMMDMMaster Data Management (MDM) is the discipline of creating and maintaining a single, consistent, trusted version of an organization's core business entities like customers, products, and suppliers.View full definition → systems clean, validate, and estimate (VEE process) this data before it reaches billing.
ISO/RTO market data comes from Independent System Operators and Regional Transmission Organizations, the entities that run wholesale electricity markets. PJM Interconnection, ERCOT, CAISO, and MISO in the US, or ENTSO-E in Europe, publish locational marginal prices, load forecasts, and generation mix data, typically on 5-minute to hourly cadences, through public APIs.
Weather APIs feed renewable forecasting and demand models. Providers like NOAA (National Oceanic and Atmospheric Administration, free) or commercial vendors like Tomorrow.io deliver temperaturetemperatureA setting that controls how random or predictable an AI model's output is: low keeps it safe and consistent, high makes it more varied and creative.View full definition →, wind speed, and irradiance data at varying spatial resolutions.
Regulatory filings are the slowest-moving layer: annual FERC Form 1 reports, EIA-923 generation and fuel data, or EU emissions filings under the Monitoring, Reporting and Verification (MRV) regulation. These arrive quarterly or annually, often as PDFs or structured but delayed datasets.
The cadence and format mismatch
| Source | Typical cadence | Common format | Owner |
|---|---|---|---|
| SCADA | 1-4 seconds | OPC-UA, proprietary binary | Asset operator / OT team |
| Smart meter (MDM) | 15 min to 1 hour | CSV, XML, MultiSpeak | Utility distribution ops |
| ISO/RTO market data | 5 min to 1 hour | JSON, CSV via REST APIREST APIApplication Programming Interface: a standardised interface that lets applications communicate and exchange data without knowing each other's internal workings.View full definition → | ISO/RTO (e.g., PJM, ERCOT) |
| Weather feeds | 15 min to 6 hours | JSON, GRIB2 | NOAA, commercial vendors |
| Regulatory filings | Monthly to annual | PDF, XBRL, flat files | FERC, EIA, national regulators |
This table is the seed of what you should build first: a data source inventory. It's a living document (a spreadsheet or a lightweight database) that catalogs every data source your organization touches, who owns it, its refresh rate, its format, and its known quality issues.
Building a data source inventory
A good inventory answers five questions for every source:
- What is it? (name, description, business purpose)
- Who owns it? (internal team or external party; include a contact)
- How often does it update? (cadence, and whether that cadence is guaranteed or best-effort)
- What format and access method? (API, SFTP, flat file, manual upload)
- What are the known gaps? (missing history, unreliable timestamps, unit inconsistencies)
Here is a minimal schemaschemaA schema is the formal blueprint that defines how data is structured, named, typed, and related within a database, file, or message.View full definition → you could use in a database or as a config file:
source_id: ercot_lmp_rt
name: "ERCOT Real-Time LMP"
owner: "Market Operations - J. Alvarez"
cadence_minutes: 5
format: "JSON via REST API"
access_url: "https://www.ercot.com/mp/data-products"
known_gaps:
- "Historical data before 2019 requires manual archive request"
- "Node-level prices occasionally delayed during grid emergencies"
last_validated: "2026-01-15"Doing this exercise across even 10 to 15 core sources typically surfaces two things fast: duplicate data being pulled by different teams in incompatible formats, and critical gaps (for example, no owner assigned to a legacy SCADA historian nobody has touched in three years).
Where the gaps usually hide
Three patterns show up repeatedly across utilities and generators:
- Orphaned systems: A SCADA historian or an old MDM instance where the original vendor contract ended and no internal team formally owns it anymore.
- Format drift: An ISO changes its API schema (this happens periodically, PJM and CAISO have both revised data formats in recent years) and downstream pipelines silently break or misparse fields.
- Cadence mismatch: Combining 5-minute market prices with hourly weather data and monthly regulatory filings without resampling correctly, producing analyses that quietly conflate different time resolutions.
A practical starting point for exploring real public data: the EIA Open Data API gives free access to US generation, consumption, and pricing datasets, a good sandbox for practicing inventory and format-mapping skills before touching proprietary utility systems.
Knowledge check
1. Why has SCADA data historically been kept air-gapped from corporate IT systems?
2. What is the fundamental reason energy sector data is so fragmented across formats and owners?
3. A data professional joining an energy company is asked to integrate SCADA telemetry with market settlement data. What should be their FIRST step, according to the mapping approach described in the lesson?
4. Select ALL correct answers about why energy data spans such different reporting frequencies and formats.
Select all the correct answers.
5. Select ALL correct answers about Meter Data Management (MDM) systems.
Select all the correct answers.
From inventory to governance
Once your inventory exists, it becomes the foundation for basic data governancedata governanceData governance is the set of policies, roles, and processes that ensure data is accurate, secure, well-defined, and used responsibly across an organization.View full definition → questions: Which sources are "system of record" versus derived copies? Which sources need a data qualitydata qualityThe degree to which data is fit for purpose: accurate, complete, consistent, timely, valid and unique. Poor quality data undermines analytics, reporting and AI.View full definition → SLA (Service Level Agreement) with an internal or external owner? Which gaps are acceptable risk versus which block a regulatory report?
For instance, if your emissions reporting under the EU's MRV regulation depends on a fuel consumption feed that has a known 3-day lag and occasional missing days, that's not just a technical footnote. It's a compliance risk that needs an escalation path, not a silent workaround in a spreadsheet.
Regulators increasingly expect this kind of traceability. FERC's data reporting requirements and the EU's Agency for the Cooperation of Energy Regulators (ACER) transaction reporting under REMIT (Regulation on Energy Market Integrity and Transparency) both assume organizations can show data lineage, that is, where a reported number came from and what transformations touched it along the way.
Key Takeaways
- Energy data comes from five structurally different layers (SCADA, MDM, ISO/RTO markets, weather, regulatory filings), each with its own cadence, format, and owner. Treat them as separate systems to map, not one unified pipeline to assume.
- Build a data source inventory early: for each source, capture owner, cadence, format, access method, and known gaps. This single artifact prevents duplicated pulls and orphaned systems.
- Watch for three recurring failure patterns: orphaned systems with no clear owner, silent format drift from API changes, and cadence mismatches when blending high-frequency market data with low-frequency regulatory data.
- Free public sources like the EIA Open Data API and ISO/RTO public feeds (ERCOT, PJM, CAISO) are excellent low-risk sandboxes for practicing inventory and integration skills.
- Data lineage is not optional in regulated reporting: REMIT and FERC-style requirements assume you can trace a reported figure back to its raw source and every transformation applied.