# Forecasting demand and generation under uncertainty
It is 2 p.m. on a summer afternoon. A grid operator watches solar output on her dashboard drop by a third in twelve minutes as a cloud bank rolls over a region packed with rooftop panels. Air conditioners are still ramping up. If she guesses wrong about how fast solar recovers, she either fires up an expensive gas peaker plant she did not need, or she scrambles to buy power on the spot market at a painful price.
This is the daily reality of forecasting in power systems: getting supply and demand to match, minute by minute, when both are uncertain.
Electricity is unusual. It must be produced and consumed at nearly the same instant. There is very little cheap, large-scale storage on most grids. So operators constantly predict two things:
Traditionally, load was the hard part and generation was easy. A coal or gas plant produces what you tell it to. But as wind and solar grow, generation itself became uncertain, driven by weather rather than a dial. That is intermittency: renewable output swings with clouds, wind gusts, and daylight.
Two related terms you will hear:
Better forecasts mean less wasted reserve, fewer emergency purchases, and lower costs passed to customers.
Old-school forecasting gave a single number: "Tomorrow at 3 p.m., load will be 42,000 megawatts." That is a point forecast. It is useless for risk decisions because it hides how wrong it might be.
Modern practice uses probabilistic forecasts: instead of one number, you get a distribution. For example, "There is a 90 percent chance load falls between 40,000 and 44,000 MW."
Why this matters: an operator does not dispatch against the average. She dispatches against the tail. If being short by 2,000 MW risks a blackout, she cares about the worst plausible case, not the expected case. Probabilistic forecasts let her size reserves to a chosen confidence level.
Machine learning (ML) models shine here because the signal comes from many messy, overlapping data sources. A production forecasting system typically ingests:
Weather data. Temperature drives cooling and heating load. Cloud cover and irradiance (the intensity of sunlight hitting a surface) drive solar. Wind speed at turbine hub height drives wind farms. Forecasters pull from numerical weather prediction models like those published by NOAA and other national services, often as ensembles (many slightly different weather scenarios).
Historical load curves. Demand follows strong daily and weekly rhythms. Monday morning looks like last Monday morning. Models learn these shapes.
Calendar and behavioral features. Holidays, school schedules, and even major sporting events shift demand. A hot day during a holiday behaves differently from a hot workday.
Real-time telemetry. Live meter data, feeder-level readings, and current renewable output let the model correct itself as conditions unfold.
The ML model's job is to mapmapUsing software to automate repetitive marketing tasks and campaigns, enabling personalisation at scale across channels like email, web, and social.View full definition → all of this to a forecast, and crucially to a range around it.
Here is the conceptual shape of quantile training in a common library:
import lightgbm as lgb
# Train separate models for each quantile you care about
quantiles = [0.1, 0.5, 0.9]
models = {}
for q in quantiles:
models[q] = lgb.LGBMRegressor(
objective="quantile", # optimize for a percentile, not the mean
alpha=q # which percentile (e.g., 0.9 = 90th)
)
models[q].fit(X_train, y_train)
# Prediction gives a low / median / high band, not a single numberThe point is not the code. It is that the model is explicitly asked to bound uncertainty, which is exactly what dispatch decisions require.
Back to our operator. A modern solar nowcasting system (short-term forecasting minutes to a few hours ahead) does not wait for the next weather bulletin. It fuses satellite cloud imagery, sky cameras at solar farms, and live output readings to predict the ramp before it fully hits.
If the model warns at 1:45 p.m. that a cloud front will cut regional solar by roughly a third within fifteen minutes, the operator can pre-position flexible resources: charge or discharge batteries, signal a fast-start gas unit, or call on demand response (paying large customers to cut usage briefly). The forecast turns a scramble into a plan.
🎬 [VIDEO: "How Grid Operators Balance Supply and Demand" — youtube.com — a clear primer on real-time grid balancing and why forecasting drives dispatch decisions]
You cannot manage what you do not measure. Two metrics dominate:
A subtle but critical concept is calibration. If your model says "90 percent confidence interval," then real outcomes should land inside that interval about 90 percent of the time. A model can be accurate on average yet badly calibrated, systematically underestimating uncertainty. That is dangerous, because operators trust the band. Utilities validate calibration continuously against actual outcomes.
Forecasts are not an academic exercise. They flow directly into money and reliability:
Knowledge check
1. Why does electricity require constant, minute-by-minute forecasting of both supply and demand in a way that most other commodities do not?
2. Traditionally, generation was considered the 'easy' side of forecasting. Why did the growth of wind and solar change this?
3. Why is a point forecast (a single number) described as 'useless for risk decisions'?
4. Select ALL correct answers about the consequences of better forecasting for a grid operator.
Select all the correct answers.
5. Select ALL correct answers that correctly define the key operational terms introduced in the lesson.
Select all the correct answers.
Distribution shift. Models learn from history. When behavior changes fast (rapid electric vehicle adoption, a new data center connecting, a heat wave outside any historical range), the past is a poor guide. Teams monitor for drift and retrain often.
Rare extreme events. The forecasts that matter most, during a severe storm or a grid emergency, are the ones with the least training data. A model that is excellent on normal days can fail exactly when the stakes are highest. This is why human operators and physics-based backup models stay in the loop.
Behind-the-meter solar. Rooftop panels reduce the load a utility sees without the utility directly metering them. From the grid's view, a sunny afternoon looks like demand mysteriously dropping. Estimating this hidden generation is a distinct, tricky forecasting task.
Correlated errors. A weather miss can throw off load and solar and wind all at once, in the same direction. Errors that stack are far more dangerous than independent ones, and reserve planning must account for that.
AI does not remove uncertainty. It quantifies it better and reacts faster. The winning setups pair ML forecasts with human judgment and physical models, rather than replacing operators. Regulators in many markets also require explainability and auditability, so pure black-box models often sit alongside interpretable baselines.
For a deeper, freely available treatment of forecasting methods and evaluation, the online textbook Forecasting: Principles and Practice is a widely respected starting point.