Data pipeline
Aussi : data pipeline, pipeline, ETL pipeline, ELT pipeline, data flow, pipeline de données, chaîne de traitement des données
An automated sequence of steps that moves data from source to destination: ingestion, transformation, validation, and loading, so it arrives clean and ready to use.
What it is
A data pipeline is an automated series of steps that carries data from where it originates (a source) to where it is consumed (a destination), applying processing along the way. The core stages are usually:
- Ingestion: pulling or receiving raw data from sources such as databases, APIs, files, event streams, or SaaS tools.
- Transformation: cleaning, reshaping, joining, and enriching the data into a usable form.
- Validation: checking quality, completeness, and conformity to expected rules before the data moves on.
- Loading: writing the result into a destination such as a data warehouse, lake, dashboard, or application.
Pipelines can run in batch (scheduled, for example hourly or nightly) or in streaming mode (continuously, near real time).
Why it matters
Without pipelines, data stays trapped in silos and every analysis becomes a manual, error-prone copy-paste exercise. A well-built pipeline delivers:
- Reliability: the same steps run the same way every time, with monitoring and alerts.
- Freshness: decisions rest on current data, not last quarter's export.
- Trust: validation catches bad data before it reaches reports or models.
- Scale: volumes that a person could never process by hand flow automatically.
For executives, the pipeline is the plumbing behind every dashboard, forecast, and AI feature. When it breaks silently, numbers drift and confidence erodes.
How it is used in practice
Teams describe pipelines as code (often called orchestration), schedule them, and monitor each run. Key practical concerns include handling failures gracefully, reprocessing when a source changes, tracking lineage (where each number came from), and controlling cost.
Worked example
A retailer wants a daily revenue-by-region dashboard:
1. Ingestion: each night the pipeline pulls orders from the e-commerce database and exchange rates from a currency API.
2. Transformation: it converts every order to euros, joins each order to its store region, and aggregates totals.
3. Validation: it checks that no region is missing and that today's total is within a plausible range of yesterday's. If a rule fails, it alerts the data team and stops.
4. Loading: clean, aggregated figures land in the warehouse, and the dashboard refreshes.
The CFO sees trustworthy numbers by 8 a.m., without anyone touching a spreadsheet.