Data Lineage
Also: Data Provenance, Lineage Tracking
Data lineage maps how data moves and transforms across systems, from origin to consumption, showing where it came from, what changed it, and where it goes.
What It Is
Data lineage is the documented map of data as it flows through an organization. It traces each data element from its source (databases, APIs, files, sensors) through every transformation, join, aggregation, and copy, all the way to its final destinations (dashboards, reports, models, downstream systems).
Lineage answers questions like: Where did this number come from? What logic produced it? What breaks if this table changes? Which reports depend on this column?
Lineage is typically captured at two levels:
- Table or dataset level: which datasets feed which other datasets.
- Column or field level: how individual fields are derived, which is more precise and more useful for impact analysis.
Why it matters
Modern data estates are sprawling and interconnected, so a single change can ripple unpredictably. Lineage provides:
- Trust: consumers can verify how a metric was calculated.
- Impact analysis: before changing a source, teams see every downstream asset affected.
- Root cause analysis: when a dashboard looks wrong, lineage helps trace the broken link upstream quickly.
- Regulatory compliance: frameworks like GDPR, BCBS 239, and SOX require proof of where sensitive or financial data originates and how it is processed.
- Governance: classifications, ownership, and quality rules can propagate along lineage paths.
For a Chief Data Officer, lineage is foundational evidence that data is governed, auditable, and reliable.
How it is used in practice
Lineage can be captured through:
- Automated parsing of SQL, ETL pipelines, and orchestration logs.
- Metadata extraction from data platforms and catalogs.
- Manual documentation for systems that cannot be parsed (often the weakest, least maintained part).
Teams then use it for impact reviews, audits, deprecation planning, and onboarding.
Concrete Example
A finance team reports Net Revenue on an executive dashboard. Lineage shows the metric is built from a `transactions` table joined to `refunds`, filtered by region, and adjusted by a currency conversion table. When the currency feed changes its schema, lineage instantly flags that the Net Revenue tile, three other reports, and a forecasting model all depend on it. The team fixes the pipeline before stakeholders see incorrect figures, turning a potential incident into a routine maintenance task.
Without lineage, that same change might silently corrupt numbers for weeks before anyone noticed.