# Data LineageData LineageData lineage maps how data moves and transforms across systems, from origin to consumption, showing where it came from, what changed it, and where it goes.Voir la définition complète → and Impact Analysis
On a Friday in 2012, Knight Capital deployed a software change that misfired and lost the firm $440 million in 45 minutes. The failure was code, not data — but the deeper lesson is the one every CDO now lives with: in a tightly coupled system, a change in one node propagates faster than any human can trace it. Now shrink that to your world. A vendor renames a column in a source feed. Eight hours later, three regulatory reports are wrong, a churn model is silently degrading, and the CFO is quoting a revenue number that no longer ties out. The question is not *whether* the break happened. It's whether you can answer, in minutes, what else just broke.
That answer is lineage. Not the diagram your architecture team drew once and framed. Living, queryable, column-level lineage that turns "we think it's fine" into "here are the 14 assets affected, ranked by blast radius." This lesson is about building lineage that earns trust and makes change safe — and about the judgement calls that separate a lineage program that pays for itself from one that becomes expensive shelfware.
There are two directions of lineage, and a CDO must be fluent in both because they answer opposite questions:
The gap that kills programs is granularity. Table-level lineage — "table A feeds table B" — is nearly worthless for impact analysis. If a source table has 60 columns and one is deprecated, table-level lineage tells you every downstream asset *might* be affected, which is functionally the same as knowing nothing. Column-level lineage is the threshold of usefulness. It lets you say: "The customer_tier field changed; that touches the pricing model and the exec churn dashboard, but not the finance close." Precision here is what converts lineage from a compliance artifact into an incident-response tool.
Consider how Airbnb approached this with their internal Dataportal work: the animating insight was that data trust is a *discovery* problem as much as a governance one. People couldn't tell which of five similarly named tables was the certified one. Lineage — combined with usage signals — became the tiebreaker. The table that feeds the CEO's dashboard and has 200 downstream dependents is self-evidently the one to trust. Lineage doesn't just protect the system; it *ranks* it.
Real lineage lives at three altitudes, and most tools only capture one:
1. Physical lineage — the actual data movement: this SQLSQLSales Qualified Lead: a prospect the sales team has validated as ready for direct outreach and a proposal, having passed clear qualification criteria.Voir la définition complète → SELECT reads these columns and writes those. Captured from query logs, ETLETLETL (Extract, Transform, Load) is a data integration process that pulls data from sources, reshapes it into a consistent format, and writes it into a target system.Voir la définition complète → job definitions, and warehouse metadatametadataDonnées sur les données, informations décrivant le contexte, la structure, la provenance et les caractéristiques d'un asset de données (auteur, date, format, source, définition)..
2. Logical lineage — the business-transformation view: "net revenue is gross revenue minus refunds and chargebacks." This is where a source column maps to a business concept.
3. Business lineage — which reports, KPIs, models, and decisions ultimately consume the asset.
The value is in the *stitching*. A break in a physical source is only alarming when you can connect it up through the logical layer to the business layer and say the words that make an executive care: "This affects the number you present to the regulator on the 15th." Lineage that stops at the physical layer generates alerts no one can prioritize.
Here is the uncomfortable truth every CDO learns: manually maintained lineage is dead lineage. The moment you rely on engineers to update a lineage document as part of their workflow, entropy wins. Within two quarters it's wrong, and wrong lineage is worse than none because it manufactures false confidence.
There are three harvesting strategies, and mature programs use all three in combination:
1. Query-log parsing (automated, physical). Modern warehouses log every executed query. Parse the SQLSQLSales Qualified Lead: a prospect the sales team has validated as ready for direct outreach and a proposal, having passed clear qualification criteria.Voir la définition complète →, extract the column-level read/write relationships, and you reconstruct physical lineage automatically with zero engineer effort. This is the backbone. Tools like OpenLineage, or the parsers inside dbt, Databricks Unity Catalog, and Snowflake's ACCESS_HISTORY, do this natively.
2. Framework-emitted lineage (automated, semantic). When transformations run through a framework that understands them — dbt models with declared ref() dependencies, or Spark jobs instrumented with OpenLineage — lineage is emitted as a byproduct of execution. This is the highest-fidelity source because it captures intent, not just observed behavior.
3. Declarative annotation (manual, for the gaps). The last mile — a business analyst pulling data into a spreadsheet, a report built on a hand-written query, a legacy system with no logs — must be captured manually. Keep this set small and treat it as technical debt to be automated away.
A clean OpenLineage event — the emerging open standard — captures the essential grammar of a lineage fact:
{
"eventType": "COMPLETE",
"job": { "namespace": "prod-etl", "name": "build_customer_revenue" },
"inputs": [
{ "namespace": "warehouse", "name": "raw.orders",
"facets": { "schema": { "fields": ["order_id","amount","refund_flag"] } } }
],
"outputs": [
{ "namespace": "warehouse", "name": "mart.customer_revenue",
"facets": { "columnLineage": {
"fields": { "net_revenue": { "inputFields": ["raw.orders.amount","raw.orders.refund_flag"] } }
} } }
]
}The point of showing this is not to make you write JSON. It's to make one decision concrete: standardize on an open lineage schema early. If every tool emits lineage in its own proprietary format, you will spend years building brittle translators. OpenLineage exists precisely so that your orchestrator, warehouse, and catalog can speak a common language and you can query lineage across the whole estate rather than tool by tool.
Executives ask "what's our lineage coverage?" and the answer they want — a single percentage — is a trap. Coverage as a raw number is meaningless because not all assets carry equal risk. Ninety percent coverage that excludes your top revenue-reporting pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.Voir la définition complète → is a failure; forty percent coverage that fully captures every regulatory and financial asset is a success.
Measure coverage weighted by criticality, not by count. The metric that matters: *of your Tier-1 assets (regulatory, financial, executive-facing, model-feeding), what fraction have complete column-level lineage to source?* Drive that to 100% before you spend a dollar chasing the long tail of ad-hoc departmental tables.
Lineage is inventory. Impact analysis is the *use* of that inventory when something is about to change — or already has. This is where a CDO demonstrates ROIROIReturn on Investment: the ratio of net profit to the cost of an investment. A 300% ROI means each dollar invested returns $3.Voir la définition complète →, because it operationalizes lineage across three distinct workflows.
Reactive: incident triage. A source feed breaks at 6 a.m. The on-call analyst queries forward lineage from the broken column and gets a ranked list of downstream assets. The ranking is the intelligence — sort by a blast-radius score that combines number of dependents, tier of those dependents, and freshness sensitivity. The output is a triage list, not a data dump: "Freeze these two dashboards, notify these three model owners, the rest can wait."
Proactive: change management. Before a schemaschemaA schema is the formal blueprint that defines how data is structured, named, typed, and related within a database, file, or message.Voir la définition complète → change ships, run impact analysis in the pull request. If an engineer proposes dropping customer_tier, the CI check surfaces every downstream consumer and blocks the merge until owners sign off. This is the single highest-leverage use of lineage, because it moves failures *left* — you prevent the 6 a.m. incident instead of triaging it. Netflix and others have built exactly this into their data-change pipelines: no breaking schemaschemaA schema is the formal blueprint that defines how data is structured, named, typed, and related within a database, file, or message.Voir la définition complète → change merges without an automated downstream-impact review.
Strategic: deprecation and cost. Forward lineage answers the question that unlocks real savings: *"If nothing consumes this table, can we kill it?"* Combine lineage with usage telemetry, and you can retire orphaned pipelines with confidence rather than leaving them running for years because no one dared touch them.
Here is where senior judgement enters, because tooling vendors will oversell this. Lineage tells you what is *connected*, not what is *correct* or what *matters*. Three failure modes to guard against:
The CDO's contribution is not running the lineage tool — it's setting the policy that makes lineage consequential: which asset tiers require pre-change impact review, what blast-radius threshold triggers a mandatory sign-off, and who is accountable when a break propagates unreviewed. Lineage without an enforcement policy is a mapmapUsing software to automate repetitive marketing tasks and campaigns, enabling personalisation at scale across channels like email, web, and social.Voir la définition complète → no one is required to read.
Vérification des acquis
1. The Knight Capital incident is used in the lesson primarily to illustrate which principle about modern data systems?
2. A vendor renames a column in a source feed, and hours later reports are wrong and a model is degrading. Which type of lineage most directly answers the CDO's urgent question 'what else just broke?'
3. According to the lesson, what distinguishes lineage that 'earns trust' from lineage that becomes 'expensive shelfware'?
4. Select ALL correct answers about how the lesson frames the relationship between lineage and data trust.
Sélectionnez toutes les réponses correctes.
5. Select ALL correct answers that correctly characterize upstream (backward) versus downstream (forward) lineage.
Sélectionnez toutes les réponses correctes.
A lineage program lives or dies on ownership, and the ownership question is subtler than "assign a data stewarddata stewardA business-side owner responsible for the quality, consistency and appropriate use of data in their domain.Voir la définition complète →." Lineage crosses every team boundary in your organization — that's the point of it — which means no single team naturally owns the whole graph.
The operating model that works assigns responsibility at the node, not the graph. Every Tier-1 asset has a named owner who is accountable for two things: that the asset's lineage is captured and current, and that they respond within an SLA when an upstream impact alert names them. The central data-governance function owns the *platform* (the catalog, the standard, the harvesting infrastructure) and the *policy* (tiering, sign-off thresholds). It does not own the correctness of every edge — that federates to node owners. This is the only model that scales past a few hundred assets.
Tie lineage into the incident-management process you already run. When a data incident is declared, the first automated action should attach the forward-lineage impact list to the incident ticket. This does two things: it accelerates triage, and it creates a feedback loop — every incident that reveals a *missing* lineage edge becomes a ticket to close the gap. Over eighteen months, this incident-driven backfill closes coverage faster than any top-down mapping project, because it prioritizes exactly the paths that break in production.
Finally, measure the program by outcomes an executive recognizes: mean time to identify impact during incidents (should drop from hours to minutes), percentage of breaking changes caught pre-deployment (should rise toward 100% for Tier-1), and Tier-1 lineage coverage. These are the three numbers that justify the investment in a budget conversation — not the size of your lineage graph.
1. Column-level or it doesn't count. Table-level lineage is too coarse for impact analysis. Set column-level, source-to-consumer lineage as the standard for all Tier-1 assets and drive that coverage to 100% before touching the long tail.
2. Automate harvesting; treat manual lineage as debt. Build on query-log parsing and framework-emitted lineage (standardize on OpenLineage). Any manually maintained edge is a decaying asset — track it and automate it away.
3. Shift impact analysis left. The highest-ROIROIReturn on Investment: the ratio of net profit to the cost of an investment. A 300% ROI means each dollar invested returns $3.Voir la définition complète → use of lineage is a pre-deployment CI check that blocks breaking schemaschemaA schema is the formal blueprint that defines how data is structured, named, typed, and related within a database, file, or message.Voir la définition complète → changes until downstream owners sign off. Preventing the 6 a.m. incident beats triaging it every time.
4. Rank blast radius by tier and usage, not edge count. A change touching one regulatory report outranks one touching fifty abandoned dashboards. Tune alerts to fire on tier-boundary crossings only, or you'll drown owners in noise and lose them.
5. Own the policy, federate the nodes. As CDO, own the platform, the standard, and the enforcement rules — which tiers require impact review and who is accountable. Push edge-level correctness to named node owners, and let production incidents drive your coverage backfill.