Your data pipelines are breaking. Not because of bugs. Because nobody agreed on what the data should look like before it was built.
A data contract is the solution, and it's one of the most practical governance innovations of the last five years.
In a typical data organization, the journey of data from source to consumer looks like this:
1. A production engineer adds a new field to the checkout APIAPIApplication Programming Interface: a standardised interface that lets applications communicate and exchange data without knowing each other's internal workings.View full definition → response
2. They forget (or don't know) that a downstream data pipelinedata pipelineETL (Extract, Transform, Load) is a data integration process that pulls data from sources, reshapes it into a consistent format, and writes it into a target system. depends on the existing
3. The pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.View full definition → breaks silently, it processes the data but produces incorrect results
4. Three weeks later, a business analyst notices revenue numbers look wrong
5. The data team spends two days debugging across six systems to find the source
This happens everywhere, every week, at every organization with a non-trivial data infrastructure. It's not a people problem. It's an architecture problem: there is no formal agreement between data producers (engineering teams that build source systems) and data consumers (data teams that build pipelines and analytics).
A data contract is that agreement. It specifies:
When the production engineer adds that new field, the data contract is updated. The change triggers automated tests. The pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.View full definition → is updated before it breaks. The analyst's dashboard never sees the problem.
Knowledge check
1. According to the lesson, what is the fundamental root cause of the pipeline-breaking scenario described (engineer adds a field, pipeline breaks silently)?
2. In the context of data contracts, what does the 'Semantics' element specify that 'Schema' does not?
3. Why does the lesson recommend storing a data contract as a YAML or JSON file in version control (Git) alongside the producing code?
4. Select ALL elements that a data contract typically specifies, according to the lesson.
Select all the correct answers.
5. Select ALL statements that correctly describe how a data contract prevents the silent-breakage scenario.
Select all the correct answers.
A data contract is typically a YAML or JSON file stored in version control (Git), alongside the code that produces the data. A simplified example would specify:
When this contract is violated, order_amount_eur has null values, or freshness exceeds 15 minutes, an alert fires. The producer team is notified. Consumers aren't surprised by degraded data qualitydata qualityThe degree to which data is fit for purpose: accurate, complete, consistent, timely, valid and unique. Poor quality data undermines analytics, reporting and AI.View full definition →; they're notified of the violation.
Shopify implemented an internal data contracts program at scale across their data platform. Their approach: any dataset served through their internal data platform must have a contract. Teams that consume data without a contract cannot hold producers accountable for quality. Teams that produce data without a contract cannot claim downstream consumers are using it correctly.
The outcome: a significant reduction in pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.View full definition → incidents caused by schemaschemaA schema is the formal blueprint that defines how data is structured, named, typed, and related within a database, file, or message.View full definition → drift, a clearer accountability model for data qualitydata qualityThe degree to which data is fit for purpose: accurate, complete, consistent, timely, valid and unique. Poor quality data undermines analytics, reporting and AI.View full definition →, and faster root-cause analysis when issues do occur.
Don't try to contract everything at once. Identify the five most business-critical data flows, the ones where a failure causes the most pain. Contract those first. Use the experience to refine your contract template and your change management process. Then scale.
Tools in this space: Soda (contract testing), Great Expectations (data qualitydata qualityThe degree to which data is fit for purpose: accurate, complete, consistent, timely, valid and unique. Poor quality data undermines analytics, reporting and AI.View full definition → validation), Gable.ai (data contract management platform), dbt contracts (built-in to dbt for transformation layer). The tooling is maturing rapidly, choose based on where in your stack you want to enforce contracts.
These actions are compiled in the role's Playbook.