measuring pipeline health: SLAs, drift and downtime

The 3 A.M. Incident

At 3:14 a.m., a mid-sized neobank's bank-feed aggregator (a service like Plaid or Tink that pulls transaction data from partner banks via APIAPIApplication Programming Interface: a standardised interface that lets applications communicate and exchange data without knowing each other's internal workings.View full definition →) pushed an update. Nobody announced it. A field called transaction_type quietly changed from a numeric code (1, 2, 3) to a text string ("debit", "credit", "transfer").

The pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.View full definition → didn't crash. It kept ingesting data all night. But every downstream job that filtered on the old numeric codes silently returned zero matching rows. By 8 a.m., 40,000 customers opened the app to see stale balances and missing transactions. No error, no alert, just wrong data delivered with total confidence.

This is a schema drift failure: the structure of incoming data changes without the pipeline breaking outright. It's one of the most common and hardest-to-catch failure modes in fintech data infrastructure, because "the system is technically running" and "the data is right" are two different claims. This lesson builds the metrics that catch the gap between them.

The data that matters: sources worth monitoring

Before defining metrics, know what you're measuring. In fintech, the highest-risk data sources are:

• Bank-feed APIs (Plaid, Tink, MX, Yodlee): third-party aggregators pulling transaction and balance data from thousands of bank connections you don't control.
• Card network and payment rails data (Visa, Mastercard, ACH, SEPA in Europe): settlement files and authorization messages with strict formats.
• Core banking system exports: ledger and account data from vendors like Mambu, Temenos, or legacy mainframes.
• Credit bureau feeds (Experian, Equifax, TransUnion in the US; Schufa in Germany, Experian in the UK): batch files scoring creditworthiness.
• KYC/AML data (Know Your Customer, Anti-Money Laundering): identity verification feeds from providers like Onfido or LexisNexis.
• Market data feeds: pricing and FX data, often from Bloomberg or Refinitiv, for trading or treasury functions.

Each has different volatility. Bank-feed APIs change schemas often because you're aggregating hundreds of underlying bank formats. Card network files are rigid and standardized (ISO 8583 messaging), so drift there is rarer but more disruptive when it happens.

SLASLAA formal commitment defining the service level a provider guarantees to a customer, with measurable targets and consequences if they are missed.View full definition → metrics: defining "working"

An SLA (Service Level Agreement) is a measurable promise about pipeline behavior. For data pipelines, the core SLA metrics are:

1. Freshness (latency)

How old is the data when it lands? Example: "Transaction data must be available in the warehouse within 15 minutes of the bank posting it."

2. Completeness

Did all expected records arrive? If you expect 50,000 daily transaction rows from a partner bank and get 12,000, that's a completeness failure even if the pipeline "ran successfully."

3. Uptime / availability

Percentage of time the pipeline or API endpoint is reachable and responding. Commonly quoted as "three nines" (99.9%) or "four nines" (99.99%).

Worked example: 99.9% uptime over a 30-day month allows:

30 days × 24 hours × 60 minutes = 43,200 minutes total
Allowed downtime = 43,200 × (1 - 0.999) = 43.2 minutes/month

Compare to 99.99% (four nines): allowed downtime = 4.32 minutes/month, roughly ten times stricter. Payment rails and card authorization systems typically target four nines or higher because each minute of downtime blocks live purchases. Internal analytics pipelines often tolerate three nines, since a delayed dashboard is inconvenient, not customer-facing.

4. Accuracy

Does the data match ground truth? Harder to measure automatically, usually sampled via reconciliation (comparing pipeline output against a trusted source, like the bank's own statement).

Data drift: the silent failure mode

SchemaSchemaA schema is the formal blueprint that defines how data is structured, named, typed, and related within a database, file, or message.View full definition → drift, like our 3 a.m. example, is one type. There are three worth distinguishing:

• Schema drift: field names, types, or structure change (numeric code becomes a string).
• Distributional drift: the shape of the data shifts even though the schema is unchanged. Example: average transaction amount jumps from $45 to $450 because a new merchant category got misrouted.
• Volume drift: row counts spike or collapse. A batch that normally delivers 2 million records suddenly delivers 200.

Detection approach: set statistical thresholds, not just presence/absence checks.

# Simplified drift check: compare today's distribution to a 30-day baseline
import numpy as np

baseline_mean, baseline_std = 45.0, 12.0  # from trailing 30-day window
today_mean = df['transaction_amount'].mean()

z_score = (today_mean - baseline_mean) / baseline_std
if abs(z_score) > 3:
    alert("Distributional drift detected: transaction amounts")

A z-score above 3 (three standard deviations from the historical mean) is a common trigger threshold for anomaly alerts, though the right number depends on how noisy your data normally is.

Open-source tools built for this include Great Expectations and Evidently AI, both free and widely used for automated data qualitydata qualityThe degree to which data is fit for purpose: accurate, complete, consistent, timely, valid and unique. Poor quality data undermines analytics, reporting and AI.View full definition → and drift checks in production pipelines.

Governance metrics: who's accountable

Data quality isn't just technical, it's organizational. Governance metrics track accountability:

• Data lineage coverage: percentage of critical datasets with a documented path from source to dashboard. Regulators like the ECB (European Central Bank) and the Federal Reserve increasingly expect lineage documentation for risk-relevant data, especially under frameworks like BCBS 239 (Basel Committee principles for risk data aggregation).
• Time to detect (TTD): how long between when a data issue occurs and when it's flagged. In the incident above, TTD was roughly 5 hours (3:14 a.m. to 8 a.m. customer complaints), which is the actual failure: not that drift happened, but that nobody caught it fast.
• Time to resolve (TTR): from detection to fix deployed.
• Incident recurrence rate: same root cause happening again within a defined window (e.g., 90 days), a signal that fixes are patches, not structural.

A useful benchmark: mature fintech data teams aim for TTD under 15 minutes for tier-1 pipelines (those touching customer balances or payments) using automated alerting, versus hours or days when relying on manual review or customer complaints.

Building the uptime dashboard that would have caught it

Return to the incident. What dashboard elements would have flagged the schema drift before customers saw it?

1. Schema validation checks running on every ingest, comparing incoming field types against a registered schema contract. Tools like Great Expectations can run this as a pre-ingestion gate.
2. Row-count anomaly alerts: even though volume didn't drop, a sudden change in the *distribution of values* within transaction_type (all values shifting from integers to strings) would trip a type-mismatch check instantly.
3. Downstream null-rate monitoring: track the percentage of null or zero-match results in dependent jobs. A spike from 0.1% to 100% null matches is an unmissable signal if you're watching it.
4. SLA burn-rate alerting: similar to how Google's Site Reliability Engineering (SRE) practice tracks "error budget burn rateburn rateBurn rate is the speed at which a company spends its cash reserves, usually measured per month, before reaching profitability or raising more funding.View full definition →," teams can alert not just on a completeness breach, but on the *speed* at which the completeness metric is degrading.

The common thread: none of these require predicting the specific failure. They require monitoring the *properties* of healthy data (schema, distribution, completeness, freshness) and alerting on deviation, regardless of cause.

Key Takeaways

• SLA metrics (freshness, completeness, uptime, accuracy) define what "healthy" means numerically. A pipeline that "runs" but silently drops data is not healthy by these definitions.
• Drift comes in three flavors: schema, distributional, and volume. Most costly incidents are distributional or schema drift, because the pipeline doesn't crash, it just becomes quietly wrong.
• Time to detect (TTD) matters more than time to resolve. A five-hour detection gap, as in the 3 a.m. scenario, is often the real root cause of customer-facing damage, not the underlying schema change itself.
• Statistical thresholds (like z-scores) and schema contracts turn "watch the data" into an automatable, testable check, not a matter of someone noticing.
• Governance metrics like lineage coverage and incident recurrence rate matter to regulators (ECB, Federal Reserve, BCBS 239 principles) as much as to engineers, because "who is accountable for this dataset" is now a supervisory question, not just an internal one.