measuring pipeline health: SLAs, drift and downtime

The pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.Voir la définition complète → didn't crash. It kept ingesting data all night. But every downstream job that filtered on the old numeric codes silently returned zero matching rows. By 8 a.m., 40,000 customers opened the app to see stale balances and missing transactions. No error, no alert, just wrong data delivered with total confidence.

This is a schema drift failure: the structure of incoming data changes without the pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.Voir la définition complète → breaking outright. It's one of the most common and hardest-to-catch failure modes in fintech data infrastructure, because "the system is technically running" and "the data is right" are two different claims. This lesson builds the metrics that catch the gap between them.

The data that matters: sources worth monitoring

Before defining metrics, know what you're measuring. In fintech, the highest-risk data sources are:

• Bank-feed APIs (Plaid, Tink, MX, Yodlee): third-party aggregators pulling transaction and balance data from thousands of bank connections you don't control.
• Card network and payment rails data (Visa, Mastercard, ACH, SEPA in Europe): settlement files and authorization messages with strict formats.
• Core banking system exports: ledger and account data from vendors like Mambu, Temenos, or legacy mainframes.
• Credit bureau feeds (Experian, Equifax, TransUnion in the US; Schufa in Germany, Experian in the UK): batch files scoring creditworthiness.
• KYC/AML data (Know Your Customer, Anti-Money Laundering): identity verification feeds from providers like Onfido or LexisNexis.
• Market data feeds: pricing and FX data, often from Bloomberg or Refinitiv, for trading or treasury functions.

Each has different volatility. Bank-feed APIs change schemas often because you're aggregating hundreds of underlying bank formats. Card network files are rigid and standardized (ISO 8583 messaging), so drift there is rarer but more disruptive when it happens.

SLA metrics: defining "working"

An SLA (Service Level Agreement) is a measurable promise about pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.Voir la définition complète → behavior. For data pipelines, the core SLA metrics are:

1. Freshness (latency)

How old is the data when it lands? Example: "Transaction data must be available in the warehouse within 15 minutes of the bank posting it."

2. Completeness

Did all expected records arrive? If you expect 50,000 daily transaction rows from a partner bank and get 12,000, that's a completeness failure even if the pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.Voir la définition complète → "ran successfully."

3. Uptime / availability

Percentage of time the pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.Voir la définition complète → or APIAPIApplication Programming Interface: a standardised interface that lets applications communicate and exchange data without knowing each other's internal workings.Voir la définition complète → endpoint is reachable and responding. Commonly quoted as "three nines" (99.9%) or "four nines" (99.99%).

Worked example: 99.9% uptime over a 30-day month allows:

Compare to 99.99% (four nines): allowed downtime = 4.32 minutes/month, roughly ten times stricter. Payment rails and card authorization systems typically target four nines or higher because each minute of downtime blocks live purchases. Internal analytics pipelines often tolerate three nines, since a delayed dashboard is inconvenient, not customer-facing.

4. Accuracy

Does the data match ground truth? Harder to measure automatically, usually sampled via reconciliation (comparing pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.Voir la définition complète → output against a trusted source, like the bank's own statement).

Data drift: the silent failure mode

SchemaSchemaA schema is the formal blueprint that defines how data is structured, named, typed, and related within a database, file, or message.Voir la définition complète → drift, like our 3 a.m. example, is one type. There are three worth distinguishing:

• Schema drift: field names, types, or structure change (numeric code becomes a string).
• Distributional drift: the shape of the data shifts even though the schemaschemaA schema is the formal blueprint that defines how data is structured, named, typed, and related within a database, file, or message.Voir la définition complète → is unchanged. Example: average transaction amount jumps from $45 to $450 because a new merchant category got misrouted.
• Volume drift: row counts spike or collapse. A batch that normally delivers 2 million records suddenly delivers 200.

Detection approach: set statistical thresholds, not just presence/absence checks.

A z-score above 3 (three standard deviations from the historical mean) is a common trigger threshold for anomaly alerts, though the right number depends on how noisy your data normally is.

Open-source tools built for this include Great Expectations and Evidently AI, both free and widely used for automated data qualitydata qualityThe degree to which data is fit for purpose: accurate, complete, consistent, timely, valid and unique. Poor quality data undermines analytics, reporting and AI.Voir la définition complète → and drift checks in production pipelines.

Governance metrics: who's accountable

Data qualityData qualityThe degree to which data is fit for purpose: accurate, complete, consistent, timely, valid and unique. Poor quality data undermines analytics, reporting and AI.Voir la définition complète → isn't just technical, it's organizational. Governance metrics track accountability:

• Data lineage coverage: percentage of critical datasets with a documented path from source to dashboard. Regulators like the ECB (European Central Bank) and the Federal Reserve increasingly expect lineage documentation for risk-relevant data, especially under frameworks like BCBS 239BCBS 239Principe du Basel Committee on Banking Supervision imposant aux grandes banques une traçabilité stricte des données de risque, ayant catalysé la création de nombreux postes de CDO dans le secteur bancaire. (Basel Committee principles for risk data aggregation).
• Time to detect (TTD): how long between when a data issue occurs and when it's flagged. In the incident above, TTD was roughly 5 hours (3:14 a.m. to 8 a.m. customer complaints), which is the actual failure: not that drift happened, but that nobody caught it fast.
• Time to resolve (TTR): from detection to fix deployed.
• Incident recurrence rate: same root cause happening again within a defined window (e.g., 90 days), a signal that fixes are patches, not structural.

A useful benchmark: mature fintech data teams aim for TTD under 15 minutes for tier-1 pipelines (those touching customer balances or payments) using automated alerting, versus hours or days when relying on manual review or customer complaints.

Building the uptime dashboard that would have caught it

Return to the incident. What dashboard elements would have flagged the schemaschemaA schema is the formal blueprint that defines how data is structured, named, typed, and related within a database, file, or message.Voir la définition complète → drift before customers saw it?

1. Schema validation checks running on every ingest, comparing incoming field types against a registered schemaschemaA schema is the formal blueprint that defines how data is structured, named, typed, and related within a database, file, or message.Voir la définition complète → contract. Tools like Great Expectations can run this as a pre-ingestion gate.

2. Row-count anomaly alerts: even though volume didn't drop, a sudden change in the *distribution of values* within transaction_type (all values shifting from integers to strings) would trip a type-mismatch check instantly.

3. Downstream null-rate monitoring: track the percentage of null or zero-match results in dependent jobs. A spike from 0.1% to 100% null matches is an unmissable signal if you're watching it.

4. SLA burn-rate alerting: similar to how Google's Site Reliability Engineering (SRE) practice tracks "error budget burn rateburn rateBurn rate is the speed at which a company spends its cash reserves, usually measured per month, before reaching profitability or raising more funding.Voir la définition complète →," teams can alert not just on a completeness breach, but on the *speed* at which the completeness metric is degrading.

The common thread: none of these require predicting the specific failure. They require monitoring the *properties* of healthy data (schemaschemaA schema is the formal blueprint that defines how data is structured, named, typed, and related within a database, file, or message.Voir la définition complète →, distribution, completeness, freshness) and alerting on deviation, regardless of cause.

Key Takeaways

• SLA metrics (freshness, completeness, uptime, accuracy) define what "healthy" means numerically. A pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.Voir la définition complète → that "runs" but silently drops data is not healthy by these definitions.
• Drift comes in three flavors: schemaschemaA schema is the formal blueprint that defines how data is structured, named, typed, and related within a database, file, or message.Voir la définition complète →, distributional, and volume. Most costly incidents are distributional or schema drift, because the doesn't crash, it just becomes quietly wrong.