# Measuring pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.View full definition → health: slas, drift, and downtime
At 3:14 a.m., a mid-sized neobank's bank-feed aggregator (a service like Plaid or Tink that pulls transaction data from partner banks via APIAPIApplication Programming Interface: a standardised interface that lets applications communicate and exchange data without knowing each other's internal workings.View full definition →) pushed an update. Nobody announced it. A field called transaction_type
123"debit""credit""transfer"The pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.View full definition → didn't crash. It kept ingesting data all night. But every downstream job that filtered on the old numeric codes silently returned zero matching rows. By 8 a.m., 40,000 customers opened the app to see stale balances and missing transactions. No error, no alert, just wrong data delivered with total confidence.
This is a schema drift failure: the structure of incoming data changes without the pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.View full definition → breaking outright. It's one of the most common and hardest-to-catch failure modes in fintech data infrastructure, because "the system is technically running" and "the data is right" are two different claims. This lesson builds the metrics that catch the gap between them.
Before defining metrics, know what you're measuring. In fintech, the highest-risk data sources are:
Each has different volatility. Bank-feed APIs change schemas often because you're aggregating hundreds of underlying bank formats. Card network files are rigid and standardized (ISO 8583 messaging), so drift there is rarer but more disruptive when it happens.
An SLA (Service Level Agreement) is a measurable promise about pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.View full definition → behavior. For data pipelines, the core SLA metrics are:
1. Freshness (latency)
How old is the data when it lands? Example: "Transaction data must be available in the warehouse within 15 minutes of the bank posting it."
2. Completeness
Did all expected records arrive? If you expect 50,000 daily transaction rows from a partner bank and get 12,000, that's a completeness failure even if the pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.View full definition → "ran successfully."
3. Uptime / availability
Percentage of time the pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.View full definition → or APIAPIApplication Programming Interface: a standardised interface that lets applications communicate and exchange data without knowing each other's internal workings.View full definition → endpoint is reachable and responding. Commonly quoted as "three nines" (99.9%) or "four nines" (99.99%).
Worked example: 99.9% uptime over a 30-day month allows:
30 days × 24 hours × 60 minutes = 43,200 minutes total
Allowed downtime = 43,200 × (1 - 0.999) = 43.2 minutes/monthCompare to 99.99% (four nines): allowed downtime = 4.32 minutes/month, roughly ten times stricter. Payment rails and card authorization systems typically target four nines or higher because each minute of downtime blocks live purchases. Internal analytics pipelines often tolerate three nines, since a delayed dashboard is inconvenient, not customer-facing.
4. Accuracy
Does the data match ground truth? Harder to measure automatically, usually sampled via reconciliation (comparing pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.View full definition → output against a trusted source, like the bank's own statement).
SchemaSchemaA schema is the formal blueprint that defines how data is structured, named, typed, and related within a database, file, or message.View full definition → drift, like our 3 a.m. example, is one type. There are three worth distinguishing:
Detection approach: set statistical thresholds, not just presence/absence checks.
# Simplified drift check: compare today's distribution to a 30-day baseline
import numpy as np
baseline_mean, baseline_std = 45.0, 12.0 # from trailing 30-day window
today_mean = df['transaction_amount'].mean()
z_score = (today_mean - baseline_mean) / baseline_std
if abs(z_score) > 3:
alert("Distributional drift detected: transaction amounts")A z-score above 3 (three standard deviations from the historical mean) is a common trigger threshold for anomaly alerts, though the right number depends on how noisy your data normally is.
Open-source tools built for this include Great Expectations and Evidently AI, both free and widely used for automated data qualitydata qualityThe degree to which data is fit for purpose: accurate, complete, consistent, timely, valid and unique. Poor quality data undermines analytics, reporting and AI.View full definition → and drift checks in production pipelines.
Data qualityData qualityThe degree to which data is fit for purpose: accurate, complete, consistent, timely, valid and unique. Poor quality data undermines analytics, reporting and AI.View full definition → isn't just technical, it's organizational. Governance metrics track accountability:
A useful benchmark: mature fintech data teams aim for TTD under 15 minutes for tier-1 pipelines (those touching customer balances or payments) using automated alerting, versus hours or days when relying on manual review or customer complaints.
Knowledge check
1. In the neobank incident, why did the schema drift in the `transaction_type` field go undetected for hours despite causing serious downstream problems?
2. What is the key distinction between 'the system is technically running' and 'the data is right,' as illustrated by the 3 a.m. incident?
3. A payments team wants to catch issues like the one in the 3 a.m. incident earlier, before customers see stale data. Which monitoring approach would most directly address this specific failure mode?
4. Select ALL correct answers about why bank-feed APIs and similar third-party data sources are especially high-risk for fintech pipelines.
Select all the correct answers.
5. Select ALL correct answers about what makes schema drift different from a typical pipeline outage or crash.
Select all the correct answers.
Return to the incident. What dashboard elements would have flagged the schemaschemaA schema is the formal blueprint that defines how data is structured, named, typed, and related within a database, file, or message.View full definition → drift before customers saw it?
1. Schema validation checks running on every ingest, comparing incoming field types against a registered schemaschemaA schema is the formal blueprint that defines how data is structured, named, typed, and related within a database, file, or message.View full definition → contract. Tools like Great Expectations can run this as a pre-ingestion gate.
2. Row-count anomaly alerts: even though volume didn't drop, a sudden change in the *distribution of values* within transaction_type (all values shifting from integers to strings) would trip a type-mismatch check instantly.
3. Downstream null-rate monitoring: track the percentage of null or zero-match results in dependent jobs. A spike from 0.1% to 100% null matches is an unmissable signal if you're watching it.
4. SLA burn-rate alerting: similar to how Google's Site Reliability Engineering (SRE) practice tracks "error budget burn rateburn rateBurn rate is the speed at which a company spends its cash reserves, usually measured per month, before reaching profitability or raising more funding.View full definition →," teams can alert not just on a completeness breach, but on the *speed* at which the completeness metric is degrading.
The common thread: none of these require predicting the specific failure. They require monitoring the *properties* of healthy data (schemaschemaA schema is the formal blueprint that defines how data is structured, named, typed, and related within a database, file, or message.View full definition →, distribution, completeness, freshness) and alerting on deviation, regardless of cause.