Data lineage and pipeline reliability for telecom reporting

The data that matters: from probe to dashboard

Telecom reporting depends on a chain of systems, each producing a distinct dataset. Understanding this chain is the foundation of "lineage": knowing where a number came from and everything that touched it along the way.

1. Network probes and switches. Raw usage events originate at cell towers, routers, and switches. A voice call, an SMS, a data session, each generates a raw event record at the network element level.

2. Mediation layer. Mediation systems collect raw records from thousands of network elements, normalize formats (a 4G record looks different from a 5G or fixed-broadband record), deduplicate, and convert usage into rated events (assigning a price or plan rule). This is where CDRs and their data equivalent, xDRs (usage detail records covering voice, data, and messaging), get standardized.

3. Billing and charging systems. The Business Support System (BSS), specifically the billing and rating engine, applies tariffs, discounts, and taxes. Prepaid networks often use an Online Charging System (OCS) that authorizes usage in real time; postpaid uses batch rating.

4. Data warehouse / lakehouse. Rated and billed records land in a central repository (Snowflake, Databricks, or a telecom-specific data platform) where they're joined with customer, network, and financial data.

5. BI and reporting layer. Dashboards (Tableau, Power BIBITechnologies and processes that turn raw data into actionable insights via reporting, dashboards and analysis, so teams can decide based on facts rather than intuition.View full definition →, or custom executive reporting) aggregate this into ARPU (Average Revenue Per User), churn, and network-quality KPIs that leadership sees.

Each handoff is a lineage point. A lineage break, where a record is lost, duplicated, or transformed incorrectly between steps, is invisible until revenue or trust disappears.

Where lineage breaks: mediation errors and revenue leakage

Revenue leakage is the industry term for revenue earned but never billed or collected, due to systems failing to capture, rate, or invoice usage correctly. Analysts and vendors (e.g., TM Forum, the telecom industry standards body) have historically estimated leakage at roughly 1 to 3% of revenue for large operators, though this is an industry estimate, not an audited figure, and varies significantly by operator maturity.

Common break points:

• Probe failure or clock drift: a cell site's timestamp is off by minutes, causing usage to be attributed to the wrong billing cycle.
• Mediation dedup errors: the same session is double-counted across two network elements (common during handovers between towers), inflating both usage and possibly revenue in the wrong direction if the customer disputes the bill.
• Rating table mismatches: a new 5G data plan launches, but the mediation layer's rating rules weren't updated, so usage is rated using an old default plan.
• Batch job failures: a nightly ETLETLETL (Extract, Transform, Load) is a data integration process that pulls data from sources, reshapes it into a consistent format, and writes it into a target system.View full definition → (Extract, Transform, LoadExtract, Transform, LoadETL (Extract, Transform, Load) is a data integration process that pulls data from sources, reshapes it into a consistent format, and writes it into a target system.View full definition →) job that moves data from billing to the warehouse fails silently, and next morning's dashboard shows a revenue dip that isn't real, it's a pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.View full definition → outage, not a business trend.

This last case is why lineage tooling matters: distinguishing "the business changed" from "the pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.View full definition → broke" is the single most valuable skill in telecom reporting literacy.

Governance metrics: how you measure pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.View full definition → health

Three metrics dominate telecom data governancedata governanceData governance is the set of policies, roles, and processes that ensure data is accurate, secure, well-defined, and used responsibly across an organization.View full definition → conversations:

1. Data completeness rate: the percentage of expected records that actually arrive. If a network element normally sends 2 million CDRs a day and mediation only receives 1.94 million, completeness is 97%. Operators typically target above 99.5% for billing-critical feeds.

2. Lineage traceability: the percentage of critical reports where you can trace every number back to its source system with a documented transformation path. Regulators and auditors (in the EU, under GDPR's Article 5 accountability principle, and in the US under Sarbanes-Oxley for public telecom carriers) increasingly expect this for financial reporting.

3. Pipeline SLA (Service Level Agreement) adherence: the agreed time window for data to move from one stage to the next. A common billing SLA is "mediation to rated record within 4 hours." Missing this delays invoicing and cash collection.

A simple worked example: if a European mobile operator has 20 million subscribers generating an average of 150 xDRs per day each, that's 3 billion records daily. Even a 0.1% completeness failure means 3 million lost or corrupted records daily, translating to potential billing disputes at scale if not caught in reconciliation.

Reconciliation is the control that catches this: comparing volumes and totals at each pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.View full definition → stage (network vs. mediation vs. billing) to confirm they match within tolerance. Most Tier 1 operators run automated reconciliation checks hourly or daily on revenue-critical flows.

For a practical framework on data qualitydata qualityThe degree to which data is fit for purpose: accurate, complete, consistent, timely, valid and unique. Poor quality data undermines analytics, reporting and AI.View full definition → dimensions (completeness, accuracy, timeliness, consistency), the DAMA-DMBOK data management framework is a widely used industry reference.

A simple lineage check, in practice

Data engineers often implement lineage checks as automated volume comparisons. Here's a simplified pseudocode version of a daily reconciliation job:

This kind of query, run automatically each morning, is the first line of defense against silent pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.View full definition → failures. When it flags a breach, someone investigates before the dashboard, and the executives reading it, get misled.

Benchmarks and what "good" looks like

Concrete numbers help calibrate expectations (these are industry estimates as of 2025-2026, not universal standards):

• Billing accuracy: mature operators target 99.9%+ accuracy on rated billing records; regulatory bodies like the FCC (Federal Communications Commission, US) and national regulators under BEREC (Body of European Regulators for Electronic Communications) coordination in the EU can penalize systemic billing errors affecting consumers.
• Mediation processing SLA: near-real-time is increasingly standard for 5G and prepaid; postpaid data can tolerate 4 to 24 hour batch windows depending on operator architecture.
• Data completeness for financial reporting: 99.5% to 99.9% is a common internal target for revenue-critical feeds; anything below signals a governance gap.
• Time-to-detect pipeline failure: leading operators aim for under 1 hour via automated monitoring; without it, failures can go unnoticed for a full billing cycle (up to 30 days), compounding leakage.

The gap between "detect in 1 hour" and "detect in 30 days" is the difference between a minor operational fix and a quarter of understated revenue that finance has to explain to investors.

Why this matters beyond IT

Executives don't need to understand mediation architecture, but they do need to ask two questions of any number on a dashboard: What's the lineage? (Can someone trace this back to source?) And what's the SLA? (How fresh and complete is this data guaranteed to be?) A dashboard number without answers to both questions is a guess dressed as a fact.

Key Takeaways

• Telecom usage data flows through a defined chain: network probes → mediation → billing/BSS → data warehousedata warehouseA central repository that consolidates data from many source systems into a structured, query-optimized store designed for analytics, reporting, and business intelligence.View full definition → → BIBITechnologies and processes that turn raw data into actionable insights via reporting, dashboards and analysis, so teams can decide based on facts rather than intuition.View full definition → dashboard. Every handoff is a potential lineage break.
• Revenue leakage from mediation and billing errors is commonly estimated at 1 to 3% of revenue for large operators (industry estimate, not audited fact); reconciliation checks between pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.View full definition → stages are the primary control.
• Track three governance metrics: data completeness rate (target 99.5%+), lineage traceability (can you trace a number to its source), and pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.View full definition → SLA adherence (agreed time windows between stages).
• Automated daily reconciliation (comparing volumes across mediation and billing) is the fastest way to catch silent pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.View full definition → failures before they reachreachThe number of unique people exposed to your message in a given period. Unlike impressions, reach counts each person once, no matter how often they see it.View full definition → executive dashboards.
• Before trusting any telecom KPIKPIKey Performance Indicator, a measurable value that shows how effectively you're achieving a specific objective, tracked over time against a target.View full definition →, ask about its lineage and its SLA. A number without a traceable source and a freshness guarantee is not reliable for decision-making.