Data lineage and pipeline reliability for telecom reporting

# Data lineageData lineageData lineage maps how data moves and transforms across systems, from origin to consumption, showing where it came from, what changed it, and where it goes.Voir la définition complète → and pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.Voir la définition complète → reliability for telecom reporting

At 9:47 PM on a Tuesday, a subscriber in Frankfurt starts a video call that runs eleven minutes over her data plan's fair-use threshold. That single event, a Call Detail Record (CDR), a structured log entry capturing who, when, how long, and how much data, now has to travel through five or six systems before it can appear correctly on a CFO's revenue dashboard three days later. If any hop in that journey drops the record, miscounts it, or delays it, two things happen: the company loses money it earned, and the dashboard tells a story that isn't true. This lesson traces that journey and shows you where it breaks.

The data that matters: from probe to dashboard

Telecom reporting depends on a chain of systems, each producing a distinct dataset. Understanding this chain is the foundation of "lineage": knowing where a number came from and everything that touched it along the way.

1. Network probes and switches. Raw usage events originate at cell towers, routers, and switches. A voice call, an SMS, a data session, each generates a raw event record at the network element level.

2. Mediation layer. Mediation systems collect raw records from thousands of network elements, normalize formats (a 4G record looks different from a 5G or fixed-broadband record), deduplicate, and convert usage into rated events (assigning a price or plan rule). This is where CDRs and their data equivalent, xDRs (usage detail records covering voice, data, and messaging), get standardized.

3. Billing and charging systems. The Business Support System (BSS), specifically the billing and rating engine, applies tariffs, discounts, and taxes. Prepaid networks often use an Online Charging System (OCS) that authorizes usage in real time; postpaid uses batch rating.

4. Data warehouse / lakehouse. Rated and billed records land in a central repository (Snowflake, Databricks, or a telecom-specific data platform) where they're joined with customer, network, and financial data.

5. BI and reporting layer. Dashboards (Tableau, Power BIBITechnologies and processes that turn raw data into actionable insights via reporting, dashboards and analysis, so teams can decide based on facts rather than intuition.Voir la définition complète →, or custom executive reporting) aggregate this into ARPU (Average Revenue Per User), churn, and network-quality KPIs that leadership sees.

Each handoff is a lineage point. A lineage break, where a record is lost, duplicated, or transformed incorrectly between steps, is invisible until revenue or trust disappears.

Where lineage breaks: mediation errors and revenue leakage

Revenue leakage is the industry term for revenue earned but never billed or collected, due to systems failing to capture, rate, or invoice usage correctly. Analysts and vendors (e.g., TM Forum, the telecom industry standards body) have historically estimated leakage at roughly 1 to 3% of revenue for large operators, though this is an industry estimate, not an audited figure, and varies significantly by operator maturity.

Common break points:

• Probe failure or clock drift: a cell site's timestamp is off by minutes, causing usage to be attributed to the wrong billing cycle.
• Mediation dedup errors: the same session is double-counted across two network elements (common during handovers between towers), inflating both usage and possibly revenue in the wrong direction if the customer disputes the bill.
• Rating table mismatches: a new 5G data plan launches, but the mediation layer's rating rules weren't updated, so usage is rated using an old default plan.
• Batch job failures: a nightly ETLETLETL (Extract, Transform, Load) is a data integration process that pulls data from sources, reshapes it into a consistent format, and writes it into a target system.Voir la définition complète → (Extract, Transform, LoadExtract, Transform, LoadETL (Extract, Transform, Load) is a data integration process that pulls data from sources, reshapes it into a consistent format, and writes it into a target system.Voir la définition complète →) job that moves data from billing to the warehouse fails silently, and next morning's dashboard shows a revenue dip that isn't real, it's a outage, not a business trend.

This last case is why lineage tooling matters: distinguishing "the business changed" from "the pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.Voir la définition complète → broke" is the single most valuable skill in telecom reporting literacy.

Governance metrics: how you measure pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.Voir la définition complète → health

Three metrics dominate telecom data governancedata governanceData governance is the set of policies, roles, and processes that ensure data is accurate, secure, well-defined, and used responsibly across an organization.Voir la définition complète → conversations:

1. Data completeness rate: the percentage of expected records that actually arrive. If a network element normally sends 2 million CDRs a day and mediation only receives 1.94 million, completeness is 97%. Operators typically target above 99.5% for billing-critical feeds.

2. Lineage traceability: the percentage of critical reports where you can trace every number back to its source system with a documented transformation path. Regulators and auditors (in the EU, under GDPR's Article 5 accountability principle, and in the US under Sarbanes-Oxley for public telecom carriers) increasingly expect this for financial reporting.

3. Pipeline SLA (Service Level Agreement) adherence: the agreed time window for data to move from one stage to the next. A common billing SLA is "mediation to rated record within 4 hours." Missing this delays invoicing and cash collection.

A simple worked example: if a European mobile operator has 20 million subscribers generating an average of 150 xDRs per day each, that's 3 billion records daily. Even a 0.1% completeness failure means 3 million lost or corrupted records daily, translating to potential billing disputes at scale if not caught in reconciliation.

Reconciliation is the control that catches this: comparing volumes and totals at each pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.Voir la définition complète → stage (network vs. mediation vs. billing) to confirm they match within tolerance. Most Tier 1 operators run automated reconciliation checks hourly or daily on revenue-critical flows.

For a practical framework on data qualitydata qualityThe degree to which data is fit for purpose: accurate, complete, consistent, timely, valid and unique. Poor quality data undermines analytics, reporting and AI.Voir la définition complète → dimensions (completeness, accuracy, timeliness, consistency), the DAMA-DMBOK data management framework is a widely used industry reference.

A simple lineage check, in practice

Data engineers often implement lineage checks as automated volume comparisons. Here's a simplified pseudocode version of a daily reconciliation job:

-- Compare record counts between mediation and billing for yesterday
SELECT
  m.record_date,
  m.record_count AS mediation_count,
  b.record_count AS billing_count,
  ROUND(100.0 * b.record_count / m.record_count, 2) AS completeness_pct
FROM mediation_daily_summary m
JOIN billing_daily_summary b
  ON m.record_date = b.record_date
WHERE m.record_date = CURRENT_DATE - 1
  AND completeness_pct < 99.5;  -- flags breaches for investigation

This kind of query, run automatically each morning, is the first line of defense against silent pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.Voir la définition complète → failures. When it flags a breach, someone investigates before the dashboard, and the executives reading it, get misled.

Benchmarks and what "good" looks like

Concrete numbers help calibrate expectations (these are industry estimates as of 2025-2026, not universal standards):

• Billing accuracy: mature operators target 99.9%+ accuracy on rated billing records; regulatory bodies like the FCC (Federal Communications Commission, US) and national regulators under BEREC (Body of European Regulators for Electronic Communications) coordination in the EU can penalize systemic billing errors affecting consumers.
• Mediation processing SLA: near-real-time is increasingly standard for 5G and prepaid; postpaid data can tolerate 4 to 24 hour batch windows depending on operator architecture.
• Data completeness for financial reporting: 99.5% to 99.9% is a common internal target for revenue-critical feeds; anything below signals a governance gap.
• Time-to-detect pipeline failure: leading operators aim for under 1 hour via automated monitoring; without it, failures can go unnoticed for a full billing cycle (up to 30 days), compounding leakage.

The gap between "detect in 1 hour" and "detect in 30 days" is the difference between a minor operational fix and a quarter of understated revenue that finance has to explain to investors.

Why this matters beyond IT

Executives don't need to understand mediation architecture, but they do need to ask two questions of any number on a dashboard: What's the lineage? (Can someone trace this back to source?) And what's the SLA? (How fresh and complete is this data guaranteed to be?) A dashboard number without answers to both questions is a guess dressed as a fact.

Key Takeaways

• Telecom usage data flows through a defined chain: network probes → mediation → billing/BSS → → dashboard. Every handoff is a potential lineage break.