Scoring data quality: completeness, accuracy and timeliness in FMCG feeds

2. Retailer/distributor inventory feeds. On-hand units by store or warehouse, usually sent via EDI (Electronic Data Interchange, a standardized format for B2B transactions) using formats like the EDI 852 (product activity data) or EDI 846 (inventory inquiry/advice).

3. Shipment and order data. EDI 850 (purchase order), 856 (advance ship notice), and 810 (invoice) transactions that tell a manufacturer what was ordered, shipped, and billed.

A fourth, growing category: loyalty and e-commerce clickstream data from retailer media networks (Walmart Connect, Kroger Precision Marketing, Tesco Clubcard data via Dunnhumby), which links purchase to shopper identity and browsing behavior.

Each feed has a different failure mode. POS feeds go stale or drop stores. Inventory feeds diverge from physical reality ("phantom inventory": the system says 12 units on shelf, the actual count is zero). Shipment data mismatches when a UPC gets remapped after a pack-size change and the old code lingers in one system.

Why this matters commercially

Phantom inventory triggers a specific, expensive loop: the system believes stock exists, so it doesn't trigger a replenishment order, the shelf stays empty, and the sale is lost. Industry research on out-of-stocks (a long-running stream from Gruen-era studies through more recent retail analytics) has repeatedly estimated retail out-of-stock rates around 5 to 8% of SKUs on any given day as an industry benchmark estimate; phantom inventory is a recognized contributor, though the exact share attributable to data error versus shelf execution varies by category and is not precisely quantified industry-wide.

The three-pillar quality scorecard

FMCG data teams score incoming feeds on three dimensions. Each gets a threshold; breach the threshold and the feed is flagged before it touches a forecast or replenishment system.

Completeness

Percentage of expected records actually received.

Formula: Completeness % = (records received / records expected) × 100

Worked example: A distributor should report inventory for 850 stores daily. Today's file contains 803 stores.

803 / 850 = 0.9447 → 94.5% complete.

Typical threshold (industry practice estimate): flag anything below 98% completeness for POS feeds used in demand forecasting; below 95% for secondary inventory feeds is a hard stop that blocks auto-replenishment triggers. These thresholds vary by company and are not universal regulation, they're operational norms set by each data governancedata governanceData governance is the set of policies, roles, and processes that ensure data is accurate, secure, well-defined, and used responsibly across an organization.View full definition → team.

Accuracy

Whether the values match ground truth. In FMCG this usually means:

• UPC match rate: percentage of scanned UPCs that correctly mapmapUsing software to automate repetitive marketing tasks and campaigns, enabling personalisation at scale across channels like email, web, and social.View full definition → to the manufacturer's master product catalog (GS1's GDSN, the Global Data Synchronization Network, is the industry backbone for this mapping).
• Inventory variance: difference between system-reported on-hand units and physical cycle count.

Worked example: Cycle count finds 40 units on shelf; the inventory feed says 58.

Variance = |58 − 40| / 40 = 45% overstatement. That's phantom inventory, and it's severe.

Typical threshold: UPC match rate below 99% is commonly treated as a red flag requiring catalog reconciliation. Inventory variance tolerance is tighter for high-velocity SKUs (often under 5%) and looser for slow movers.

Timeliness

How fast data arrives relative to when the event happened.

Common benchmarks (estimates, vary by retailer agreement):

• POS scan data: expected within 24 to 48 hours of the sale (T+1 or T+2).
• EDI 852 inventory feeds: daily, often overnight batch.
• EDI 856 advance ship notices: within hours of shipment departure.

Worked example: A retailer's scan-out data for Saturday sales arrives Wednesday instead of Monday. That's a 2-day lag against a T+2 standard, a straightforward SLA (service level agreement) breach that should suppress that store's data from the current forecast run rather than let it silently skew the model.

A simple flagging rule in practice

Here's a minimal pseudocode pattern data teams use to auto-flag a feed before it enters a demand-planning model:

This is deliberately simple. Real production systems (often built on platforms like Snowflake, Databricks, or retailer-specific EDI middleware) add weighting, store-level history, and statistical outlier detection, but the logic tree is the same: measure, threshold, quarantine, escalate.

Governance: who owns the fix

Scoring is useless without an escalation path. Most mature FMCG data organizations run a data governance council (cross-functional, usually sales operations, IT, and category management) that:

1. Owns the master data (product catalog, UPC assignments) in line with GS1 standards.

2. Reviews weekly quality scorecards by retailer/distributor partner.

3. Escalates chronic offenders (a retailer consistently below 95% completeness) to the commercial team for a data-quality conversation, since this is a trading-partner relationship issue, not just an IT ticket.

GS1's data qualitydata qualityThe degree to which data is fit for purpose: accurate, complete, consistent, timely, valid and unique. Poor quality data undermines analytics, reporting and AI.View full definition → guidance and the Voluntary Interindustry Commerce Solutions Association (VICS, now largely folded into GS1 US initiatives) historically provided shared standards for EDI transaction sets that make cross-retailer scorecards comparable at all. Without a shared standard, a "complete" feed from Retailer A and Retailer B might not even mean the same thing structurally.

Benchmarking your scorecard

There's no single global regulator setting FMCG data-quality thresholds (unlike, say, financial reporting under IFRS or GAAP). Benchmarks come from:

• Trading partner agreements: SLAs negotiated directly between manufacturer and retailer/distributor.
• Industry syndicators: NielsenIQ and Circana publish methodology notes on panel completeness and projection accuracy, useful as external reference points.
• GS1 standards: for master data structure and UPC integrity, not for statistical thresholds.

A practical starting point for a new scorecard in 2026: completeness ≥ 98%, UPC match rate ≥ 99%, timeliness within T+2, inventory variance ≤ 5% for top-velocity SKUs. Treat these as reasonable industry-practice defaults to calibrate against your own historical data, not as fixed external mandates.

🎬 [VIDEO: "How Retailers and Suppliers Share Data (EDI Explained)" - youtube.com - a walkthrough of EDI transaction types (850, 856, 810, 852) that underpin most FMCG data feeds]

Key Takeaways

• FMCG data qualitydata qualityThe degree to which data is fit for purpose: accurate, complete, consistent, timely, valid and unique. Poor quality data undermines analytics, reporting and AI.View full definition → rests on three feed families: POS/scan data, inventory feeds (EDI 852/846), and shipment/order data (EDI 850/856/810). Each fails differently.
• Score every feed on completeness (records received vs. expected), accuracy (UPC match rate, inventory variance vs. cycle count), and timeliness (lag vs. T+1/T+2 SLA).
• Phantom inventory (system says stock exists, shelf says otherwise) is a leading cause of false "no reorder needed" signals; a variance threshold around 5% for high-velocity SKUs is a common industry practice estimate for flagging it.
• Thresholds are operational norms set by trading partners and internal governance councils, not government regulation; GS1's GDSN standard is the closest thing to a shared industry backbone for master data.
• Flag-and-quarantine logic (exclude bad feeds from forecasts, route to a data stewarddata stewardA business-side owner responsible for the quality, consistency and appropriate use of data in their domain.View full definition →) is more valuable than trying to "fix" bad data silently inside a model.