# Turning loyalty and household panel data into shopper insight
A shopper scans a loyalty card at a UK grocery checkout on a Tuesday evening. That single beep tells the retailer what she bought, at what price, alongside what else, and how that basket compares to the 40 she filled over the past year. Multiply that by millions of households, and you have the richest behavioral dataset in consumer goods. The problem is that most brand teams stare at it and see only sales going up or down. The insight lives one layer deeper.
This lesson shows how to read loyalty and household panel data the way a category manager does: separating *why* sales moved, spotting who is trading up from cheaper products, and telling the difference between a launch that grows the category and one that just steals from your own shelf.
Loyalty-card data (retailer first-party data): every transaction tied to a card, from one retailer. Enormous scale, but only that retailer's stores. You do not see what the shopper bought at a competitor.
Household panel data: a recruited, representative sample of households (often 15,000 to 30,000 in a large market) who record *all* their grocery purchases across every retailer, usually by scanning barcodes at home. Providers like Kantar Worldpanel and NielsenIQ run these. Smaller sample, but total-market view.
Rule of thumb: loyalty data is deep but narrow. Panel data is complete but sampled. Serious FMCG teams use both.
Here is the single most useful decomposition in FMCG. Any brand's sales can be broken into:
Why this matters: two brands with identical sales can be in completely different health.
Imagine Brand A grew sales 6% last year. Before you celebrate, decompose it.
Decades of research summarized by the Ehrenberg-Bass Institute show that brands almost always grow by increasing penetration, not loyalty. Big brands are big mostly because more people buy them, not because their buyers are more devoted. If you want the underlying science, the Institute's research overview is a good free starting point.
So the first question you ask a loyalty dataset is never "did sales go up?" It is "did *more households* buy us, or did the *same households* buy more?"
A raw loyalty extract looks like a giant table:
| card_id | date | store | product | price_paid | promo_flag |
|---------|------|-------|---------|-----------|------------|
To get from this to penetration and frequency, you group by household. A simple illustration in SQLSQLSales Qualified Lead: a prospect the sales team has validated as ready for direct outreach and a proposal, having passed clear qualification criteria.View full definition →:
SELECT
COUNT(DISTINCT CASE WHEN brand = 'OurBrand'
THEN card_id END) * 1.0
/ COUNT(DISTINCT card_id) AS penetration,
COUNT(*) FILTER (WHERE brand = 'OurBrand') * 1.0
/ COUNT(DISTINCT CASE WHEN brand = 'OurBrand'
THEN card_id END) AS purchases_per_buyer
FROM transactions
WHERE category = 'Yogurt'
AND date BETWEEN '2025-01-01' AND '2025-12-31';Penetration is buyers divided by all category shoppers. Frequency is total purchases divided by buyers. Two lines of logic, and you already know more than a sales chart tells you.
Private label (a retailer's own-brand product, often cheaper) is the biggest competitive threat in most FMCG categories. Loyalty data lets you find exactly which shoppers move between private label and your brand, because you can follow the same card over time.
Build a switching matrix. Take every household that bought the category in two consecutive periods and cross-tabulate what they bought:
| | Bought your brand (period 2) | Bought private label (period 2) |
|---|---|---|
| Bought your brand (period 1) | Loyal to you | Lost to private label |
| Bought private label (period 1) | Won from private label | Loyal to private label |
Now segment the "won from private label" cell. Are they younger, more promo-sensitive, buying larger packs? That profile tells you *what* pulled them across: a price cut, a new format, a health claim.
A concrete example: a category manager notices that shoppers switching from private label into a premium brand overwhelmingly bought on a multibuy promotion and skewed toward smaller households. Read that carefully. Those switchers may not stick once the promotion ends. Contrast that with switchers who came in at full price and repeated: those are real, durable gains.
The behavioral question underneath: did you change *who* buys the category, or just *where* existing buyers spend this week?
Now the hardest and most valuable use. You launch a new variant, say a high-protein version of an existing yogurt. Sales appear. But some of those sales came from people who would have bought your regular yogurt anyway. That is cannibalization. The sales that are genuinely new to your brand portfolio are the incremental lift.
Getting this wrong destroys value. A launch that looks successful (the new variant sells) can leave total brand sales flat because it hollowed out the existing line.
How to measure it with household data:
1. Identify households who bought the new variant.
2. Look at their purchases of your *other* products before and after launch.
3. If a buyer's spend on your regular yogurt dropped by roughly the amount they now spend on the variant, that is cannibalized.
4. If their total category spend with your brand rose, that portion is incremental.
A cleaner method is a test-versus-control design: compare stores or regions where the variant launched against matched ones where it did not, and measure the difference in *total brand* sales, not just variant sales. Panel data supports this because you see each household's full basket.
Decision rule people actually use: a launch is worth keeping if incrementality (new sales to the brand plus category expansion) clearly exceeds the margin lost to cannibalization. A variant that is 80% cannibalistic can still be justified if it defends against a competitor about to launch something similar, but you should *know* that is the trade you are making, not discover it a year later.
Knowledge check
1. A brand's sales have stayed flat, but a category manager wants to understand what is actually happening beneath that number. Why is the penetration × frequency × spend-per-trip decomposition more useful than the headline sales figure?
2. A brand team wants to know whether a new product launch is genuinely growing the category or simply pulling sales away from its own existing products. Which data capability is most essential for answering this?
3. Loyalty-card data is described as 'deep but narrow.' What does the 'narrow' limitation specifically refer to?
4. Select ALL correct answers about the differences between loyalty-card data and household panel data.
Select all the correct answers.
5. Select ALL correct answers about correctly interpreting penetration and frequency.
Select all the correct answers.
Survivorship in loyalty programs. Loyalty-card holders are not the whole population. They tend to be more engaged, more frequent shoppers. Do not treat them as the average shopper without adjusting.
Missing baskets. People forget their card, or shop at a competitor. Loyalty data undercounts. Panel data corrects the total-market picture but at smaller sample sizes, so tiny segmentssegmentsDividing a market into distinct groups of customers who share similar needs, characteristics or behaviours, so each group can be served with a tailored approach.View full definition → get noisy fast.
Confusing correlation with cause. If heavy buyers also buy your new variant, that does not mean the variant created heavy buyers. It may just be that loyal customers try everything you make.
Promotion masking. A frequency spike during a deep discount is not new loyalty. Always flag promoted versus full-price volume before drawing conclusions.
Suppose your yogurt brand's annual sales are up 4%. You decompose:
Your recommendation writes itself: the growth is real and penetration-led, the variant is worth keeping because it recruits new buyers even with meaningful cannibalization, and the private-label wins are worth protecting with continued full-price availability rather than deeper discounts.