Mapping the public data landscape: registries, admin records, and survey data, MBA Training, MBA Training
1/5+150 XP
Mapping the public data landscape: registries, admin records, and survey data
# Mapping the public data landscape: registries, admin records, and survey data
A family applies for rental assistance in Cook County, Illinois. The housing agency's caseworker pulls up three numbers: the household's income from a tax record, its size and composition from a Census Bureau survey, and its eviction history from a court case-management system. The three sources disagree on almost every detail. The tax record is a year old. The survey estimate covers a neighborhood, not this address. The case file has the family's name spelled two different ways. The caseworker has to decide, in real time, which number to trust. This is not an edge case. It is the daily reality of public sector data work, and it is why understanding the landscape of registries, administrative records, and surveys matters more than any single dashboard.
The three families of public data
Public sector data mostly comes from three structurally different sources, and confusing them is the single most common analytical mistake in the sector.
Registries are official, legally mandated lists: birth and death registries, business registries, property (land title) registries, voter rolls. They aim for completeness within a legal population and are usually maintained continuously.
Administrative records are the exhaust of running a program: tax filings, benefits case files, school enrollment systems, hospital intake records. They exist because someone had to process a transaction, not because someone designed a research dataset. In the EU, these are increasingly linked under frameworks like Eurostat's administrative data guidelines, which favor reusing government records over new data collection.
Survey data
is collected by sampling a population and asking questions, then weighting the answers to represent the whole. The
is the workhorse example: about 3.5 million households sampled annually (Census Bureau, as of 2024 methodology) to produce estimates for the full US population between the once-a-decade full Census counts.
Each family has a different relationship to truth. Registries claim to be the truth for a legal category (who owns this parcel). Admin records reflect what happened in a specific transaction (who received a benefit). Surveys estimate a population parameter with a margin of error attached, by design.
Why they disagree, and how to arbitrate
Back to the housing agency. The tax record undercounts informal or gig income. The ACS estimate is a modeled average for a census tract (a small geographic unit of roughly 1,200 to 8,000 people), not this household. The court record may reflect a resolved eviction that no longer describes current risk.
The rule of thumb caseworkers and analysts use:
For eligibility (a legal determination about this specific person), admin records and registries win. They are the closest thing to ground truth for an individual case, even if imperfect.
For prevalence (how many people, in general, face a condition), survey data wins. It is designed to generalize, admin data is not, because admin data only reflects people who already interacted with a system (a known problem called "administrative undercount" or, in benefits research, the "take-up gap").
For trend and targeting (where is need growing), triangulate. Overlaying ACS poverty estimates with property-tax delinquency records and eviction-filing databases (many US courts publish these; see the Eviction Lab at Princeton) gives a picture no single source provides alone.
Data qualityData qualityThe degree to which data is fit for purpose: accurate, complete, consistent, timely, valid and unique. Poor quality data undermines analytics, reporting and AI.Voir la définition complète → metrics that actually get used
Non-technical staff often assume "data qualitydata qualityThe degree to which data is fit for purpose: accurate, complete, consistent, timely, valid and unique. Poor quality data undermines analytics, reporting and AI.Voir la définition complète →" means "no errors." In practice, public sector analysts track a handful of concrete dimensions:
Completeness: percentage of required fields populated. A housing case-management system with 15% missing Social Security numbers has a completeness problem that blocks matching to tax records.
Timeliness: lag between an event and its appearance in the dataset. Unemployment insurance claims data in the US is near real-time; poverty estimates from the ACS lag by about one year for the 1-year estimates (Census Bureau, standard release cadence).
Coverage / undercount: the gap between the target population and who actually appears in the source. The 2020 US Census had an estimated net undercount of the Black population of about 3.3% and of the Hispanic/Latino population of about 4.9% (US Census Bureau Post-Enumeration Survey, 2022 release, figures are Bureau estimates with margins of error).
Match rate: when linking two sources (say, tax records to housing case files) by name and birthdate, what percentage link successfully. Below roughly 85 to 90% match rate, analysts should treat the linked dataset as biased, not just smaller.
Duplication rate: same entity appearing multiple times under different identifiers, a chronic issue in registries without a unique national ID (the US has no single national identifier equivalent to Nordic personal ID numbers, which complicates registry linkage compared to, say, Denmark's CPR system).
A simple worked example of match rate:
Housing case files: 10,000 households
Successfully matched to tax records: 8,200
Match rate = 8,200 / 10,000 = 82%
An 82% match rate means 1,800 households are invisible to any analysis that requires both datasets, and that missing 18% is rarely random. It tends to skew toward informal renters, recent movers, and undocumented residents, exactly the population many housing programs most want to reachreachThe number of unique people exposed to your message in a given period. Unlike impressions, reach counts each person once, no matter how often they see it.Voir la définition complète →.
Governance: who is allowed to see and join what
Data qualityData qualityThe degree to which data is fit for purpose: accurate, complete, consistent, timely, valid and unique. Poor quality data undermines analytics, reporting and AI.Voir la définition complète → is only half the story. Governance determines whether triangulation is even legal.
In the US, the Privacy Act of 1974 restricts how federal agencies share records containing personal information across systems without consent or a specific legal exemption. The Confidential Information Protection and Statistical Efficiency Act (CIPSEA), updated in 2018, governs how statistical agencies like the Census Bureau can (and mostly cannot) share identifiable survey data with other agencies, even for good causes like fraud detection.
In the EU, the General Data Protection Regulation (GDPR) sets the baseline: personal data can only be processed for the purpose it was collected for, unless a specific legal basis (like a public interest task) justifies reuse. This is why EU statistical offices lean heavily on anonymized, aggregated linkage rather than case-level joins across agencies.
Practically, this means the housing agency in our opening example likely cannot simply pull an individual's IRS (Internal Revenue Service) tax return. It can use aggregate ACS estimates freely, request specific admin data matches only under a data-sharing agreement with defined legal authority, and must document the legal basis for every linkage, an audit trail regulators and inspectors general will eventually check.
Vérification des acquis
1. What fundamentally distinguishes a registry from an administrative record dataset?
2. Why did the tax record, survey estimate, and court case file in the Cook County example disagree with each other?
3. A policy analyst needs to estimate the number of households below the poverty line in a specific census tract, but no complete registry of income exists. Which data family is best suited to this task, and why?
CHOIX MULTIPLES
4. Select ALL correct answers about administrative records as a data source.
Sélectionnez toutes les réponses correctes.
CHOIX MULTIPLES
5. Select ALL correct answers about why a caseworker must decide 'in real time' which data source to trust when sources conflict.
Sélectionnez toutes les réponses correctes.
Benchmarks analysts should know
A few reference points worth memorizing, all flagged as estimates given methodology and year of collection:
ACS margin of error for small geographies (a census tract) can exceed 20 to 30% of the estimate itself for less common characteristics, meaning tract-level poverty rates should be treated as a range, not a point figure.
Eurostat's EU Statistics on Income and Living Conditions (EU-SILC) survey samples roughly 130,000 households across member states annually (Eurostat, as of recent survey years), the EU's rough analog to the ACS for poverty and income metrics.
A commonly cited data-linkage benchmark: match rates above 90% are considered strong for probabilistic linkage (matching without a shared unique ID) in public health and social service contexts; below 70% generally signals the linked file should not be used for population-level inference.
🎬 [VIDEO: "How the Census Bureau Protects Your Data" - youtube.com/@uscensusbureau - a short official explainer on differential privacy and disclosure avoidance methods used in the 2020 Census, useful for understanding why published counts are deliberately noised]
Key Takeaways
Public sector data comes in three structurally different families: registries (legal completeness), administrative records (transaction exhaust), and surveys (sampled estimates with margins of error). Know which one you're using before trusting a number.
For individual eligibility decisions, trust admin records and registries. For population-level prevalence and trend estimates, trust surveys. Triangulate for targeting decisions.
Track completeness, timeliness, coverage/undercount, match rate, and duplication rate as concrete, calculable data-quality metrics, not vague notions of "clean data."
Governance rules (Privacy Act and CIPSEA in the US, GDPR in the EU) often restrict linkage before data qualitydata qualityThe degree to which data is fit for purpose: accurate, complete, consistent, timely, valid and unique. Poor quality data undermines analytics, reporting and AI.Voir la définition complète → even becomes the limiting factor. Legal basis for a join is a prerequisite, not an afterthought.
A match rate below roughly 70 to 85% when linking datasets usually signals systematic bias in who gets left out, not just a smaller sample.