Building a de-identification and re-identification risk workflow

A researcher published an "anonymized" oncology dataset. Zip code, admission date, and rare tumor subtype were all still there. Within days, a graduate student cross-referenced it against a public obituary and a hospital press release, and named one patient. The data was technically stripped of names. It was not anonymous.

This lesson shows you how to avoid that outcome: how to de-identify a real oncology cohort under US and EU rules, then attack your own dataset to prove the anonymization holds.

The two US methods you must know

Under the US HIPAA (Health Insurance Portability and Accountability Act, the 1996 federal law governing protected health information), the HHS (Department of Health and Human Services) recognizes exactly two ways to de-identify data. This is set out in the Privacy Rule, and the reference document is worth bookmarking: HHS De-identification Guidance.

Method 1: safe harbor

Remove 18 specified identifier types. Examples relevant to an oncology dataset:

• Names
• All geographic units smaller than a state (so no zip codes, except the first 3 digits under certain population rules)
• All date elements except year (no admission date, no date of surgery), and ages over 89 must be aggregated into a "90+" bucket
• Medical record numbers, device identifiers, biometric identifiers
• Any other unique identifying number or code

Safe Harbor is mechanical. You either removed the 18 categories or you did not. No statistician required.

The catch: it can destroy the science. Oncology research often needs precise dates (time from diagnosis to progression) and granular geography (cancer cluster analysis). Safe Harbor strips both.

Method 2: expert determination

A qualified statistician certifies that the re-identification risk is "very small." This lets you keep more detail (for example, exact intervals between events instead of just years) in exchange for a documented, defensible risk analysis.

This is where re-identification risk math lives. And it is the method serious biotech and medtech teams use for high-value clinical datasets.

Europe is stricter, and different in kind

Under the EU GDPRGDPREU regulation governing how organizations collect, store and use personal data, with fines tied to global revenue for breaches.View full definition → (General Data Protection Regulation), the standard is higher. GDPR distinguishes:

• Pseudonymized data: identifiers replaced with a key, but re-linkage is still possible. This is STILL personal data and STILL regulated.
• Anonymized data: irreversible, no reasonable means to re-identify. This falls OUTSIDE GDPR.

The key regulatory reference is the Article 29 Working Party opinion on anonymization techniques (the Working Party is now succeeded by the EDPB, the European Data Protection Board). GDPR judges anonymization against three risks: singling out, linkability, and inferenceinferenceThe moment a trained AI model is put to work: it takes a new input and produces an answer, prediction or generated output.View full definition →. Your workflow must address all three, not just remove names.

Practical takeaway: HIPAA Safe Harbor is NOT automatically GDPR-compliant. A dataset legal to release in Boston may still be personal data in Berlin.

The re-identification model: kkThe average number of new users each existing user generates through referrals. Above 1.0, growth compounds on itself and becomes exponential.View full definition →-anonymity

k-anonymity is the core metric. A dataset is k-anonymous if every record is indistinguishable from at least k-1 other records, based on its quasi-identifiers.

Quasi-identifiers are fields that are not direct IDs but combine to fingerprint a person: age, gender, zip code, admission year, diagnosis. A famous cited estimate (Latanya Sweeney's work, widely referenced) suggests roughly 87 percent of the US population was uniquely identifiable in 1990 census data using just ZIP + birth date + gender. Treat that as an often-cited estimate, not a current figure, but the lesson stands: three "harmless" fields can be a fingerprint.

If k=1 for a record, that record is unique on its quasi-identifiers. That is your re-identification risk.

A worked example

Suppose an oncology cohort has these quasi-identifiers per patient: `age_band`, `sex`, `state`, `cancer_type`.

Count how many patients share each combination:

The first two rows are safe. The third row has k=1: one 40-something woman in Wyoming with a rare BRCA1 ovarian case. Even with her name gone, she is a unique record. Combine that with a local news story and she is re-identifiable.

The fix: generalize (state becomes region, cancer_type becomes Ovarian) or suppress the row entirely until k reaches your threshold. A common target is k=5 or higher for released health data, though the right threshold depends on your risk assessment, not a universal rule.

Running the attack in code

Here is a minimal k-anonymity check. This is the "audit" step: you attack your own data before anyone else does.

import pandas as pd

quasi = ["age_band", "sex", "state", "cancer_type"]

# group by quasi-identifiers, count records per combination
group_sizes = df.groupby(quasi).size().reset_index(name="k")

# flag high-risk records where fewer than k=5 people share the profile
risky = group_sizes[group_sizes["k"] < 5]

print(f"Vulnerable equivalence classes: {len(risky)}")
print(f"Patients at risk: {risky['k'].sum()}")
print(risky.sort_values("k").head(10))

If risky is non-empty, you are not done. You generalize or suppress, then re-run until every class meets your threshold.

Beyond k-anonymity: two known weaknesses

k-anonymity alone is not enough. Two attacks defeat it:

• Homogeneity attack: a group of 5 people (k=5) all have the SAME sensitive value. If all five 60-69 male NY lung patients are marked "deceased within 6 months," you learn the outcome without needing to single anyone out. The fix is l-diversity (each group must contain at least *l* distinct sensitive values).
• Background knowledge attack: an attacker knows something external (a neighbor's approximate diagnosis date). The refinement here is t-closeness, which keeps each group's distribution close to the overall distribution.

For most oncology releases, aim for k-anonymity PLUS l-diversity on the sensitive outcome field.

🎬 [VIDEO: "Data Anonymization: k-anonymity, l-diversity and t-closeness explained" - youtube.com - a concise walkthrough of the three privacy models with worked examples]

The governance wrapper

The math is half the job. The other half is documentation and process, because regulators and IRBs (Institutional Review Boards, the ethics committees that approve human-subjects research) will ask for it.

A defensible workflow has these steps:

1. Data inventory. List every field. Classify each as direct identifier, quasi-identifier, or non-identifying clinical variable.
2. Method choice. Safe Harbor for low-value routine releases; Expert Determination for research datasets needing granular dates or geography.
3. Risk measurement. Run the k-anonymity and l-diversity checks. Record thresholds and results.
4. Transformation. Generalize, suppress, or add controlled noise. Log every transformation.
5. Expert sign-off. For Expert Determination, the statistician documents assumptions and certifies "very small" risk. Keep this document; it is your audit trail.
6. Data Use Agreement (DUA). Even de-identified data usually ships with a contract barring re-identification attempts. HIPAA's Limited Data Set pathway (which keeps dates and some geography) REQUIRES a DUA.
7. Re-assessment trigger. Anonymization decays. New public datasets (genealogy databases, leaked breaches) can re-enable linkage later. Schedule periodic re-review.

Where this bites in the real world

Genomic data is the hardest case. A genome is inherently identifying: you cannot "de-identify" DNA the way you redact a zip code, because the sequence itself is unique. This is why controlled-access repositories like the NIH's dbGaP exist: the data is not released openly at all, it is gated behind approved-researcher agreements. When someone tells you their genomic dataset is "anonymized," be skeptical.

Medtech adds device telemetry. A pacemaker or continuous glucose monitor produces a data stream with timing patterns that can act as a behavioral fingerprint. Treat device serial numbers and high-resolution timestamps as quasi-identifiers.

Key Takeaways

• HIPAA gives you two paths. Safe Harbor (remove 18 identifier categories, no statistician needed, but destroys granular dates and geography) or Expert Determination (documented risk analysis, keeps scientific value).
• GDPR is a higher bar. Pseudonymized data is still regulated; true anonymization must defend against singling out, linkability, and inference. Safe Harbor compliance does not equal GDPR anonymization.
• k-anonymity is measurable, so measure it. Group by quasi-identifiers, flag any class below your threshold (k=5 is a common target), then generalize or suppress. Attack your own dataown dataData collected directly from your own customers and prospects through your own channels: your most reliable and privacy-compliant source.View full definition → before release.
• k-anonymity alone is not enough. Add l-diversity for sensitive outcomes and watch for homogeneity and background-knowledge attacks.
• Anonymization decays and needs governance. Keep the expert determination document, ship a Data Use Agreement, and schedule re-review, because new public datasets can re-enable linkage you thought was closed.

This lesson is educational and not legal advice. Consult qualified counsel and a certified statistical expert before releasing any patient-level dataset.