# Consent, de-identification and the limits of anonymous data
In 2018, researchers demonstrated that a "de-identified" health dataset covering a small number of patients with a rare genetic disorder could be re-identified in minutes. The method: cross-referencing the de-identified claims data with a public patient registry where families had voluntarily shared their child's age, condition, and state of residence.
No names. No Social Security numbers. Just three or four indirect details, layered together, were enough to point to one household.
This is the core problem of this lesson: de-identification removes direct identifiers, but it does not remove uniqueness. And in pharma real-world evidence (RWE, meaning clinical insights drawn from data generated outside traditional clinical trials, such as claims, EHRs, and registries), uniqueness is exactly what rare-disease data has in abundance.
De-identification is a process of removing or altering data elements so that a data set is no longer reasonably linkable to a specific person. It does not mean "impossible to re-identify." It means "not reasonably identifiable given current methods and reasonably available data."
That last clause is doing a lot of work. "Reasonably available data" changes every year as more datasets get published, breached, or sold. A dataset de-identified safely in 2020 may be re-identifiable in 2026 because a new registry or genomic database now exists to link against.
Two regulatory frameworks define the bar differently:
Practical consequence for RWE teams: a claims dataset that is HIPAAHIPAAHealth Insurance Portability and Accountability Act, loi américaine imposant la protection des données de santé (PHI). Violations : amendes jusqu'à 1,9M$ par catégorie de violation. Safe Harbor de-identified for a US study is very likely still "pseudonymized personal data" under GDPR if any EU patients are in scope. Same file, different legal status, depending on jurisdiction.
Rare diseases (in the US, defined under the Orphan Drug Act as affecting fewer than 200,000 people nationally; in the EU, a prevalence of no more than 5 in 10,000) create a structural re-identification risk:
1. Small population size means each patient contributes more "identifying weight" to the dataset. If a condition affects 300 people in a state, knowing age, sex, and county narrows the pool dramatically.
2. Registry overlap is common. Patient advocacy groups (e.g., disease-specific foundations) often run public or semi-public registries with enrollment details that overlap with claims or EHR fields: diagnosis date, treatment center, sometimes even first name initials in newsletters or fundraising pages.
3. Linkage attacks don't need a name field. They need enough quasi-identifiers (attributes that aren't identifying alone but become identifying combined, like ZIP code, birth date, and sex) to shrink the anonymity set to one.
This is precisely the mechanism in the opening scene. Claims data with year-of-birth, state, and rare diagnosis code, linked to a registry with age, condition, and location, and the anonymity set is not large. It's often one.
Before a rare-disease RWE dataset moves to an analytics team or a CROCROConversion Rate Optimization (CRO) is the systematic practice of increasing the percentage of users who complete a desired action, using data, testing, and user research.Voir la définition complète → (contract research organization), run these checks:
K-anonymity means every combination of quasi-identifiers in the dataset matches at least *kkThe average number of new users each existing user generates through referrals. Above 1.0, growth compounds on itself and becomes exponential.Voir la définition complète →* other records. If kkThe average number of new users each existing user generates through referrals. Above 1.0, growth compounds on itself and becomes exponential.Voir la définition complète →=1, that record is unique and identifiable.
# Simplified k-anonymity check (pandas)
import pandas as pd
quasi_ids = ['birth_year', 'state', 'diagnosis_code', 'sex']
group_sizes = df.groupby(quasi_ids).size()
k_min = group_sizes.min()
unique_records = (group_sizes == 1).sum()
print(f"Minimum group size (k): {k_min}")
print(f"Number of unique (k=1) records: {unique_records}")For rare diseases, expect this to fail often. Standard fix: generalize fields (birth year to birth decade, state to region) until kkThe average number of new users each existing user generates through referrals. Above 1.0, growth compounds on itself and becomes exponential.Voir la définition complète → rises to an acceptable threshold, commonly kkThe average number of new users each existing user generates through referrals. Above 1.0, growth compounds on itself and becomes exponential.Voir la définition complète →≥5 as a working industry norm, though no single legal standard fixes this number.
Ask: what public or semi-public registries exist for this condition? Patient advocacy organizations (e.g., the National Organization for Rare Disorders, NORD) maintain lists of active disease registries. Cross-check whether your quasi-identifier set overlaps meaningfully with fields those registries publish or share with members.
Consent is not binary. A patient may have consented to:
Consent scope creep happens when a licensed dataset gets repurposed for an analysis beyond what the original informed consent form (ICF) authorized. In the US, this is scrutinized under the Common Rule (45 CFR 46) for federally funded research and by Institutional Review Boards (IRBs). In the EU, GDPR Article 6 and Article 9 (special category data, including health data) require a specific lawful basis, and "broad consent" for future research is narrower than many assume.
Audit question for any RWE project: does the consent language actually cover *this* use case, with *this* data granularity, shared with *this* third party?
Legitimate RWE data licenses (from vendors like IQVIA, Komodo Health, or claims aggregators) typically include contractual bans on re-identification attempts. This is a legal backstop, not a technical one. Contract review should confirm the DUA explicitly restricts linkage to outside datasets.
Vérification des acquis
1. The 42-patient re-identification example illustrates which core limitation of de-identification?
2. Why does the definition of 'de-identified' shift over time even if the dataset itself never changes?
3. For a rare-disease dataset with only a handful of patients per condition, why is de-identification especially challenging compared to a common-condition dataset with millions of patients?
4. Select ALL correct answers about the two HIPAA de-identification routes described.
Sélectionnez toutes les réponses correctes.
5. Select ALL correct answers about why 'reasonably available data' is a moving target in de-identification standards.
Sélectionnez toutes les réponses correctes.
In a mature pharma RWE program, de-identification risk isn't just a statistician's job. It sits across three functions:
A recurring real failure mode: a dataset is technically de-identified and legally licensed, but no one checks whether a *newly published* external registry now creates a linkage path that didn't exist when the original risk assessment was done. De-identification is a snapshot judgment, not a permanent state. Best practice is periodic re-assessment, particularly before any new data partnership or publication.
🎬 [VIDEO: "De-identification of Protected Health Information" - youtube.com - a walkthrough of HIPAAHIPAAHealth Insurance Portability and Accountability Act, loi américaine imposant la protection des données de santé (PHI). Violations : amendes jusqu'à 1,9M$ par catégorie de violation. Safe Harbor vs Expert Determination methods for health data teams]