# Measuring evidence quality: from trial data integrity to RWE fit-for-purpose scores
A health plan denies coverage for a $180,000 gene therapy, citing "insufficient real-world evidence of durability." The manufacturer's evidence package: a claims database analysis of 4,200 patients pulled from Optum's de-identified claims. The payer's pharmacy director asks one question before reading a single result: "Where did this data come from, and can I trust the pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.Voir la définition complète → that touched it?" That question, not the p-value, decides whether the study moves the reimbursement needle.
This lesson walks through how pharma and payers actually score evidence quality, using the FDA's real-world evidence (RWE) framework as the backbone.
RWE means data collected outside a traditional randomized controlled trial (RCT), typically from electronic health records (EHRs), insurance claims, registries, or wearables. The FDA does not ask "is this data good?" in the abstract. It asks whether the data is fit-for-purpose: adequate for the *specific* regulatory or reimbursement question being asked.
The FDA's 2018 Framework for RWE (FDA.gov) and the follow-up guidance on data standards lay out two pillars:
1. Relevance: does the dataset capture the right population, exposure, and outcome?
2. Reliability: is the data accurate, complete, and traceable from source to analysis?
A claims database might be perfectly reliable (clean, well-governed) but irrelevant (missing lab values needed to define your outcome). Both pillars must clear the bar.
Captured under Good Clinical Practice (GCP), with case report forms (CRFs), electronic data capture (EDC) systems like Medidata Rave, and audit trails. High reliability by design, but relevance is narrow: trial populations exclude comorbid, elderly, and polypharmacy patients who dominate real-world use.
Sources like IQVIA, Optum Clinformatics, and CMS Medicare claims. Strength: huge scale (Medicare fee-for-service covers over 30 million beneficiaries, CMS estimate). Weakness: claims are billing artifacts, not clinical facts. A diagnosis code (ICD-10) reflects what was billed, not necessarily confirmed disease. No lab values, no genomic data, no cause of death detail.
Sources like Epic's Cosmos network or Flatiron Health (oncology-specific). Richer clinical detail (labs, vitals, physician notes via natural language processing) but fragmented across health systems, and structured/unstructured data qualitydata qualityThe degree to which data is fit for purpose: accurate, complete, consistent, timely, valid and unique. Poor quality data undermines analytics, reporting and AI.Voir la définition complète → varies wildly by site.
Disease-specific, often prospective (e.g., the CIBMTR for stem cell transplant outcomes). High relevance for niche populations, but smaller sample sizes and slower accrual.
Take that claims-based gene therapy durability study. Run it through an FDA-aligned fit-for-purpose checklist:
1. Data provenance
Where did each record originate? Is there a documented chain from the health plan's adjudication system to the analytic dataset? A study lacking a data lineagedata lineageData lineage maps how data moves and transforms across systems, from origin to consumption, showing where it came from, what changed it, and where it goes.Voir la définition complète → diagram (source system → ETLETLETL (Extract, Transform, Load) is a data integration process that pulls data from sources, reshapes it into a consistent format, and writes it into a target system.Voir la définition complète → → analytic file) fails this immediately.
2. Transformation transparency
Every claims dataset requires cleaning: mapping National Drug Codes (NDC) to generic drug names, deduplicating claims, defining the "index date" for treatment start. If the methods section cannot reproduce these steps, reviewers cannot assess bias introduced during transformation.
3. Validation rate
What percentage of algorithmically-defined outcomes were confirmed against a gold standard (chart review)? A commonly cited benchmark in pharmacoepidemiology: claims-based algorithms for conditions like myocardial infarction typically show positive predictive values (PPV) in the 85 to 95% range when validated against medical records (this varies by algorithm and is study-specific, always check the validation paper cited).
4. Missing data handling
Claims data has no "missing" lab value field, it simply has no record. Did the study distinguish "test not done" from "test result was normal and not billed separately"? Mishandling this is a classic claims-data trap.
5. Linkage quality
If claims are linked to a mortality registry (like the National Death Index) or lab data (via a vendor like Datavant tokenization), what is the match rate? A 70% linkage rate versus 95% materially changes confidence in survival endpoints.
Say the study reports 4,200 patients, with an outcome validated by chart review in a random subsample of 300.
If FDA precedent or the therapeutic area norm expects PPV above 90% for a primary endpoint used in a label claim, this study sits just below threshold, a reviewer would flag it as supportive evidence, not confirmatory.
Payers and regulators increasingly score datasets on standing metrics, independent of any single study:
| Metric | What it measures | Typical benchmark (estimate) |
|---|---|---|
| Completeness rate | % of key fields non-null | greater than 95% for core demographic fields is common industry expectation |
| Concordance rate | Agreement between two independent data sources for same patient | 80 to 90% for diagnosis codes across payer/EHR, estimate, varies by condition |
| Audit trail coverage | % of records with traceable source-to-report lineage | Required at or near 100% under 21 CFR Part 11 for regulated submissions |
| Refresh latency | Time lag between real-world event and data availability | Claims: 1 to 3 months typical lag (estimate); EHR: days |
Data standards bodies matter here too. CDISC (Clinical Data Interchange Standards Consortium) standards, like SDTM (Study Data Tabulation Model), are mandatory for FDA trial submissions. For RWE, no single mandatory standard exists yet, but OMOP (Observational Medical Outcomes Partnership) common data model, coordinated by OHDSI, is becoming a de facto standard for multi-database federated studies in both the US and Europe.
import pandas as pd
df = pd.read_csv("claims_extract.csv")
completeness = df[["patient_id", "ndc_code", "diagnosis_code", "service_date"]].notna().mean()
print(completeness)
# patient_id 1.00
# ndc_code 0.94
# diagnosis_code 0.99
# service_date 1.00A 94% completeness rate on drug codes might be acceptable for a descriptive study, but too low if NDC is the variable defining treatment exposure in a comparative effectiveness claim.
Vérification des acquis
1. A claims database is well-governed, complete, and traceable from source to analysis, but lacks the lab values needed to define the study outcome. How would this dataset be characterized under the FDA's RWE framework?
2. Why does the FDA frame RWE evaluation as 'fit-for-purpose' rather than asking whether data is simply 'good'?
3. A pharmacy director's first question about a claims-based study is 'where did this data come from, and can I trust the pipeline that touched it?' This question primarily addresses which pillar of evidence quality?
4. Select ALL correct answers about why clinical trial data collected under Good Clinical Practice (GCP) is described as having high reliability but narrow relevance.
Sélectionnez toutes les réponses correctes.
5. Select ALL correct answers about evaluating a real-world evidence package for a reimbursement decision.
Sélectionnez toutes les réponses correctes.
The EU's counterpart infrastructure is DARWIN EU (Data Analysis and Real World Interrogation Network), coordinated by the European Medicines Agency (EMA), which federates observational databases across member states using the OMOP common data model rather than pooling raw data centrally. This mirrors the FDA Sentinel System's federated query approach in the US (used originally for drug safety surveillance, now expanding to effectiveness questions).
Both systems solve the same governance problem: you cannot move patient-level data across institutions or borders easily under GDPR (General Data Protection Regulation) in Europe or HIPAAHIPAAHealth Insurance Portability and Accountability Act, loi américaine imposant la protection des données de santé (PHI). Violations : amendes jusqu'à 1,9M$ par catégorie de violation. (Health Insurance Portability and Accountability Act) in the US, so the query goes to the data instead of the data coming to the analyst.
🎬 [VIDEO: "Real-World Evidence: What It Is and Why It Matters" - youtube.com/results?search_query=FDA+real+world+evidence+explained - search for FDA or Duke-Margolis explainer videos covering the RWE framework basics and Sentinel System architecture]