A sepsis prediction tool hits 92% accuracy in the lab. The hospital deploys it. Six months later, clinicians ignore its alerts because it fires constantly on patients who never deteriorate. The model was never broken on accuracy. It was broken on the benchmarks that matter: it caught few true cases and cried wolf on the rest.
This lesson is about the measurement standards that separate a diagnostic model you can trust from one that quietly harms patients. We will cover four benchmark families: sensitivity and specificity, calibration, and data conformance to CDISC and OMOP standards. All are data-discipline questions, not clinical opinions.
Accuracy is the share of predictions that are correct. In medicine it lies, because disease is rare.
Imagine a rare cancer present in 1% of a screened population. A model that predicts "no cancer" for everyone is 99% accurate and catches zero cancers. Useless.
So clinical data teams use two paired metrics instead.
Suppose you validate a diagnostic model on 1,000 patients. 100 truly have the disease.
| | Model says Positive | Model says Negative |
|---|---|---|
| Truly has disease (100) | 90 (true positives) | 10 (false negatives) |
| Truly healthy (900) | 45 (false positives) | 855 (true negatives) |
Notice sensitivity and specificity tell you where the pain is (10 missed patients, 45 false alarms) while accuracy hides it.
There is no universal cutoff. Thresholds depend on the clinical use. A confirmatory test tolerates lower sensitivity; a screening test that rules disease out demands very high sensitivity so it does not miss cases.
The single most-cited summary metric is AUC-ROC (area under the receiver operating characteristic curve), which ranges from 0.5 (coin flip) to 1.0 (perfect). As a rough field convention (an estimate, not a regulation), AUC below 0.7 is weak, 0.7 to 0.8 acceptable, 0.8 to 0.9 good, above 0.9 excellent. Do not treat these as pass marks. The FDA and Europe's regulators evaluate intended use, not a magic number.
For the tradeoff intuition:
Sensitivity and specificity ask whether the ranking is right. Calibration asks whether the probabilities are honest.
If your model says "70% risk of stroke" for a group of patients, roughly 70% of them should actually have a stroke. If only 40% do, the model is overconfident and miscalibrated, even if its AUC is high.
Calibration matters enormously in medicine because clinicians act on the probability, not the rank. A 30% versus 80% risk changes whether you operate.
Common tools:
A well-known field lesson: many published COVID-19 prognostic models scored decent AUCs but were poorly calibrated and unusable in practice. A useful reference framework for reporting is TRIPOD, the guideline for transparent reporting of prediction models.
# Quick calibration + discrimination check with scikit-learn
from sklearn.metrics import roc_auc_score, brier_score_loss
from sklearn.calibration import calibration_curve
auc = roc_auc_score(y_true, y_prob) # discrimination (ranking)
brier = brier_score_loss(y_true, y_prob) # calibration + accuracy combined
frac_pos, mean_pred = calibration_curve(y_true, y_prob, n_bins=10)
# Plot mean_pred vs frac_pos; closer to the diagonal = better calibratedA model is only as trustworthy as the datasets feeding it. This is where data conformance standards enter. Two dominate biotech and medtech.
CDISC (Clinical Data Interchange Standards Consortium) sets the data formats for clinical trials. The key ones:
Since December 2016, the US FDA requires CDISC-conformant data for new drug and biologic submissions (NDAs, BLAs) under its data standards catalog. Japan's PMDA has parallel requirements. So for a therapeutic, CDISC conformance is not optional; a non-conformant submission gets a Refuse to File.
Conformance is checked with validation rules (for example via the open Pinnacle 21 / CDISC CORE engine). A conformance report flags errors (must fix) and warnings. Benchmark: zero unexplained errors before submission.
OMOP CDM (Observational Medical Outcomes Partnership Common Data Model), maintained by the OHDSI community, standardizes real-world datareal-world dataRWD, données collectées en dehors des essais cliniques contrôlés : dossiers médicaux, claims d'assurance, données de dispositifs connectés, base des Real-World Evidence (RWE).: electronic health records, claims, registries. It lets a query written once run across hospitals in the US, Europe, and Asia without reworking each database's quirks.
OMOP matters for real-world evidence (RWE), increasingly used to support device approvals and post-market surveillance. The EU's EHDS (European Health Data Space), which entered into force in 2025 with phased application through the late 2020s, pushes hard toward standardized, interoperable health data across member states, and OMOP is a common target model.
OHDSI provides an open tool, Achilles / Data Quality Dashboard, that runs thousands of automated checks (plausibility, conformance, completeness) against an OMOP dataset. Teams report a data-quality pass rate; a common working target is that failed checks stay under a small single-digit percentage, though the exact bar is study-specific.
If two hospitals code "hemoglobin A1c" differently, your multi-site model trains on garbage. Conformance to a shared vocabulary (OMOP uses standard vocabularies like SNOMED, LOINC, RxNorm) is what makes the sensitivity and calibration numbers portable. Skip it and your 0.9 AUC evaporates at the next site.
Vérification des acquis
1. Why is accuracy considered a misleading headline metric for diagnostic models detecting rare diseases?
2. A sepsis model 'fires constantly on patients who never deteriorate,' causing clinicians to ignore it. Which benchmark failure does this describe?
3. In the worked example, sensitivity was 90% and specificity was 95%. What is the primary advantage of reporting these two metrics instead of the single 94.5% accuracy figure?
4. Select ALL correct answers about sensitivity and specificity.
Sélectionnez toutes les réponses correctes.
5. Select ALL correct answers about the framing of benchmarks in this lesson.
Sélectionnez toutes les réponses correctes.
A credible diagnostic model validation reports all four families. Here is a concrete scorecard structure a data team would present to a regulatory or clinical committee.
| Benchmark family | Metric | Example target (illustrative, not a regulatory cutoff) |
|---|---|---|
| Discrimination | Sensitivity | High for screening use (context-set) |
| Discrimination | Specificity | Balanced against false-alarm cost |
| Discrimination | AUC-ROC with 95% confidence interval | Reported, not just point estimate |
| Calibration | Calibration plot + Brier score | On diagonal; low Brier |
| Data conformance | CDISC validation report | Zero unexplained errors |
| Data conformance | OMOP DQDQThe degree to which data is fit for purpose: accurate, complete, consistent, timely, valid and unique. Poor quality data undermines analytics, reporting and AI.Voir la définition complète → Dashboard pass rate | High pass rate, documented failures |
Two rules that separate professionals from amateurs:
1. Always report on an external validation set, meaning data from a site or time period the model never trained on. In-sample metrics are marketing.
2. Report subgroup performance. A model with 90% overall sensitivity can be 60% sensitive in one demographic. US FDA guidance and EU frameworks increasingly expect subgroup breakdowns to detect bias.
The practical takeaway: your benchmark scorecard is the evidence these bodies read. Weak calibration or unexplained conformance errors are not statistical footnotes; they are approval blockers.
*This lesson is educational and not medical, legal, or regulatory advice. All illustrative thresholds are field conventions, not binding standards; confirm current requirements with the relevant authority.*