Evaluating vendor and build-versus-buy AI options

# Evaluating vendor and build-versus-buy AI options

A radiology group signs a two-year contract for an AI tool that flags suspected lung nodules. Six months in, sensitivity on their patient population (heavily rural, older, more emphysema) is noticeably worse than the vendor's published numbers. The training data came mostly from urban academic centers. Switching costs are high because the model is wired into their PACS (Picture Archiving and Communication System, the software that stores and displaysdisplaysThe total number of times an ad or piece of content is displayed, regardless of clicks. Each display counts as one impression, even to the same person.Voir la définition complète → medical images). They are stuck.

This lesson gives you the scorecard to avoid that trap.

Why build-versus-buy is different in biotech and medtech

In most industries, "buy" wins by default: vendors have scale. In this sector, three things complicate that.

1. Your patients are not their patients. Model performance is population-specific. A sepsis prediction model trained on one health system routinely degrades elsewhere.

2. Regulation follows the model. In the US, many clinical AI tools are regulated by the FDA (Food and Drug Administration) as SaMD (Software as a Medical Device). In Europe, they fall under the MDR (Medical Device Regulation, EU 2017/745) and, increasingly, the EU AI Act, which classifies most medical AI as "high-risk." Who holds the regulatory clearance matters.

3. Integration is the real cost. The model is 10% of the work. The other 90% is EHR (Electronic Health Record) integration, LIMS (Laboratory Information Management System) connections, and clinical workflow.

The short version of build-versus-buy

• Buy when the task is generic and a vendor already holds regulatory clearance (nodule detection, diabetic retinopathy screening, transcription).
• Build only when you have proprietary data, in-house ML and regulatory talent, and the use case is core to your differentiation (for example, a pharma company building target-discovery models on its own screening libraries).
• Partner/fine-tune is the common middle path: license a base model, adapt it on your data under a clear contract.

Most organizations overestimate their ability to build. Building a compliant clinical model means owning the entire quality management system under standards like IEC 62304 (the software lifecycle standard for medical devices). That is a permanent staffing commitment, not a project.

The vendor scorecard

Score each vendor 1 to 5 on the dimensions below. Weight the categories to your context (a diagnostic lab weights integration higher; a drug-discovery team weights data provenancedata provenanceData lineage maps how data moves and transforms across systems, from origin to consumption, showing where it came from, what changed it, and where it goes.Voir la définition complète → higher).

1. Training data provenancedata provenanceData lineage maps how data moves and transforms across systems, from origin to consumption, showing where it came from, what changed it, and where it goes.Voir la définition complète →

Ask for specifics, in writing.

• What populations? Age, sex, race/ethnicity, geography, comorbidities, and scanner/device manufacturers.
• How many patients and sites? A model validated on 3 sites is not the same as one validated on 30.
• Consent and licensing. Was the data legally usable? Under GDPR (General Data Protection Regulation) in Europe and HIPAAHIPAAHealth Insurance Portability and Accountability Act, loi américaine imposant la protection des données de santé (PHI). Violations : amendes jusqu'à 1,9M$ par catégorie de violation. (Health Insurance Portability and Accountability Act) in the US, tainted training data is your liability once you deploy.
• Labeling method. Radiologist consensus? Pathology confirmation? Billing codes (weak)?

Red flag: "proprietary dataset, cannot share details."

2. Performance on YOUR population

Published metrics describe the vendor's test set, not your clinic. Demand a local validation before signing, or a contractual right to validate during a pilot.

Key metrics, defined:

• Sensitivity: of patients who truly have the condition, what fraction the model catches. Critical for screening (you do not want to miss disease).
• Specificity: of healthy patients, what fraction the model correctly clears. Low specificity means alert fatigue.
• PPV (Positive Predictive Value): of the cases the model flags, what fraction are truly positive. This depends on how common the disease is in YOUR population.

That last point trips people up. Here is why.

A worked example: why prevalence changes everything

A vendor advertises sensitivity 95%, specificity 90%. Sounds great. Now apply it to a screening population where the disease prevalence is 1% (a plausible screening scenario). Take 10,000 patients:

Diseased: 100
  Detected (95% sens):        95 true positives
  Missed:                      5 false negatives

Healthy: 9,900
  Flagged (10% false pos):   990 false positives
  Correctly cleared:       8,910 true negatives

PPV = TP / (TP + FP) = 95 / (95 + 990) = 8.8%

So 91% of the model's alerts are false alarms, even with excellent sensitivity and specificity. If clinicians must review every flag, you have created a workload problem, not a solution. Always recompute PPV at YOUR prevalence. The vendor's marketing rarely does this.

(These figures are illustrative, not vendor claims.)

3. Integration with lab and clinical systems

• Standards support. Does it speak HL7 FHIR (Fast Healthcare Interoperability Resources, the modern data-exchange standard) and DICOM (the imaging standard)? Custom point-to-point integrations age badly.
• Where does inference run? On-premise, in your cloud tenant, or the vendor's cloud? This affects data residency (a hard GDPR requirement in Europe).
• Workflow fit. Does the alert appear inside the radiologist's or pathologist's existing tool, or in a separate portal nobody opens?

4. Lock-in risk

• Data export. Can you export your data, labels, and model outputs on exit, in a usable format?
• Model portability. If you fine-tuned on your data, who owns the fine-tuned weights?
• Regulatory dependency. If you built your clinical workflow around their FDA clearance and they exit the market, you may have to stop using the tool.
• Contract term and price escalation for AI compute. Ask how per-study or per-seat pricing changes at scale.

🎬 [VIDEO: "How to Evaluate Healthcare AI Vendors" - youtube.com/results?search_query=evaluating+healthcare+AI+vendors - practical walkthrough of due-diligence questions for clinical AI procurement]

5. Monitoring and drift

Clinical AI degrades over time as patient populations, scanners, and protocols change. This is model drift. Ask:

• Does the vendor monitor real-world performance post-deployment?
• Under the FDA's framework for a PCCP (Predetermined Change Control Plan), has the vendor pre-specified how the model will be updated without a new full submission?
• Who is accountable when performance drops? Get it in the contract.

Turning the scorecard into a decision

Give each category a weight and a score. Example weighting for a hospital diagnostic deployment:

| Category | Weight | Vendor A (1-5) | Vendor B (1-5) |

|---|---|---|---|

| Data provenanceData provenanceData lineage maps how data moves and transforms across systems, from origin to consumption, showing where it came from, what changed it, and where it goes.Voir la définition complète → | 20% | 4 | 2 |

| Local performance | 25% | 3 | 4 |

| Integration | 25% | 2 | 4 |

| Lock-in risk | 15% | 2 | 3 |

| Monitoring/drift | 15% | 4 | 3 |

Weighted score, Vendor A:

(4*.20)+(3*.25)+(2*.25)+(2*.15)+(4*.15)
= .80+.75+.50+.30+.60 = 2.95

Vendor B:

(2*.20)+(4*.25)+(4*.25)+(3*.15)+(3*.15)
= .40+1.00+1.00+.45+.45 = 3.30

Vendor B wins on integration and local performance, which dominate a real deployment, despite weaker data documentation. The scorecard makes the tradeoff explicit instead of letting the slickest demo decide.