Guardrails and pre-deployment checks

Legally, most of these are medical devices. In the US they fall under the FDA (Food and Drug Administration), typically as SaMD (Software as a Medical Device). In Europe they are regulated under the MDR (Medical Device Regulation, EU 2017/745), and separately as high-risk AI under the EU AI Act, which entered into force in 2024 with obligations phasing in through 2026 and 2027.

That regulatory status sets the floor. Good guardrails go above it.

The pre-deployment checklist

Run these gates in order. A failure at any gate blocks release.

Gate 1: Clinical validation

Before anything else, prove the model works on the population that will actually use it.

• Retrospective validation: test on held-out historical data the model never saw during training.
• External validation: test on data from a *different* hospital or region. Models that hit 0.92 AUROC in-house often drop sharply elsewhere. (AUROC, area under the ROC curve, is a common accuracy measure where 1.0 is perfect and 0.5 is a coin flip.)
• Prospective validation: run silently alongside real clinicians (a "shadow mode" deployment) and compare predictions to outcomes without acting on them.

Concrete threshold example: a stroke-triage tool might require sensitivity above a pre-registered floor (say 0.90) because a missed stroke is catastrophic, accepting more false positives as the trade.

Gate 2: Bias and subgroup audit

Aggregate accuracy hides harm. Break performance down by subgroup and check the gaps.

Report metrics per group: age band, sex, race and ethnicity, insurance status, primary language. Look at both false negative rate (missed urgent cases) and false positive rate (unnecessary escalation) per group.

Here is a minimal audit in code:

The rule matters more than the code: define, before you look, what gap is unacceptable. A common approach is to require that no protected subgroup's false negative rate exceeds the best-performing group by more than a set margin. If it does, you retrain, reweight, or restrict the intended-use population, and you document why.

The EU AI Act requires this kind of data governancedata governanceData governance is the set of policies, roles, and processes that ensure data is accurate, secure, well-defined, and used responsibly across an organization.View full definition → and bias examination for high-risk systems. The FDA's Good Machine Learning Practice principles, issued jointly with UK and Canadian regulators, name representativeness of data as a core expectation.

Gate 3: Human-in-the-loop thresholds

"Human-in-the-loop" means a person reviews or approves the model's output before it affects care. The design question is *when*.

Set thresholds by risk, not convenience:

• Automate: low-stakes, high-confidence outputs (routing a stubbed toe to a nurse line).
• Recommend, human decides: the default for triage. The model suggests acuity; a clinician confirms.
• Force human review: high-acuity flags, low-confidence scores, and any case near the decision boundary.

A trap here is automation bias: clinicians rubber-stamp the model because it is usually right. Counter it by showing the model's confidence, surfacing the top drivers of the score, and periodically injecting cases where the model is wrong to keep reviewers alert (used in some radiology QA programs).

Gate 4: Fallback modes

Ask: what happens when the model is unavailable, uncertain, or fed garbage?

• Uncertainty abstention: if confidence is below a floor, the system returns "unable to score" and routes to standard human triage rather than guessing.
• Out-of-distribution detection: flag inputs unlike anything in training (a pediatric patient hitting an adult-only model) and refuse to score.
• Graceful degradation: if the model service goes down, the workflow falls back to the existing manual protocol, not a blank screen. Clinicians should never be blocked from care because an APIAPIApplication Programming Interface: a standardised interface that lets applications communicate and exchange data without knowing each other's internal workings.View full definition → timed out.

Write the fallback into the clinical workflow and *test it* by deliberately taking the model offline during a drill.

Gate 5: Documentation and traceability

Regulators and your own risk team need a paper trail.

• Model card: a short document stating intended use, training data, performance by subgroup, and known limitations.
• Intended-use statement: exactly what the model is cleared to do, and what it is not (for example: "adult ED triage support; not for pediatric or obstetric use").
• Audit log: every prediction, the input version, the model version, and the human decision, retained so you can reconstruct any case.

The EU AI Act mandates technical documentation and logging for high-risk systems. Treat it as a design requirement, not paperwork bolted on at the end.

Guardrails that hold in production

Passing pre-deployment gates is necessary but not sufficient. Models decay.

Monitor for drift

Data drift is when the incoming patient population shifts away from the training data (a new variant, a new referral pattern, a merged hospital system). Performance drift is when accuracy degrades over time. Track both.

Practical setup: monitor input feature distributions weekly, and where outcomes are available (did the flagged patient actually deteriorate?), track false negative rate on a rolling window. Set an alert threshold. If the rolling FNR breaches it, the model auto-reverts to shadow mode and pages the responsible clinician-owner.

Assign an owner

Every deployed model needs a named accountable owner, usually a clinician plus a data scientist. Under both FDA and EU frameworks, post-market surveillance is not optional. Someone has to watch the dashboards and have authority to pull the model.

Plan for change

Models get retrained. The FDA supports a Predetermined Change Control Plan (PCCP): you specify in advance what kinds of updates you may make (retraining on new data, threshold tweaks) and how you will validate them, so routine improvements do not require a fresh submission each time. Define your PCCP before launch.

A simple worked example

Suppose a sepsis-warning tool is deployed across 20,000 ED visits per year, with a true sepsis rate of 2 percent, so 400 real cases. At 0.90 sensitivity the model catches 360 and misses 40. If a subgroup audit shows sensitivity for non-English-speaking patients is only 0.75, then within that subgroup the model misses one in four true cases. That gap, not the headline 0.90, is what your guardrail must catch. (Illustrative figures.)

Key Takeaways

• Triage algorithms are usually regulated medical devices (FDA SaMD in the US, MDR plus the EU AI Act in Europe). Regulation is the floor; engineer guardrails above it.
• Never trust aggregate accuracy. Audit false negative and false positive rates by subgroup, set an unacceptable-gap threshold before you look, and block release if it breaks.
• Design human-in-the-loop by risk tier and actively fight automation bias. Build fallback modes (abstention, out-of-distribution detection, graceful degradation) and drill them.
• Ship with a model card, an intended-use statement, and full audit logging. Then monitor for data and performance drift with an alert that can auto-revert the model.
• Name an accountable clinician-plus-data-scientist owner and define a Predetermined Change Control Plan before launch, so updates stay safe and traceable.

*This lesson is educational and not legal, regulatory, or medical advice.*