Model governance

In one line

Model governance answers three questions before a model touches a patient: who is accountable when it's wrong, how would anyone find out, and what makes it stop? Everything else is detail.

Why this isn't bureaucracy

There's a reflex among engineers that governance is what slows good work down. In clinical AI the opposite is true, and the reason is structural.

A model has no professional registration. It cannot be sued, struck off, or asked what it was thinking. When a clinician makes an error, an entire apparatus exists — registration, indemnity, review, learning. When a model makes an error, that apparatus finds nobody at the other end. Governance is how you attach the model to a human who can be held responsible, and without it a hospital simply cannot deploy the thing responsibly.

So the question is never "do we need governance?" It's "who is the human?"

The three questions

Who is accountable? Not the vendor — they have a contract, not a duty of care to your patient. Not "the AI committee" — a committee cannot be accountable, only individuals can. There must be a named clinical owner who can be asked, in a review, why this model was running on this ward. If you cannot name that person today, you are not ready to deploy.

How would you know it's wrong? This is the one teams skip. A model that quietly degrades produces the same confident output as a model that works — see dataset shift. So you need pre-defined monitoring: what metric, measured how often, compared against what baseline, reviewed by whom. "We'll notice" is not monitoring. Nobody noticed the sepsis model for years.

What makes it stop? A kill switch with a named owner and a threshold agreed in advance. Agreeing the threshold before go-live matters enormously, because after go-live the model has a budget, a champion and a sunk cost, and every argument for switching it off becomes an argument against someone's project.

The lifecycle, briefly

• Before build — what decision does this change? If it changes no decision, stop; you have a demo, not a tool.
• Before deploy — validate on your population, not the vendor's. Document intended use, known limits, and who it was NOT validated on. A model card is the standard shape for this, and shipping one with the model rather than after an incident is what "AI-native" actually means.
• In production — monitor performance, drift, alert burden, and override rate.
• Retirement — models should be allowed to die. A model nobody owns and nobody monitors is a liability generating numbers.

The metrics nobody watches

Two signals that tell you more than accuracy:

Override rate. If clinicians never override the model, they have stopped thinking — that's automation bias, and it's a harm. If they always override it, the model is noise with a maintenance cost. Both extremes are alarms; neither shows up in an AUC.

Alert burden. Every alert spends a clinician's attention, which is finite and shared across every other alert in the building. A model that improves one outcome by degrading attention everywhere else may be net-negative and look excellent in its own evaluation.

Frameworks worth knowing

NIST's AI Risk Management Framework organises this into Govern → Map → Measure → Manage, and it's a genuinely useful spine for a health system's policy. WHO's guidance adds the ethical core: autonomy, transparency, accountability, equity — and it insists that AI must not widen the gap between those who have care and those who don't, which is a live risk when models are trained on data from well-resourced populations and deployed on everyone else. The EU AI Act classifies most clinical AI as high-risk, with obligations that follow from that.

None of these will tell you whether your model is safe. They tell you what questions you are obliged to have answered.

The honest summary

If you take one thing: the hard part of clinical AI was never the model. It is the fairness work, the validation, the monitoring, the named owner, and the willingness to turn something off that a lot of people worked hard on. Teams that find governance boring tend to be the ones who discover it after an incident, when it is called an inquiry instead.