Reliability / SRE

In one line

SRE treats reliability as a feature with an explicit budget: decide how much downtime is acceptable, measure it, and spend the remainder on shipping. The framework is excellent. In healthcare, the arithmetic underneath it needs re-reading.

The core machinery

• SLI — a measurement: the proportion of requests served successfully in under 300ms.
• SLO — a target: 99.9% of them, over 30 days.
• Error budget — the leftover: 0.1%, which is ~43 minutes a month. That's not failure, it's allowance. Spend it on risky deploys, experiments, and moving fast.
• The rule — burn the budget and you stop shipping features until reliability recovers.

The genuine insight here is political, not technical. The error budget converts an endless argument ("dev wants to ship, ops wants stability") into a number both sides agreed to in advance. That's why SRE spread — it settled a fight, not a bug.

Where healthcare breaks the model

The error-budget logic rests on an assumption that is usually unstated: the cost of downtime is roughly linear and roughly financial. An e-commerce site down for 43 minutes loses 43 minutes of sales. Unpleasant, measurable, survivable.

Now price 43 minutes of your EHR being down during a Monday morning ward round.

The cost isn't revenue. It's a nurse reconstructing a drug chart from memory, a clinician ordering blind, a lab result that doesn't reach anyone, and — this is the part that doesn't fit the spreadsheet — a small probability of a serious harm that no dashboard will attribute to you. The harm is real, delayed, and statistically invisible in your incident review.

So take SRE's machinery and be honest about two adjustments:

Not all downtime is equal. Your SLO should distinguish the medication administration path from the reporting dashboard. Averaging them produces a number that looks fine while the dangerous thing is broken. Tier your services by clinical consequence, not by traffic.

Timing dominates. 43 minutes at 03:00 on a Sunday and 43 minutes at 09:00 on a Monday are the same SLO and completely different events. Health load is brutally predictable — round times, shift changes, OPD opening — which means you can plan around it, and have no excuse not to.

Downtime procedures — the part software people forget

Hospitals have run without computers for most of their history, and they still know how. There are downtime forms, paper drug charts, and a procedure for reconciling everything afterwards.

Two things follow that engineers routinely miss:

• Your system's failure mode should be degraded, not gone. Read-only access to the chart during an outage is enormously more useful than a friendly error page. If clinicians can still see the allergy list, you have prevented most of the harm.
• The reconciliation afterwards is the dangerous part. Hours of paper notes get typed back in, out of order, by tired people. That's where the medication errors cluster — not during the outage itself. Design for the recovery, not just the incident.

Blamelessness, and why it's not softness

SRE's blameless postmortem is often mistaken for niceness. It isn't. It's a claim about information: if people are punished for reporting, you stop learning what happened.

Healthcare arrived at this independently, decades earlier, and calls it patient-safety culture. The same finding, twice, from two directions: you can have accountability or you can have information, and the systems that pick information get safer.

That's the sentence worth carrying — and the reason an SRE and a patient-safety officer, who have never met, are doing recognisably the same job.