Backup & DR

In one line

An untested backup is a rumour. The only backup that exists is one you have restored — and in a hospital, the question isn't whether you can restore, it's how many hours the ward can run on paper while you do.

The two numbers

• RPO — Recovery Point Objective. How much data may you lose? Nightly backups = up to 24 hours of clinical documentation gone.
• RTO — Recovery Time Objective. How long until you're back? Four hours? Two days?

Both must be agreed with clinicians, not chosen by IT. And the clinical answer differs wildly by system: losing a day of the dietary module is inconvenient; losing a day of medication administration records is a serious patient-safety event and possibly unrecoverable, because nobody can reconstruct what was given.

Which means one RPO/RTO for "the hospital" is meaningless. Tier by clinical consequence, the same way you'd tier your SLOs.

Ransomware changed the job

Backups used to be insurance against a failed disk. Now they are the primary defence against the most likely catastrophic event a hospital faces, and healthcare is heavily targeted for a brutally rational reason: hospitals cannot tolerate downtime, so they pay.

Modern ransomware knows this, and it changed the rules:

• It hunts your backups first. Encrypting production while backups survive doesn't pay. So attackers dwell, find the backup server, and destroy or encrypt it before triggering.
• It waits. Dwell time is often weeks. Your backups from the last month may already contain the payload — so restoring reinfects you.
• It exfiltrates before encrypting. Double extortion: pay to decrypt, and pay again not to publish the patient data. Your backup does not save you from the second demand. That's a breach, not an outage, and no restore fixes it.

Which is why backup is no longer a storage topic. It's a security one.

3-2-1, and the word that matters

Three copies, two media, one offsite — and now a fourth requirement that is doing all the work:

One copy must be immutable or offline.

Write-once storage, object lock, or genuinely disconnected media. If your backup can be deleted by a credential in your network, then it can be deleted by the attacker holding that credential, and you don't have a backup — you have a second target.

Test the restore, or you have nothing

The failures are boringly consistent, and every one is discovered during the incident:

• The backup ran for two years and backed up an empty directory. Nobody checked.
• The restore works but takes five days, because nobody measured against 8TB over that link.
• You can restore the database but not the encryption keys, so you have ciphertext.
• You restore, and discover the dependency order was never documented.
• The runbook lives on the wiki. The wiki is down.

The discipline: restore drills, on a schedule, timed, with the runbook on paper. A restore you have never performed is a plan, not a capability.

The part that isn't technology

The hospital's real DR plan is downtime procedures — paper drug charts, downtime forms, and a rehearsed way of working without computers. Most hospitals have these; most have not opened them in years, and the junior staff have never seen one.

Two things follow, and they're the ones that hurt:

• Your system should fail degraded, not gone. Read-only access to the chart during an outage prevents most harm — clinicians can still see the allergy list.
• The reconciliation afterwards is the dangerous part. Hours of paper notes typed back in, out of order, by exhausted people. That's where the medication errors cluster — not during the outage.

Recovery is not the moment the system comes back. It's the moment the record is trustworthy again, and that's days later.