Bulk Data & Subscriptions

In one line

FHIR's REST API is beautifully designed for one patient at a time, on request. Two common needs break that assumption — give me a million patients and tell me when something happens — and Bulk Data and Subscriptions are the answers.

Why the normal API fails at scale

Suppose you want every diabetic patient's HbA1c for a population health programme. With plain FHIR you'd page through search results, resource by resource, over HTTPS. For 500,000 patients that is millions of requests. Your API falls over, the EHR vendor rate-limits you, and the job takes days.

The problem isn't that FHIR is slow. It's that you're using a transactional interface for an analytical job — the same category error as running a report against a live OLTP database.

Bulk Data (Flat FHIR)

The Bulk Data specification takes a different shape entirely:

1. You kick off an export — $export at system, patient or group level.
2. The server says "working on it" and gives you a status URL. The request is asynchronous — this is the key move.
3. You poll the status until it's done.
4. You get back a set of NDJSON files — newline-delimited JSON, one resource per line — usually as signed URLs to cloud storage.

NDJSON is the tell. It's not a pretty API response; it's a bulk file format that streams cleanly into a data warehouse, Spark job, or OMOP ETL. Bulk Data isn't "FHIR but faster" — it's FHIR admitting that analytics is a different problem and handing you a file.

This is what powers population health, quality measures, research extracts, and in the US the payer-to-payer and provider directory obligations behind the interoperability rules.

Subscriptions

The other gap: polling is a terrible way to learn that something happened. Asking "any new results?" every thirty seconds is expensive, slow, and mostly returns nothing.

A Subscription inverts it: you register a criterion — "tell me when an Observation with this code is created for this patient" — and the server notifies you. Push, not pull. It is FHIR's answer to what HL7 v2 always did naturally, because v2 was event-driven from birth. Subscriptions are, in a real sense, FHIR rediscovering why ADT feeds exist.

Worth knowing: the R4 Subscription resource was reworked substantially for R5 into a topic-based model, and the Subscriptions R5 Backport IG brings that design to R4 servers. So "does it support Subscriptions?" is an ambiguous question — ask which model, and pin the IG version like the dependency it is.

The security bit that gets skipped

Bulk export is a de facto data-extraction pipe for an entire population. That deserves more thought than a normal API call, and rarely gets it:

• SMART Backend Services authorisation exists precisely because there's no user clicking "allow" — it's system-to-system, so the trust is entirely in the client credential.
• The output URLs are frequently pre-signed cloud links. Anyone holding the link holds the data. Expiry and scope are load-bearing.
• Exports land in a warehouse, and the warehouse now holds identifiable health data — usually under weaker controls than the EHR it came from. That's where breaches happen.
• Under DPDP or a consent-artefact regime, "we exported the population" needs a lawful basis per patient. A bulk pipe makes it very easy to move data at a scale the consent never contemplated.

The rule of thumb: the easier the extraction, the more the governance has to carry. Bulk Data made the extraction easy.