Apache Kafka

In one line

Kafka is a distributed, append-only event log: producers write events to partitioned topics, the log persists them for days or forever, and any number of consumer groups read at their own pace — including re-reading history.

The problem it solves

A simple MQTT broker routes a message to whoever's subscribed right now and forgets it. But at hospital scale you want many downstream systems — analytics, alerting, the EHR, an ML pipeline, an audit store — each consuming the same stream independently, and you want to replay history when you add a new one. That's a durable log, not a transient queue — which is Kafka.

How it works

• Unlike a message queue that deletes on delivery, Kafka retains events; each consumer just tracks its offset (position in the log).
• Partitions give parallelism and ordering per key — all events for device-42 stay in order.
• Replication gives durability; consumer groups give horizontal scale-out.
• The replay property is the killer feature: deploy a new analytics service today and let it consume last month's stream as if it were live.

Where it shows up in digital health

• Hospital-scale device telemetry — MQTT at the edge, bridged into Kafka in the core.
• HL7/FHIR event backbones feeding many downstream systems independently.
• Audit pipelines (an immutable log is a natural audit trail) and ML feature streams.

The honest sizing note

Kafka is powerful and operationally heavy (brokers, partitions, replication, tuning). The sizing rule from this platform's own cost discipline: Kafka earns its weight at high volume — do not run it for ten devices. A small deployment is better served by a managed queue or even a TSDB directly.

Common pitfalls

• Reaching for Kafka too early — the operational cost outweighs the benefit at low volume.
• Poor partition-key choice — wrong keys break ordering guarantees or create hot partitions.
• Unbounded retention — "keep forever" is a storage (and PHI-governance) decision, not a default.

Key takeaways

• Kafka = a durable, partitioned, replayable event log — not a transient queue.
• Consumers read independently at their own offset; replay lets new services see history.
• The backbone for high-volume health event streams (telemetry, FHIR events, audit).
• Heavy by design — adopt it at real scale, not for small fleets.