Messaging systems decouple services so they can fail and scale independently. The three you'll meet most are Kafka, RabbitMQ, and Pulsar. This guide helps you choose, then install each for high availability on Kubernetes — and prove it with a node-failure drill.

Prerequisites

  • A Kubernetes cluster with 3+ worker nodes (ideally across zones) and a dynamic StorageClass backed by SSD.
  • kubectl and helm installed.
  • Familiarity with pods, StatefulSets, and PersistentVolumeClaims.

Which one, and why

Kafka RabbitMQ Pulsar
Model Distributed log (replayable) Broker/queue (smart routing) Log + queue, tiered storage
Best for High-throughput streams, event sourcing, analytics Task queues, complex routing, RPC Multi-tenant, geo-replication, streaming+queuing
Ordering Per partition Per queue Per partition
Consumption Pull, offset-based, replayable Push/pull, ack, then gone Both, with cursors
Gotcha Partition planning, rebalances Throughput ceiling per queue More components (brokers + BookKeeper + metadata)

Rule of thumb: streams and replay → Kafka; flexible routing and task queues → RabbitMQ; multi-tenant or geo-replicated at scale → Pulsar.

What "HA" actually requires (all three)

  1. Replication — every message exists on more than one node (replication factor ≥ 3).
  2. Spread — replicas land on different nodes/zones (pod anti-affinity + topology spread).
  3. Quorum — a majority must acknowledge writes so a single failure loses nothing (with RF 3, keep min in-sync replicas = 2).
  4. Disruption budget — a PodDisruptionBudget so node drains/upgrades never break quorum.

Kafka HA with the Strimzi operator

Strimzi makes Kafka declarative. Modern Kafka uses KRaft (no ZooKeeper):

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata: {name: prod}
spec:
  kafka:
    replicas: 3
    config:
      default.replication.factor: 3
      min.insync.replicas: 2
    storage: {type: persistent-claim, size: 100Gi, class: fast-ssd}
    template:
      pod:
        affinity:
          podAntiAffinity:                 # keep brokers on separate nodes
            requiredDuringSchedulingIgnoredDuringExecution:
              - topologyKey: kubernetes.io/hostname
                labelSelector:
                  matchLabels: {strimzi.io/name: prod-kafka}

Install the operator (helm install strimzi strimzi/strimzi-kafka-operator), apply the manifest, and Strimzi builds the StatefulSets, storage, and listeners. Add a KafkaTopic with replicas: 3.

RabbitMQ HA with the Cluster Operator

apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata: {name: prod}
spec:
  replicas: 3                    # odd number → clean quorum
  persistence: {storage: 50Gi}

Then use quorum queues (Raft-based), not classic mirrored queues — they're the modern, correct way to survive node loss:

rabbitmqadmin declare queue name=orders durable=true arguments='{"x-queue-type":"quorum"}'

Pulsar HA

Pulsar separates serving (brokers, stateless) from storage (BookKeeper bookies). For HA: brokers ≥ 3, bookies ≥ 3 with ensemble=3, write-quorum=3, ack-quorum=2, and a 3-node metadata quorum (ZooKeeper or etcd). The Apache Pulsar Helm chart or the StreamNative operator wires this up. It's more moving parts — the payoff is tiered storage (offload cold data to S3) and true multi-tenancy.

Consumers, partitions & delivery guarantees

HA keeps the broker up; delivery guarantees decide whether you lose or duplicate messages. This is where most real bugs live, and the concepts map across all three systems.

Partitions are the unit of parallelism and ordering (Kafka/Pulsar). A topic is split into partitions; each is consumed by one member of a consumer group at a time, so ordering is only guaranteed within a partition. More partitions = more parallel consumers, but also more overhead and weaker global ordering. Pick a partition key (e.g. user_id) so related messages stay ordered together.

The three delivery semantics:

  • At-most-once — commit the offset before processing. Fast, but a crash loses the in-flight message. Rarely what you want.
  • At-least-once — process then commit. The default and the right choice for most systems; a crash re-delivers, so your consumer must be idempotent (processing the same message twice is safe).
  • Exactly-once — Kafka offers it via the idempotent producer (enable.idempotence=true) plus transactions that atomically write output and commit input offsets. Powerful but with throughput cost; reserve it for money-movement-grade needs.

Durability on the write side is the other half: acks=all plus min.insync.replicas=2 (with RF 3) means a write is only acknowledged once it's safely on two brokers. Set acks=1 and a broker crash right after ack loses data. RabbitMQ's equivalent is publisher confirms on quorum queues; Pulsar's is ack-quorum.

Rebalances are the operational gotcha: when a consumer joins or leaves, partitions are reassigned and consumption briefly pauses. Frequent rebalances (from crashing or slow consumers hitting max.poll.interval) look like "the pipeline keeps stalling." Right-size consumers and processing time.

Design rule: assume redelivery, make consumers idempotent, and require quorum acks. That single sentence prevents most data-loss and duplicate-processing incidents.

Schemas & the schema registry

Messages are a contract between producers and consumers, and that contract drifts — someone renames a field, and every downstream consumer breaks. A schema registry (Confluent Schema Registry, Apicurio, or Pulsar's built-in schema store) fixes this by storing versioned schemas (Avro, Protobuf, JSON Schema) and enforcing compatibility rules on every new version:

  • Backward compatible — new consumers can read old messages (safe to add optional fields).
  • Forward compatible — old consumers can read new messages.
  • Full — both.

Producers register a schema; the registry rejects an incompatible change before it reaches the topic, so a bad deploy can't silently corrupt the stream. Messages carry a small schema ID instead of the full schema, which also shrinks payloads. If you take streaming to production, a schema registry is not optional — it's the thing that lets many teams share a topic without breaking each other.

Security: encryption, auth & authorization

An open broker is a data breach waiting to happen. Three layers, in every serious deployment:

  • Encryption in transit — TLS between clients and brokers, and broker-to-broker. Kafka, RabbitMQ, and Pulsar all support it; the operators make cert management manageable (Strimzi issues them for you).
  • Authentication — who is connecting. Kafka uses SASL (SCRAM or mutual-TLS); RabbitMQ has users and vhosts; Pulsar has token/JWT and mTLS. No anonymous producers in production.
  • Authorization (ACLs) — what each client may do. Restrict producers to their topics and consumers to their groups, so a compromised service can't read or write everything. Kafka ACLs, RabbitMQ permissions per vhost, Pulsar namespace policies.

Multi-tenancy raises the stakes: Pulsar's tenant/namespace model is built for it; Kafka relies on ACLs and quotas. Either way, treat the message bus like a database — it holds your data in motion.

Capacity planning: partitions, throughput & retention

Sizing is where streaming projects succeed or thrash. The key levers:

  • Partitions set your maximum consumer parallelism (one active consumer per partition per group). Estimate target throughput ÷ per-consumer throughput, add headroom, and pick a partition count you won't need to change often (increasing partitions later reshuffles keys). Too few = a bottleneck; too many = more overhead and longer failovers.
  • Throughput is governed by message size, batching, and compression. Enable producer batching and compression (lz4/zstd) — it dramatically raises throughput and cuts network/disk cost.
  • Retention decides disk. Kafka keeps messages for a time or size window regardless of consumption (retention.ms, retention.bytes) — a replayable log — so size disks for the retention window, not just the backlog. RabbitMQ queues are drained on consume (size for peak backlog). Pulsar can offload cold segments to S3, decoupling retention from local disk.

What to monitor

Each system has a "tells you everything" signal:

  • Kafka — consumer lag (how far consumers trail the log head) is the number one health metric, plus under-replicated partitions and offline partitions.
  • RabbitMQ — queue depth and unacked message count (a growing queue = consumers can't keep up), plus memory/disk alarms.
  • Pulsar — backlog per subscription, plus bookie storage.

All three expose Prometheus metrics; wire them into the kube-prometheus-stack and alert on lag/backlog growth before it becomes an outage.

Client tuning: producers & consumers

The brokers are only half the system; the clients decide throughput and safety. A few settings matter most.

Producers. For durability, acks=all with enable.idempotence=true (the safe default in modern clients). For throughput, batch and compress: linger.ms (wait a few ms to fill a batch), batch.size, and compression.type=lz4 (or zstd) together can multiply throughput and cut cost. Choose a partition key deliberately — it sets both ordering and how evenly load spreads (a bad key creates a hot partition).

Consumers. The big decision is offset commits: auto-commit is easy but can lose or double-process messages around a crash; manual commit after processing gives you at-least-once with control. Size max.poll.records and keep per-message processing fast, or a slow consumer blows max.poll.interval.ms and triggers a rebalance (the "pipeline keeps stalling" symptom). Scale consumers up to — but not beyond — the partition count, since extra consumers in a group sit idle.

The mental model: producers trade latency for throughput and durability; consumers trade simplicity for delivery guarantees. Know which you're choosing.

Architecture patterns you'll actually use

Messaging shines in a few well-worn patterns worth recognizing:

  • Event-driven / pub-sub. Services publish events ("order placed") and any number of consumers react (email, inventory, analytics) without the publisher knowing about them. This is the core decoupling win — add a new consumer without touching the producer.
  • Event sourcing. Store the log of events as the source of truth (Kafka's replayable log is ideal), and derive current state by replaying them. You get a full audit trail and can rebuild state or new views at will.
  • CQRS. Separate the write model from read models built by consuming events — each read model optimized for its queries.
  • The transactional outbox. The classic "how do I update my database and publish an event atomically" problem: write the event to an outbox table in the same DB transaction, then a relay (or CDC like Debezium) publishes it to Kafka. This avoids the dual-write bug where the DB commits but the publish fails (or vice-versa) — a subtle data-integrity trap worth knowing by name.

You don't need all of these, but recognizing them helps you design systems that stay decoupled and recoverable.

Dead letter queues & poison messages

Sooner or later a consumer hits a message it can't process — malformed data, a bug, a downstream that's down. Without a plan, that poison message either blocks the queue (the consumer retries it forever, stalling everything behind it) or gets dropped silently (data loss). The fix is a dead letter queue (DLQ): after N failed attempts, the message is moved aside to a separate queue/topic for later inspection, and the consumer moves on.

  • RabbitMQ has first-class DLQ support — configure a x-dead-letter-exchange, and messages that are rejected or expire are routed there automatically.
  • Kafka has no built-in DLQ, but the pattern is standard: on repeated failure, produce the message to a something.DLT topic with the error and original metadata, then commit the offset so the consumer proceeds. Kafka Connect and many frameworks implement this for you.
  • Pulsar supports DLQ policies and negative acknowledgements natively.

Then alert on DLQ depth — a growing DLQ means a real bug is throwing messages away, and someone should look. A DLQ turns "one bad message halts the pipeline" into "one bad message is quarantined for review" — essential for any system that must keep flowing.

Verify HA with a failure drill

This is the step people skip — and regret. Prove it:

# 1. Watch broker/queue health
kubectl get pods -l strimzi.io/cluster=prod -w

# 2. Kill a broker
kubectl delete pod prod-kafka-1

# 3. Confirm: producers/consumers keep working, the pod is recreated,
#    and no partition is under-replicated once it rejoins.
kubectl exec prod-kafka-0 -- bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 --describe --under-replicated-partitions

An empty under-replicated list after recovery means your replication and spread are correct. Do the equivalent for RabbitMQ (quorum queue stays available) and Pulsar (bookie loss, backlog drains).

Troubleshooting

  • "3 replicas" all landed on one node → anti-affinity/topology spread missing or unsatisfiable. Add podAntiAffinity on kubernetes.io/hostname and spread across zones.
  • Writes hang after one node dies → min.insync.replicas equals the replica count, so losing one breaks quorum. With RF 3, set it to 2.
  • Pods stuck Pending → no PVs / wrong StorageClass, or not enough nodes for anti-affinity. Check kubectl describe pod.
  • Data lost on restart → you used emptyDir instead of a PVC. Stateful messaging must use persistent volumes.
  • Drain takes everything down → add a PodDisruptionBudget (minAvailable: 2).

Where to go next

Our free course installs all three on Kubernetes, configures replication and anti-affinity, and runs the kill-a-node drill so you see HA work. Start with Kafka via Strimzi, then compare RabbitMQ and Pulsar hands-on in the lab.