Kubernetes moves pods around all the time. A node gets drained for a kernel patch, the cluster autoscaler removes an under-used node to save money, a managed node pool rolls to a new version. For a stateless web deployment that's a non-event — a replica dies here, another starts there. For a stateful, quorum-based system such as HashiCorp Consul, ClickHouse Keeper or a MinIO cluster, losing the wrong pod at the wrong moment can take the whole thing read-only, stall every client that depends on it, or — in the worst case — put your data at risk.

The small, frequently-forgotten object that prevents this is the PodDisruptionBudget (PDB). This guide explains exactly what a PDB does, why stateful workloads need one more than anything else on your cluster, and gives you copy-pasteable configurations for Consul, ClickHouse Keeper and MinIO — plus the traps that silently break node upgrades and stop your cluster from scaling down.

Two kinds of disruption — and only one that PDBs govern

Every pod eviction falls into one of two buckets:

  • Involuntary disruptions — a node's hardware fails, the kernel panics, the kubelet OOM-kills a pod, a network partition isolates a node. Nothing you configure prevents these. You survive them with replication and quorum, not with policy.
  • Voluntary disruptions — things Kubernetes or an operator does on purpose: kubectl drain, a node-pool upgrade, the cluster autoscaler (or Karpenter / GKE NAP) removing a node, a controller rebalancing pods. These all go through the Eviction API, and the Eviction API is the one thing that respects PodDisruptionBudgets.

That distinction is the whole point of a PDB: it lets you tell Kubernetes, "when you choose to move my pods, never take more than this many down at once." It does not protect you from a spot instance being reclaimed or a node dying — those are involuntary.

What a PodDisruptionBudget actually is

A PDB selects a set of pods by label and declares one of two limits:

  • minAvailable: N — at least N pods (an absolute number or a percentage) must stay available, or
  • maxUnavailable: N — at most N pods (number or percentage) may be unavailable at once.

When something calls the Eviction API, the API server checks every PDB whose selector matches the target pod. If evicting it would breach the budget, the eviction is refused (HTTP 429) and the drain blocks and retries until it is safe again. In practice this means a node drain evicts your cluster members one at a time, waiting for each replacement pod to become Ready before it touches the next one.

The simplest possible PDB:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: web

Why quorum systems are a different animal

Consul, ClickHouse Keeper, etcd, Kafka controllers and MinIO all share a property: they need a quorum — a majority of members alive and talking — to accept writes and stay consistent. Lose quorum and the system stops electing a leader, refuses writes, and typically goes read-only or fully unavailable until enough members return.

The math is simple and worth memorising. For a cluster of n members using Raft (Consul, Keeper, etcd):

  • quorum = ⌊n/2⌋ + 1
  • tolerable losses = n − quorum
Members (n) Quorum Can lose Safe maxUnavailable
3 2 1 1
5 3 2 2
7 4 3 3

Now picture a node-pool upgrade with no PDB. The upgrade cordons and drains two nodes at once, and by bad luck those two nodes host two of your three Consul servers. Both get evicted simultaneously. You are now at one of three servers — below quorum. Consul is leaderless, service discovery and KV reads/writes stall, and every application doing a Consul lookup feels it. A one-line PDB (maxUnavailable: 1) would have forced the drain to take those servers serially, keeping two of three alive the entire time.

Example 1 — HashiCorp Consul

Consul servers form a Raft cluster, deployed as a StatefulSet of 3 (small/medium) or 5 (large). The budget that keeps quorum during any voluntary disruption:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: consul-server
  namespace: consul
spec:
  maxUnavailable: 1        # 3 servers → quorum 2 → only one may be down at a time
  selector:
    matchLabels:
      app: consul
      component: server

The official Consul Helm chart ships this automatically — server.disruptionBudget.enabled=true computes maxUnavailable as ⌊(n-1)/2⌋ (1 for three servers, 2 for five). Do not disable it. If you run Consul from raw manifests, add the PDB yourself. Note the selector must match the server pods only — you do not want the same budget accidentally covering Consul clients or the mesh sidecars.

Example 2 — ClickHouse Keeper

ClickHouse Keeper is the Raft-based coordination service (a drop-in ZooKeeper replacement) that manages replication and distributed DDL for ClickHouse. It is almost always a StatefulSet of 3. The quorum rules are identical to Consul, so the budget is too:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: clickhouse-keeper
  namespace: clickhouse
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: clickhouse-keeper

Why it matters: if Keeper loses quorum, ClickHouse cannot coordinate replicated tables — INSERTs into ReplicatedMergeTree tables block, ON CLUSTER DDL hangs, and replicas drift. The Altinity Operator provisions Keeper for you, but you should still confirm a PDB exists. And remember Keeper is only half the picture: the ClickHouse server replicas themselves deserve their own PDB sized to your replication factor, so a drain never removes every replica of a shard at once:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: clickhouse-server
  namespace: clickhouse
spec:
  maxUnavailable: 1        # keep at least one replica of every shard serving
  selector:
    matchLabels:
      app: clickhouse
      component: server

Example 3 — MinIO

MinIO in distributed mode spreads objects across drives grouped into erasure sets, protecting data with erasure coding rather than full replication. Reads need a read quorum of drives and writes need a write quorum (quorum + parity). Crucially, taking one MinIO pod offline removes several drives at once (every drive on that node), so it is far easier to cross the write-quorum threshold than with a simple replicated system. The safe default is therefore strictly serial disruption:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: minio
  namespace: minio
spec:
  maxUnavailable: 1        # never take two nodes' worth of drives down together
  selector:
    matchLabels:
      app: minio

For a 4-node cluster this keeps writes healthy through any rolling operation. On larger clusters you can sometimes allow more, but only after you have done the parity math for your specific erasure-set layout — when in doubt, maxUnavailable: 1 is the answer that never corrupts a write.

How PDBs interact with the rest of your platform

  • kubectl drain and node-pool upgrades honour PDBs and evict serially. A managed upgrade will simply take longer, member by member — which is exactly what you want.
  • Cluster Autoscaler / Karpenter / GKE NAP will not scale a node down if doing so would violate a PDB. This is a double-edged sword: it protects your quorum, but a too-strict PDB (see below) can pin an empty-ish node open and quietly cost you money.
  • StatefulSet rolling updates are governed by the StatefulSet's own updateStrategy (OrderedReady, partition), not the PDB. A common misconception is that the PDB paces your kubectl rollout — it does not. The PDB is about node-level / eviction disruptions.
  • Spot / preemptible reclamation is involuntary — a PDB does not stop a spot node being taken back. You survive that with enough replicas, fast rescheduling, and graceful SIGTERM handling, not with a budget. Never assume a PDB makes a quorum safe to run entirely on spot.

The traps that bite people

  1. The deadlock budget. Setting minAvailable equal to the replica count (e.g. minAvailable: 3 on a 3-replica StatefulSet) allows zero disruptions — a node drain then hangs forever and your cluster upgrade stalls. Use maxUnavailable: 1 instead so exactly one member can move at a time.
  2. Singletons. A 1-replica StatefulSet with maxUnavailable: 0 can never be voluntarily moved, permanently blocking upgrades of its node. For true singletons, either accept brief downtime (maxUnavailable: 1) or run the workload HA.
  3. Selector mismatch. If the PDB's selector doesn't match the real pod labels, it silently protects nothing. Always verify ALLOWED DISRUPTIONS.
  4. Confusing PDBs with replication. A PDB is not a substitute for running enough replicas — it only paces voluntary moves. You still need 3/5 members to survive involuntary failures.
  5. Over-tight budgets throttling the autoscaler. A budget that never allows an eviction stops scale-down and inflates your bill.

Verify it actually works

kubectl get pdb -n consul
# NAME            MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
# consul-server   N/A             1                 1                     5m

ALLOWED DISRUPTIONS: 1 means one pod can be evicted right now without breaching quorum. If it reads 0 unexpectedly, a member is unhealthy or the budget is too tight — investigate before you upgrade anything. Then prove it on a test cluster with a real drain and watch the pods leave one at a time:

kubectl drain <node> --ignore-daemonsets --delete-emptydir-data
kubectl get pods -n consul -w    # exactly one server should go Terminating at a time

Spread members across nodes and zones — or the budget is a lie

A PDB quietly assumes your members live on different nodes. If two Consul servers happen to schedule onto the same node, that node cannot be drained at all without breaching the budget (the second eviction is refused), so your upgrade stalls — and far worse, an involuntary loss of that one node kills two members at once, straight through quorum. Always pair a quorum StatefulSet with anti-affinity and topology spread so each member lands on its own node (and ideally its own zone):

spec:
  template:
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels: { app: consul, component: server }
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - topologyKey: kubernetes.io/hostname
              labelSelector:
                matchLabels: { app: consul, component: server }

With members guaranteed one-per-node, maxUnavailable: 1 behaves exactly as intended and a single node failure only ever costs you one member. Add topology.kubernetes.io/zone as a second spread key and a whole-zone outage still leaves quorum intact.

Three members or five?

Three is the right default for most coordination clusters: it tolerates one failure, keeps Raft elections fast, and minimises write latency (every commit must reach a majority). Move to five only when you genuinely need to survive two simultaneous failures — for example, spanning three zones where you want to lose an entire zone and still tolerate a node failure. Five members mean more replication traffic and slightly slower commits, so don't reach for it by default. And never run an even number: four or six raise the quorum requirement without raising fault tolerance, trading availability for nothing. Keep member counts odd.

Percentage budgets, and when to use them

For large stateless fleets a percentage reads better and scales with the deployment (minAvailable: 75%). For small quorum clusters, stick to absolute numbers — a percentage over three members rounds in ways that surprise you, and the entire point here is precise control over a handful of critical pods.

Quick FAQ

  • Does a PDB slow down kubectl rollout restart? No — Deployment/StatefulSet rollouts follow the workload's own update strategy. PDBs only gate the eviction path: drains, autoscaler, node upgrades.
  • My drain says "cannot evict pod as it would violate the budget" — bug? No, that's the PDB doing its job: a member is currently unavailable, so no further eviction is allowed. Wait for it to become Ready (or fix why it isn't) and the drain proceeds on its own.
  • Do I need a PDB with only one replica? A PDB can't manufacture availability you don't have. Run the workload HA first; then the budget protects it.

The one-line takeaway

For every quorum-based StatefulSet you run: keep the member count odd, spread members one-per-node with anti-affinity, add a PDB of maxUnavailable: 1 (or ⌊(n-1)/2⌋ for larger clusters), and confirm ALLOWED DISRUPTIONS reads at least 1. It's a few lines of YAML standing between a routine node upgrade and a 2 a.m. outage.

Where to go next

Our Docker & Kubernetes course runs this as a hands-on lab — deploying Consul and MinIO as StatefulSets, adding PDBs, then draining nodes and watching the budget hold the line while quorum survives. It's the difference between knowing what a PDB is and trusting one in production.