Chapter 39 of 47 · More workload types

Singletons and disruptions

Usually more copies are better. But some jobs must be done by exactly one copy at a time: send the daily newsletter once, not three times. And the opposite worry: when you take a node down for maintenance, you want enough copies to keep running. This chapter covers both: singletons, and PodDisruptionBudgets.

When "only one" matters

Imagine the posts service gets a new feature: every night at 08:00 it emails "today's top posts" to all users. With 3 replicas, three Pods wake up at 08:00 and every user gets three emails. Other examples:

• A worker that reads events in strict order, where two readers would process things twice.
• A job that talks to an outside system allowing only one connection.
• A scheduler inside your app ("every 5 minutes, clean up old sessions").

There are two ways to guarantee one active copy: from the outside (Kubernetes runs only one) or from the inside (many run, but they agree on one boss).

Option 1: let Kubernetes keep it at one

The easiest fix: move the nightly email into its own small workload with one replica. Kubernetes keeps exactly one running and replaces it if it crashes. Your app code stays simple.

If the task runs on a schedule, an even better fit is a CronJob, which starts one Pod at 08:00 and lets it finish (chapter 36).

The replicas: 1 trap

"One replica" sounds like "never two". It is not, with a normal Deployment. Pick an update strategy and see:

08:00:00  newsletter-old   Running
08:00:01  newsletter-new   Starting      ← rollout begins (maxSurge)
08:00:05  newsletter-new   Running
          newsletter-old   Running       ← two copies at once!
08:00:06  newsletter-old   Terminating

spec:
  replicas: 1
  strategy:
    type: Recreate          # stop the old Pod first, then start the new one

kind: StatefulSet
spec:
  replicas: 1

Option 2: leader election inside the app

Sometimes you want several copies running, so a standby can take over in seconds, but only one may act. They hold an election using a shared lock. In Kubernetes that lock is a small object called a Lease. It is like a library book: whoever has it borrowed is the leader, and the loan must be renewed regularly or someone else can take it.

Kubernetes uses this itself: only one controller manager and one scheduler are active at a time, even when three control-plane nodes run them (chapter 44).

PodDisruptionBudget: keep enough running

Now the opposite problem. You have 3 posts Pods. An admin drains nodes to upgrade them. If all three posts Pods are evicted at once, your site is down. A PodDisruptionBudget (PDB) says how many may be missing at the same time because of planned work.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: posts-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels: { app: posts }

spec:
  maxUnavailable: 1
  selector:
    matchLabels: { app: posts }

Try it: drain a node

Three posts Pods, one per node. Pick a budget and drain a node. With minAvailable: 2, the drain waits until a replacement is Ready before evicting the next Pod. Then try minAvailable: 3.

$ kubectl get pdb
NAME        MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
posts-pdb   2               N/A               1                     3d

ALLOWED DISRUPTIONS is the number to watch: 0 means a drain will wait.

Which tool?

Recap

• Some work must be done by exactly one copy: scheduled emails, ordered processing, single-connection jobs.
• Outside: run one replica. With a Deployment use strategy: Recreate, or use a StatefulSet for a strict "at most one".
• Inside: run several copies and elect a leader with a Lease; standbys take over quickly.
• A PodDisruptionBudget limits how many Pods planned maintenance may take down at once.
• It does not help against crashes, and a too-strict PDB blocks node drains.

Quick check