Chapter 38 of 47 · More workload types

StatefulSets: apps with identity and data

A Deployment treats its Pods like identical cups from a stack: take any one, throw away a broken one, grab a new one. That works for posts and comments. A database is different. Each copy has its own data and its own job (primary, replica), so each one needs a fixed name and its own disk. A StatefulSet gives Pods exactly that.

Why a Deployment is not enough

Try running PostgreSQL with 3 replicas in a Deployment and three things go wrong:

The four promises

1. Stable name: Pods are db-0, db-1, db-2. A replacement for db-1 is called db-1 again.
2. Stable network address: with a headless Service, each Pod gets its own DNS name, db-1.db-headless (chapter 26).
3. Its own disk, forever: each Pod gets its own PVC, data-db-1, and always gets that same one back, even on another node.
4. Order: started 0, 1, 2 one after another; stopped 2, 1, 0.

Try it: scale, delete, scale again

Scale up and watch the Pods start one by one. Delete a Pod and see it come back with the same name and disk. Scale down, then up again, and watch the old disks get reused. Then switch to Parallel and compare.

The YAML

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: db
spec:
  serviceName: db-headless
  replicas: 3
  podManagementPolicy: OrderedReady
  selector:
    matchLabels: { app: db }
  template:
    metadata:
      labels: { app: db }
    spec:
      containers:
        - name: postgres
          image: postgres:16
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: [ReadWriteOnce]
        resources:
          requests:
            storage: 10Gi
  persistentVolumeClaimRetentionPolicy:
    whenScaled: Retain
    whenDeleted: Retain

$ kubectl get pods,pvc -l app=db
NAME       READY   STATUS
pod/db-0   1/1     Running
pod/db-1   1/1     Running
pod/db-2   1/1     Running

NAME                              STATUS   CAPACITY
persistentvolumeclaim/data-db-0   Bound    10Gi
persistentvolumeclaim/data-db-1   Bound    10Gi
persistentvolumeclaim/data-db-2   Bound    10Gi

Order of start and stop

Updates and staged rollouts

Change the image and apply. A StatefulSet updates from the highest number down: db-2, then db-1, then db-0, each only after the previous is Ready again. That way the primary is updated last.

To test a new version on just a few Pods first, use a partition: only Pods with a number greater than or equal to the partition get the new version.

updateStrategy:
  type: RollingUpdate
  rollingUpdate:
    partition: 2

db-0  postgres:16      (old)
db-1  postgres:16      (old)
db-2  postgres:16.4    ← only this one

db-0  postgres:16      (old)
db-1  postgres:16.4    ← new
db-2  postgres:16.4    ← new

db-0  postgres:16.4
db-1  postgres:16.4
db-2  postgres:16.4

The other strategy, OnDelete, waits for you to delete each Pod by hand, like for DaemonSets (chapter 37).

Scaling down safely

• Disks stay behind (with the default Retain). Scale back up and db-2 gets data-db-2 again, data included. Disks you no longer want must be deleted by hand: kubectl delete pvc data-db-2.
• The app must agree. Kubernetes does not move data. Before removing db-2, the database must stop counting on it (leave the cluster, hand over its data). Otherwise you may lose data or break a quorum.
• A stuck node blocks you. If the node running db-1 disappears, Kubernetes will not start a second db-1 elsewhere until it is sure the first one is gone. Two db-1 Pods writing the same data would be worse than waiting.

What a StatefulSet does not do

A StatefulSet gives names, disks and order. It knows nothing about databases. It does not:

• copy data from the primary to replicas,
• pick a new primary when db-0 dies,
• take backups, or resize the cluster safely.

The database does the first two itself, and the rest is your job, or an operator's: a controller that knows how to run one specific database (chapter 46). In practice, most people install PostgreSQL, MySQL or Kafka with a Helm chart or an operator, which creates the StatefulSet for them. Or they use a managed cloud database and keep the cluster stateless.

StatefulSet or Deployment?

Recap

• StatefulSet Pods have fixed names (db-0, db-1…) that survive replacement.
• A headless Service gives each Pod a fixed DNS name.
• volumeClaimTemplates give each Pod its own PVC, which follows it and is kept on scale-down.
• Pods start in order 0, 1, 2 and stop in reverse; Parallel turns that off.
• Updates go from the highest number down; a partition lets you roll out step by step.
• Replication, failover and backups are the database's or an operator's job, not the StatefulSet's.

Quick check