Chapter 45 of 47 · Under the hood

Controllers: the reconcile loop

Almost every feature you have used so far, from Deployments and ReplicaSets to Services and Jobs, is made by a controller: a small program that keeps looking at what you asked for, looks at what really exists, and fixes the difference. Once you see this loop, all of Kubernetes makes sense, and you can even write your own.

A thermostat for your cluster

A thermostat does not care why the room is cold: someone opened a window, the sun went down, whatever. It only compares two numbers, "wanted 21°" and "now 18°", and turns the heating on until they match. Then it keeps watching.

A Kubernetes controller is exactly that, for objects. You write the wanted state in YAML; the controller makes reality match, forever.

Observe, compare, act

You saw this loop in chapter 4 as "desired vs current". Try it once more, thinking "controller" this time: crash Pods, change the number, and watch the controller close the gap.

Controllers you already use

The controller manager (chapter 44) is a bundle of dozens of these loops. A few you know:

Even the scheduler and the kubelet follow the same idea: watch for Pods, and make the world match them.

Watching, not polling

Asking the API server "anything new?" every second would be wasteful. Instead a controller opens a watch: a long-lived connection where the API server pushes every change as it happens.

$ kubectl get configmaps -n blog --watch
NAME          DATA   AGE
app-config    3      2d
app-config    3      2d      ← someone just edited it
app-config    4      2d      ← and again

Real controllers also keep a local copy (a cache) of everything they watch, so they can compare without asking the API server again and again.

Look at the state, not the event

This is the one idea that makes controllers robust. Compare two ways to handle "a Pod was deleted":

on "Pod deleted":   create 1 Pod
on "Pod created":   do nothing

on any change (or every few minutes anyway):
  wanted = spec.replicas          # 3
  have   = count(matching Pods)   # whatever is true right now
  create or delete (wanted - have) Pods

Owners and garbage collection

When a controller creates an object, it writes its own name into the new object's ownerReferences. That is how Kubernetes knows what belongs to what:

$ kubectl get pod posts-7d9f8c6b5d-4kq2p -o yaml
metadata:
  ownerReferences:
    - apiVersion: apps/v1
      kind: ReplicaSet
      name: posts-7d9f8c6b5d
      controller: true

Write a tiny controller

A controller is just a program that watches and acts. Here is a real, working one in a few lines of shell. The goal: when a ConfigMap labelled restart-on-change=true changes, restart every Deployment that uses it. This fixes the "env vars never update" problem from chapter 18.

#!/bin/sh
# config-watcher: restart Deployments when their ConfigMap changes
kubectl get configmaps -n blog -l restart-on-change=true \
  --watch-only -o jsonpath='{.metadata.name}{"\n"}' |
while read -r cm; do
  for d in $(kubectl get deploy -n blog \
      -l uses-config="$cm" -o name); do
    kubectl rollout restart "$d" -n blog
    echo "ConfigMap $cm changed → restarted $d"
  done
done

To run it in the cluster: put the script in a container, give it a ServiceAccount with a small Role (watch ConfigMaps; get and patch Deployments, see chapter 43), and run one replica as a Deployment. Real controllers are usually written in Go (client-go, controller-runtime), but libraries exist for most languages, including Node (@kubernetes/client-node) and Python.

Rules for good controllers

Which of these does a well-behaved controller do?

Recap

• A controller is a loop: observe, compare with the desired state, act. Then repeat.
• Deployments, ReplicaSets, Services, Jobs and namespaces all work through controllers.
• Controllers watch the API server instead of polling, and keep a local cache.
• They look at the whole current state, not single events, so missed events do no harm.
• Owner references link children to parents; the garbage collector removes orphans.
• You can write your own: a watch, a comparison, an idempotent action, and a small RBAC role.

Quick check