Chapter 14 of 47 · Running well on Kubernetes

Managed lifecycle: starting and stopping politely

On your laptop, you decide when your app starts and stops. On Kubernetes, the platform decides, at any moment: during a deploy, when scaling down, when a node is drained. It always tells your container first. A well-behaved app listens to these messages and reacts. This chapter is about how.

The idea: a shop at closing time

A good shop does not lock the door on customers mid-purchase. It announces "we close in 10 minutes", stops letting new people in, serves the people already inside, then turns off the lights. If someone is still dawdling at the deadline, security walks them out.

Health probes (chapter 13) let Kubernetes ask your app how it is doing. Lifecycle events are the opposite direction: Kubernetes tells your app what is about to happen. Reacting is optional, but apps that do react deploy and scale without errors.

When Kubernetes stops a container

• A rolling update replaces old Pods with new ones (chapter 10).
• You scale down, or kubectl delete pod.
• A node is drained for maintenance or an upgrade, or the cluster autoscaler removes it.
• A failed liveness probe makes Kubernetes restart the container.
• The node runs short of memory and evicts Pods (chapter 22).

This happens all the time. Each time, the same polite sequence runs.

SIGTERM: "please finish up"

A signal is a tiny message the operating system delivers to a process. SIGTERM means "please shut down". Your app should: stop accepting new work, finish what it is doing, close database connections, then exit. In Node:

// posts/index.js
const server = app.listen(4000, () => console.log('Posts listening on 4000'));

process.on('SIGTERM', () => {
  console.log('SIGTERM received, shutting down');
  server.close(async () => {          // stop new connections, wait for open requests
    await db.end();                   // close the database pool
    process.exit(0);                  // 0 = clean exit
  });
});

CMD npm start                 # shell form: PID 1 is /bin/sh, then npm, then node
# or
CMD ["npm", "start"]          # PID 1 is npm, which does not forward SIGTERM reliably

CMD ["node", "index.js"]      # exec form: PID 1 is node itself

SIGKILL: "time's up"

Kubernetes does not wait forever. After SIGTERM it waits a grace period, 30 seconds by default. If the process is still running, it sends SIGKILL. That one cannot be caught or ignored: the process is stopped instantly, mid-request. The container's exit code is then 137 (128 + 9, the number of SIGKILL).

spec:
  terminationGracePeriodSeconds: 60     # per Pod; default 30
  containers:
    - name: posts
      image: micro-posts:1.0

• Set it a little longer than your slowest request or cleanup.
• It is not a guarantee: kubectl delete pod --grace-period=0 --force skips it. Design the app so a sudden kill does not corrupt anything.
• Best of all: make startup and shutdown fast. Quick to start, quick to stop is what makes Pods truly disposable (chapter 7).

Try the shutdown timeline

What actually happens between "delete this Pod" and the container being gone? Change the settings and read the verdict below the timeline. One detail surprises everyone: removing the Pod from the Service and sending SIGTERM happen at the same time, and the Service update takes a few seconds to reach every node. Meanwhile, new requests still arrive.

Find the setting where everything is green: the app handles SIGTERM, a short preStop sleep covers the Service delay, and the grace period is long enough for both.

The preStop hook

A hook is a command or HTTP call Kubernetes runs at a specific moment in a container's life. preStop runs before SIGTERM is sent, and SIGTERM waits until it finishes.

containers:
  - name: posts
    image: micro-posts:1.0
    lifecycle:
      preStop:
        exec:
          command: ["sleep", "5"]     # needs a 'sleep' binary in the image

lifecycle:
  preStop:
    httpGet:
      path: /shutdown
      port: 4000

containers:
  - name: proxy
    image: nginx:1.27
    lifecycle:
      preStop:
        exec:
          command: ["nginx", "-s", "quit"]   # nginx's own "finish requests, then stop"

• Its time counts toward the grace period: preStop + shutdown must fit inside it.
• If preStop fails or hangs, the container is still stopped at the deadline. It cannot prevent shutdown.

The postStart hook

postStart runs right after the container is created, at the same time as your app starts, not before it.

lifecycle:
  postStart:
    exec:
      command: ["sh", "-c", "echo started at $(date) >> /tmp/lifecycle.log"]

Honest advice: most apps never need postStart. Warm-up code usually belongs in the app's own startup, and "wait for X before starting" belongs in an init container.

Hooks vs init containers

An init container is a container that runs to completion before the main containers start. If there are several, they run one after another. Example: the posts service should not start until the event bus answers DNS:

spec:
  initContainers:
    - name: wait-for-event-bus
      image: busybox:1.36
      command: ["sh", "-c", "until nslookup event-bus-srv; do echo waiting; sleep 2; done"]
  containers:
    - name: posts
      image: micro-posts:1.0

Advanced: wrapping the entrypoint

Tools that build Pods automatically, such as CI pipelines like Tekton, sometimes need even more control: start containers in a strict order while sidecars keep running. Init containers cannot do that, because the main containers only start after all init containers have finished. The trick is to wrap each container's start command:

1. An init container copies a small "supervisor" program into a shared emptyDir volume.
2. The main container's command is replaced by that supervisor, and the original command becomes its arguments.
3. The supervisor starts the real program when it is told to, and handles signals, ordering and results.

spec:
  volumes:
    - name: tools
      emptyDir: {}
  initContainers:
    - name: install-runner
      image: example/runner:1.0
      command: ["cp", "/runner", "/tools/runner"]
      volumeMounts: [{ name: tools, mountPath: /tools }]
  containers:
    - name: build
      image: node:22
      command: ["/tools/runner"]                  # the wrapper starts first
      args: ["npm", "run", "build"]               # the original command, now its arguments
      volumeMounts: [{ name: tools, mountPath: /tools }]

You will rarely write this yourself; it is good to recognise when you read the Pods a CI tool creates.

From least to most control, and effort: signal handling in your app → lifecycle hooks → init containers → entrypoint wrapping. Start with the first; it is enough for most services.

Checklist for the blog app

1. Dockerfiles use CMD ["node", "index.js"], not npm start.
2. Every service handles SIGTERM: close the server, finish requests, close connections, exit 0.
3. A preStop sleep of about 5 seconds, so no requests hit a closing Pod.
4. terminationGracePeriodSeconds covers preStop + the slowest request.
5. Optional: an init container that waits for event-bus-srv.

Recap

• Kubernetes, not you, decides when containers start and stop, and it announces it.
• SIGTERM: finish up and exit. SIGKILL: forced, after the grace period (30s default).
• Start Node with the exec form so SIGTERM actually reaches it.
• preStop runs before SIGTERM; its classic job is a short sleep while traffic drains.
• postStart runs alongside startup with weak guarantees; init containers are the reliable way to run setup first.

Quick check