Chapter 40 of 47 · Scheduling and scaling

Scheduling: where Pods run

When you create a Pod, nobody tells it which machine to go to. The scheduler picks a node for it, a bit like a hotel receptionist picking a room. Most of the time its choice is fine. This chapter shows how it decides, and the knobs you can turn when you need a say: "only on SSD nodes", "never two copies on the same machine", "keep normal apps off the GPU nodes".

How the scheduler decides

The scheduler is a control-plane program (chapter 44). It watches for Pods that have no node yet, and for each one it does two rounds:

• Filter is strict: not enough free requested CPU or memory (chapter 22), a missing label, an untolerated taint → that node is out.
• Score is a preference: spread Pods evenly, prefer nodes with room left, prefer the image already downloaded, and your own "preferred" rules.
• If no node passes the filter, the Pod stays Pending and the scheduler keeps retrying.

Try it: where can this Pod go?

Four nodes, each with labels and maybe a taint. Add rules to the Pod and watch nodes drop out:

Everything below is just a different way of feeding the filter or the score step.

nodeSelector: the simple filter

Nodes carry labels, just like Pods. Some come for free (kubernetes.io/hostname, topology.kubernetes.io/zone, kubernetes.io/arch); you can add your own:

$ kubectl label node node-1 disktype=ssd
$ kubectl get nodes -L disktype,topology.kubernetes.io/zone
NAME       STATUS   DISKTYPE   ZONE
node-1     Ready    ssd        zone-a
node-2     Ready    hdd        zone-b

Then ask for it in the Pod spec. Only nodes with all these labels pass:

spec:
  nodeSelector:
    disktype: ssd
  containers:
    - name: query
      image: micro-query:1.0

Simple, and enough for most cases. Its limit: it can only say "must equal". No "or", no "prefer".

Node affinity: must vs prefer

Node affinity is nodeSelector with more words. Its two flavours have long but honest names:

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions:
            - key: topology.kubernetes.io/zone
              operator: In
              values: [zone-a, zone-b]       # either zone is fine
            - key: kubernetes.io/arch
              operator: NotIn
              values: [arm64]

affinity:
  nodeAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 80                            # 1–100: how much you care
        preference:
          matchExpressions:
            - key: disktype
              operator: In
              values: [ssd]

"IgnoredDuringExecution" means: the rule is only checked when the Pod is placed. If someone later removes the label from the node, the running Pod stays where it is.

Pod affinity and anti-affinity

Sometimes the question is not "which node?" but "next to whom?". These rules look at the other Pods already on a node.

affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchLabels: { app: posts }
        topologyKey: kubernetes.io/hostname

affinity:
  podAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 50
        podAffinityTerm:
          labelSelector:
            matchLabels: { app: redis }
          topologyKey: kubernetes.io/hostname

topologyKey decides what "the same place" means: kubernetes.io/hostname = same node, topology.kubernetes.io/zone = same zone (data centre). Same rule, different scale.

Topology spread: share them out fairly

Anti-affinity only knows "together" or "apart". Topology spread constraints say "keep the numbers balanced": for example 6 comments Pods across 3 zones → 2, 2, 2.

spec:
  topologySpreadConstraints:
    - maxSkew: 1                               # zones may differ by at most 1 Pod
      topologyKey: topology.kubernetes.io/zone
      whenUnsatisfiable: ScheduleAnyway        # or DoNotSchedule (hard)
      labelSelector:
        matchLabels: { app: comments }

Taints and tolerations: "keep out" signs

Everything so far is the Pod choosing nodes. A taint is the opposite: the node pushes Pods away. Only Pods with a matching toleration ("I'm OK with that") may come in. Think of a "staff only" sign: everyone stays out except people with a badge.

# the node says: keep out unless you need a GPU
$ kubectl taint nodes gpu-node gpu=true:NoSchedule

# the Pod carries the badge
spec:
  tolerations:
    - key: gpu
      operator: Equal
      value: "true"
      effect: NoSchedule

You see taints every day without noticing: control-plane nodes carry node-role.kubernetes.io/control-plane:NoSchedule, which is why your apps never land there.

$ kubectl describe node gpu-node | grep Taints
Taints:             gpu=true:NoSchedule

Priority and preemption: who goes first

When the cluster is full, some Pods matter more than others. A PriorityClass gives Pods a rank:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: business-critical
value: 100000              # higher = more important
description: "Core blog APIs: posts and comments"
---
spec:
  priorityClassName: business-critical
  containers:
    - name: posts
      image: micro-posts:1.0

Use priorities sparingly: if everything is "critical", nothing is. Kubernetes' own components use the built-in classes system-cluster-critical and system-node-critical. Priority is separate from QoS (chapter 22): priority decides who gets scheduled, QoS who gets evicted when a node runs out of memory.

Which tool when?

Placement is only decided once. As the cluster changes, Pods can end up badly spread. The optional descheduler add-on evicts such Pods now and then so the scheduler can place them again, better.

Recap

• The scheduler filters out nodes that cannot work, then scores the rest.
• nodeSelector: simple "node must have this label".
• Node affinity: required (filter) or preferred (score), with In/NotIn/Exists.
• Pod affinity / anti-affinity: near or away from other Pods; topologyKey sets node or zone.
• Topology spread: keep replica counts balanced across nodes or zones.
• Taints keep Pods out; tolerations let specific Pods in.
• PriorityClass + preemption: important Pods can push less important ones out when the cluster is full.

Quick check