Chapter 22 of 47 · Resource management, part 1

CPU and memory: requests, limits and QoS

Many Pods share each node. Without rules, one hungry app can slow down or crash its neighbours. Every container can declare two numbers per resource: a request (what it needs, reserved for it) and a limit (the most it may use).

Request vs limit

Think of a shared flat. The request is the room you rent: it is yours, nobody else can rent it. The limit is the house rule: you may use the living room too, but never more than this much.

	Request	Limit
Means	"I need at least this much"	"Never let me use more than this"
Used by	The scheduler, to pick a node with room	The node (kernel), while the container runs
If exceeded	Fine, as long as the node has spare capacity	CPU: slowed down. Memory: container killed.

Units

CPU

1      = 1 CPU core (or vCPU)
1000m  = 1 core  ("m" = millicores)
500m   = half a core
100m   = a tenth of a core

Memory

128Mi  = 128 mebibytes
1Gi    = 1024Mi
128M   = 128 megabytes (≈122Mi)
128m   = 0.128 bytes (!)

Use Mi and Gi. Lower-case m on memory is a classic typo.

The YAML

Set per container, under resources:

containers:
  - name: posts
    image: micro-posts:1.0
    resources:
      requests:
        cpu: 250m          # reserve a quarter of a core
        memory: 128Mi      # reserve 128Mi
      limits:
        cpu: 500m          # never more than half a core
        memory: 256Mi      # killed if it goes above 256Mi

A Pod's total is the sum of its containers. A Deployment applies these to every replica.

Requests decide placement

The scheduler only looks at requests, never at real usage. A node is "full" when the requests of its Pods add up to its capacity, even if they are all idle. Create Pods and see:

$ kubectl describe pod posts-5
Events:
  Warning  FailedScheduling  0/2 nodes are available: 2 Insufficient cpu.

A Pod stuck in Pending with "Insufficient cpu/memory" means: requests are too big for any node's free room. Lower the requests, delete other Pods, or add nodes.

Limits: throttled or killed

CPU and memory behave very differently at the limit. Move the sliders past the black line:

CPU is "compressible"

Over the limit, the container just gets less CPU time. Requests take longer, but nothing crashes. Too-low CPU limits are a common hidden cause of slow responses.

Memory is not

Memory cannot be taken back gradually, so the kernel kills the process: OOMKilled, exit code 137. Node.js tip: keep the V8 heap under the limit, e.g. --max-old-space-size about 75% of the memory limit.

Quality of Service classes

When a node runs out of memory, the kubelet must evict some Pods. Kubernetes sorts every Pod into a QoS class based on its requests and limits, and evicts the least protected ones first. Try the combinations:

Class	When	Good for
Guaranteed	Every container has CPU and memory limits, and requests = limits	Databases, critical services
Burstable	At least one request or limit set, but not Guaranteed	Most web apps and APIs
BestEffort	No requests or limits at all	Throwaway jobs you do not mind losing

$ kubectl get pod posts-4kq2p -o jsonpath='{.status.qosClass}'
Burstable

Seeing it with kubectl

$ kubectl top pods
NAME                    CPU(cores)   MEMORY(bytes)
posts-depl-4kq2p        12m          74Mi
comments-depl-x7m2d     8m           61Mi
$ kubectl top nodes
NAME     CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
node-1   420m         21%    2380Mi          61%

Actual usage right now. Needs the metrics-server add-on (on minikube: minikube addons enable metrics-server). Use it to choose sensible requests.

$ kubectl describe node node-1
...
Allocated resources:
  Resource   Requests      Limits
  --------   --------      ------
  cpu        1750m (87%)   3 (150%)
  memory     3Gi (78%)     5Gi (128%)

What is reserved on the node. Limits may add up to more than 100% ("overcommit"): it works as long as not everyone bursts at once.

Choosing numbers

Run the app, put some load on it, and watch kubectl top.
CPU request ≈ normal usage. Memory request ≈ normal usage plus a margin.
Memory limit: always set it, a bit above peak usage. Many teams set memory request = limit to avoid surprises.
CPU limit: optional and debated. Without one, the app may use spare CPU; with one that is too low, it is throttled. If you set it, give generous headroom.
Revisit after real traffic. Numbers copied from a blog post are a guess.

Recap

Requests are reserved and decide which node a Pod lands on.
Limits cap usage: CPU is throttled, memory is OOMKilled.
CPU in cores or m, memory in Mi/Gi.
QoS classes (Guaranteed, Burstable, BestEffort) decide who is evicted first when a node runs out of memory.
Measure with kubectl top, then set the numbers.

Quick check

A Pod restarts every few minutes. kubectl describe shows Reason: OOMKilled, Exit Code: 137. What do you change?

Show answer

It hit its memory limit. Raise limits.memory (and probably requests.memory), or fix a memory leak in the app. Raising the CPU limit would not help.

Your nodes are only 20% busy in kubectl top, but new Pods stay Pending with "Insufficient cpu". How?

Show answer

The scheduler counts requests, not usage. Existing Pods requested far more CPU than they use. Lower their requests.

Resource requests and limits: common questions

What is the difference between requests and limits?

A request is reserved for the container and used by the scheduler to place the Pod. A limit is the maximum it may use: over the CPU limit it is throttled, over the memory limit it is killed.

What does OOMKilled exit code 137 mean?

The container used more memory than its limit and the kernel killed it. Raise the memory limit or fix the memory use.