Chapter 41 of 47 · Scheduling and scaling

Autoscaling: HPA, VPA and cluster autoscaling

Traffic is never flat. Your blog is quiet at night and busy when a post goes viral. Paying for peak capacity all day is wasteful; running too little at peak means slow pages. Kubernetes can grow and shrink things automatically, at three levels: more Pods, bigger Pods, or more machines.

Three ways to grow

Think of a restaurant on a busy evening. You can call in more waiters (horizontal), give each waiter a bigger tray (vertical), or open the terrace for more tables (more room).

Manual scaling first

Before automating, remember the manual way (chapter 10):

$ kubectl scale deployment comments-depl --replicas=5     # imperative
# or change replicas: 5 in the YAML and kubectl apply     # declarative

Autoscaling simply does this for you, all day, based on numbers it measures.

HPA: more Pods when busy

The HPA checks every 15 seconds or so: "how busy are the Pods, compared to what I want?" and changes replicas on the Deployment. Click the parts:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: comments
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: comments-depl
  minReplicas: 2
  maxReplicas: 8
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

It needs the metrics-server add-on to read CPU numbers (the same one behind kubectl top). Quick start without YAML:

$ kubectl autoscale deployment comments-depl --cpu-percent=70 --min=2 --max=8
$ kubectl get hpa
NAME       REFERENCE                  TARGETS   MINPODS   MAXPODS   REPLICAS
comments   Deployment/comments-depl   92%/70%   2         8         3

Try the HPA maths

The rule is simple: new replicas = current replicas × (current use ÷ target), rounded up, then kept between min and max. Move the sliders:

Calm, not jumpy

CPU numbers wiggle. An autoscaler that reacts to every wiggle would add and remove Pods all the time ("flapping"). The HPA protects you in two ways:

• A 10% tolerance: 74% against a 70% target is "close enough", no change.
• A stabilization window: scale up fast, scale down only after load has stayed low for 5 minutes (default).

spec:
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0        # react to spikes at once
      policies:
        - type: Pods
          value: 4                         # but add at most 4 Pods
          periodSeconds: 60                #   per minute
    scaleDown:
      stabilizationWindowSeconds: 300      # wait 5 minutes of calm before removing Pods

Remember: a new Pod needs time to start and become ready (chapter 13). Autoscaling smooths out the day; it cannot absorb a spike that arrives in one second. Keep some headroom with minReplicas.

Scaling on other numbers

CPU is not always the best signal. A comments service waiting on a database barely uses CPU, but its queue grows. autoscaling/v2 can scale on several kinds of metrics:

- type: Resource
  resource:
    name: memory
    target: { type: Utilization, averageUtilization: 80 }

- type: Pods
  pods:
    metric:
      name: http_requests_per_second
    target: { type: AverageValue, averageValue: "100" }

- type: External
  external:
    metric:
      name: queue_messages_ready
      selector: { matchLabels: { queue: comment-events } }
    target: { type: AverageValue, averageValue: "30" }

With several metrics, the HPA computes a replica count for each and takes the highest.

KEDA and scale to zero

The HPA never goes below 1 replica. But a worker that processes moderation events has nothing to do at 3 a.m. KEDA (Kubernetes Event-Driven Autoscaling) is an add-on that watches event sources (queues, Kafka topics, databases, cron schedules…) and can scale a Deployment to zero and back:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: moderation-worker
spec:
  scaleTargetRef:
    name: moderation-depl
  minReplicaCount: 0                 # nothing running when the queue is empty
  maxReplicaCount: 10
  triggers:
    - type: rabbitmq
      metadata:
        queueName: comments-to-moderate
        value: "20"                  # one Pod per 20 waiting messages

Under the hood KEDA feeds an HPA for the 1-to-N part and handles 0-to-1 itself. Knative offers similar scale-to-zero for HTTP services: requests wait briefly while the first Pod starts.

VPA: bigger Pods instead of more

Getting requests right is hard (chapter 22). The Vertical Pod Autoscaler (an add-on) watches real usage over days and recommends, or sets, better requests.

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: query
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: query-depl
  updatePolicy:
    updateMode: "Off"          # only recommend, change nothing

Cluster autoscaling: more nodes

The HPA can ask for 8 Pods, but if the nodes are full, the new Pods stay Pending. The Cluster Autoscaler (or Karpenter on AWS and others) watches for exactly that:

Cluster autoscaling reacts to Pending Pods and requests, not to CPU usage. Correct requests matter here too.

All three together

1. Set honest requests, using VPA in Off mode as an advisor.
2. Let the HPA add and remove Pods as traffic moves, with sensible min/max.
3. Let the cluster autoscaler add nodes when Pods are Pending and remove idle ones.
4. Use KEDA for queue workers that can sleep at zero.

Recap

• HPA changes the number of Pods; VPA changes their size; the cluster autoscaler changes the number of nodes.
• HPA CPU targets are a percentage of the request, so requests must be set; metrics-server must be installed.
• Scale up fast, scale down slowly; the defaults already do this.
• Custom and external metrics (requests per second, queue length) often beat CPU.
• KEDA can scale workers to zero; mind the cold start.
• Do not let HPA and VPA both control CPU.

Quick check