Chapter 19 of 47 · Configuration and security, part 2

Secrets: handling sensitive data

A Secret works almost exactly like a ConfigMap, but is meant for passwords, API keys and certificates. Kubernetes treats it more carefully. It is not magic encryption, though, and knowing exactly what it does and does not protect is the whole point of this chapter.

Secret vs ConfigMap

ConfigMapSecret
ForSettings: URLs, log levels, config filesPasswords, tokens, keys, certificates
Values stored asPlain textBase64 (encoded, not encrypted)
Used in Pods asenv vars or filesenv vars or files, the same way
Extra care by KubernetesNoneSent only to nodes that run a Pod using it; mounted in memory (tmpfs), never written to the node's disk; can be encrypted at rest in etcd; separate access rules
Size limit1 MiB1 MiB

Base64 is not encryption

Secret values in YAML look scrambled, like aHVudGVyMg==. That is base64: a way to write any bytes as plain letters. There is no key, so anyone can reverse it. Try it, and try the classic newline mistake:

So never commit a Secret YAML to Git thinking it is protected. Base64 in a public repo is the same as the password in a public repo.

Writing a Secret

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
type: Opaque
data:
  DB_USER: cG9zdHM=              # "posts"
stringData:
  DB_PASSWORD: hunter2          # plain text, encoded for you

stringData

Write values as plain text and Kubernetes encodes them on save. Easier and fewer mistakes. When you read the Secret back, everything appears under data, encoded.

Using it in a Pod

Exactly like a ConfigMap, with secret in place of configMap:

containers:
  - name: posts
    image: micro-posts:1.0
    env:
      - name: DB_PASSWORD
        valueFrom:
          secretKeyRef:
            name: db-credentials
            key: DB_PASSWORD
    # or every key at once:
    envFrom:
      - secretRef:
          name: db-credentials

Your app reads process.env.DB_PASSWORD, already decoded. Simple, and fine for many apps.

Downside: env vars leak easily. They show up in crash reports, in kubectl describe-style debugging, in child processes, and in any "print all config" log line someone adds. They also never change until the Pod restarts.

Secret types

TypeHoldsUsed by
OpaqueAnything (default)Your app: DB passwords, API keys
kubernetes.io/dockerconfigjsonRegistry loginThe kubelet, to pull images from a private registry (imagePullSecrets)
kubernetes.io/tlstls.crt + tls.keyIngress, for HTTPS certificates
kubernetes.io/basic-authusername + passwordTools that expect that shape
kubernetes.io/ssh-authssh-privatekeye.g. cloning private Git repos
kubernetes.io/service-account-tokenAn API tokenLegacy; modern clusters give Pods short-lived tokens automatically

Pulling your micro-posts image from a private registry, for example:

$ kubectl create secret docker-registry regcred \
    --docker-server=ghcr.io --docker-username=me --docker-password=$TOKEN

# in the Pod spec
spec:
  imagePullSecrets:
    - name: regcred
  containers:
    - name: posts
      image: ghcr.io/me/micro-posts:1.0

Creating with kubectl

  1. $ kubectl create secret generic db-credentials \
        --from-literal=DB_USER=posts \
        --from-literal=DB_PASSWORD='hunter2'
    secret/db-credentials created

    The easiest way; encoding is done for you, no newline bug. Note: the password ends up in your shell history. --from-file avoids that.

  2. $ kubectl get secret db-credentials
    NAME             TYPE     DATA   AGE
    db-credentials   Opaque   2      5s
    $ kubectl describe secret db-credentials
    Data
    ====
    DB_PASSWORD:  7 bytes
    DB_USER:      5 bytes

    get and describe show only key names and sizes, not values.

  3. $ kubectl get secret db-credentials -o jsonpath='{.data.DB_PASSWORD}' | base64 -d
    hunter2

    …but anyone allowed to get Secrets can read them. That permission is what you must guard.

  4. $ kubectl create secret tls posts-tls --cert=tls.crt --key=tls.key
    secret/posts-tls created

    A TLS Secret for HTTPS, used by Ingress later.

What actually keeps it safe

  1. Access control (RBAC). Only people and apps that truly need it may get or list Secrets. Note: anyone who can create a Pod in a namespace can mount any Secret in that namespace.
  2. Encryption at rest. Turn on etcd encryption so a stolen etcd backup does not reveal every password. Managed clusters (EKS, GKE, AKS) offer this, often with a cloud key service (KMS).
  3. Keep plain Secrets out of Git. Use a tool built for this: Sealed Secrets (encrypt for the cluster, safe to commit), SOPS, or the External Secrets Operator pulling from AWS Secrets Manager, Vault, etc.
  4. Prefer files over env vars for highly sensitive values, and mount them read-only.
  5. Separate namespaces for separate teams or apps, so a Secret is only visible where it belongs.

Safe or not?

Recap

Secrets: common questions

Are Kubernetes Secrets encrypted?

Not by default. Values are only base64-encoded. Protect them with RBAC, encryption at rest in etcd, and by keeping plain Secret files out of Git.