The Complete Kubernetes Guide

This is part one of three. It covers everything you need to run real applications on Kubernetes, not a tour of the logo. By the end you can start a cluster on your laptop, deploy an application from a YAML file you wrote yourself, give it a stable address, configure it without rebuilding the image, update it with zero downtime, roll it back when the update is bad, and read the error messages well enough to fix the common failures on your own. Mid-level and Senior take the same topics further; nothing here is thrown away.

The guide is written against Kubernetes 1.37 (the current minor, released on 26 August 2026; the latest patch is v1.37.1). Everything a beginner touches has been stable for years, so if your cluster is a version or two older, the commands still work. Where something changed recently, the guide says so.

Each section ends with a Try it task. Do them as you go. Kubernetes only makes sense once you have deleted a pod with your own hands and watched another one appear in its place.

What Kubernetes is, and the problem it solves

Kubernetes is a system for running containers across a group of machines and keeping them running. You tell it what you want ("three copies of this image, reachable on port 80, with 256 MiB of memory each"), and it decides which machine each copy runs on, starts them, watches them, and replaces any copy that dies. You describe the result; Kubernetes does the work of getting there and staying there.

To see why that matters, look at what came before. The Docker guide ends with a project running under Docker Compose on one machine. That is a fine place to be for development. Now imagine taking it to production. You rent three servers so that one failure does not take the service down. You SSH into each and run docker run. Then the questions start.

• One server reboots at 3 a.m. Who starts the containers again?
• The app runs out of memory and exits. Who notices, and who restarts it?
• Traffic doubles. Who decides which server has room for two more copies, and who updates the load balancer to include them?
• You ship version 2. Who replaces the old containers one at a time so users never see an outage, and who puts version 1 back when version 2 turns out to be broken?
• A server dies for good. Who moves its containers to the other two?

Before orchestrators, the answer to every one of those questions was "a person, or a script a person wrote". Teams built piles of shell scripts, cron jobs and runbooks around docker run. Every team's pile was different, and every pile had gaps that showed up during an incident. Kubernetes takes all of those jobs (placement, restarting, scaling, rolling updates, service discovery, load balancing) and makes them the platform's responsibility, done the same way in every company that runs it.

Kubernetes came out of Google in 2014, drawing on its experience running containers internally, and is now maintained by a large open-source community under the Cloud Native Computing Foundation. It ships three minor versions a year, and each is supported with patches for about a year. The name is Greek for "helmsman", and people shorten it to K8s (K, eight letters, s).

Two consequences of the design explain most of what follows, so notice them now.

You describe the end state, not the steps. You never tell Kubernetes "start a container on server 2". You tell it "there should be three copies", and it works out the rest. That is called being declarative, and it is the single most important idea in this guide.

Individual containers are disposable. Kubernetes does not repair a broken container; it throws it away and starts a fresh one from the image. If you come from the world of hand-tended servers, this feels careless at first. It is the opposite: because nothing is precious, nothing needs rescuing at 3 a.m.

What teams use it for, especially in machine learning:

You need little to follow along: a laptop with 8 GB of RAM or more, Docker (or a compatible runtime) installed, a terminal and an internet connection. You will run a real cluster on your own machine, so every experiment is free and every mistake is private.

Desired state and the reconcile loop

Every part of Kubernetes follows one pattern, and once you have it, the rest of the system stops being mysterious.

You write down the desired state: what should exist. Kubernetes keeps observing the actual state: what does exist. Small programs called controllers each run a loop, forever:

1. Look at the desired state for the objects I am responsible for.
2. Look at the actual state.
3. If they differ, take one step to make the actual state closer to the desired state.
4. Go back to step 1.

This is called the reconcile loop (or control loop). A thermostat is the classic analogy: you set 21 °C, the thermostat measures the room, and it switches the heating on or off until the two match. You do not tell the thermostat "run the heater for ten minutes". You tell it the temperature you want.

Here is what that means in practice. Suppose you ask for three copies of a web server and one of them crashes. Nobody has to notice. The controller responsible for those copies sees "desired: 3, actual: 2" and starts a new one. Suppose a whole machine disappears. Same thing: the copies that were on it no longer exist, the numbers no longer match, and new copies start on the surviving machines. Suppose you change your mind and ask for five. The numbers differ again, and two more copies start. Recovery, scaling and deployment are all the same mechanism: change the desired state, or let reality drift from it, and the loop closes the gap.

Three properties of this design will explain behaviour that otherwise looks strange.

It is eventually consistent. When you submit a change, the command returns as soon as the desired state is stored, not when the containers are running. kubectl apply saying deployment.apps/web created means "your request was accepted", not "your app is up". You check the actual state separately, and this guide shows you how.

It keeps trying. If a container cannot start (a wrong image name, a missing configuration value), Kubernetes does not give up and report failure once. It retries, with increasing delays, for as long as the desired state says the container should exist. That is why failures show up as statuses like ImagePullBackOff and CrashLoopBackOff: the "back-off" is the loop waiting longer between attempts.

Fighting it is pointless. If you delete a pod that a controller manages, the controller creates a new one, because you changed the actual state, not the desired state. To make something go away for good, you change or delete the object that declares it. Beginners who try to "kill" a misbehaving app by deleting its pods over and over are arguing with a thermostat.

The cluster: control plane and nodes

A cluster is the set of machines Kubernetes manages, split into two roles. The control plane is the brain: it stores the desired state, makes decisions and runs the controllers. The nodes (worker machines, physical or virtual) are the muscle: they run your containers. On your laptop both roles live in one place; in production the control plane runs on its own dedicated machines, or your cloud provider runs it for you.

Meet each component once, briefly; Mid-level goes inside them.

• kube-apiserver is the front door to everything. Every request, from you, from a controller or from a node, goes through it. It checks who is asking (authentication), whether they are allowed (authorization), whether the request is valid, and then stores the result.
• etcd is a small, highly available database that holds the entire state of the cluster. Only the API server talks to it. If etcd is lost without a backup, the cluster's memory of what should be running is lost too, which is why operators back it up and why managed services hide it from you.
• kube-scheduler watches for new pods that have no node yet, filters out nodes that cannot run them (not enough free CPU, for example), scores the rest and assigns the best one. It only decides; it starts nothing.
• kube-controller-manager runs the built-in controllers: the one that keeps the right number of copies running, the one that tracks which pods belong behind which Service, the one that notices dead nodes, and many more.
• kubelet runs on every node. It watches the API server for pods assigned to its node, asks the container runtime to start them, runs their health checks and reports their status back.
• The container runtime actually pulls images and runs containers. Kubernetes talks to it through a standard interface called the CRI. Since Kubernetes 1.24 that is containerd or CRI-O rather than Docker Engine directly, which surprises people but changes nothing for you: images built with Docker run unchanged, because they are standard OCI images.
• kube-proxy implements Service networking on each node. Some network plugins replace it entirely; you will rarely think about it at this level.

Notice how the reconcile loop from the previous section runs through this picture. You send a desired state to the API server; it lands in etcd; the scheduler notices an unplaced pod and assigns it; the kubelet on that node notices a pod assigned to it and starts it; everybody reports back through the API server. No component gives orders to another. Each watches the API server for the state it cares about and acts on its own. That is why the system survives any single part restarting.

Pods, Deployments and Services: the core nouns

Kubernetes has dozens of object types. Three of them carry nearly all of a beginner's work, and a few supporting ideas hold them together.

A Pod is the smallest thing Kubernetes runs. Usually it holds exactly one container, your app. Sometimes it holds a main container plus a helper, such as a log shipper, that must live and die with it. All containers in a pod share one IP address and can share files through volumes, and they always land on the same node. The key fact about pods is that they are temporary. A pod that dies is not restarted as the same pod on another node; it is replaced by a new pod with a new name and a new IP address. Plan around that and most networking questions answer themselves.

A Deployment is how you actually run a stateless application. You tell it the pod you want (the image, ports and settings, called the pod template) and how many copies (replicas). It creates the pods, replaces any that die, and when you change the template it rolls out the new version gradually. Under the hood a Deployment manages an intermediate object called a ReplicaSet, whose only job is "keep exactly N pods matching this template". You will see ReplicaSets in command output, and you will almost never create one directly.

A Service solves the problem the Deployment creates. Pods come and go, and their IP addresses change every time. Anything that wants to call your app needs one address that never changes. A Service provides it: a fixed virtual IP and a DNS name, like web, that load-balances across whichever pods currently match. Pods are replaced; the Service stays put.

Four supporting ideas tie those three together.

• Labels are key/value tags on any object, such as app=web or tier=frontend. They mean nothing to Kubernetes by themselves.
• Selectors are queries over labels, such as "every pod with app=web". This is how a Deployment knows which pods are its own and how a Service knows where to send traffic. Objects are connected by labels, not by names, and a mismatched label is the most common reason a beginner's Service sends traffic nowhere.
• Namespaces divide one cluster into named areas, such as dev, staging or team-a. Names must be unique within a namespace, not across the cluster. They are an organisational boundary, and the unit for permissions and quotas, but on their own they are not a security wall.
• Nodes, as above, are the machines. You can list them like any other object.

A handful of other objects appear later in this guide and in the next two levels. ConfigMaps and Secrets hold configuration. Jobs run a task to completion, and CronJobs run one on a schedule. StatefulSets run databases and other apps that need stable identities and storage. DaemonSets run one pod on every node, typically for agents like log shippers. You can recognise each by what it controls, and every one of them follows the same reconcile loop.

Installing kubectl and a local cluster

You need two things: kubectl, the command-line client, and a cluster to point it at. For learning, the cluster runs on your laptop.

Installing kubectl

kubectl is a single binary. Install it with your platform's package manager, or download it from the official release server.

# macOS (Homebrew)
brew install kubectl

# Windows (winget)
winget install -e --id Kubernetes.kubectl

On Linux, download the binary, check its checksum and install it. These are the official steps; the stable.txt file always names the current stable release, v1.37.1 at the time of writing.

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl.sha256"
echo "$(cat kubectl.sha256)  kubectl" | sha256sum --check
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

The checksum line should print kubectl: OK. On an ARM machine, replace amd64 with arm64 in both URLs. If you prefer apt on Debian or Ubuntu, use the community package repository at pkgs.k8s.io, which has one repository per minor version.

sudo apt-get update && sudo apt-get install -y apt-transport-https ca-certificates curl gnupg
sudo mkdir -p -m 755 /etc/apt/keyrings    # only needed on Debian < 12 and Ubuntu < 22.04
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.37/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.37/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update && sudo apt-get install -y kubectl

Check what you installed:

kubectl version --client

Client Version: v1.37.1
Kustomize Version: v5.x.x

The exact Kustomize line varies. What matters is the client version. kubectl supports API servers one minor version either side of itself, so a 1.37 kubectl works with 1.36, 1.37 and 1.38 clusters. When you use a managed cluster that runs an older version, match your kubectl to it.

Choosing a local cluster

The Kubernetes project lists two tools for a learning cluster on your own machine, plus kubeadm for real multi-machine clusters. Either of the first two works for this whole guide.

Docker Desktop and Rancher Desktop can also switch on a single-node cluster from their settings, which is convenient if you already use them. This guide uses kind in its examples because it needs nothing beyond Docker and creates a cluster in under a minute. Install it from its quick-start page (for example brew install kind on macOS), then create a cluster:

kind create cluster --name learn

Creating cluster "learn" ...
 ✓ Ensuring node image (kindest/node:...)
 ✓ Preparing nodes
 ✓ Writing configuration
 ✓ Starting control-plane
 ✓ Installing CNI
 ✓ Installing StorageClass
Set kubectl context to "kind-learn"

Read the last line: kind has already configured kubectl to talk to the new cluster, under a context called kind-learn. The next section explains what that means. The Kubernetes version your kind cluster runs depends on your kind release; each kind release pins a default node image, and kubectl version will tell you which one you got.

If you chose minikube instead, minikube start does the equivalent and names the context minikube. Everything else in this guide is the same.

Check your setup: kubeconfig, contexts and namespaces

Before deploying anything, confirm which cluster kubectl is talking to. This habit sounds fussy until the day you run a delete command against production because your terminal still pointed there from the morning.

kubectl reads its connection details from a file called the kubeconfig, by default ~/.kube/config. It holds three lists and one pointer:

A context is the thing you switch between. When you have a local cluster, a staging cluster and a production cluster, you have (at least) three contexts, and current-context decides which one your next command hits. You can point kubectl at a different file with the KUBECONFIG environment variable.

kubectl config get-contexts          # list contexts; * marks the current one
kubectl config current-context       # print just the current one
kubectl config use-context kind-learn
kubectl cluster-info                 # where is the API server, and is it answering?
kubectl get nodes                    # the machines in this cluster

Kubernetes control plane is running at https://127.0.0.1:52341
CoreDNS is running at https://127.0.0.1:52341/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

NAME                  STATUS   ROLES           AGE   VERSION
learn-control-plane   Ready    control-plane   3m    v1.37.1

Read the node line from left to right: the node's name, its status (Ready means the kubelet is healthy and reporting in), its role, how long it has existed, and the kubelet's version. On a kind cluster you see one node that plays both roles; on a real cluster you see several, and any node that is not Ready is worth investigating.

kubectl version without --client prints both the client and the server version. The server line is the one that matters when you read documentation, because features depend on the cluster's version, not your laptop's.

Namespaces you already have

A fresh cluster is not empty. List its namespaces:

kubectl get namespaces

NAME                 STATUS   AGE
default              Active   5m
kube-node-lease      Active   5m
kube-public          Active   5m
kube-system          Active   5m
local-path-storage   Active   5m

default is where your objects go when you do not say otherwise. kube-system holds the cluster's own components: DNS, the network plugin and, on kind, the control plane itself running as pods. Look inside it once with kubectl get pods -n kube-system so you know what "healthy" looks like, then leave it alone. local-path-storage is kind's storage helper and will not exist on other clusters.

Every command that reads or changes namespaced objects accepts -n <namespace> (or --namespace). -A (--all-namespaces) looks across all of them. To stop typing -n all day, set a default namespace for the current context:

kubectl create namespace demo
kubectl config set-context --current --namespace=demo

Your first Deployment, step by step

Now run something. This section uses kubectl's quick imperative commands, which create objects straight from the command line. They are the fastest way to see the moving parts. The next section shows the declarative YAML way you will use for real work.

# 1. Create a Deployment running three copies of nginx
kubectl create deployment web --image=nginx:1.27 --replicas=3

# 2. Watch the pods appear
kubectl get pods -w

NAME                   READY   STATUS              RESTARTS   AGE
web-5d8c7f9b6d-2xkqp   0/1     ContainerCreating   0          2s
web-5d8c7f9b6d-8hj4w   0/1     ContainerCreating   0          2s
web-5d8c7f9b6d-tq7mz   0/1     ContainerCreating   0          2s
web-5d8c7f9b6d-2xkqp   1/1     Running             0          9s
web-5d8c7f9b6d-8hj4w   1/1     Running             0          10s
web-5d8c7f9b6d-tq7mz   1/1     Running             0          10s

Press Ctrl+C to stop watching. The pod names tell a story: web is the Deployment, 5d8c7f9b6d identifies the ReplicaSet (it is a hash of the pod template, so it changes when the template changes), and the last five characters are random, one per pod. READY 1/1 means one of one containers in the pod is ready. STATUS walks from Pending (waiting for a node) through ContainerCreating (pulling the image, starting) to Running.

Look at the three layers you just created:

kubectl get deployment,replicaset,pods

NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/web   3/3     3            3           40s

NAME                             DESIRED   CURRENT   READY   AGE
replicaset.apps/web-5d8c7f9b6d   3         3         3       40s

NAME                       READY   STATUS    RESTARTS   AGE
pod/web-5d8c7f9b6d-2xkqp   1/1     Running   0          40s
pod/web-5d8c7f9b6d-8hj4w   1/1     Running   0          40s
pod/web-5d8c7f9b6d-tq7mz   1/1     Running   0          40s

The ReplicaSet line is the reconcile loop in a table: DESIRED 3, CURRENT 3, READY 3. Now break it on purpose. Delete one pod and look again immediately:

kubectl delete pod web-5d8c7f9b6d-2xkqp
kubectl get pods

NAME                   READY   STATUS    RESTARTS   AGE
web-5d8c7f9b6d-8hj4w   1/1     Running   0          2m
web-5d8c7f9b6d-tq7mz   1/1     Running   0          2m
web-5d8c7f9b6d-w9rcs   1/1     Running   0          3s

A new pod, with a new name, three seconds old. You changed the actual state; the desired state still said three; the controller fixed the difference. That is self-healing, and you just watched it.

Giving it an address

The three pods each have an IP address, but those addresses change whenever a pod is replaced. Put a Service in front of them:

kubectl expose deployment web --port=80 --target-port=80
kubectl get service web

NAME   TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
web    ClusterIP   10.96.142.17   <none>        80/TCP    5s

ClusterIP is the default Service type: an address reachable only from inside the cluster. To reach it from your laptop while developing, use port-forwarding, which tunnels a local port through the API server:

kubectl port-forward service/web 8080:80

Forwarding from 127.0.0.1:8080 -> 80
Forwarding from [::1]:8080 -> 80

Open http://localhost:8080 and nginx answers. The command keeps running until you press Ctrl+C. Port-forwarding is a developer's tool for poking at a service, not a way to serve real users; you will meet the real ways in the Services section.

Scaling and cleaning up

kubectl scale deployment web --replicas=5
kubectl get pods
kubectl delete service web
kubectl delete deployment web

Scaling is just changing the desired count; two pods appear. Deleting the Deployment deletes its ReplicaSet and its pods with it, because Kubernetes tracks which objects own which (the pods carry an "owned by" reference). Deleting the pods alone would have been pointless, as you saw.

Manifests: describing objects in YAML

The commands in the last section were quick, but they leave no record. Nobody can review them, nobody can re-run them exactly, and in a month you will not remember what flags you used. Real work uses manifests: YAML files that describe each object, stored in Git next to your code, and applied with kubectl apply. This is the declarative model in its practical form: the file is the desired state.

Every Kubernetes object, in every manifest you will ever read, has the same four top-level fields, plus a fifth that Kubernetes writes for you.

Here is the Deployment from the last section written as a manifest. Create a folder called k8s-hello and save this in it:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  labels:
    app: web
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: nginx
          image: nginx:1.27
          ports:
            - containerPort: 80

Read it top to bottom, because the nesting is the part that confuses people.

• apiVersion: apps/v1 and kind: Deployment: Deployments live in the apps API group, version v1. Core objects such as Pods, Services and ConfigMaps use plain v1.
• metadata.name: web: the Deployment's name, unique within its namespace.
• spec.replicas: 3: how many pods.
• spec.selector.matchLabels: which pods this Deployment owns. It claims every pod labelled app: web.
• spec.template: the pod template, a complete pod description minus the name. It has its own metadata (the labels each pod gets) and its own spec (the containers).
• containers[].name, image, ports: one container, called nginx, from the nginx:1.27 image, listening on port 80. containerPort is informational, much like EXPOSE in a Dockerfile; it does not open or publish anything by itself.

Apply it:

kubectl apply -f deployment.yaml

deployment.apps/web created

Run the same command again without changing the file and you get deployment.apps/web unchanged. Change replicas to 4, apply again, and you get configured. That is the heart of declarative management: apply means "make the cluster look like this file", however many times you run it, whatever state the cluster was in. You can apply a single file, a whole directory (kubectl apply -f k8s-hello/) or a URL.

A Pod on its own, for comparison

You will see bare Pod manifests in documentation and tutorials. They look like the inside of a Deployment's template, promoted to the top level:

apiVersion: v1
kind: Pod
metadata:
  name: hello
  labels:
    app: hello
spec:
  containers:
    - name: hello
      image: busybox:1.36
      command: ["sh", "-c", "echo Hello from $(hostname); sleep 3600"]

Useful for a quick experiment; not for anything that must stay up, because nothing replaces it when it dies.

Let kubectl write the first draft

Nobody types manifests from memory. Two tricks produce a correct starting point:

# Generate YAML instead of creating the object
kubectl create deployment web --image=nginx:1.27 --replicas=3 --dry-run=client -o yaml > deployment.yaml

# Look up any field, with documentation, from the cluster itself
kubectl explain deployment.spec.replicas
kubectl explain deployment.spec.strategy --recursive

--dry-run=client -o yaml prints what the command would create and creates nothing. The output includes a few empty fields such as creationTimestamp: null and status: {} that you can delete. kubectl explain reads the schema from your cluster, so it is always right for your version, unlike a blog post from 2019.

Labels and selectors: how objects find each other

In the manifest, the Deployment found its pods through a label, not a name. That is not a detail of Deployments; it is how nearly everything in Kubernetes connects to everything else. Services find pods by label. Deployments find pods by label. Later, network policies, autoscalers and monitoring tools find pods by label. Get labels wrong and objects silently fail to connect, with no error message, because "a selector that matches nothing" is a perfectly valid state.

A label is a key/value pair in metadata.labels. Kubernetes gives no meaning to specific keys; teams agree on their own. Common choices include app, tier, env and version, and the project recommends a standard set with an app.kubernetes.io/ prefix, such as app.kubernetes.io/name and app.kubernetes.io/version, which many tools understand.

You can select by label from the command line:

kubectl get pods --show-labels                 # show every pod's labels
kubectl get pods -l app=web                    # equality
kubectl get pods -l 'env in (prod,staging)'    # set-based
kubectl get pods -l app=web,tier!=cache        # combine with a comma (AND)
kubectl label pod <pod-name> tier=frontend     # add a label to a live object
kubectl label pod <pod-name> tier-             # remove it (note the trailing dash)

Annotations look similar (key/value pairs under metadata.annotations) but are never used for selection. They hold notes for people and tools: a description, a Git commit, the reason for a change. Use a label if you will ever want to select on it; use an annotation for everything else.

Here is an experiment that shows how literal the matching is. Take a running pod from your web Deployment and change its app label:

kubectl label pod <one-web-pod> app=debug --overwrite
kubectl get pods --show-labels

The Deployment now counts only two pods with app=web, so it creates a third. The relabelled pod keeps running, orphaned, because nothing selects it any more. That is a real debugging technique: pull a misbehaving pod out of rotation to inspect it while its replacement serves traffic. It is also a warning: the connection between objects is exactly as strong as the labels.

Services: a stable address for pods that come and go

Pods are replaced all the time: during updates, when they crash, when a node is drained for maintenance. Each replacement gets a new IP address. If your frontend called your API by pod IP, it would break every few hours. A Service gives a group of pods one address that never changes and spreads traffic across whichever pods currently match its selector and are ready.

apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  selector:
    app: web
  ports:
    - port: 80
      targetPort: 80

Two port fields, two different meanings:

They are often the same number, but they need not be. An app listening on 8000 behind a Service on port 80 is common and perfectly normal: port: 80, targetPort: 8000.

Finding a Service by name

Every cluster runs a DNS server (CoreDNS) that gives each Service a name. From any pod in the same namespace, the Service above is simply web. From another namespace it is web.demo (service name, then namespace), and its full name is web.demo.svc.cluster.local. This is what makes Kubernetes configuration portable: your app connects to http://web, and it works in every cluster and every environment where a Service by that name exists.

Test it from inside the cluster with a throwaway pod:

kubectl run tmp --image=busybox:1.36 --rm -it --restart=Never -- sh

Then, at the prompt inside that pod:

wget -qO- http://web | head -n 4
nslookup web
exit

--rm -it --restart=Never means "run one interactive pod and delete it when I exit", the Kubernetes equivalent of docker run --rm -it. It is the quickest way to test networking from the inside.

Where the traffic actually goes

A Service keeps a live list of the pod addresses behind it, stored in objects called EndpointSlices. When a Service seems dead, this list is the first thing to check:

kubectl get endpointslices -l kubernetes.io/service-name=web

NAME        ADDRESSTYPE   PORTS   ENDPOINTS                            AGE
web-7xk2p   IPv4          80      10.244.0.12,10.244.0.13,10.244.0.14   5m

Three addresses: three ready pods. An empty ENDPOINTS column means the selector matches no ready pods: either a label mismatch (the previous section) or pods that are not passing their readiness check (later in this guide).

The four Service types

On a cloud cluster, type: LoadBalancer makes the provider create a real load balancer and put its address in the EXTERNAL-IP column. On kind or minikube there is no cloud to ask, so the column stays at <pending> (minikube offers minikube tunnel to fill it in). For local learning, ClusterIP plus kubectl port-forward is all you need.

The commands you will use, grouped by task

kubectl has a lot of subcommands. Day to day, you use about twenty, and they fall into a few groups by what you are trying to do. Nearly all of them follow the pattern kubectl <verb> <type> <name> [flags], and types have short forms: po for pods, deploy for deployments, svc for services, ns for namespaces, cm for configmaps. kubectl api-resources lists every type and its short name.

Looking around

kubectl get pods                         # list; add -o wide for node and IP
kubectl get deploy,svc,pods              # several types at once
kubectl get pods -A                      # every namespace
kubectl get pod <pod> -o yaml            # the full object, spec and status
kubectl get pods -w                      # watch for changes
kubectl describe pod <pod>               # human-readable detail plus recent Events

get answers "what exists and what state is it in". describe answers "why": it includes the Events at the bottom, a timeline of what the cluster did to that object (scheduled, pulled the image, started, failed a health check). When anything goes wrong, describe is almost always your second command.

Kubernetes 1.37 also made a newer output format, -o kyaml, generally available. It is a stricter YAML style designed to avoid YAML's classic ambiguities. Any KYAML output is still valid YAML. You do not need it yet, but you will start seeing it in newer documentation.

Reading what the app says

kubectl logs <pod>                       # the container's stdout and stderr
kubectl logs <pod> -f                    # follow live
kubectl logs <pod> --previous            # the PREVIOUS container, the one that crashed
kubectl logs <pod> --tail=100 --since=1h
kubectl logs deploy/web                  # logs from one pod of the Deployment
kubectl logs <pod> -c <container>        # pick a container when the pod has several

--previous is the flag that matters most during a crash loop. After a restart, plain kubectl logs shows the new container, which may have printed nothing yet. --previous shows the one that died, including its last words.

Getting inside

kubectl exec -it <pod> -- sh             # a shell in a running container
kubectl exec <pod> -- env                # run one command and print the result
kubectl port-forward svc/web 8080:80     # tunnel a local port to a Service
kubectl port-forward pod/<pod> 8080:80   # or straight to one pod
kubectl run tmp --image=busybox:1.36 --rm -it --restart=Never -- sh   # a throwaway debug pod

The -- separates kubectl's own flags from the command to run inside the container. Leave it out and kubectl tries to interpret your command's flags as its own.

Creating and changing things

kubectl apply -f deployment.yaml         # create or update from a file
kubectl apply -f k8s/                    # a whole directory
kubectl delete -f deployment.yaml        # delete what the file describes
kubectl scale deploy/web --replicas=5
kubectl set image deploy/web nginx=nginx:1.28
kubectl create deployment web --image=nginx:1.27 --dry-run=client -o yaml   # generate YAML
kubectl explain pod.spec.containers      # field documentation from the cluster

For anything that lasts longer than an experiment, prefer apply with a file you commit. Commands such as scale and set image change the live object but not your file, so the next apply of an outdated file silently puts the old value back.

Rollouts

kubectl rollout status deploy/web        # wait for an update to finish
kubectl rollout history deploy/web       # list revisions
kubectl rollout undo deploy/web          # back to the previous revision
kubectl rollout restart deploy/web       # replace every pod, gradually, same spec

The next section covers these properly.

Resource usage

kubectl top pods
kubectl top nodes

top shows live CPU and memory use, and needs an add-on called metrics-server. kind does not install it, so kubectl top fails there with Metrics API not available until you add it (minikube has minikube addons enable metrics-server). Its API, metrics.k8s.io/v1, became stable in Kubernetes 1.37.

Updating an app: rolling updates and rollbacks

Shipping a new version is where Kubernetes pays for itself. When you change a Deployment's pod template (a new image tag, a new environment variable, anything under spec.template), the Deployment does not stop everything and restart. It performs a rolling update: it creates a new ReplicaSet for the new template and shifts pods over gradually, starting new pods and removing old ones, so that there are always enough ready pods to serve traffic.

Two settings control the pace, under spec.strategy.rollingUpdate. Both default to 25%:

• maxSurge: how many pods above the desired count may exist during the update. More surge means a faster rollout and more temporary resource use.
• maxUnavailable: how many pods below the desired count are allowed. Zero means "never drop below full capacity".

The alternative strategy, Recreate, deletes all old pods before starting new ones. That means downtime, and it exists for apps that genuinely cannot run two versions at once.

Watch an update happen. Change the image in deployment.yaml from nginx:1.27 to nginx:1.28, then:

kubectl apply -f deployment.yaml
kubectl rollout status deploy/web

Waiting for deployment "web" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "web" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "web" rollout to finish: 1 old replicas are pending termination...
deployment "web" successfully rolled out

kubectl get rs now shows two ReplicaSets: the new one with three pods and the old one scaled to zero. The old one is kept on purpose, because it is how rollback works.

When the new version is broken

Now ship a mistake. Set the image to a tag that does not exist:

kubectl set image deploy/web nginx=nginx:9.99-does-not-exist
kubectl get pods

NAME                   READY   STATUS             RESTARTS   AGE
web-6b9f5c8d47-5kzvn   1/1     Running            0          3m
web-6b9f5c8d47-m2l8q   1/1     Running            0          3m
web-6b9f5c8d47-x7rjt   1/1     Running            0          3m
web-7f6d4b9c88-qp2dd   0/1     ImagePullBackOff   0          40s

The rolling update protected you. The first new pod could not start, so it never became ready, so the Deployment never removed an old pod. Your three working pods are still serving. kubectl rollout status would wait here (and eventually report that the rollout exceeded its progress deadline). Undo it:

kubectl rollout history deploy/web
kubectl rollout undo deploy/web

deployment.apps/web rolled back

undo switches the Deployment back to the previous ReplicaSet's template. kubectl rollout undo deploy/web --to-revision=2 picks a specific revision from the history. The CHANGE-CAUSE column in the history is empty unless you record why each change happened; set the kubernetes.io/change-cause annotation with kubectl annotate deploy/web kubernetes.io/change-cause="upgrade to nginx 1.28" and it appears there.

kubectl rollout restart deploy/web is the third command worth knowing. It replaces every pod with a fresh one, using the same rolling process, without changing the spec. It is the standard way to make pods pick up a changed ConfigMap or Secret they only read at startup.

Configuration: ConfigMaps and Secrets

An image should be the same in development, staging and production. What differs between them (the database address, the log level, the model version to load, the API key) is configuration, and it belongs outside the image. Kubernetes gives you two objects for it. A ConfigMap holds non-confidential key/value settings or whole files. A Secret holds confidential values such as passwords and tokens. Both are limited to 1 MiB, and pods consume them in the same two ways: as environment variables or as files in a mounted volume.

Create them from the command line:

kubectl create configmap app-config \
  --from-literal=LOG_LEVEL=info \
  --from-literal=MODEL_NAME=sentiment-v1

kubectl create secret generic app-secrets \
  --from-literal=API_TOKEN=change-me-please

Or declare them in YAML, which is what you commit:

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  LOG_LEVEL: "info"
  MODEL_NAME: "sentiment-v1"
  settings.ini: |
    [server]
    workers = 2
    timeout_seconds = 30

Notice that ConfigMap values are always strings, so quote numbers and booleans. The | on settings.ini starts a multi-line value: that key holds a whole file.

Using them in a pod

This fragment of a pod template shows all three common patterns at once:

    spec:
      containers:
        - name: app
          image: nginx:1.27
          envFrom:
            - configMapRef:
                name: app-config          # every key becomes an env var
          env:
            - name: API_TOKEN
              valueFrom:
                secretKeyRef:
                  name: app-secrets
                  key: API_TOKEN          # one key, one env var
          volumeMounts:
            - name: config-files
              mountPath: /etc/app         # settings.ini appears as /etc/app/settings.ini
              readOnly: true
      volumes:
        - name: config-files
          configMap:
            name: app-config
            items:
              - key: settings.ini
                path: settings.ini

• envFrom imports every key of the ConfigMap as an environment variable. Quick, but keys that are not valid variable names (such as settings.ini) are skipped.
• env with valueFrom picks one key and names the variable yourself. Use it for Secrets, so you know exactly which values land in the environment.
• A configMap volume turns keys into files. Use it for whole configuration files.

What Secrets do give you is separation: a distinct object type with its own permissions, kept out of your image and out of your ConfigMaps, which Mid-level and Senior build on with encryption at rest, tight access control and external secret stores such as Vault. The type you create with generic is Opaque; others exist for specific jobs, such as kubernetes.io/tls for certificates and kubernetes.io/dockerconfigjson for registry credentials.

When changes take effect

This catches everyone once. Environment variables are read when the container starts. Edit a ConfigMap and running pods keep the old values until they are replaced. Files mounted from a ConfigMap volume are updated in place after a short delay, but that only helps if your app re-reads them. The reliable habit: after changing configuration, run kubectl rollout restart deploy/<name>.

If a pod references a ConfigMap or Secret that does not exist, it does not start at all. Its status shows CreateContainerConfigError, and kubectl describe names the missing object, for example secret "app-secrets" not found. You can mark a reference optional: true if the pod should start without it.

Resources and health checks

Two small additions to every container spec turn a demo into something a cluster can run responsibly: telling Kubernetes how much CPU and memory the container needs, and telling it how to check the container is healthy.

Requests and limits

          resources:
            requests:
              cpu: "100m"
              memory: "128Mi"
            limits:
              memory: "256Mi"

Units first. CPU is measured in cores, and m means thousandths: 100m is a tenth of a core, 1 or 1000m is one core. Memory uses binary suffixes: Mi (mebibytes) and Gi (gibibytes). Write 128Mi, not 128M (which means megabytes) and certainly not 128m, which Kubernetes reads as 0.128 bytes.

The two fields do different jobs:

The scheduler places pods using requests, not actual usage. A node with 4 CPUs can hold pods requesting 4 CPUs in total, however idle they are. If no node has enough unrequested capacity, a new pod waits in Pending with an event like 0/3 nodes are available: 3 Insufficient cpu. That message means "nothing has room for what you asked", not that the cluster is busy.

Kubernetes also sorts pods into three quality of service classes based on these fields. Guaranteed pods set requests equal to limits for every container. Burstable pods set some values. BestEffort pods set none, and they are the first to be evicted when a node runs short of memory. A pod with no requests at all is therefore both invisible to the scheduler's planning and first in line to be thrown off a struggling node.

Probes: how Kubernetes knows your app is healthy

By default Kubernetes considers a container healthy as long as its process is running. That misses a lot: an app that is still loading a 2 GB model, an app that is deadlocked but still alive, an app that has lost its database connection. Probes let you describe health properly. There are three, and they do different things.

          readinessProbe:
            httpGet:
              path: /
              port: 80
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /
              port: 80
            periodSeconds: 10
            failureThreshold: 3

Each probe can make an HTTP request (httpGet, success is a 2xx or 3xx status), open a TCP connection (tcpSocket) or run a command inside the container (exec, success is exit code 0). periodSeconds sets how often it runs, and failureThreshold how many consecutive failures count as failure.

The readiness probe is the one that makes rolling updates safe. A new pod joins the Service only once it reports ready, and the Deployment only removes an old pod once the new one is ready. Without a readiness probe, "ready" simply means "the process started", so users can be sent to an app that is still warming up. For ML services that load a model at startup, a readiness probe on an endpoint that answers only once the model is loaded is essential.

Keeping data: volumes and PersistentVolumeClaims

A container's filesystem is temporary. When the container restarts, anything it wrote outside a volume is gone, exactly as in Docker. Kubernetes has two broad kinds of storage to fix that, and at this level you need to recognise both.

Pod-scoped volumes live as long as the pod. An emptyDir volume starts empty when the pod is created, survives container restarts, and is deleted with the pod. It is useful for scratch space, caches and files shared between containers in one pod. The configMap and secret volumes you already used are also this kind.

Persistent storage outlives pods. You ask for it with a PersistentVolumeClaim (PVC): "I need 1 GiB of storage that one node can write to". The cluster satisfies the claim with a PersistentVolume (PV), a real piece of storage such as a cloud disk. A StorageClass describes what kind of storage to create automatically when a claim arrives; kind and minikube ship a default one that uses a folder on the node.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

A pod then mounts the claim by name:

      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: data

ReadWriteOnce means one node at a time can mount the volume read-write, which is what most cloud disks support. ReadWriteMany (many nodes) needs a shared file system and is not available everywhere. kubectl get pvc shows whether a claim is Bound (storage attached) or Pending.

Reading pod status and fixing the common errors

Sooner or later a pod will not start, and the STATUS column will show a word you have not seen. The good news is that a beginner meets the same half-dozen failures over and over, and each has a recognisable signature. The skill to build is a fixed routine for reading them, so you never have to guess.

The debugging routine

1. Getkubectl get pods. Read STATUS, READY and RESTARTS. A climbing RESTARTS count means the container starts and then dies.
2. Describekubectl describe pod <pod>. Scroll to Events at the bottom and read the last few lines. Also read State and Last State for each container: the reason and exit code of the last crash live there.
3. Logskubectl logs <pod>, and kubectl logs <pod> --previous if the container has restarted. Application errors (a stack trace, a missing setting) show up here, not in Events.
4. Get insidekubectl exec -it <pod> -- sh if the container stays up long enough, or a throwaway busybox pod to test networking and DNS from next to it.
5. Widen the viewkubectl get events --sort-by=.lastTimestamp for the whole namespace, and kubectl describe node <node> if the problem looks like capacity.

Events and logs answer different questions. Events are the cluster talking about your pod: scheduling, image pulls, probe failures, kills. Logs are your application talking. If the image never pulled, there are no logs to read; if the app crashed on a bad setting, the events only say "back-off", and the reason is in the logs. Check both before concluding anything.

The statuses, and what each one means

Three of these deserve a closer look, because they account for most beginner time lost.

ImagePullBackOff. The events show the exact reason, and there are only three common ones. not found or manifest unknown means the name or tag is wrong; check for typos and remember that tags are case-sensitive. pull access denied or unauthorized means the registry is private and the node has no credentials: create a registry Secret with kubectl create secret docker-registry regcred --docker-server=... --docker-username=... --docker-password=... and reference it in the pod spec under imagePullSecrets. toomanyrequests means you have hit a registry's rate limit. One kind-specific trap: an image you built locally with docker build is not visible inside the kind cluster, because kind's node has its own image store. Load it with kind load docker-image myapp:dev --name learn, or push it to a registry.

CrashLoopBackOff. This is not an error in itself. It means "the container exited, Kubernetes restarted it, it exited again, and Kubernetes is now waiting before the next try". The wait starts at 10 seconds and doubles each time up to 5 minutes, which is why a crash-looping pod seems to sit still. The real error is always in the application output, and kubectl logs <pod> --previous is the command that shows it. The usual causes are a missing environment variable, a wrong command, a failed connection to a dependency at startup, or a liveness probe killing an app that was simply slow.

Pending. Read the FailedScheduling event word for word; it lists why each node was rejected. Insufficient cpu or Insufficient memory means the requests do not fit anywhere. node(s) had untolerated taint {node-role.kubernetes.io/control-plane: } means the only nodes available are control-plane nodes, which refuse ordinary workloads by default. node(s) didn't match Pod's node affinity/selector means you asked for a node label that no node has. pod has unbound immediate PersistentVolumeClaims means the storage is not ready.

Exit codes

When a container dies, describe shows its exit code under Last State: Terminated. The same codes you met in Docker apply:

Errors that come from kubectl itself

Not every error is about pods. These come back straight from the command line:

• The connection to the server localhost:8080 was refused means no kubeconfig was found. The connection to the server <host>:6443 was refused means the API server at that address is not answering; for a local cluster, it is probably not running.
• error: You must be logged in to the server (Unauthorized) means your credentials have expired or belong to another cluster. Refresh your login and check kubectl config current-context.
• Error from server (Forbidden): pods is forbidden: User "jane" cannot list resource "pods" in API group "" in the namespace "dev" means you are authenticated but not permitted. kubectl auth can-i list pods -n dev tells you what you can do.
• Error from server (NotFound): deployments.apps "web" not found most often means you are in the wrong namespace. Add -n or check your context's default.
• no matches for kind "Deployment" in version "extensions/v1beta1" means you copied a manifest from an old tutorial. That API version was removed years ago; Deployments are apps/v1.

Letting users in: Ingress and the Gateway API

A ClusterIP Service is reachable only inside the cluster. For real users on the internet you need a way in. You have seen one already, type: LoadBalancer, which gives each Service its own cloud load balancer. That works, but one load balancer per service gets expensive, and it only understands ports, not URLs. Most teams want one entry point that routes by hostname and path: api.example.com/predict to one Service, example.com/ to another, with TLS handled in one place.

Kubernetes has two APIs for that HTTP-level routing, and you need to know which is which, because the landscape changed in 2026.

Ingress (networking.k8s.io/v1) is the older one. An Ingress object lists routing rules, and a separately installed Ingress controller reads them and does the actual routing. The API still works and is not being removed, but it is frozen: it gets no new features. The most widely used controller, the community's Ingress NGINX project, was retired on 24 March 2026, and it no longer receives releases or security fixes. Existing installations keep running, but you should not choose it for a new cluster, whatever older tutorials say.

Gateway API (gateway.networking.k8s.io/v1) is the successor, and the project's recommendation for new work. It splits the job into roles: a GatewayClass names the implementation (provided by your platform), a Gateway is the actual entry point (run by the cluster operator), and an HTTPRoute holds the routing rules for one application (written by the app team, which is you). Like Ingress, it needs an implementation installed in the cluster; several projects provide one.

A route for your web Service looks like this, assuming the platform team has already created a Gateway named public:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: web
spec:
  parentRefs:
    - name: public
  hostnames:
    - "web.example.com"
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: web
          port: 80

Read it as a sentence: "attach to the Gateway called public; for requests to web.example.com whose path starts with /, send them to the Service web on port 80." Setting up the Gateway itself, TLS certificates and choosing an implementation are Mid-level topics. At this level, it is enough to recognise both APIs, know that Ingress NGINX is retired, and know that new routing is written as an HTTPRoute.

Putting it all together

Everything above, in one small project. Nothing here is new. It is a web service with its own namespace, configuration kept out of the image, a Secret, resource requests, health checks, a Service in front and a rolling update strategy that never drops below full capacity. Read it as a whole and you should recognise every line and be able to say why it is there.

The app is deliberately simple: nginx serving a page that comes from a ConfigMap, so you can see configuration reach a running container without building an image. Swap in your own image later and the structure stays the same. Create a folder called k8s-project and put four files in it.

apiVersion: v1
kind: Namespace
metadata:
  name: shop
  labels:
    app.kubernetes.io/part-of: shop

apiVersion: v1
kind: ConfigMap
metadata:
  name: web-config
  namespace: shop
data:
  LOG_LEVEL: "info"
  index.html: |
    <h1>Hello from Kubernetes</h1>
    <p>Served by a Deployment, configured by a ConfigMap.</p>

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  namespace: shop
  labels:
    app: web
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: nginx
          image: nginx:1.27
          ports:
            - containerPort: 80
          env:
            - name: LOG_LEVEL
              valueFrom:
                configMapKeyRef:
                  name: web-config
                  key: LOG_LEVEL
            - name: API_TOKEN
              valueFrom:
                secretKeyRef:
                  name: web-secrets
                  key: API_TOKEN
          resources:
            requests:
              cpu: "100m"
              memory: "64Mi"
            limits:
              memory: "128Mi"
          readinessProbe:
            httpGet:
              path: /
              port: 80
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /
              port: 80
            periodSeconds: 10
            failureThreshold: 3
          volumeMounts:
            - name: site
              mountPath: /usr/share/nginx/html
              readOnly: true
      volumes:
        - name: site
          configMap:
            name: web-config
            items:
              - key: index.html
                path: index.html

apiVersion: v1
kind: Service
metadata:
  name: web
  namespace: shop
spec:
  selector:
    app: web
  ports:
    - port: 80
      targetPort: 80

The numeric prefixes are a convention, not a Kubernetes feature: kubectl apply -f on a directory processes files in name order, so the namespace exists before anything tries to go into it. The Secret is deliberately not in the folder, because a real one never belongs in Git. Create it by hand, then apply the rest.

kubectl apply -f k8s-project/00-namespace.yaml
kubectl create secret generic web-secrets -n shop --from-literal=API_TOKEN=change-me-please
kubectl apply -f k8s-project/
kubectl rollout status deploy/web -n shop
kubectl get deploy,rs,pods,svc -n shop
kubectl get endpointslices -n shop -l kubernetes.io/service-name=web
kubectl port-forward -n shop svc/web 8080:80

Open http://localhost:8080 and you should see the page from the ConfigMap. The second apply reports the namespace as unchanged and everything else as created, which is the declarative model showing through: applying the same file twice is harmless.

Now run the three exercises that prove the setup does its job. First, self-healing: delete a pod and watch a replacement appear. Second, a safe update: change the image to nginx:1.28 in 20-deployment.yaml, apply, and watch kubectl get pods -n shop -w; because maxUnavailable is 0, a new pod must pass its readiness probe before an old one goes. Third, a configuration change: edit the page in 10-config.yaml, apply, and reload. The mounted file updates by itself after a short delay, while LOG_LEVEL in the environment only changes after kubectl rollout restart deploy/web -n shop.

Ten decisions carry the lesson of this page, and each maps to a section above:

When you are done, one command removes the lot, because deleting a namespace deletes everything inside it:

kubectl delete namespace shop

What you can now do, and what comes next

You can explain what Kubernetes is for and what came before it, describe the reconcile loop and why it makes recovery, scaling and deployment the same mechanism, name the parts of a cluster and what each one does, install kubectl and run a local cluster, check which cluster and namespace you are talking to, write Deployment and Service manifests and apply them, connect objects with labels, reach a service by DNS name, configure an app with ConfigMaps and Secrets, set requests, limits and probes, keep data in a PersistentVolumeClaim, ship and roll back an update safely, and diagnose the common pod failures from their status, events and logs. That is enough to deploy and operate a real stateless service on a cluster someone else runs, and to hold your own in a team that uses Kubernetes every day.

Mid-level takes each of those topics further. It explains what happens inside the API server, the scheduler and the kubelet precisely enough to predict their behaviour. It covers packaging with Helm and Kustomize, server-side apply, StatefulSets, Jobs and CronJobs for ML batch work, autoscaling with the HorizontalPodAutoscaler, network policies, RBAC, Pod Security, setting up the Gateway API with TLS, and debugging beyond the basics with ephemeral containers and node-level tools. It also connects Kubernetes to CI/CD and the cloud.

Senior then covers owning a cluster as a platform for other teams: control-plane architecture and failure modes, which component breaks first as you scale, the security model from authentication to supply chain, multi-tenancy, cost, the upgrade treadmill of three releases a year, GPUs and Dynamic Resource Allocation for ML workloads, and when Kubernetes is the wrong tool.

In this catalogue, Kubernetes sits in a chain. Terraform is how teams create the clusters themselves, and Helm and Argo CD are how they package and deliver what runs on them. On the machine-learning side, KServe serves models on Kubernetes using exactly the Deployments, Services and probes you learned here.

Sources

• Kubernetes documentation home
• Kubernetes releases
• Kubernetes v1.37 release announcement
• Version skew policy
• Kubernetes components
• Install tools
• Install and set up kubectl on Linux
• Install and set up kubectl on macOS
• Install and set up kubectl on Windows
• kubectl quick reference
• Pod lifecycle
• Resource management for pods and containers
• Secrets
• Good practices for Kubernetes Secrets
• Ingress
• Gateway API
• Node-pressure eviction
• Debug pods
• Deprecated API migration guide
• Kubernetes v1.36 release announcement
• Deprecation and removal of Service externalIPs

Part one of three. This is a thirty-minute review of the Beginner Detailed track, followed by the questions interviewers actually ask a junior engineer about Kubernetes. Mid-level reviews this plus its own material; Senior reviews all three. Everything here matches Kubernetes 1.37, the current minor as of August 2026. If an interviewer quotes an older tutorial, you will be able to say what changed.

Interviewers at this level are not checking whether you have memorised flags. They want to know if you understand why Kubernetes behaves the way it does. A candidate who can explain why a deleted pod comes back will be trusted with a cluster. One who can only recite kubectl commands will not.

The thirty-second answer

Kubernetes is an open-source system for running containers across a group of machines and keeping them running. You declare the state you want, such as "three copies of this image behind one address", and controllers keep comparing that desired state with what actually exists and act to close the gap: they place, restart, replace, scale and roll out containers for you.

Then add the line that shows you have used it: "the key idea is the reconcile loop. Recovery, scaling and deployment are all the same mechanism, which is why you change the manifest rather than fighting individual pods."

If they ask for one more sentence, give the division of labour: Docker (or any OCI tool) builds the image; Kubernetes runs it at scale. Kubernetes does not replace your Dockerfile. It starts where docker push ends.

The model

Know these terms well enough to define each in one breath. Interviewers often ask for two of them side by side, so the contrasts matter as much as the definitions.

The architecture in one minute

Interviewers love "what happens when I run kubectl apply?" because the answer tests the whole model at once. Walk it like this:

1. kubectl sends an HTTPS request to the kube-apiserver. The API server authenticates you, checks you are authorised, validates the object and stores it in etcd. The command returns now, before anything runs.
2. The Deployment controller (in the kube-controller-manager) sees a new Deployment and creates a ReplicaSet. The ReplicaSet controller sees the ReplicaSet wants three pods and creates three pod objects.
3. The kube-scheduler sees three pods with no node assigned. It filters out nodes that cannot fit them, scores the rest, and writes a node name onto each pod.
4. The kubelet on each chosen node sees a pod assigned to it, asks the container runtime (containerd or CRI-O) to pull the image and start the containers, runs the probes, and reports status back to the API server.

The line that earns credit: no component gives orders to another. Each one watches the API server and acts on its own. Only the API server talks to etcd. That is why a single component can restart without losing anything.

A follow-up you should expect: "Does Kubernetes use Docker?" Answer that the dockershim was removed in Kubernetes 1.24, so nodes use containerd or CRI-O through the Container Runtime Interface. Images built with Docker still run unchanged, because they are standard OCI images. Only running Docker Engine as the node's runtime went away, and even that is possible through the separate cri-dockerd adapter.

Pod versus Deployment versus Service

The reason all three exist is that each solves the problem the previous one creates. A pod runs your container but dies alone. A Deployment keeps pods alive and replaces them, so their IPs keep changing. A Service gives the changing set one address that never changes.

The commands to have ready

kubectl config get-contexts                  # which clusters can I reach? * marks the current one
kubectl config set-context --current --namespace=dev
kubectl get pods -A -o wide                  # every namespace, with node and IP
kubectl describe pod <pod>                   # details plus Events at the bottom
kubectl logs <pod> --previous                # output of the container that crashed
kubectl exec -it <pod> -- sh                 # shell inside a running container
kubectl apply -f k8s/                        # make the cluster match these files
kubectl create deployment web --image=nginx:1.27 --dry-run=client -o yaml > web.yaml
kubectl expose deployment web --port=80 --target-port=80
kubectl port-forward svc/web 8080:80         # local access while developing
kubectl scale deploy/web --replicas=5
kubectl set image deploy/web nginx=nginx:1.28
kubectl rollout status deploy/web
kubectl rollout undo deploy/web
kubectl explain deployment.spec.strategy     # field docs from the cluster itself
kubectl get endpointslices -l kubernetes.io/service-name=web

Two version notes interviewers sometimes probe. kubectl get endpoints now prints a deprecation warning, because the v1 Endpoints API was deprecated in 1.33 in favour of EndpointSlices. And kubectl run -f was deprecated in 1.37: use kubectl run NAME --image=... for one-off pods and kubectl apply -f for files.

Manifests and labels

Every object has apiVersion, kind, metadata and spec, and the cluster adds status. Say plainly that spec is what you want and status is what the controllers observed. It is the reconcile loop in data form.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web            # which pods this Deployment owns
  template:
    metadata:
      labels:
        app: web          # must match the selector
    spec:
      containers:
        - name: web
          image: nginx:1.27
          ports:
            - containerPort: 80

Three facts to state unprompted. The selector must match the template's labels, or the API server rejects the Deployment. The selector is immutable after creation, so changing it means deleting and recreating. And objects connect by labels, not names: a Service with a selector that matches nothing is valid and silently routes nowhere.

Services and DNS

port is where the Service listens; targetPort is where the container listens. From the same namespace a Service is reachable as web; from another as web.<namespace>; fully qualified as web.<namespace>.svc.cluster.local. The Service only sends traffic to pods that match its selector and are ready, which is where readiness probes enter the story.

For HTTP routing by host and path, know that there are two APIs. Ingress is frozen (it still works, gets no new features), and the widely used Ingress NGINX controller was retired on 24 March 2026 with no further security patches. Gateway API (Gateway plus HTTPRoute) is the recommended successor. Saying this unprompted signals that your knowledge is current.

Configuration, resources and probes

The traps, and why they happen

Most beginner failures trace back to three misunderstandings: forgetting that the system is declarative (you fight the controller instead of changing the spec), forgetting that objects connect by labels (so a typo breaks the connection silently), and forgetting that apply returning means accepted, not running (so you assume success too early).

The debugging routine

1. Getkubectl get pods. Read STATUS, READY and RESTARTS.
2. Describekubectl describe pod <pod>. Read the Events at the bottom, and Last State for the exit code and reason.
3. Logskubectl logs <pod>, and --previous after a restart. Application errors live here, not in Events.
4. Insidekubectl exec -it <pod> -- sh, or a throwaway busybox pod to test DNS and networking.
5. Widerkubectl get events --sort-by=.lastTimestamp and kubectl describe node for capacity problems.

Say the distinction that separates good answers from weak ones: events are the cluster talking; logs are your application talking. An image that never pulled has no logs. An app that crashed on a bad setting shows only "back-off" in events. You need both.

Common interview questions

1. What is Kubernetes, and why would a team use it?It runs containers across many machines and keeps them in the state you declare: placing them, restarting and replacing failed ones, scaling them, giving them stable addresses and rolling out new versions without downtime. Before it, those jobs were done by people and scripts, differently in every team. Kubernetes makes them a platform responsibility, done the same way everywhere.
2. What does "declarative" mean here?You describe the end state, such as three replicas of an image, not the steps to reach it. Controllers keep comparing desired and actual state and act to close the gap. Applying the same manifest twice changes nothing, and drift is corrected automatically.
3. What is a pod, and why not just run containers?A pod is the smallest unit Kubernetes schedules: one or more containers that share a network namespace (one IP), can share volumes, and always land on the same node. It lets a helper such as a log shipper or proxy live and die with the main container. Most pods hold one container.
4. Pod versus Deployment?A pod is one running instance with no self-healing of its own. A Deployment manages a ReplicaSet that keeps N identical pods alive and rolls out template changes gradually. For anything that must stay up, create a Deployment.
5. What does a ReplicaSet do, and why do you rarely create one?It keeps exactly N pods matching its template running. A Deployment creates a new ReplicaSet for each template change and shifts pods between them, which is how rolling updates and rollbacks work. Creating one directly loses that.
6. Why does a pod I delete come back?Its ReplicaSet still wants N pods. Deleting a pod changes the actual state, not the desired state, so the controller creates a replacement. To remove it for good, scale the Deployment down or delete it.
7. What is a Service, and why is it needed?Pods are replaced with new IPs all the time. A Service gives a set of pods, chosen by label selector, one stable virtual IP and DNS name, and load-balances across the ready ones. Clients call the Service, never pod IPs.
8. Explain the Service types.ClusterIP is internal only and the default. NodePort opens a port in 30000–32767 on every node. LoadBalancer asks the cloud for an external load balancer. ExternalName is a DNS alias to something outside the cluster.
9. My Service returns nothing. How do you debug it?Check its EndpointSlices with kubectl get endpointslices -l kubernetes.io/service-name=web. If they are empty, the selector matches no ready pods: compare the selector with kubectl get pods --show-labels, then check the pods' readiness. If endpoints exist, check targetPort against the port the container actually listens on.
10. How do pods find each other?Cluster DNS (CoreDNS) gives every Service a name. From the same namespace it is just web; from another it is web.namespace, fully web.namespace.svc.cluster.local.
11. ConfigMap versus Secret, and is a Secret secure?Both hold configuration outside the image and are consumed as environment variables or files. A Secret is for confidential values and has its own permissions, but its data is only base64-encoded, and it is not encrypted in etcd unless the operator enables encryption at rest. Anyone who can read it can decode it. Never commit real Secret manifests to Git.
12. I changed a ConfigMap and the app did not notice. Why?Environment variables are read once, at container start. Mounted files update after a delay, but only help if the app re-reads them. Run kubectl rollout restart deploy/NAME to replace the pods.
13. Requests versus limits?Requests are what the scheduler reserves when placing the pod. Limits are the ceiling enforced at run time. Going over a CPU limit throttles the container; going over a memory limit gets it OOM-killed with exit code 137. A pod with neither is BestEffort and first to be evicted under memory pressure.
14. Liveness versus readiness probe?Readiness decides whether a pod gets traffic: failing removes it from the Service without restarting it. Liveness decides whether it is stuck: failing restarts the container. A liveness probe that checks a shared dependency like a database can restart every pod at once, so keep it local and cheap, and use a startup probe for slow starters.
15. How does a rolling update work, and how do you roll back?Changing the pod template makes the Deployment create a new ReplicaSet and shift pods over gradually, limited by maxSurge and maxUnavailable (both 25% by default), waiting for new pods to become ready. If the new version never becomes ready, old pods keep serving. kubectl rollout undo switches back to the previous ReplicaSet; then fix the manifest so the next apply does not reintroduce the bug.
16. A pod is in CrashLoopBackOff. What do you do?It means the container keeps exiting and Kubernetes is waiting longer between restarts, from 10 seconds doubling to 5 minutes. Read kubectl logs POD --previous for the crashed container's output, and describe for the exit code and reason. Common causes are a missing setting, a wrong command, a dependency not reachable at startup, or a liveness probe killing a slow app.
17. A pod is stuck in Pending. Why?The scheduler cannot place it. The FailedScheduling event says why for every node: Insufficient cpu or memory, an untolerated taint, a node selector no node matches, or a PersistentVolumeClaim that is not bound.
18. What is a namespace, and is it a security boundary?A named scope inside a cluster: names must be unique within it, and permissions, quotas and policies are applied per namespace. It is not a security boundary on its own; isolation needs RBAC, network policies and Pod Security on top.
19. What is a kubeconfig context, and why does it matter?A named pairing of cluster, user and default namespace. current-context decides which cluster your next command hits. Checking it before any destructive command is the habit that keeps people from deleting things in production.
20. How do you keep data when a pod is replaced?Use a PersistentVolumeClaim, which the cluster satisfies with a PersistentVolume, usually provisioned through a StorageClass. For databases with several replicas, use a StatefulSet, which gives each replica its own claim and a stable name.
21. How do you expose an HTTP app to the internet today?A LoadBalancer Service for a single service, or HTTP routing with Gateway API: an HTTPRoute attached to a Gateway the platform team runs. Ingress still works but is frozen, and Ingress NGINX was retired in March 2026, so I would not pick it for a new cluster.
22. Which version of Kubernetes is current, and how often does it change?1.37, released 26 August 2026. There are three minor releases a year, and each gets about a year of patches, so clusters must be upgraded regularly and one minor at a time. kubectl supports API servers one minor version either side of itself.

A scenario to rehearse out loud

Many junior interviews include one small live scenario. A common one: "You deployed an API, the pods are Running, but calls to the Service time out. Walk me through it." A strong answer moves from the outside in and says what each step rules out.

First, confirm the context and namespace, because the most embarrassing failure is debugging the wrong cluster. Next, check the EndpointSlices for the Service. Empty means the selector matches no ready pods, so compare labels and look at the READY column. If endpoints exist, compare the Service's targetPort with the port the app listens on, visible in the pod spec and in the app's startup log. Then test from inside the cluster with a throwaway busybox pod: wget -qO- http://api:80 isolates the Service from anything outside it, and wget to a pod IP isolates the pod from the Service. Finally, if the app listens on 127.0.0.1 rather than 0.0.0.0, nothing from outside its own pod can reach it, which is the same trap as in Docker.

The interviewer is less interested in which step finds the bug than in whether each step has a reason. Say the reason as you go.

Sixty-second self-test

• Give the thirty-second answer, including the sentence about the reconcile loop.
• Trace kubectl apply of a Deployment from your terminal to a running container, naming each component.
• Explain pod, ReplicaSet, Deployment and Service, and the problem each one solves.
• Say why a deleted pod comes back and how to make it go away for good.
• Explain how a Service finds its pods, and the first command you run when it has none.
• Tell port from targetPort, and give a Service's DNS name from another namespace.
• Say whether a Secret is encrypted, and what to do after changing a ConfigMap.
• Explain requests versus limits, and what exit code 137 means.
• Explain readiness versus liveness, and how a bad liveness probe causes an outage.
• Describe a rolling update, and the step people forget after rollout undo.
• Name the command for a crash-looping pod's real error, and where to read why a pod is Pending.
• Say what happened to Ingress NGINX and what replaces it.

Part one of three. Most beginner trouble with Kubernetes comes from three misunderstandings. The first is fighting the controller instead of changing the desired state. The second is forgetting that objects find each other by labels, so one typo disconnects them without an error. The third is reading "created" as "running", when kubectl apply only means the request was accepted. Start with the table of real mistakes below, then the habits, the practice cards and the checklist. Everything matches Kubernetes 1.37.

The mistakes beginners actually make

Each row is a failure that shows up in almost every newcomer's first weeks. Read the symptom column first, because that is what you will see. The cause column is the part to remember.

Three of those deserve a longer explanation, because they cost beginners the most hours.

The silent selector. Kubernetes connects Deployments to pods and Services to pods through label selectors. A selector that matches nothing is a perfectly valid state, so there is no error. The symptom shows up somewhere else, often as a timeout in an app calling the Service. The fix is a habit, not a command: whenever a Service misbehaves, look at its EndpointSlices first. An empty list tells you in one second that the problem is labels or readiness, not networking.

Reading the wrong logs. When a container crashes and restarts, kubectl logs shows the new container, which may have printed nothing yet. The explanation you need is in the previous container's output. Beginners who do not know about --previous stare at an empty screen and conclude the app is not logging. Also remember that some failures produce no logs at all: an image that never pulled never ran, so the explanation is in kubectl describe, under Events.

Fighting the reconcile loop. Deleting a misbehaving pod feels like the obvious move. It works once, because the replacement is fresh. Then it fails the same way, because the replacement comes from the same template. Deleting pods treats the symptom. Change the template (the image, the configuration, the probe) and let the Deployment roll out the fix. That is also what gets recorded in Git.

Reading output without guessing

Most debugging time goes on reading the wrong part of the output, or reading only the first few words. These are the fields that answer the question.

# The five commands that settle most "why is it broken?" questions
kubectl get pods -o wide
kubectl describe pod <pod>
kubectl logs <pod> --previous
kubectl get events --sort-by=.lastTimestamp
kubectl get endpointslices -l kubernetes.io/service-name=<svc>

Exit codes carry the same meaning they had in Docker. 0 means the process finished, which in a Deployment still counts as a crash, because a server is not supposed to finish. 1 is a generic application error, so read the logs. 127 is command not found. 137 is SIGKILL, usually the memory limit. 143 is SIGTERM, a normal shutdown request.

The habits that pay off most

Two setup steps are worth ten minutes on the first day. Turn on shell completion, so pod names with random suffixes are one Tab away:

# bash (needs bash-completion v2 and Bash 4.1 or newer)
echo 'source <(kubectl completion bash)' >> ~/.bashrc
echo 'alias k=kubectl' >> ~/.bashrc
echo 'complete -o default -F __start_kubectl k' >> ~/.bashrc

# zsh (the default shell on macOS, whose Bash 3.2 is too old)
echo 'source <(kubectl completion zsh)' >> ~/.zshrc

And set a default namespace per context, so you stop typing -n and stop creating things in default by accident:

kubectl create namespace demo
kubectl config set-context --current --namespace=demo

Imperative versus declarative, in practice

The quick commands (create, scale, set image, expose) are fine for learning and for emergencies. The trouble starts when you mix them with files. The file says three replicas; you run kubectl scale --replicas=5 during a traffic spike; next week a colleague applies the file and quietly takes you back to three. The live cluster and Git have drifted, and nobody can tell which is right.

A good compromise while you learn: use imperative commands with --dry-run=client -o yaml to write files, then apply the files. You get the speed of the quick commands and the record of the declarative way.

Practice cards

Short, self-contained exercises on a kind or minikube cluster. Each takes a few minutes and ends in something you can see on screen. Do them in a throwaway namespace and delete it afterwards.

1. Watch self-healingCreate a Deployment with three replicas and run kubectl get pods -w. In another terminal, delete a pod. Note how many seconds pass before the replacement is Running, and that its name is new.
2. Orphan a pod with a labelRelabel one pod to app=debug --overwrite. Watch the Deployment create a fourth pod to replace it, while the orphan keeps running untouched. Delete the orphan with -l app=debug.
3. Empty a Service on purposePut a Service in front of the Deployment and list its EndpointSlices. Change the Service selector to a label nothing has, apply, and list them again. This is what "no endpoints" looks like.
4. Break an image and roll backSet the image to a tag that does not exist. Confirm the old pods keep serving, read the ImagePullBackOff event, then kubectl rollout undo and check kubectl get rs.
5. Find a crash's last wordsRun kubectl run crasher --image=busybox:1.36 -- sh -c "echo missing DATABASE_URL; exit 1". Compare kubectl logs crasher with kubectl logs crasher --previous, and watch the gap between restarts grow.
6. Get OOM-killedAdapt the official docs' memory example: a pod running the polinux/stress image with command: ["stress"], args: ["--vm", "1", "--vm-bytes", "250M", "--vm-hang", "1"] and a memory limit of 100Mi. Find OOMKilled and exit code 137 in describe.
7. Ask for too muchCreate a pod requesting 64 CPUs. Read the FailedScheduling event and count how many nodes were rejected and why.
8. Prove env vars are read at startLoad a ConfigMap value as an environment variable, change the ConfigMap, and read the variable with kubectl exec. It is unchanged until kubectl rollout restart.
9. Decode a SecretCreate a Secret with --from-literal, then decode it with kubectl get secret NAME -o jsonpath='{.data.KEY}' | base64 -d. That one command is why Secret manifests stay out of Git.
10. Call a Service by nameStart a throwaway pod with kubectl run tmp --image=busybox:1.36 --rm -it --restart=Never -- sh, then wget -qO- http://web and nslookup web. Repeat from another namespace using web.<namespace>.

Debugging a pod that will not start

When a pod is unhealthy, follow the same order every time. The order matters because each step rules out a class of problem before you spend effort on the next.

1. Confirm where you arekubectl config current-context and the namespace. A surprising share of "missing" objects are in another namespace or another cluster.
2. Read the statuskubectl get pods. Pending points at scheduling, ImagePullBackOff at the image, CreateContainerConfigError at configuration, CrashLoopBackOff at the app itself, and Running with 0/1 READY at the readiness probe.
3. Read the Eventskubectl describe pod <pod>, bottom section. The cluster's explanation is almost always there in one line.
4. Read the app's outputkubectl logs <pod> --previous for a crash. The stack trace or "missing setting" message lives here.
5. Get next to itkubectl exec -it <pod> -- sh if it stays up, or a busybox pod in the same namespace to test DNS and ports.
6. Compare with the filekubectl get deploy <name> -o yaml shows what the cluster actually has. It settles "but I set that" in seconds.

Things old tutorials still get wrong

Kubernetes changes three times a year, and search results lag behind it. If you copy from a tutorial, check it against this list first.

A starter layout to copy

Keep every project's manifests in one folder with a predictable shape, applied in name order. This is the same structure as the Beginner Detailed "Putting it all together" project.

k8s/
  00-namespace.yaml     # the project's namespace
  10-config.yaml        # ConfigMap: non-secret settings
  20-deployment.yaml    # the app: replicas, image, env, resources, probes
  30-service.yaml       # the stable name in front of it

kubectl apply -f k8s/00-namespace.yaml
kubectl create secret generic app-secrets -n myapp --from-literal=API_TOKEN=...   # never in Git
kubectl apply -f k8s/
kubectl rollout status deploy/web -n myapp

When you are finished experimenting, kubectl delete namespace myapp removes the lot. On kind, kind delete cluster --name learn removes the cluster itself, and kind create cluster --name learn gives you a clean one in under a minute. A fresh cluster is often faster than cleaning up a confused one.

Day-one checklist

• kubectl installed, and kubectl version shows a client within one minor of the cluster
• A local cluster (kind or minikube) you can create and destroy freely
• Shell completion on, and the current context visible before you run anything destructive
• A default namespace set for your context, and one namespace per project
• Every long-lived app runs as a Deployment, never a bare pod
• selector.matchLabels equal to the pod template's labels
• Image tags pinned, never :latest
• CPU and memory requests on every container, and a memory limit
• A readiness probe on every server; a startup probe for slow starters
• Liveness probes that check only the container itself
• Configuration in ConfigMaps, credentials in Secrets, and no Secret with real values in Git
• kubectl rollout restart after changing configuration read from the environment
• Manifests in Git, changed through files and kubectl apply, not live edits
• After any rollout undo, the file reverted too
• The debugging order memorised: get, describe, logs --previous, exec, events

Mid-level tips take each of these further: errors you only meet once apps grow up, such as quota rejections, Pod Security violations, RBAC denials, server-side apply conflicts and stuck namespaces. They also cover the practices with the most leverage (Helm or Kustomize, autoscaling, network policies, resource right-sizing) and operational recipes for draining nodes, debugging with ephemeral containers and rolling out safely.

This is part two of three. Beginner got a Deployment running, exposed it with a Service, and taught you to read kubectl describe. That is enough to ship something. It is not enough to be the person on the team who can say why a rollout stalled at 3 of 5 pods, why a pod was evicted from a node that looked half empty, or why a perfectly healthy service drops a few requests on every deploy. Those questions all have precise answers, and the answers come from knowing how the machinery behaves rather than from memorising more commands.

Everything here is written against Kubernetes 1.37 (latest patch v1.37.1). Where a behaviour changed in the last few releases, the section says so, because a lot of blog posts and Stack Overflow answers you will meet at work describe a cluster that no longer exists.

Where this picks up

Each section starts with the problem it solves and ends with a Try it you can run on a local kind or minikube cluster in a few minutes. If you do not have one, kind create cluster gives you a single-node cluster inside Docker, and every exercise here works on it unless the section says otherwise.

The control loop, precisely

Beginner described Kubernetes as "declarative": you describe the state you want and the cluster makes it so. That sentence hides the mechanism, and the mechanism is what lets you predict behaviour.

Every change you make goes to one place, the kube-apiserver. It authenticates you, checks authorisation, runs admission (the stage where policies can reject or modify your object), and then persists the object to etcd, the key-value store that holds all cluster state. Nothing else in the cluster talks to etcd directly. The scheduler, the controller manager, every kubelet and every operator you install are all clients of the API server, exactly like kubectl is.

Those clients do not poll. They open a watch: a long-lived request that says "send me every change to Pods (or Deployments, or Nodes) after version X". Every object carries a metadata.resourceVersion, an opaque string that changes whenever the object is written. A controller lists the objects it cares about once, remembers the resourceVersion, and then receives a stream of ADDED, MODIFIED and DELETED events from that point on. It keeps an in-memory copy of what it has seen, usually called an informer cache.

A controller is a loop over that cache. For each object it compares the desired state (spec) with the observed state (status, plus whatever other objects exist) and takes one step towards closing the gap. The important word is level-triggered. The controller does not react to "a pod was deleted"; it reacts to "there are currently 2 pods and there should be 3". If it misses an event, crashes, or is restarted, the next pass still computes the right action from the current level. This is why Kubernetes recovers from its own component failures so gracefully, and why "just delete the pod" works as a fix so often: you are not repairing anything, you are letting a controller observe a gap and fill it.

The chain of controllers behind a single kubectl apply of a Deployment looks like this:

No component calls the next one. Each writes an object, and the next component notices that object through its own watch. That decoupling is the whole design, and it has three consequences you will meet constantly.

Ownership and garbage collection. When the ReplicaSet controller creates a Pod, it sets metadata.ownerReferences on the Pod pointing at the ReplicaSet, which in turn points at the Deployment. The garbage collector uses those references: delete the Deployment and, by default, its ReplicaSets and Pods are deleted in the background. It is also why deleting a Pod that belongs to a ReplicaSet just produces a replacement. The Pod is not the thing you declared; it is a consequence of something you declared.

Finalizers. A finalizer is a string in metadata.finalizers that tells the API server "do not actually remove this object until someone clears this entry". Deletion sets metadata.deletionTimestamp and waits. Controllers use finalizers to clean up external resources first, such as a cloud load balancer or a storage volume. When an object is stuck in Terminating, the question is always which finalizer is still present and which controller was supposed to remove it.

Generation and observedGeneration. Every write to an object's spec increments metadata.generation. Controllers record the generation they last acted on in status.observedGeneration. If the two differ, the controller has not caught up with your latest change yet, and any status you read describes the previous spec. Since 1.35 Pods also carry metadata.generation and status.observedGeneration, which matters for in-place resize. When a script checks "is the rollout done?", it must check observedGeneration first, and kubectl rollout status does exactly that for you.

kubectl get deploy web -o jsonpath='{.metadata.generation} {.status.observedGeneration}{"\n"}'
kubectl get pod -l app=web -o jsonpath='{range .items[*]}{.metadata.name}{"  owner="}{.metadata.ownerReferences[0].kind}/{.metadata.ownerReferences[0].name}{"\n"}{end}'

4 4
web-7c9d8f6b5d-2xk9p  owner=ReplicaSet/web-7c9d8f6b5d
web-7c9d8f6b5d-q8lrm  owner=ReplicaSet/web-7c9d8f6b5d

One recent change is worth knowing here. Since 1.37 the API server may answer with HTTP 429 and a Retry-After header while its watch cache is warming up after a restart. Client libraries handle this, but a homegrown script that calls the API with curl in a tight loop needs to back off rather than treat 429 as a failure.

How a Deployment rollout really works

A Deployment does not manage Pods. It manages ReplicaSets, and each ReplicaSet manages Pods for exactly one version of your pod template. When you change anything inside spec.template (an image, an env var, an annotation), the Deployment controller hashes the new template, creates a new ReplicaSet whose name ends in that hash, and labels its pods with pod-template-hash. Changes outside the template, such as replicas, do not create a new ReplicaSet.

A rollout is then just the controller moving replicas from the old ReplicaSet to the new one in steps, bounded by two numbers in spec.strategy.rollingUpdate:

• maxSurge: how many pods above the desired count may exist during the rollout. Default 25%, rounded up.
• maxUnavailable: how many pods below the desired count may be unavailable. Default 25%, rounded down.

With 4 replicas and the defaults, maxSurge is 1 and maxUnavailable is 1, so the controller may run up to 5 pods and must keep at least 3 available. With 3 replicas, maxSurge rounds up to 1 and maxUnavailable rounds down to 0, so the rollout proceeds strictly one new pod at a time and can never go below 3. That rounding is why small Deployments roll out more slowly than you expect, and why a Deployment with 1 replica and the defaults never has a moment with zero pods.

"Available" has a precise meaning: the pod is Ready (its readiness probe passes) and has stayed Ready for minReadySeconds (default 0). The rollout only advances as new pods become available. This single fact explains most stuck rollouts. If the new version never becomes Ready, because of a crash, a failing readiness probe, a missing Secret or an image that cannot be pulled, the controller creates as many new pods as maxSurge allows and then waits. The old pods keep serving. Nothing breaks, and nothing progresses.

It waits for spec.progressDeadlineSeconds, default 600 seconds. After that the Deployment gets a condition Progressing=False with reason ProgressDeadlineExceeded. Kubernetes does not roll back automatically. The condition exists so that your CI or GitOps tool can notice and act.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  labels:
    app: web
spec:
  replicas: 4
  revisionHistoryLimit: 5
  progressDeadlineSeconds: 300
  minReadySeconds: 5
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - name: web
        image: nginx:1.27
        ports:
        - containerPort: 80
        readinessProbe:
          httpGet:
            path: /
            port: 80
          periodSeconds: 5

maxUnavailable: 0 with maxSurge: 1 is the conservative choice for a service that must never lose capacity: add one new pod, wait for it to be available, remove one old pod, repeat. It needs one pod's worth of spare room in the cluster, so on a tightly packed cluster the surge pod can sit in Pending, which stalls the rollout just as surely as a crash does.

The commands that go with this:

kubectl rollout status deploy/web --timeout=5m     # exits non-zero if it does not complete
kubectl rollout history deploy/web
kubectl rollout history deploy/web --revision=3    # shows the template of that revision
kubectl rollout undo deploy/web --to-revision=3
kubectl rollout pause deploy/web                   # batch several edits into one rollout
kubectl rollout resume deploy/web
kubectl get rs -l app=web                          # old ReplicaSets are kept at 0 replicas

Undo works because old ReplicaSets are kept, scaled to zero, up to revisionHistoryLimit (default 10). An undo is simply the Deployment controller scaling an old ReplicaSet back up. That also means an undo is a new rollout with the same surge rules, not an instant switch.

kubectl rollout restart deserves a note. It sets the annotation kubectl.kubernetes.io/restartedAt in the pod template, which changes the template hash and therefore triggers a normal rolling update with brand-new pods. It is the correct way to pick up a changed ConfigMap that is consumed as environment variables, which, as a later section explains, never update in place.

When you need more than rolling updates, such as canaries with traffic weights or automatic rollback on error rates, that is the job of a progressive delivery controller like Argo Rollouts or Flagger, usually alongside a GitOps tool. See the Argo CD guide for the GitOps side.

Scheduling, precisely

A pod whose spec.nodeName is empty is unscheduled. The kube-scheduler watches for those, picks a node, and writes a binding. It makes the choice in two phases.

Filtering removes every node that cannot run the pod at all. A node fails the filter if the pod's requests do not fit in the node's allocatable capacity minus the requests of pods already on it, if the node has a taint the pod does not tolerate, if the pod's nodeSelector or required node affinity does not match the node's labels, if a required volume cannot attach there, or if a hard topology constraint would be violated.

Scoring ranks the survivors, preferring nodes that spread replicas, satisfy preferred affinities, and balance resource use. The highest score wins.

The phrase that trips people up is "requests, not usage". The scheduler never looks at how much CPU a node is actually using. A node running pods that request 3.5 of its 4 allocatable CPUs is nearly full to the scheduler even if those pods are idle. Equally, a node whose pods have no requests at all looks empty even if it is at 100% CPU. This is why unset requests are the root of both "Pending with half the cluster idle" and "everything landed on one node and fell over".

When nothing passes the filter, the pod stays Pending and the scheduler records an event whose message lists why each node was rejected:

Warning  FailedScheduling  default-scheduler  0/3 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }, 2 Insufficient cpu. preemption: 0/3 nodes are available: 1 Preemption is not helpful for scheduling, 2 No preemption victims found for incoming pod.

Read it as a tally: three nodes, one excluded by a taint, two by CPU requests. The preemption: suffix says the scheduler also tried evicting lower-priority pods and could not make room.

The tools for steering placement, from bluntest to most expressive:

Taints and tolerations are the one people most often get backwards. A taint on a node repels pods. A toleration on a pod lets it ignore that repulsion, but does not pull it towards the node. To dedicate a GPU pool you therefore need both: a taint on the GPU nodes so ordinary pods stay off, and a nodeSelector or node affinity on the training pods so they go there.

For spreading replicas, prefer topologySpreadConstraints over pod anti-affinity. Anti-affinity with requiredDuringScheduling is all-or-nothing: with three zones and a fourth replica, the fourth pod can never schedule. A spread constraint expresses the actual intent, "keep the counts within one of each other":

spec:
  topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app: web
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: web

The first constraint spreads across zones but will still schedule if a zone is full (ScheduleAnyway). The second insists on spreading across nodes. On a managed cluster in a Middle East region that offers several availability zones, the zone constraint is what makes "we are multi-AZ" true for your workload and not only for the node pool.

Priority and preemption close the loop. A PriorityClass gives pods an integer priority. When a high-priority pod cannot be scheduled, the scheduler may evict lower-priority pods to make room. This is valuable (your API should beat a nightly batch job) and dangerous (a mis-set priority can evict production). Keep the number of classes small and document them.

kubectl get pods --field-selector=status.phase=Pending -A
kubectl events --for pod/web-7c9d8f6b5d-2xk9p --types=Warning
kubectl describe node <node> | sed -n '/Allocated resources/,/Events/p'
kubectl get nodes -L topology.kubernetes.io/zone,node.kubernetes.io/instance-type

The Allocated resources block of kubectl describe node is the scheduler's view: the sum of requests and limits on that node as a percentage of allocatable. Compare it with kubectl top node, which is actual usage, and the gap between the two is your waste or your risk.

Resources, QoS and eviction

Beginner said "set requests and limits". The reason to set them carefully is that the two numbers do completely different jobs, enforced by different components, with different failure modes.

Requests are a scheduling promise. The scheduler uses them to decide placement, and the kubelet uses them to divide CPU time under contention: a container that requests 500m (half a core) is guaranteed roughly that share when the node is busy. A container may use more than its request whenever spare capacity exists.

Limits are a ceiling enforced by the Linux kernel through cgroups, and the enforcement differs by resource:

• CPU is compressible. A container that hits its CPU limit is throttled: the kernel stops scheduling it for the rest of each accounting period. Nothing crashes. Latency rises, often in a spiky pattern that looks like a slow dependency.
• Memory is not compressible. A container that exceeds its memory limit is killed by the kernel OOM killer. kubectl describe pod shows Last State: Terminated, Reason: OOMKilled, Exit Code: 137, and the restart count goes up.

That asymmetry drives a common production stance: always set memory requests and limits, usually equal, because an unbounded memory leak on a shared node hurts everyone; set CPU requests accurately, and treat CPU limits as a deliberate choice rather than a default, because they turn spare capacity into self-inflicted latency. Teams differ here, and a namespace LimitRange or policy may force CPU limits on you. What matters is that you can explain the trade-off.

resources:
  requests:
    cpu: 250m
    memory: 512Mi
  limits:
    memory: 512Mi

QoS classes

From the requests and limits of every container in a pod, Kubernetes derives a Quality of Service class, visible at status.qosClass:

The class matters when a node runs short, which is a different event from a container hitting its own limit. The kubelet watches node signals such as memory.available and nodefs.available against eviction thresholds. The defaults on Linux are memory.available<100Mi, nodefs.available<10% and imagefs.available<15%. When a threshold is crossed, the kubelet evicts pods to reclaim resources, starting with pods using the most above their requests. The pod ends in phase Failed with reason Evicted and a message such as The node was low on resource: memory. A controller then creates a replacement elsewhere.

Two cluster-side objects enforce discipline per namespace. A LimitRange sets default requests and limits for containers that do not declare them, plus minimums and maximums. A ResourceQuota caps the namespace total. Once a quota covers a resource, every pod in that namespace must declare it, or admission rejects it with must specify limits.memory or similar; a LimitRange with defaults is what keeps that from breaking every hand-written manifest.

apiVersion: v1
kind: LimitRange
metadata:
  name: defaults
  namespace: team-a
spec:
  limits:
  - type: Container
    defaultRequest:
      cpu: 100m
      memory: 128Mi
    default:
      memory: 256Mi
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute
  namespace: team-a
spec:
  hard:
    requests.cpu: "8"
    requests.memory: 16Gi
    limits.memory: 32Gi
    pods: "50"

When the quota is exhausted, the error names every number you need: pods "x" is forbidden: exceeded quota: compute, requested: requests.cpu=500m, used: requests.cpu=7800m, limited: requests.cpu=8. Note that a Deployment will not show this error on its own status line; the ReplicaSet will. Run kubectl describe rs when a Deployment has fewer pods than expected and no pods are Pending.

In-place resize

Until recently, changing a pod's resources meant replacing the pod. Since 1.35, in-place pod resize is GA: you patch the resize subresource and the kubelet adjusts the cgroup without restarting the container, where the runtime and resource allow it.

kubectl patch pod web-7c9d8f6b5d-2xk9p --subresource resize \
  -p '{"spec":{"containers":[{"name":"web","resources":{"requests":{"cpu":"800m"}}}]}}'
kubectl get pod web-7c9d8f6b5d-2xk9p -o jsonpath='{.status.containerStatuses[0].resources}'

This is most useful for single long-running pods such as a notebook server or a training job that needs more CPU for a phase. For Deployments, you still change the template, because a resized pod is replaced by the original template at the next rollout. In-place resize, like memory QoS, needs cgroup v2, which since 1.35 the kubelet requires by default anyway.

Probes, termination and zero-downtime deploys

The problem this section solves is the one every team notices eventually: a service with healthy pods and a clean rolling update still returns a handful of 502s or connection resets on each deploy. The cause is the pod shutdown sequence, and the fix is small once you see it.

First, the three probes, precisely:

The defaults are periodSeconds: 10, timeoutSeconds: 1, failureThreshold: 3, successThreshold: 1. That one-second timeout is tighter than it looks for an endpoint that touches a database.

Two rules keep probes from causing outages. A liveness probe should check only the process itself, never a dependency: if the database is down and every pod's liveness probe checks the database, Kubernetes restarts your entire fleet in a loop and turns a database incident into an application outage. And a slow-starting app, a model server loading a few gigabytes of weights for example, needs a startupProbe with a generous budget rather than a large initialDelaySeconds on liveness, because the startup probe stops mattering as soon as it succeeds.

The termination sequence

When a pod is deleted, whether by a rollout, a scale-down or a drain, two things begin at the same time:

1. The pod is marked terminating, the EndpointSlice controller removes it from the Service's endpoints, and every node's kube-proxy (and every ingress or gateway) eventually updates its routing.
2. The kubelet runs the container's preStop hook, if any, and then sends SIGTERM to the main process.

The grace period, terminationGracePeriodSeconds (default 30), covers both the preStop hook and the time after SIGTERM. When it runs out, the kubelet sends SIGKILL.

The race is between step 1 and step 2. Endpoint removal is propagated across the cluster asynchronously and takes a moment. If your application handles SIGTERM by closing its listener immediately, it stops accepting connections while some load balancers are still sending it traffic. Those are your 502s.

The fix is to delay SIGTERM by a few seconds so routing catches up first, then shut down gracefully. Since 1.34 there is a built-in sleep action for exactly this, so you no longer need a sleep binary in the image:

spec:
  terminationGracePeriodSeconds: 45
  containers:
  - name: api
    image: registry.example.com/api:1.8.2
    lifecycle:
      preStop:
        sleep:
          seconds: 10
    readinessProbe:
      httpGet:
        path: /ready
        port: 8080
      periodSeconds: 5
    livenessProbe:
      httpGet:
        path: /live
        port: 8080
      periodSeconds: 10
    startupProbe:
      httpGet:
        path: /live
        port: 8080
      periodSeconds: 5
      failureThreshold: 60

Budget the grace period as preStop plus your longest request plus a margin. Here: 10 seconds of drain, up to 30 seconds to finish in-flight requests, 45 in total. The application still has to handle SIGTERM properly, which means running as PID 1 in exec form; the Docker guide covers why a shell-form CMD swallows the signal.

Native sidecars

A sidecar, such as a log shipper, a proxy or a token refresher, used to be just a second container, with two chronic problems: it could start after the app needed it, and in a Job it kept running after the main container finished, so the Job never completed. Since 1.33 the proper form is GA: an init container with restartPolicy: Always. It starts before the app containers, keeps running alongside them, is restarted if it dies, and is stopped after the app containers during termination.

spec:
  initContainers:
  - name: log-shipper
    image: fluent/fluent-bit:3.2
    restartPolicy: Always
    volumeMounts:
    - name: logs
      mountPath: /var/log/app
  containers:
  - name: trainer
    image: registry.example.com/trainer:2.1.0
    volumeMounts:
    - name: logs
      mountPath: /var/log/app
  volumes:
  - name: logs
    emptyDir: {}

For ML batch jobs this is the difference between a Job that completes and one that hangs at 1/2 containers ready forever.

Service networking under the hood

A Service is a stable name and virtual IP in front of a changing set of pods. Beginner used it; here is how the traffic actually gets there, because every "the Service does not work" ticket is a failure at one of these steps.

Selection. The Service's spec.selector is a label query. The EndpointSlice controller watches pods matching it and writes EndpointSlice objects listing the IP and port of each one, plus whether each is ready. Only ready pods receive traffic. The older v1 Endpoints API is deprecated since 1.33 and kubectl get endpoints prints a warning; use EndpointSlices.

kubectl get endpointslices -l kubernetes.io/service-name=web -o wide
kubectl get pods -l app=web -o wide          # compare IPs with the slice

If the slice is empty, the selector matches nothing, or nothing it matches is Ready. That is the whole diagnosis for most "connection refused on the Service" incidents: a label typo or a failing readiness probe.

Routing. On every node, kube-proxy watches Services and EndpointSlices and programs the kernel so that packets to the Service's ClusterIP are rewritten to one of the pod IPs. Its modes are iptables, nftables (the recommended mode on modern Linux) and ipvs, which is deprecated since 1.35 and will be removed in a future release. Some CNI plugins, Cilium for example, replace kube-proxy entirely with their own datapath. The ClusterIP is not bound to any interface anywhere, which is why you cannot ping it and why a successful ping never proves a Service works. Test with the actual protocol.

Naming. CoreDNS gives each Service the name <service>.<namespace>.svc.cluster.local. A pod's /etc/resolv.conf contains search domains for its own namespace and options ndots:5, so the short name web resolves inside the same namespace, and web.payments from another. The ndots:5 setting means any name with fewer than five dots is first tried against every search domain. A lookup of api.openai.com from a pod therefore tries several cluster-local names before the real one. For chatty outbound clients this adds DNS load and latency; writing the name fully qualified with a trailing dot, or lowering ndots in spec.dnsConfig.options, avoids it.

Headless Services (clusterIP: None) skip the virtual IP. DNS returns the individual pod IPs instead, and with a StatefulSet each pod gets its own stable name such as postgres-0.postgres.db.svc.cluster.local. This is what clustered databases and distributed training frameworks use to find their peers.

Topology. Traffic to a ClusterIP is spread across all ready endpoints, including those in other zones, which costs latency and, on most clouds, cross-zone transfer fees. spec.trafficDistribution: PreferSameZone asks the dataplane to keep traffic in the client's zone when healthy endpoints exist there. The older value PreferClose means the same thing and is deprecated; PreferSameNode goes further.

Getting traffic in: Gateway API

Here the ecosystem changed under everyone's feet. The community ingress-nginx controller was retired on 24 March 2026 and no longer receives releases or security patches. Existing installs still run, but it is not something to deploy in a new cluster. The Ingress API itself is frozen, not removed. The recommended path for new work is Gateway API (gateway.networking.k8s.io/v1, latest release v1.6.2), which splits responsibility by role:

• A GatewayClass names an implementation (Envoy Gateway, Istio, Cilium, NGINX Gateway Fabric, Traefik, Kong and others). Installed by the platform team.
• A Gateway is an actual listener: ports, hostnames, TLS certificates. Owned by the cluster operator.
• An HTTPRoute (or GRPCRoute, TLSRoute, TCPRoute, UDPRoute) attaches to a Gateway and routes to Services. Owned by the application team.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: web
  namespace: shop
spec:
  parentRefs:
  - name: public
    namespace: gateway-system
  hostnames:
  - shop.example.com
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /api
    backendRefs:
    - name: api
      port: 8080
      weight: 90
    - name: api-canary
      port: 8080
      weight: 10
  - backendRefs:
    - name: web
      port: 80

The weighted backendRefs give you a basic canary with no extra tooling, something Ingress could only do through controller-specific annotations. Whether a route attached successfully is recorded in its status; kubectl describe httproute web and look for Accepted and ResolvedRefs conditions. A route in one namespace attaching to a Gateway in another only works if the Gateway's allowedRoutes permits it, which is the most common reason a first HTTPRoute is silently ignored.

Configuration management that scales past one environment

Once an application runs in dev, staging and production, the questions change from "how do I set an env var" to "how do I keep three environments' worth of YAML from drifting apart", "why didn't my config change take effect", and "why did my deploy just undo someone's fix". Each has a mechanical answer.

How ConfigMap and Secret changes propagate

The behaviour depends entirely on how the pod consumes the object:

So editing a ConfigMap is not a deploy. For env-var consumption you must restart the pods (kubectl rollout restart), and for files the application must notice the change and reload, which most do not. The robust pattern is to treat configuration as immutable and versioned: give each ConfigMap a name with a content hash, reference that name from the pod template, and let the name change trigger a normal rolling update. Kustomize's configMapGenerator does exactly this, appending a hash suffix and rewriting references automatically, and Helm charts commonly put a checksum of the config in a pod annotation for the same effect.

Kustomize overlays

Kustomize is built into kubectl (kubectl apply -k, kubectl kustomize). It keeps one base of plain YAML and small overlays per environment that patch it. No templating language, so the base stays valid Kubernetes YAML you can apply directly.

deploy/
├── base/
│   ├── kustomization.yaml
│   ├── deployment.yaml
│   └── service.yaml
└── overlays/
    ├── staging/
    │   └── kustomization.yaml
    └── prod/
        ├── kustomization.yaml
        └── replicas.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: shop-prod
resources:
- ../../base
images:
- name: registry.example.com/api
  newTag: sha-3f9c2e1
configMapGenerator:
- name: api-config
  literals:
  - LOG_LEVEL=info
  - FEATURE_RECS=true
patches:
- path: replicas.yaml

kubectl kustomize deploy/overlays/prod | less     # render without applying
kubectl diff -k deploy/overlays/prod              # what would change in the cluster
kubectl apply -k deploy/overlays/prod

Kustomize fits applications your team owns end to end. Helm fits software you install from others, or when you need real parameterisation and a release history. Many teams use both: Helm for third-party charts, Kustomize for their own services. The Kustomize guide and Helm guide go deep on each.

Server-side apply and field ownership

Classic kubectl apply is client-side: kubectl stores your last applied YAML in an annotation (kubectl.kubernetes.io/last-applied-configuration), computes a three-way diff locally, and sends a patch. It works, but it has no idea that other actors, such as an HPA managing replicas or a controller setting annotations, also own parts of the object. The classic symptom: your manifest says replicas: 3, the HPA has scaled to 12, and your next deploy slams it back to 3 during peak traffic.

Server-side apply moves the merge to the API server and records, per field, which field manager set it, in metadata.managedFields. If you apply a value for a field another manager owns, you get a conflict instead of a silent overwrite:

error: Apply failed with 1 conflict: conflict with "kubectl-client-side-apply" using apps/v1: .spec.replicas
Please review the fields above--they currently have other managers.

kubectl apply --server-side --field-manager=ci -f deploy/
kubectl apply --server-side --field-manager=ci --force-conflicts -f deploy/   # take ownership deliberately
kubectl get deploy web -o yaml --show-managed-fields | less

The practical rules: give each automated writer its own --field-manager name so ownership is legible; remove replicas from the manifest of any Deployment that an HPA controls, so nobody owns it but the autoscaler; and use --force-conflicts only when you have read the conflict and mean to take the field. Since 1.36, the kubectl preferences file ~/.kube/kuberc (Beta, on by default) can make server-side apply your default so you do not have to remember the flag:

apiVersion: kubectl.config.k8s.io/v1beta1
kind: Preference
defaults:
- command: apply
  options:
  - name: server-side
    default: "true"

kuberc holds preferences only, never credentials; those stay in kubeconfig.

Autoscaling and disruption

Scaling in Kubernetes happens at three layers, and people routinely mix them up. The HorizontalPodAutoscaler changes the number of pods. The VerticalPodAutoscaler, an add-on, recommends or sets requests per pod. A node autoscaler (Cluster Autoscaler or Karpenter, both add-ons, or your managed service's equivalent) adds nodes when pods are Pending and removes underused ones. An HPA that scales pods into a full cluster only produces Pending pods unless the node layer follows.

The HPA algorithm

The HPA controller runs every 15 seconds. For each metric it computes:

desiredReplicas = ceil( currentReplicas × currentMetricValue / targetValue )

With CPU, the metric is utilisation as a percentage of the request, averaged across pods. Four pods at 90% against a 60% target gives ceil(4 × 90/60) = 6. Two consequences: without CPU requests, utilisation is undefined and the HPA reports <unknown>; and a request set far too low makes normal load look like 300% utilisation. It ignores changes within a tolerance of 10% to avoid flapping; since 1.37 that tolerance is configurable per HPA. With several metrics, it computes a desired count for each and takes the largest.

Scaling speed is shaped by spec.behavior. By default scale-up is immediate (the larger of doubling or adding 4 pods per 15 seconds) and scale-down waits for a 5-minute stabilisation window, using the highest recommendation seen in that window. That is why a load test shows a fast scale-up and a scale-down that seems to be stuck.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 3
  maxReplicas: 30
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 65
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 600
      policies:
      - type: Percent
        value: 20
        periodSeconds: 60

This scales down at most 20% of pods per minute, after ten minutes of consistently lower demand. For a service with bursty traffic after prayer times or at the end of a Ramadan fast, a slow scale-down is cheap insurance against scaling in just before the next spike.

kubectl get hpa api -w
kubectl describe hpa api        # conditions: AbleToScale, ScalingActive, ScalingLimited
kubectl top pods -l app=api     # needs metrics-server

CPU and memory metrics come from metrics-server through metrics.k8s.io, which went GA as v1 in 1.37. Queue depth, requests per second or GPU utilisation need a custom or external metrics adapter, often Prometheus Adapter or KEDA.

Scale to zero. Since 1.37 HPA scale-to-zero is Beta and on by default: minReplicas: 0 is accepted, but only with Object or External metrics, not CPU or memory, because a Deployment with no pods has no CPU to measure. The HPA sets a ScaledToZero condition. This is useful for queue consumers and GPU inference workers that sit idle overnight; KEDA remains the more complete option for event-driven scaling.

PodDisruptionBudgets

Autoscalers, node upgrades and kubectl drain all remove pods on purpose. A PodDisruptionBudget (policy/v1) tells the cluster how many of your pods may be down at once for such voluntary disruptions. It does nothing for involuntary ones such as a node crashing.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: api

kubectl drain evicts pods through the Eviction API, which checks PDBs. When an eviction would breach one, drain retries and prints Cannot evict pod as it would violate the pod's disruption budget. That is the budget working. It becomes a problem when the budget can never be satisfied: minAvailable: 3 on a 3-replica Deployment, or maxUnavailable: 0, blocks every node drain forever and stalls cluster upgrades. Prefer maxUnavailable: 1 (or a percentage), and make sure there is room elsewhere for the evicted pod to land.

kubectl get pdb -A
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data
kubectl uncordon <node>

Stateful workloads and batch jobs

Most application pods are interchangeable. Databases, message brokers and distributed training are not, and Kubernetes has distinct controllers for them.

StatefulSets and storage

A StatefulSet gives each pod a stable, ordinal identity (postgres-0, postgres-1), a stable DNS name through a headless Service, and its own PersistentVolumeClaim created from volumeClaimTemplates. If postgres-1 is rescheduled to another node, it comes back with the same name and reattaches the same volume. Pods are created in order and, with the default RollingUpdate strategy, updated in reverse order one at a time.

apiVersion: v1
kind: Service
metadata:
  name: postgres
spec:
  clusterIP: None
  selector:
    app: postgres
  ports:
  - port: 5432
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  serviceName: postgres
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
      - name: postgres
        image: postgres:17
        env:
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: pg
              key: password
        - name: PGDATA
          value: /var/lib/postgresql/data/pgdata
        volumeMounts:
        - name: data
          mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: standard
      resources:
        requests:
          storage: 20Gi

Three storage behaviours catch people. First, deleting or scaling down a StatefulSet does not delete its PVCs by default; the data survives, and so does the bill. Since 1.37 PVCs carry an Unused condition that makes orphaned claims easy to find. Second, a StorageClass with volumeBindingMode: WaitForFirstConsumer delays provisioning until the pod is scheduled, so the volume is created in the zone where the pod landed. With Immediate, the volume can be created in a zone the pod then cannot reach. Third, a zonal block volume ties the pod to that zone forever; if the zone has no capacity, the pod stays Pending.

kubectl get pvc,pv
kubectl get storageclass            # the one marked (default) is used when none is named
kubectl get pvc -o 'custom-columns=NAME:.metadata.name,UNUSED:.status.conditions[?(@.type=="Unused")].reason'

Running production databases on Kubernetes is possible, usually through an operator such as CloudNativePG, but whether you should is a senior-level trade-off. At this level, know how the pieces behave.

Jobs for training and batch

A Job runs pods to completion. The fields that matter in practice:

podFailurePolicy is what makes Jobs sensible for ML training. A training script that exits with a code meaning "bad hyperparameters" should fail the Job immediately, not retry six times on expensive GPUs. A pod killed because its spot node was reclaimed should be retried without counting against the limit.

apiVersion: batch/v1
kind: Job
metadata:
  name: train-resnet
spec:
  backoffLimit: 4
  activeDeadlineSeconds: 21600
  ttlSecondsAfterFinished: 86400
  podFailurePolicy:
    rules:
    - action: FailJob
      onExitCodes:
        containerName: trainer
        operator: In
        values: [42]
    - action: Ignore
      onPodConditions:
      - type: DisruptionTarget
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: trainer
        image: registry.example.com/trainer:2.1.0
        args: ["--epochs", "20"]
        resources:
          requests:
            cpu: "4"
            memory: 16Gi
          limits:
            memory: 16Gi

A CronJob (batch/v1) wraps a Job template with a schedule. Set timeZone explicitly (for example Asia/Riyadh or Africa/Cairo) rather than doing UTC arithmetic in your head, and set concurrencyPolicy: Forbid for jobs that must not overlap, such as a nightly feature-table rebuild.

For multi-pod distributed training, where all workers must start together or not at all, plain Jobs are not enough. Look at JobSet, Kueue for queueing shared GPU capacity, the gang-scheduling Workload API (Beta in 1.37), and the higher-level operators covered in Kubeflow and Ray.

Access and isolation for a team

On a shared cluster, three questions come up the moment a second team arrives: who may do what through the API, what a pod may do on its node, and which pods may talk to which. Kubernetes answers each with a separate mechanism, and none of them is on by default in a useful form.

ServiceAccounts and RBAC

Humans authenticate with whatever the cluster is configured for, usually your cloud identity through an exec plugin in kubeconfig. Pods authenticate as a ServiceAccount. Every namespace has a default ServiceAccount, and pods use it unless told otherwise. Tokens are projected into the pod, short-lived and bound to an audience; long-lived token Secrets are no longer created automatically.

RBAC grants permissions. A Role lists verbs on resources inside one namespace; a ClusterRole does the same cluster-wide or as a reusable template. A RoleBinding or ClusterRoleBinding grants a role to users, groups or ServiceAccounts. RBAC is purely additive: there are no deny rules, so the only way to restrict is to grant less.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: deployer
  namespace: shop
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: deployer
  namespace: shop
rules:
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["get", "list", "watch", "create", "update", "patch"]
- apiGroups: [""]
  resources: ["services", "configmaps"]
  verbs: ["get", "list", "watch", "create", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: deployer
  namespace: shop
subjects:
- kind: ServiceAccount
  name: deployer
  namespace: shop
roleRef:
  kind: Role
  name: deployer
  apiGroup: rbac.authorization.k8s.io

kubectl auth can-i create deployments -n shop --as=system:serviceaccount:shop:deployer
kubectl auth can-i --list -n shop --as=system:serviceaccount:shop:deployer
kubectl auth whoami
kubectl create token deployer -n shop --duration=1h

kubectl auth can-i answers exactly the question the API server will ask, which makes it the first step for any Forbidden error. The error itself names the user, verb, resource, API group and namespace; read all five before changing anything.

Some permissions are more powerful than they look. Being able to create pods in a namespace means being able to mount any Secret in that namespace. list or watch on Secrets returns their contents. pods/exec gives a shell in running workloads. Grant these deliberately.

Pod Security Admission

The pod's own powers on the node, such as running as root, using host namespaces, or adding Linux capabilities, are governed by Pod Security Admission, the built-in replacement for the PodSecurityPolicy removed in 1.25. It enforces one of three Pod Security Standards per namespace, set by labels:

• privileged: no restrictions, for system components.
• baseline: blocks the known privilege escalations (host namespaces, privileged containers, most capabilities).
• restricted: current hardening best practice; non-root, no privilege escalation, all capabilities dropped, a seccomp profile.

kubectl label namespace shop \
  pod-security.kubernetes.io/enforce=restricted \
  pod-security.kubernetes.io/warn=restricted \
  pod-security.kubernetes.io/audit=restricted

A pod that breaks the rule is rejected with a message naming every violation, for example pods "x" is forbidden: violates PodSecurity "restricted:latest": allowPrivilegeEscalation != false .... The security context that satisfies restricted is short enough to put in every template:

spec:
  securityContext:
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  containers:
  - name: api
    image: registry.example.com/api:1.8.2
    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      capabilities:
        drop: ["ALL"]

Roll it out by setting warn and audit first, fixing what they report, and only then setting enforce. Pod Security Admission checks pods at creation, so enforcing on a namespace does not kill running pods, but their replacements will be rejected at the next rollout, which is a bad time to find out.

NetworkPolicy

By default every pod can reach every other pod in the cluster. A NetworkPolicy selects pods and lists the traffic allowed to or from them; once any policy selects a pod for a direction, everything not allowed in that direction is denied. NetworkPolicies are only enforced if your CNI plugin supports them (Calico and Cilium do; some simple CNIs do not, and the objects are then silently ignored).

The standard starting point is a namespace-wide default deny, followed by explicit allows:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: shop
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: shop
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
      podSelector:
        matchLabels:
          k8s-app: kube-dns
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-from-web
  namespace: shop
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes: [Ingress]
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: web
    ports:
    - protocol: TCP
      port: 8080

Debugging beyond logs

Beginner's loop was get, describe, logs. It still starts every investigation. This section adds the tools for when that loop runs dry: the container has no shell, the problem is on the node, or you need to query fifty objects at once.

Events, filtered

Events are the cluster's narrative: scheduling decisions, image pulls, probe failures, kills, evictions. They expire after about an hour by default, so read them early.

kubectl events -n shop --types=Warning
kubectl events --for deploy/api -n shop
kubectl get events -n shop --sort-by=.lastTimestamp
kubectl get events -A --field-selector reason=FailedScheduling

Ephemeral debug containers

Distroless and minimal images have no shell, so kubectl exec -- sh fails with executable file not found in $PATH. kubectl debug adds an ephemeral container to the running pod with whatever image you choose. With --target, it shares the target container's process namespace, so you can see its processes and read its filesystem through /proc/1/root (if the target runs as a different non-root user, that read can fail with permission denied).

kubectl debug -it api-5d8f7c9b6-xk2lp --image=busybox:1.36 --target=api
# inside:
ps aux
ls /proc/1/root/app
wget -qO- localhost:8080/ready

When the pod crashes too fast to attach to, debug a copy instead. The copy can override the command so it starts a shell rather than the crashing entrypoint:

kubectl debug api-5d8f7c9b6-xk2lp -it --copy-to=api-dbg --container=api -- sh
kubectl delete pod api-dbg     # clean up afterwards

For node-level problems, kubectl debug node/<node> starts a pod on that node with the host filesystem mounted at /host. From there you can read kubelet logs or inspect containers with crictl through chroot /host. The --profile flag (general, baseline, restricted, netadmin, sysadmin) sets how much privilege the debug container gets; netadmin is the one for packet captures.

kubectl debug node/worker-2 -it --image=busybox:1.36 --profile=sysadmin
# inside:
chroot /host journalctl -u kubelet --since "15 min ago"
chroot /host crictl ps -a

Querying many objects

When the question is "which pods in this namespace have restarted" or "which nodes are my GPU pods on", scrolling YAML does not scale. JSONPath and custom columns do.

kubectl get pods -n shop -o 'custom-columns=NAME:.metadata.name,NODE:.spec.nodeName,RESTARTS:.status.containerStatuses[0].restartCount,QOS:.status.qosClass'
kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded
kubectl get pods -n shop --sort-by=.status.containerStatuses[0].restartCount
kubectl get pods -n shop -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.containerStatuses[0].lastState.terminated.reason}{"\n"}{end}'
kubectl get deploy api -o kyaml       # KYAML output, GA in 1.37

And for scripts and pipelines that need to wait for a state rather than poll:

kubectl wait --for=condition=Available deploy/api -n shop --timeout=180s
kubectl wait --for=condition=complete job/train-resnet --timeout=6h
kubectl wait --for=delete pod/api-dbg --timeout=60s

Work downwards. Most incidents are solved at the object layer; the node layer is for when several unrelated pods on the same node misbehave.

Kubernetes in the delivery pipeline

A mid-level engineer is often the one who wires the cluster into CI/CD. The goal is that what reaches production is exactly what was tested, that a failed deploy is detected by the pipeline and not by users, and that the cluster's state can be reconstructed from Git.

Deploy by immutable reference. Build an image once, tag it with the commit SHA, and deploy that tag or, better, its digest (registry.example.com/api@sha256:...). A mutable tag like latest or prod means two nodes can run different code under the same name, and imagePullPolicy behaviour then decides which one you get.

Validate before you apply. Three levels of checking, each catching more:

kubectl apply --dry-run=client -f deploy/            # schema-level, no cluster needed
kubectl apply --dry-run=server -f deploy/            # full admission, defaulting, policy, quota
kubectl diff -f deploy/                              # exactly what will change

The server-side dry run is the one that catches Pod Security rejections, quota exhaustion and removed API versions, because it goes through the real API server's admission chain without persisting anything.

Gate on the rollout. kubectl apply returns as soon as the API server accepts the object, long before any pod is running. A pipeline that stops there reports success for a deploy that is about to fail. Always follow it with kubectl rollout status --timeout (or kubectl wait) and fail the job on a non-zero exit.

The workflow below shows the shape. It assumes an earlier step has already given kubectl credentials for your cluster; that step is provider-specific (for example the cloud's own login action using OIDC), so it is left out here.

name: deploy
on:
  push:
    branches: [main]
jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: production
    steps:
    - uses: actions/checkout@v4
    - name: Set image
      working-directory: deploy/overlays/prod
      run: kustomize edit set image registry.example.com/api=registry.example.com/api:sha-${GITHUB_SHA::7}
    - name: Diff
      run: kubectl diff -k deploy/overlays/prod || true
    - name: Apply
      run: kubectl apply --server-side --field-manager=ci -k deploy/overlays/prod
    - name: Wait for rollout
      run: kubectl rollout status deploy/api -n shop-prod --timeout=5m
    - name: Roll back on failure
      if: failure()
      run: kubectl rollout undo deploy/api -n shop-prod

Cluster credentials for the job should come from short-lived federation (for example GitHub OIDC to your cloud's Kubernetes service) bound to a narrowly scoped ServiceAccount or RBAC group like the deployer role above, never a copied admin.conf. The GitHub Actions guide covers the OIDC side.

Or pull instead of push. In the GitOps model, CI only builds images and commits the new tag to a Git repository of manifests; an in-cluster agent such as Argo CD or Flux notices the commit and applies it. The cluster never hands credentials to CI, drift is detected and corrected automatically, and "what is deployed" is answered by reading Git. Most teams that run several clusters end up here. The same rules still apply: immutable tags, server-side validation, and health checks that decide whether a sync succeeded.

Test against a real API server. Unit tests do not catch a bad selector or a missing RBAC rule. A kind cluster starts in under a minute on a CI runner, so an integration job can apply your manifests, wait for rollout, run a smoke test through kubectl port-forward, and tear down. It is the cheapest way to catch "works in my head, rejected by the API server".

kind create cluster --name ci --wait 120s
kubectl apply -k deploy/overlays/ci
kubectl rollout status deploy/api -n shop-ci --timeout=180s
kubectl port-forward -n shop-ci svc/api 8080:8080 &
sleep 3 && curl -fsS localhost:8080/ready
kind delete cluster --name ci

Watch deprecations before they bite. Each Kubernetes minor removes or deprecates something. The API server exposes which deprecated APIs are still being called:

kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis

On a managed cluster in a Gulf region, where data-residency rules often mean you run in one provider's local region rather than choosing freely, the provider's Kubernetes version calendar sets your upgrade rhythm. Three minors a year with about a year of support each means an upgrade roughly every four months, and every one of them is easier if your manifests were already clean.

Putting it all together

Here is one service, a model-inference API, with every control from this guide applied. It is written as a Kustomize base so the overlays can change the tag, replica bounds and hostname per environment.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  labels:
    app: api
spec:
  revisionHistoryLimit: 5
  progressDeadlineSeconds: 300
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      serviceAccountName: api
      terminationGracePeriodSeconds: 45
      securityContext:
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: ScheduleAnyway
        labelSelector:
          matchLabels:
            app: api
      containers:
      - name: api
        image: registry.example.com/api
        ports:
        - containerPort: 8080
        envFrom:
        - configMapRef:
            name: api-config
        resources:
          requests:
            cpu: 500m
            memory: 1Gi
          limits:
            memory: 1Gi
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          capabilities:
            drop: ["ALL"]
        lifecycle:
          preStop:
            sleep:
              seconds: 10
        startupProbe:
          httpGet:
            path: /live
            port: 8080
          periodSeconds: 5
          failureThreshold: 60
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          periodSeconds: 5
        livenessProbe:
          httpGet:
            path: /live
            port: 8080
          periodSeconds: 10
        volumeMounts:
        - name: tmp
          mountPath: /tmp
      volumes:
      - name: tmp
        emptyDir:
          sizeLimit: 1Gi
---
apiVersion: v1
kind: Service
metadata:
  name: api
spec:
  selector:
    app: api
  trafficDistribution: PreferSameZone
  ports:
  - port: 8080
    targetPort: 8080
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: api
automountServiceAccountToken: false
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 3
  maxReplicas: 20
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 65
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 600
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: api

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- api.yaml
- httproute.yaml
- netpol.yaml
configMapGenerator:
- name: api-config
  literals:
  - LOG_LEVEL=info

kubectl diff -k deploy/overlays/prod
kubectl apply --server-side --field-manager=ci -k deploy/overlays/prod
kubectl rollout status deploy/api -n shop-prod --timeout=5m
kubectl get deploy,rs,hpa,pdb,endpointslices -n shop-prod -l app=api

Note what is absent: there is no replicas field, because the HPA owns it. The ServiceAccount does not mount an API token, because this service never calls the Kubernetes API. The writable /tmp is a size-limited emptyDir, because the root filesystem is read-only. Every choice has a reason from an earlier section:

Where you are now

You can explain how a change travels from kubectl apply through watches and controllers to a running container, read a stalled rollout from its ReplicaSets and conditions, predict where the scheduler will place a pod and why it refuses to, tell a CPU throttle from an OOM kill from an eviction, deploy without dropping requests, debug a Service from its EndpointSlices, manage configuration across environments without drift or ownership fights, scale on metrics without fighting your own manifests, run stateful and batch workloads with the right failure semantics, lock a namespace down with RBAC, Pod Security and NetworkPolicy, debug pods without shells and nodes without SSH, and wire all of it into a pipeline that fails honestly.

Senior takes every one of these further, with a platform, scale or security dimension. Control-plane architecture and which component breaks first under load; etcd, API Priority and Fairness and the 1.37 watch-cache changes; cluster upgrades without skipping minors; encryption at rest and external secret stores; admission policy with CEL; multi-tenancy from namespaces to virtual clusters; cost and capacity for GPU fleets; and when Kubernetes is the wrong tool altogether.

Sources

• Kubernetes documentation home
• Kubernetes v1.37 release announcement
• Kubernetes v1.36 release announcement
• Kubernetes v1.35 release announcement
• Kubernetes v1.34 release announcement
• Deprecation and removal of Service externalIPs
• Gateway API v1.6 release
• Kubernetes components
• Pod lifecycle
• Container lifecycle hooks
• Resource management for pods and containers
• Node-pressure eviction
• About cgroup v2
• Autoscaling workloads
• Horizontal Pod Autoscaling
• Secrets
• Good practices for Kubernetes Secrets
• Pod Security Standards
• Role-based access control good practices
• Multi-tenancy
• Ingress
• Gateway API
• kubectl quick reference
• kuberc: kubectl user preferences
• Debug pods
• Troubleshooting clusters
• Deprecated API migration guide
• Version skew policy

Part two of three. This is a cumulative review of the Beginner and Mid-level material, organised the way a mid-level interview actually runs. There are a few quick checks that you know the vocabulary, then scenarios where the interviewer describes a broken cluster and listens to how you reason, then trade-offs with no single right answer. Budget about forty minutes. Senior reviews all three levels.

At this level, interviewers are not testing whether you can type kubectl get pods. They want to know whether you can predict what the system will do. They want to hear that you check the cheap, likely causes before the expensive, unlikely ones, and that you can say why a fix works. A candidate who says "I would restart the pod" and stops has given a beginner answer. A candidate who says "deleting the pod works because the ReplicaSet controller is level-triggered and will recreate it, but if it crashes again the problem is in the spec, so first I read logs --previous" has given a mid-level one.

Where this picks up

The model in one screen

Kubernetes is a declarative control system built around one API. You write desired state as objects. Controllers watch those objects and take one step at a time to make the observed state match. No component calls another directly; each one writes objects and the next one notices through a watch. Only the API server talks to etcd.

If you can say that in thirty seconds and then explain one consequence of it, you have set the level of the interview. Three consequences are worth having ready:

Deleting a managed pod "fixes" it because controllers are level-triggered. The ReplicaSet controller sees two pods where there should be three and creates one. That does not repair a bad spec, so a crash that returns means the problem is in the template, the config or a dependency.

Stuck deletions are almost always finalizers. Deletion sets deletionTimestamp and waits for every entry in metadata.finalizers to be cleared by the controller that owns it. If that controller is gone, or a webhook it depends on is down, the object waits forever.

Status can describe the previous spec. If metadata.generation is ahead of status.observedGeneration, the controller has not processed your latest change. Any script that checks "is it done?" must look at observedGeneration first, which kubectl rollout status does for you.

Commands to have ready

kubectl get pods -o wide --show-labels
kubectl describe pod <pod>                         # read Events at the bottom first
kubectl logs <pod> --previous                      # why the last container died
kubectl events --for pod/<pod> --types=Warning
kubectl get rs -l app=web                          # which template is Ready?
kubectl rollout status deploy/web --timeout=5m     # non-zero exit on failure
kubectl rollout undo deploy/web
kubectl get endpointslices -l kubernetes.io/service-name=web
kubectl auth can-i list pods -n dev --as=jane
kubectl debug -it <pod> --image=busybox:1.36 --target=<container>
kubectl debug node/<node> -it --image=busybox:1.36
kubectl apply --server-side --field-manager=ci -f deploy/
kubectl diff -k overlays/prod
kubectl apply --dry-run=server -f deploy/
kubectl top pods --sort-by=memory                  # needs metrics-server

Scenario questions: rollouts and deploys

1. A deploy has been "in progress" for twenty minutes. What do you check first?I would run kubectl get rs -l app=<name> to see whether the new ReplicaSet has any Ready pods, then describe one of its pods. A rollout only advances as new pods become available, so a stall almost always means the new pods never become Ready: a crash, a failing readiness probe, a missing Secret, an image that cannot be pulled, or a surge pod stuck in Pending because the cluster has no room. The old pods keep serving throughout, so there is no outage yet. After progressDeadlineSeconds (600 by default) the Deployment gets Progressing=False with reason ProgressDeadlineExceeded, but Kubernetes does not roll back by itself. I would undo or fix forward, and I would not delete the old pods, because they are the only healthy capacity.
2. Every deploy causes a handful of 502s even though all pods are healthy. Why?It is the termination race. When a pod is deleted, endpoint removal and the SIGTERM to the container start at the same time. Endpoint removal propagates to every kube-proxy and gateway asynchronously, so for a moment traffic still arrives at a pod that has already closed its listener. The fix is a preStop hook that sleeps for a few seconds (the built-in sleep action, GA since 1.34, so you do not need a sleep binary in the image), then graceful SIGTERM handling in the app, with terminationGracePeriodSeconds sized to cover the sleep plus the longest in-flight request. I would also check the app actually receives SIGTERM, which means running in exec form as PID 1.
3. You changed a ConfigMap and nothing happened.It depends on how the pod consumes it. Environment variables are read once at container start and never update. Files from a mounted volume update eventually after the kubelet syncs, unless they are mounted with subPath, which never updates. Even when files change, most apps do not reload them. So a ConfigMap edit is not a deploy. For an immediate fix, kubectl rollout restart. The durable fix is to version config by name, with Kustomize's configMapGenerator hash suffix or a checksum annotation in a Helm chart, so a config change changes the pod template and triggers a normal rollout that can be undone.
4. After every deploy the HPA's replica count drops back to 3 during peak traffic.The manifest still sets spec.replicas: 3, so each apply resets the field the HPA is managing. Remove replicas from the manifest or the Helm values once an HPA exists. With server-side apply, ownership of each field is tracked in managedFields, so a mismatch shows up as a conflict instead of a silent overwrite, which is one reason I would use --server-side with a named --field-manager in CI.
5. The CI job says the deploy succeeded, but users see errors.kubectl apply returns as soon as the API server accepts the object, long before any pod runs. The pipeline must gate on kubectl rollout status --timeout or kubectl wait and fail on a non-zero exit. I would also add a server-side dry run before applying, because it runs the real admission chain and catches Pod Security rejections, quota exhaustion and removed API versions that a client-side dry run cannot see.
6. A pod shows CrashLoopBackOff but kubectl logs is empty.The current container may have just restarted and not written anything yet. kubectl logs --previous shows the output of the container that died. If that is also empty, the process probably failed before logging: describe will show the last state, the exit code and the reason. RunContainerError with executable file not found in $PATH means the command is wrong. If I need a shell and the image has none, I would use kubectl debug --copy-to with a sh command so the copy starts without the crashing entrypoint.

Scenario questions: scheduling and resources

1. Pods are Pending but the dashboards show the cluster at 30% CPU.The scheduler places pods on requests, not on usage. A node whose pods request 3.5 of 4 allocatable cores is full to the scheduler even if they are idle. I would read the FailedScheduling event, which tallies every node's rejection reason, then compare kubectl describe node's "Allocated resources" block with kubectl top node. The gap is inflated requests. The fix is right-sizing requests, using VPA recommendations or in-place resize for experiments, not adding nodes.
2. Read this event: 0/3 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }, 2 Insufficient cpu. preemption: 0/3 nodes are available: ...Three nodes were considered. One was excluded because it is a control-plane node with a taint the pod does not tolerate. The other two did not have enough unrequested CPU for the pod's request. The preemption: part says the scheduler also tried to evict lower-priority pods to make room and found nothing it could evict. So I would either lower the request, add capacity, or check why the node autoscaler did not react.
3. A service's latency has spiky p99s but its pods never restart.That pattern suggests CPU throttling. When a container hits its CPU limit, the kernel stops scheduling it for the rest of each accounting period, so requests stall without anything crashing. I would compare usage with the limit and look at the container's throttling metrics in the monitoring stack. The usual fix is an accurate CPU request with a higher limit or no CPU limit, which is a deliberate policy choice, not an oversight. Memory is different: exceeding a memory limit gets the container OOM-killed with exit code 137.
4. Pods were evicted from a node that has plenty of free memory now.An eviction is the kubelet reacting to node pressure, not a container exceeding its own limit. When memory.available or disk signals cross the eviction thresholds (defaults on Linux: memory.available<100Mi, nodefs.available<10%, imagefs.available<15%), the kubelet evicts pods, starting with those using the most above their requests. Once pods are evicted the pressure is gone, so the node looks fine afterwards. I would check the pod's eviction message, which names the resource, and look for BestEffort pods and pods with no ephemeral-storage limits writing large files to /tmp.
5. You tainted the GPU nodes and added a toleration to the training pods, but they still land on CPU nodes.A toleration allows a pod onto a tainted node; it does not attract the pod there. A dedicated pool needs both halves: a taint to keep other pods off, and a nodeSelector or required node affinity on the training pods to send them there. For newer GPU setups I would also check whether the cluster uses Dynamic Resource Allocation, where devices are requested through a ResourceClaim against a DeviceClass rather than an extended resource in limits.
6. With four replicas and three zones, the fourth pod stays Pending.Probably a required pod anti-affinity on the zone key. It is all-or-nothing: once every zone has one replica, there is nowhere legal for the fourth. topologySpreadConstraints with maxSkew: 1 expresses the real intent, keeping counts within one of each other, and whenUnsatisfiable: ScheduleAnyway lets the pod schedule even when the spread cannot be perfect.

Scenario questions: networking, access and data

1. A Service returns "connection refused". Walk me through it.Start with the EndpointSlices: kubectl get endpointslices -l kubernetes.io/service-name=<svc>. If there are no endpoints, the selector matches no pods or none of them is Ready, so I compare the selector with kubectl get pods --show-labels and check readiness. If endpoints exist, I check that the Service's targetPort matches the port the container actually listens on, and that the app binds 0.0.0.0, not localhost. Then DNS from a debug pod, and if the namespace has a default-deny NetworkPolicy, whether an allow rule exists. I would not use ping on the ClusterIP, because it is not bound to any interface; a failed ping proves nothing.
2. After a new default-deny policy, every outbound call fails with a name-resolution error.A default-deny egress policy also blocks DNS to CoreDNS in kube-system. The symptom looks like a DNS bug, which sends people to the wrong layer. The fix is an egress rule allowing UDP and TCP port 53 to the k8s-app: kube-dns pods, shipped in the same change as the deny. I would also mention that NetworkPolicies only take effect if the CNI enforces them; on a CNI that does not, they are silently ignored.
3. A developer gets Forbidden: pods is forbidden: User "jane" cannot list resource "pods" in API group "" in the namespace "dev".The message names all five things the API server checked: user, verb, resource, API group and namespace. I would reproduce it with kubectl auth can-i list pods -n dev --as=jane, then find the RoleBindings in dev. RBAC is additive, so the fix is a Role granting exactly that verb and a RoleBinding, not a ClusterRoleBinding to edit or cluster-admin.
4. Pods in a namespace were fine yesterday; today new ones are rejected with violates PodSecurity "restricted:latest".Someone set the namespace's pod-security.kubernetes.io/enforce label. Pod Security Admission only checks pods at creation, so running pods survived and the next rollout's pods are rejected. The error lists each violation, and the fix is the standard restricted security context: runAsNonRoot, allowPrivilegeEscalation: false, drop ALL capabilities, and a RuntimeDefault seccomp profile. For the process lesson, roll out with warn and audit labels first.
5. You scaled a StatefulSet down to zero and the storage bill did not change.Deleting or scaling down a StatefulSet does not delete the PVCs created from volumeClaimTemplates by default, which protects the data and keeps billing for it. Since 1.37, PVCs carry an Unused condition, so orphaned claims are easy to list. I would delete them deliberately after confirming nothing needs the data.
6. A training Job retried six times on expensive GPUs because of a config typo.The default backoffLimit is 6, and every failure counts. A podFailurePolicy can say "exit code 42 means bad configuration, fail the whole Job now" with FailJob, and "a pod with the DisruptionTarget condition was preempted or its node drained, retry without counting it" with Ignore. I would add activeDeadlineSeconds as a wall-clock cap and ttlSecondsAfterFinished for clean-up.

Trade-off questions

These have no single correct answer. The interviewer wants the factors, a default, and when you would switch.

CPU limits or no CPU limits? Without a CPU limit, a container can use idle CPU on the node, and its request still guarantees its share under contention; the cost is less predictable performance across nodes. With a limit, behaviour is predictable but spare capacity is wasted and bursts are throttled. A common stance is accurate CPU requests, no CPU limits for latency-sensitive services, and memory request equal to limit. A multi-tenant cluster may enforce CPU limits through a LimitRange, which is a reasonable policy for fairness.

Kustomize or Helm? Kustomize keeps plain YAML and patches it per environment, with no templating language and no release state; it is built into kubectl. Helm is a package manager with templating, values and a release history, which suits third-party software and heavily parameterised charts. Many teams use Helm for things they install and Kustomize for their own services. The weak answer is "Helm because everyone uses it".

Push-based CI deploys or GitOps? Push is simpler to start: the pipeline holds credentials and runs kubectl apply. GitOps with Argo CD or Flux pulls from Git, so CI never holds cluster credentials, drift is corrected automatically, and "what is deployed" is answered by reading the repository. The cost is another component to run and a slightly longer path to production. Past two or three clusters, GitOps usually wins.

Deployment or StatefulSet for a database? A StatefulSet gives stable names, ordered rollout and a volume per pod, which a database needs. But running a production database well also needs backups, failover and upgrades, which is why teams use an operator such as CloudNativePG, or a managed database. The honest answer weighs the team's operational capacity against the benefit of keeping everything in one platform.

HPA on CPU or on a custom metric? CPU is available out of the box through metrics-server and works when CPU tracks load. For queue consumers, inference servers limited by GPU, or I/O-bound services, CPU is a poor proxy; requests per second or queue depth through a metrics adapter or KEDA is better. Since 1.37, scaling to zero with the built-in HPA is Beta and on by default, but only with Object or External metrics, because a Deployment with zero pods has no CPU to measure.

Ingress or Gateway API for a new cluster? Gateway API. The community ingress-nginx controller was retired on 24 March 2026 and gets no more security patches, and the Ingress API is frozen. Gateway API separates the infrastructure (GatewayClass), the listener (Gateway) and the routes (HTTPRoute), supports weighted backends natively, and has many implementations. Existing Ingress resources keep working, so migration can be gradual, but nothing new should start on ingress-nginx.

Liveness probe or no liveness probe? A liveness probe helps only when the process can get stuck in a way that a restart fixes, such as a deadlock. A bad one causes outages: a liveness check that calls the database restarts the whole fleet during a database incident. Many services are better with only readiness and startup probes. If you add liveness, make it check the process itself and give it a generous threshold.

Requests equal to limits (Guaranteed) or Burstable? Guaranteed pods are evicted last and get predictable resources, but you pay for the peak all the time. Burstable pods pack more densely and can use spare capacity, at the cost of eviction risk under node pressure. Critical stateful pods and latency-critical services lean Guaranteed for memory; batch and dev workloads lean Burstable.

What a strong answer covers

Three habits separate strong candidates. First, they read error messages aloud and interpret each part. A FailedScheduling event or a Forbidden message contains the answer, and walking through it shows you know that. Second, they name the layer: object, container or node. "It is an object-layer problem, the selector does not match" is a stronger sentence than "something is wrong with the Service". Third, they mention how they would prevent recurrence: a server-side dry run in CI, a PDB, a podFailurePolicy, a restricted namespace label. Interviewers at this level are hiring someone who will leave the system better than they found it.

It also helps to be current. Mentioning in passing that kubectl get endpoints is deprecated in favour of EndpointSlices, that sidecars are now init containers with restartPolicy: Always, that in-place resize is GA since 1.35, or that the kubelet refuses to start on cgroup v1 by default since 1.35, tells the interviewer your knowledge is from this year and not from a 2021 tutorial.

Common interview questions

1. What is the difference between a ReplicaSet and a Deployment?A ReplicaSet keeps N identical pods running for one pod template. A Deployment manages ReplicaSets: each template change creates a new ReplicaSet, and the rollout moves replicas from old to new. Old ReplicaSets are kept at zero, up to revisionHistoryLimit, which is what makes rollout undo possible.
2. What do maxSurge and maxUnavailable do, and how do they round?Surge is how many extra pods may exist during a rollout, rounded up; unavailable is how many below the desired count may be unavailable, rounded down. Both default to 25%. With three replicas that gives one surge and zero unavailable, so small Deployments roll one pod at a time.
3. Explain the three probes.Startup holds off the other two until the app has started once, which is how you handle slow starters like model servers. Readiness failing removes the pod from Service endpoints without restarting it. Liveness failing restarts the container. Defaults: period 10 s, timeout 1 s, failure threshold 3.
4. What are QoS classes?Guaranteed (every container's requests equal its limits for CPU and memory), Burstable (something set but not Guaranteed) and BestEffort (nothing set). Under node pressure BestEffort goes first and Guaranteed last.
5. What is a native sidecar?An init container with restartPolicy: Always, GA since 1.33. It starts before the app containers, runs alongside them, and stops after them. It fixes the old problem of a sidecar keeping a Job from completing.
6. What is a headless Service for?With clusterIP: None, DNS returns the pod IPs directly instead of a virtual IP. With a StatefulSet, each pod gets a stable name like db-0.db.ns.svc.cluster.local, which databases and distributed training use to find peers.
7. What does ndots:5 in a pod's resolv.conf cause?Names with fewer than five dots are tried against every search domain first, so an external lookup like api.example.com generates several failed cluster-local queries before the real one. Use a trailing dot or lower ndots in dnsConfig for chatty clients.
8. What is server-side apply?The API server performs the merge and records which field manager owns each field in managedFields. Writing a field someone else owns gives a conflict rather than a silent overwrite; --force-conflicts takes ownership deliberately.
9. How does the HPA compute replicas?ceil(currentReplicas × current / target), every 15 seconds, per metric, taking the largest. CPU utilisation is a percentage of the request. Scale-down waits for a five-minute stabilisation window by default, and the 10% tolerance is configurable per HPA since 1.37.
10. What does a PodDisruptionBudget protect against, and what does it not?Voluntary disruptions through the Eviction API: drains, autoscaler scale-downs. Not node crashes. A PDB that can never be satisfied, such as minAvailable equal to the replica count, blocks every drain.
11. How do you debug a pod that has no shell?kubectl debug -it <pod> --image=busybox:1.36 --target=<container> adds an ephemeral container sharing the target's process namespace, so you can see its processes and its filesystem through /proc/1/root.
12. Are Secrets encrypted?Not by default. They are base64-encoded in etcd. Encryption at rest must be configured on the API server, and access should be limited with RBAC, remembering that anyone who can create a pod in a namespace can mount its Secrets.
13. What changed about GPUs in recent releases?Dynamic Resource Allocation's core became GA in 1.34 and 1.37 added device taints, extended resources and device status as GA. Devices are requested through ResourceClaims against DeviceClasses, which allows sharing and richer selection than a count in limits.
14. How do you keep cluster credentials out of CI?Short-lived federation such as OIDC from the CI system to the cloud's Kubernetes service, bound to a narrowly scoped role, or GitOps so CI never touches the cluster at all. Never a copied admin kubeconfig.

Final self-test

• Explain "level-triggered" and use it to say why deleting a pod recreates it.
• Say what generation versus observedGeneration tells you.
• Compute the surge and unavailability for 3 and for 10 replicas at the defaults.
• Name four reasons a rollout stalls, and say what Kubernetes does at the progress deadline.
• Read a FailedScheduling message line by line.
• Explain why a toleration alone does not put a pod on a GPU node.
• Say which resource throttles and which kills, and how an eviction differs from both.
• Describe the termination race and the three settings that fix it.
• Give the four ways a ConfigMap can be consumed and when each sees an update.
• Explain how server-side apply stops a deploy from resetting an HPA.
• State the HPA formula and why scale-to-zero needs an external metric.
• Diagnose an empty EndpointSlice in two commands.
• Write a default-deny policy and the rule it needs so DNS still works.
• Say what podFailurePolicy adds to backoffLimit.
• Explain why a new cluster should use Gateway API, not ingress-nginx.

Part two of three. At this level your workloads run, so the problems change character. They are no longer "why won't my pod start" but "why does every deploy drop requests", "why is this pod Pending when the cluster is idle", "why did my config change do nothing", and "why did the node upgrade stall for an hour". Most of them come from a small number of misunderstandings about how controllers, the scheduler and the kubelet behave. Start with the error table, then the practices and practice cards, then the recipes you will reach for in the middle of an incident.

Everything here assumes Kubernetes 1.37. Where behaviour changed recently, the tip says so, because an answer from an older blog post can now be wrong.

Common errors at this level

Cumulative: Beginner's errors still apply, and these are the ones that appear once things work.

The practices that pay off most

In rough order of leverage: the first few prevent the most incidents per minute of effort.

The order matters. Teams often start with the security items because they are visible in audits, and those are important, but the first four prevent the incidents that actually page people. If you have one afternoon, set requests from real usage data and add a rollout gate to the pipeline. If you have a second, add preStop hooks and fix any liveness probe that calls a database.

Practice cards

1. Stall a rollout on purposeSet a Deployment's image to a tag that does not exist. Watch kubectl get rs -w, confirm the old pods keep serving, wait for ProgressDeadlineExceeded, then undo. Note that nothing rolled back until you did it.
2. Measure the termination raceLoop curl against a Service from a debug pod while running kubectl rollout restart several times. Count non-200s. Add preStop.sleep.seconds: 5 and repeat.
3. Read a scheduling failureRequest cpu: "50" and read the FailedScheduling event word by word. Then taint a node and read it again with the new reason.
4. Throttle and killRun one pod against a 100m CPU limit with a busy loop and one against a 64Mi memory limit that allocates 200 MB. Describe both and compare what each one's "failure" looks like.
5. Prove env vars do not updateConsume one ConfigMap as envFrom and as a volume. Edit it, wait a minute, and compare env with the mounted file.
6. Watch the HPA fight a manifestCreate an HPA, load the Deployment until it scales, then re-apply a manifest that sets replicas: 2 with client-side apply, then with --server-side. Compare what happens.
7. Block a drainCreate a PDB with minAvailable equal to the replica count and drain the node. Read the retry message, then fix the PDB and watch the drain finish.
8. Break DNS with a policyOn a cluster with a policy-enforcing CNI, apply only a default-deny policy and run nslookup kubernetes.default from a pod. Add the DNS allow and try again.
9. Fail a Job fastGive a Job a podFailurePolicy that fails on exit code 42. Run it with exit 42 and with exit 1 and count the pods each time.
10. Debug without a shellDeploy a distroless image, confirm exec fails, and attach with kubectl debug --target. List the target's processes and read a file through /proc/1/root.

Recipe: a stalled rollout in five minutes

When a deploy hangs, the goal is to find out which new pod is unhappy and why, without touching the old pods that are still serving.

kubectl rollout status deploy/api -n shop --timeout=10s   # confirm it is really stuck
kubectl get rs -n shop -l app=api                          # new RS: desired vs ready
kubectl get pods -n shop -l app=api -o wide                # which pods are new, and where
kubectl describe pod -n shop <new-pod>                     # Events: pull, schedule, probe
kubectl logs -n shop <new-pod> --previous                  # if it is crashing
kubectl get deploy api -n shop \
  -o jsonpath='{.status.conditions[?(@.type=="Progressing")].reason}{"\n"}'

Four outcomes cover almost every case. ImagePullBackOff means a wrong tag or a missing pull secret. CrashLoopBackOff means the new version fails at start, so read the previous logs. Running but not Ready means the readiness probe is failing, often because the port or path changed. Pending means the surge pod has no room. Decide between kubectl rollout undo deploy/api -n shop and a fix forward, and never delete the old pods to "unstick" things.

Recipe: a pod that will not schedule

kubectl get pods -A --field-selector=status.phase=Pending
kubectl events -n shop --for pod/<pod> --types=Warning
kubectl describe node <node> | sed -n '/Allocated resources/,/Events/p'
kubectl top node
kubectl get nodes -L topology.kubernetes.io/zone,node.kubernetes.io/instance-type
kubectl describe node <node> | grep -A3 Taints

Read the FailedScheduling tally first; it names every rejection reason with a node count. Insufficient cpu or Insufficient memory means requests do not fit, so compare the pod's requests with the largest free allocatable on any node. untolerated taint means the pod is missing a toleration or should not go there at all. didn't match Pod's node affinity/selector points to a label typo. pod has unbound immediate PersistentVolumeClaims means storage, not compute: check the StorageClass and the CSI driver. If the reasons are all capacity and a node autoscaler is installed, check its logs next, because that is the component that should have reacted.

Recipe: a Service with no traffic

kubectl get svc api -n shop -o wide                                  # selector and ports
kubectl get endpointslices -n shop -l kubernetes.io/service-name=api -o wide
kubectl get pods -n shop -l app=api -o wide                          # same labels? Ready?
kubectl run tmp -n shop --image=busybox:1.36 --rm -it --restart=Never -- sh
# inside the debug pod:
nslookup api
wget -qO- -T 3 http://api:8080/ready
wget -qO- -T 3 http://<pod-ip>:8080/ready

Work from the object layer outwards. An empty EndpointSlice is a selector mismatch or pods that are not Ready. A full slice that still times out usually means targetPort does not match the container's port, or the app binds 127.0.0.1. If calling the pod IP directly works but the Service name fails, look at DNS and at NetworkPolicies in the namespace. If the pod IP fails too, the problem is the app itself.

Recipe: memory kills and evictions

kubectl get pods -n shop -o 'custom-columns=NAME:.metadata.name,QOS:.status.qosClass,RESTARTS:.status.containerStatuses[0].restartCount,LAST:.status.containerStatuses[0].lastState.terminated.reason'
kubectl describe pod -n shop <pod> | grep -A5 'Last State'
kubectl top pods -n shop --sort-by=memory
kubectl describe node <node> | grep -A8 Conditions
kubectl get pods -A --field-selector=status.phase=Failed | grep -i evicted

Separate the two cases before changing anything. OOMKilled with exit 137 in the last state means the container hit its own limit; the node was fine. Evicted means the node was under pressure and the kubelet chose victims, beginning with the pods furthest above their requests. For OOM kills, check whether the runtime sizes its heap from the node rather than the cgroup, which older JVMs and Node versions do, before raising the limit. For evictions, find the pods with no requests on that node, and look at MemoryPressure or DiskPressure in the node's conditions.

Recipe: rolling config safely

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deployment.yaml
configMapGenerator:
- name: api-config
  literals:
  - LOG_LEVEL=info
  - MODEL_VERSION=2026-09-15

kubectl kustomize . | grep 'name: api-config'      # api-config-<hash>
kubectl diff -k .
kubectl apply --server-side --field-manager=ci -k .
kubectl rollout status deploy/api --timeout=5m

The generator appends a hash of the content to the ConfigMap name and rewrites every reference in the pod template. A config change therefore changes the template, which triggers a normal rolling update with readiness gates, a history entry and rollout undo. The old ConfigMap stays, so pods from the previous ReplicaSet keep their config during the rollout. Clean up old generated ConfigMaps periodically, or use --prune with care.

Recipe: draining a node without surprises

kubectl get pdb -A                                  # any with ALLOWED DISRUPTIONS 0?
kubectl cordon <node>
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data --timeout=15m
# maintenance happens here
kubectl uncordon <node>
kubectl get pods -A -o wide --field-selector spec.nodeName=<node>

Check PDBs before you start. Any budget showing 0 allowed disruptions will block the drain until something changes, and drain will keep printing Cannot evict pod as it would violate the pod's disruption budget. Fix the budget or scale up first. --delete-emptydir-data acknowledges that emptyDir contents will be lost, which is what you want for caches and scratch space but not for anything you forgot to put on a PVC. After uncordoning, remember that existing pods do not move back on their own; the node fills again as new pods are scheduled.

Recipe: debugging a node from inside the cluster

When several unrelated pods on the same node misbehave, the problem is probably the node, and you may not have SSH access to it, which is common on managed clusters.

kubectl describe node <node>                        # conditions, pressure, taints
kubectl debug node/<node> -it --image=busybox:1.36 --profile=sysadmin
# inside the debug pod:
chroot /host journalctl -u kubelet --since "30 min ago" | tail -50
chroot /host crictl ps -a
chroot /host df -h /var/lib/containerd

The debug pod mounts the host filesystem at /host. chroot /host gives you the node's own tools, if the node image has them. Look for disk usage under the container runtime's directory, kubelet errors about the runtime, and containers stuck in created or exited states. Delete the debug pod when you are done, because it runs with elevated privileges.

Recipe: finding waste

kubectl top pods -A --sort-by=cpu | head -20
kubectl get pods -A -o 'custom-columns=NS:.metadata.namespace,NAME:.metadata.name,CPU_REQ:.spec.containers[0].resources.requests.cpu,MEM_REQ:.spec.containers[0].resources.requests.memory'
kubectl get pvc -A -o 'custom-columns=NS:.metadata.namespace,NAME:.metadata.name,SIZE:.spec.resources.requests.storage,UNUSED:.status.conditions[?(@.type=="Unused")].reason'
kubectl get deploy -A -o 'custom-columns=NS:.metadata.namespace,NAME:.metadata.name,REPLICAS:.spec.replicas' | awk '$3=="0"'

Compare actual usage with requests per workload; a request several times larger than sustained usage is money reserved for nothing. Since 1.37, PVCs carry an Unused condition, so claims left behind by deleted StatefulSets are easy to list. Deployments scaled to zero for months, with their claims and Services still in place, are the other common leftover. This audit often pays for itself in the first month.

Recipe: auditing who can do what in a namespace

Before handing a namespace to a new team member, or after a Forbidden error, check the permissions as the API server sees them rather than reading YAML and guessing.

kubectl auth whoami
kubectl auth can-i --list -n shop
kubectl auth can-i --list -n shop --as=system:serviceaccount:shop:deployer
kubectl auth can-i get secrets -n shop --as=jane
kubectl get rolebindings -n shop -o wide
kubectl get clusterrolebindings -o wide | grep -i shop

Look for three things. Wildcards (* in verbs or resources) grant far more than anyone intended, including verbs that future API versions add. list or watch on Secrets returns their contents, not only their names. And create on pods in a namespace means the holder can mount any Secret in that namespace, so treat it as roughly equal to reading them. A ServiceAccount that only needs to update one Deployment should have a Role that says exactly that, bound in exactly one namespace.

Recipe: moving a route off ingress-nginx

The community ingress-nginx controller was retired on 24 March 2026 and receives no further security patches. Existing Ingress objects still work, so the migration can be gradual: run a Gateway API implementation alongside, move one host at a time, and switch DNS when the route reports healthy.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: api
  namespace: shop
spec:
  parentRefs:
  - name: public
    namespace: gateway-system
  hostnames:
  - api.example.com
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: api
      port: 8080

kubectl get gatewayclass
kubectl get gateway -A
kubectl apply -f httproute.yaml
kubectl describe httproute api -n shop       # Accepted=True and ResolvedRefs=True

Translate annotations carefully. Most ingress-nginx behaviour lived in controller-specific annotations such as rewrites, timeouts and canary weights, and Gateway API expresses much of it as first-class fields like filters and weighted backendRefs. Anything without an equivalent is an implementation-specific policy, so check your chosen implementation's documentation rather than assuming. The Helm guide and Argo CD guide cover packaging and rolling this change out through GitOps.

Habits worth adopting now

Run kubectl diff before every manual apply. It shows exactly what will change and catches the wrong context or namespace before it matters. Put the current context in your shell prompt for the same reason.

Give each writer its own field manager. --field-manager=ci, --field-manager=hotfix-jane. When a conflict appears, the name tells you who to talk to.

Keep one namespace per application per environment. RBAC, quotas, Pod Security labels and NetworkPolicies all work at namespace scope, so the namespace is your unit of isolation and of clean-up.

Label everything with the recommended labels. app.kubernetes.io/name, app.kubernetes.io/instance, app.kubernetes.io/version, app.kubernetes.io/part-of. Selectors, dashboards and cost reports all become easier.

Pin versions narrowly. Image tags by commit SHA or digest, Helm chart versions exact, and your kubectl within one minor of the API server, which the version skew policy requires.

Set timeZone on every CronJob. Asia/Riyadh, Asia/Dubai or Africa/Cairo explicitly, instead of converting to UTC in your head and getting it wrong at a daylight-saving change.

Stop teaching old habits. Use EndpointSlices, not Endpoints. Use native sidecars, not a second container with a shutdown hack. Use Gateway API for new routing, not ingress-nginx, which was retired in March 2026. Use kubectl apply -f rather than the deprecated kubectl run -f.

Senior tips go further: control-plane failure modes and which component breaks first at scale, cluster upgrades one minor at a time, encryption at rest and external secret stores, admission policy with CEL, multi-tenancy beyond namespaces, GPU fleet cost, and the review checklists for running Kubernetes as a platform for other teams.

This is part three of three. Beginner got a Deployment running. Mid explained the machinery precisely enough to predict it: watches and controllers, rollouts, scheduling, eviction, probes, Services, RBAC for a team. This level is about the cluster itself, and about the moment it stops being your tool and becomes something other teams depend on.

The questions change. It is no longer "why is my pod Pending" but "why is every kubectl command in the company timing out", "who can read the production database password, and how would we know if they had", "how do we get from 1.35 to 1.37 without an outage window", and "the data science team wants forty GPUs and a guarantee, what do we tell them". Those questions have answers, but the answers come from the architecture, the trust model and the economics of a shared cluster rather than from any single object's YAML.

Everything here is written against Kubernetes 1.37 "Garhwal" (released 26 August 2026, latest patch v1.37.1). Several behaviours a platform owner cares about changed in 1.35, 1.36 and 1.37, and the sections say so where they did.

Where this picks up

Each section starts with the problem, then the mechanism, then the decision you would actually make. Most end with a Try it that runs on a disposable kind cluster or a kubeadm lab; the few that need a managed cloud cluster say so. None of them should be tried first on a cluster anyone else uses.

The control plane as a distributed system

At Mid level the control plane was a set of boxes that cooperate. For an owner, it is a distributed system with a single source of truth, and the useful exercise is to walk through what happens when each piece fails, because that ordering is what your alerts and runbooks should reflect.

etcd is the only component that holds state. It is a Raft-based key-value store: a cluster of members elects a leader, every write goes through the leader, and a write is committed only when a majority (the quorum, n/2 + 1) has persisted it. Three members tolerate one failure; five tolerate two. Four members tolerate only one, the same as three, while adding another machine that must agree on every write, which is why member counts are always odd. Lose quorum and etcd refuses writes, and without writes the entire cluster becomes read-only at best.

kube-apiserver is stateless. It authenticates, authorises, runs admission, validates, and persists to etcd; nothing else in the cluster talks to etcd. Because it is stateless you run several behind a load balancer, and any of them can serve any request. Each instance keeps a watch cache: an in-memory copy of recent objects per resource type, fed by a watch on etcd, from which it serves most lists and watches without touching etcd at all. The watch cache is the reason a cluster with thousands of controllers watching pods does not melt etcd, and it is also why an API server that has just started is briefly fragile: it has to fill that cache before it can serve cheaply.

kube-scheduler and kube-controller-manager are also stateless, but they must not act twice. They run several replicas and use leader election through a Lease object: one replica holds the lease and works, the others wait. cloud-controller-manager, where you have one, does the same.

kubelets on every node watch for pods bound to their node and drive the container runtime through CRI. They hold no cluster state beyond what they need to run their own pods.

That last point is the most important fact in this section, and it shapes incident response. When the API server is unreachable, your applications are usually still up. The danger is that nothing reconciles: a node that dies is not replaced, a crashing pod on a healthy node is restarted by its kubelet but a deleted pod is not recreated, endpoints do not update when pods become unready, and autoscalers freeze. A control-plane outage is a clock, not an immediate crisis, and the right first move is rarely a drastic one.

The failure table is worth memorising, because each row points at a different first command.

Notice that a missing scheduler produces no event. The event FailedScheduling is written by the scheduler, so when the scheduler is gone, a pod sits in Pending with an empty events list. An empty events list on a Pending pod is itself a diagnosis.

Topology decisions. A highly available self-managed control plane is at least three control-plane nodes behind a load balancer, spread across failure zones, with etcd either stacked (an etcd member on each control-plane node, simpler, and what kubeadm does by default) or external (a separate etcd cluster, more machines but independent failure and scaling). Stacked is fine for most clusters. External etcd earns its cost when the etcd disk profile or the blast radius of a control-plane node failure starts to matter. With kubeadm, the --control-plane-endpoint flag must be set at kubeadm init time if you might ever want more than one control-plane node, because it becomes the address baked into every certificate and kubeconfig; adding it later is a painful migration.

On a managed service (EKS, GKE, AKS and similar) the provider runs etcd and the API servers, and this whole section becomes a question of what the provider exposes: API server metrics, audit logs, a control-plane tier with a stated SLA. You still own everything from admission policy down, and you still suffer the failure modes, just with less ability to fix them.

etcd, the component you cannot lose

Everything else in the control plane can be rebuilt from scratch in minutes. etcd cannot, which is why it gets its own section and its own on-call attention.

Performance. etcd commits every write with an fsync to its write-ahead log, so it is sensitive to disk latency far more than to CPU. The practical rules are a fast SSD, ideally dedicated to etcd, and low, stable network latency between members. When etcd is slow, the symptom appears upstream: API requests take seconds, leader elections flap because leases cannot be renewed in time, and controllers log timeouts. The etcd metrics that tell you it is the disk are the WAL fsync and backend commit duration histograms; if their high percentiles climb into tens of milliseconds, look at the disk before anything else.

Size. etcd has a storage quota (the --quota-backend-bytes flag, 2 GiB by default). When the database hits it, etcd raises a NOSPACE alarm and refuses writes until someone compacts, defragments and disarms the alarm. The API server compacts old revisions automatically, but compaction only marks space as free; defragmentation returns it. The objects that fill etcd in practice are Events on a busy cluster, large ConfigMaps and Secrets (the 1 MiB object limit exists to protect etcd), and custom resources from an operator that writes status too often. That last one is common and worth checking whenever the database grows faster than your workload does.

# The certificates kubeadm generates for etcd (endpoints are passed per command)
export ETCDCTL_API=3
TLS="--cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key"
EP="--endpoints=https://127.0.0.1:2379"

# Health, leader, DB size, raft index per member
etcdctl $TLS $EP endpoint status --cluster -w table
etcdctl $TLS $EP endpoint health --cluster

# Any alarms raised (NOSPACE, CORRUPT)?
etcdctl $TLS $EP alarm list

# Reclaim space on one member at a time, never all at once
etcdctl $TLS --endpoints=https://10.0.0.11:2379 defrag

Defragmentation blocks the member it runs on while it works, so you run it one member at a time and never against the whole cluster in parallel.

Backups. A snapshot is a consistent copy of the whole keyspace, taken from one member:

etcdctl $TLS $EP snapshot save /var/backups/etcd-$(date +%Y%m%dT%H%M).db
etcdutl snapshot status /var/backups/etcd-20260930T0200.db -w table

Restore uses etcdutl, not etcdctl: etcdctl snapshot restore has been deprecated in favour of etcdutl since etcd 3.5. A restore writes a new data directory, which you then point the etcd static pod (or service) at:

etcdutl snapshot restore /var/backups/etcd-20260930T0200.db \
  --data-dir /var/lib/etcd-restored

A backup you have never restored is a hope, not a backup. Two properties make etcd backups special. First, they contain every Secret in the cluster, so if encryption at rest is not enabled, the snapshot file holds every password in readable base64 and must be treated as the most sensitive file your organisation owns. Second, they are data subject to residency rules: an etcd snapshot of a cluster serving Saudi or Egyptian customers contains their personal data wherever it lands, so the backup bucket belongs in the same region as the cluster, not in whatever region your backup tooling defaults to.

On managed services you do not get etcd snapshots at all. The equivalent is backing up the objects (Velero, or simply the Git repository your GitOps tool deploys from) and the persistent volumes, and accepting that anything not in Git or a backup is lost with the cluster. For most platform teams that is the better model anyway: if the cluster can be rebuilt from Git plus volume snapshots, etcd stops being a single point of failure for your organisation's knowledge.

Scaling: which component breaks first

The published limits for 1.37 are 5,000 nodes, 110 pods per node, 150,000 pods and 300,000 containers in one cluster. Hardly anyone reaches them. Clusters fail long before that, and the order in which things break is predictable enough that you can plan for it.

First: the API server, from list and watch traffic. The most expensive thing a client can do is list a large collection without a limit, especially across all namespaces, and especially with a label selector the watch cache cannot help with. One misbehaving controller that lists every pod every ten seconds, or a DaemonSet whose agent on each of 800 nodes lists all pods at startup, can consume more API server capacity than every human user combined. The fix is almost never "a bigger API server"; it is finding the client. The API server's own metrics break request counts and latency down by verb, resource and client, and the audit log records the user agent.

Second: admission webhooks. Every write to a matched resource calls every matching webhook, synchronously, before it is persisted. A webhook with a five-second latency makes every pod creation take five seconds; a webhook that is down with failurePolicy: Fail blocks every write it matches. Webhooks scale with write traffic, so they break during exactly the events that create the most writes: a large rollout, a node pool replacement, a cluster upgrade.

Third: CRD conversion webhooks, which matter more in 1.37. When a custom resource is stored in one version and read in another, the API server calls the operator's conversion webhook. In 1.37 ConcurrentWatchObjectDecode is on by default, so while filling its watch cache the API server can make up to ten conversion calls concurrently. A conversion webhook that caps its own concurrency below ten, or that runs as a single small replica, becomes the bottleneck of API server startup. After an upgrade, a restart of the API servers can then be slow in a way that looks like a control-plane bug.

Fourth: etcd, from database size or write latency, covered above.

Fifth: the cloud around the cluster. Adding nodes in large batches hits cloud API rate limits and quotas: instances, IP addresses, load balancers, disks. With most CNI plugins on the big clouds, every pod takes a real VPC address, so pod density per node is bounded by the network plan you made on day one. Running out of IP space in a subnet is one of the most common reasons a cluster "cannot scale" when CPU is plentiful.

API Priority and Fairness

The API server's defence against a noisy client is API Priority and Fairness (APF, flowcontrol.apiserver.k8s.io/v1). Each request is classified by a FlowSchema into a PriorityLevelConfiguration; each priority level gets a share of the server's concurrency, and within a level, requests are grouped into flows (by user, or by namespace) and queued fairly, so one flow cannot starve the others. When a level's queues are full, the request is rejected with HTTP 429. Built-in levels already protect system traffic, leader election and node heartbeats from ordinary workload traffic.

The platform move is to give a known-heavy client its own small level, so that when it misbehaves it throttles itself instead of everyone:

apiVersion: flowcontrol.apiserver.k8s.io/v1
kind: PriorityLevelConfiguration
metadata:
  name: batch-controllers
spec:
  type: Limited
  limited:
    nominalConcurrencyShares: 10     # a small share of total concurrency
    lendablePercent: 50              # may lend unused capacity to others
    limitResponse:
      type: Queue
      queuing:
        queues: 16
        handSize: 4
        queueLengthLimit: 50
---
apiVersion: flowcontrol.apiserver.k8s.io/v1
kind: FlowSchema
metadata:
  name: batch-controllers
spec:
  priorityLevelConfiguration:
    name: batch-controllers
  matchingPrecedence: 1000            # lower numbers are matched first
  distinguisherMethod:
    type: ByUser
  rules:
  - subjects:
    - kind: ServiceAccount
      serviceAccount:
        name: pipeline-controller
        namespace: ml-pipelines
    resourceRules:
    - verbs: ["list", "watch", "get"]
      apiGroups: ["*"]
      resources: ["*"]
      namespaces: ["*"]

# Which priority level is rejecting, and for whom?
kubectl get --raw /metrics | grep apiserver_flowcontrol_rejected_requests_total
kubectl get --raw /debug/api_priority_and_fairness/dump_priority_levels
kubectl get flowschemas

The 1.37 change to know. While an API server is initialising its watch cache (right after it starts, or after an upgrade), it now answers overload with HTTP 429 plus a Retry-After header instead of passing expensive requests through to etcd. Well-behaved clients, including anything built on client-go, back off and retry. Home-grown scripts and clients in other languages that treat 429 as fatal will fail during every API server restart. If a team's tool breaks "only during upgrades", this is the first thing to check.

The trust model

Mid taught RBAC as "give the team the verbs it needs in its namespace". An owner has to think about it from the other side: given everything that has been granted, what is the worst thing each identity could do? Kubernetes permissions compose in ways that are not obvious from reading a Role, and several ordinary-looking verbs are equivalent to much larger ones.

A request passes three gates in order. Authentication establishes who you are: an X.509 client certificate, a bearer token (a ServiceAccount's projected token, or an OIDC ID token), or an exec credential plugin that fetches one. Authorisation decides whether that identity may perform that verb on that resource; in practice this is the Node authoriser (which limits each kubelet to objects related to its own pods) followed by RBAC. Admission can then reject or modify the request, which is the subject of the next section. RBAC is additive only: there are no deny rules, so the only way to take a permission away is to find and remove every binding that grants it.

Escalation paths hiding in ordinary verbs

These are the grants that look modest and are not. They come from the official RBAC good-practices page, and every one of them appears in real clusters.

Two recent changes shrink this table. Fine-grained kubelet authorisation (GA in 1.36) lets monitoring agents read kubelet metrics and pod lists through narrow subresources instead of the all-powerful nodes/proxy, so a metrics agent no longer needs a permission that includes exec. Constrained impersonation (Beta since 1.36) lets you grant impersonation for specific actions rather than "become this identity for everything". If a vendor's Helm chart still asks for nodes/proxy, that is a question for the vendor.

You audit the result with the API rather than by reading YAML:

# What can this identity do, anywhere?
kubectl auth can-i --list --as=system:serviceaccount:ml-pipelines:pipeline-controller

# Who can read Secrets in production? (bindings that grant it)
kubectl get rolebindings,clusterrolebindings -A -o wide | grep -E 'admin|edit|secret'

# The dangerous ones specifically
kubectl auth can-i create pods/exec -n payments --as=jane@example.com
kubectl auth can-i impersonate users --as=system:serviceaccount:ci:deployer

Third-party tools such as rbac-tool or kubectl-who-can answer the reverse question ("who can do X") across all bindings, which is the one you want during a review.

Human identity and break-glass

Humans should authenticate through your identity provider, not with certificates. Certificates cannot be revoked in Kubernetes: a client certificate is valid until it expires, and there is no revocation list. If an employee leaves with a kubeconfig containing a client certificate valid for a year, the only reliable fix is rotating the cluster CA. OIDC tokens, by contrast, expire in minutes and are revoked when the person is removed from the identity provider.

The structured authentication configuration (GA in 1.34) replaced the old single-issuer --oidc-* flags with a file passed to --authentication-config. It supports several issuers, claim validation rules and CEL claim mappings, and it is reloaded without restarting the API server:

apiVersion: apiserver.config.k8s.io/v1
kind: AuthenticationConfiguration
jwt:
- issuer:
    url: https://login.example.com/realms/platform
    audiences:
    - kubernetes-prod
  claimValidationRules:
  - expression: "claims.email_verified == true"
    message: "email must be verified"
  claimMappings:
    username:
      claim: email
      prefix: "oidc:"
    groups:
      claim: groups
      prefix: "oidc:"

Groups arrive with an oidc: prefix, so a ClusterRoleBinding can target oidc:platform-admins and the group can never collide with a built-in system: group.

Then there is the credential that bypasses all of this. Members of the group system:masters are authorised before RBAC is consulted, and nothing in RBAC can restrict them. kubeadm writes /etc/kubernetes/super-admin.conf with exactly that group, as a break-glass credential, and binds the ordinary admin.conf through RBAC (the kubeadm:cluster-admins group) so that it can be revoked. The rule is simple: super-admin.conf goes offline, into a sealed store with access that pages someone, and is never copied onto a laptop. Nobody's everyday identity should be in system:masters.

Audit logging is the other half of the trust model: without it, you cannot answer "did anyone read that Secret?". An audit policy chooses, per rule, how much to record. The sensible pattern records metadata only for Secrets (never their bodies), full requests for RBAC and admission changes, and nothing for high-volume read noise:

apiVersion: audit.k8s.io/v1
kind: Policy
omitStages: ["RequestReceived"]
rules:
- level: None                       # kube-proxy's constant watches are noise
  users: ["system:kube-proxy"]
  verbs: ["watch"]
- level: Metadata                   # who touched a Secret, never its contents
  resources:
  - group: ""
    resources: ["secrets", "configmaps"]
- level: RequestResponse            # every change to who-can-do-what
  resources:
  - group: "rbac.authorization.k8s.io"
  - group: "admissionregistration.k8s.io"
  verbs: ["create", "update", "patch", "delete"]
- level: Metadata                   # everything else, lightly

The API server takes it with --audit-policy-file and writes to --audit-log-path (or a webhook backend); ship the output to a SIEM outside the cluster, where someone with cluster-admin cannot edit it. On managed services the provider exposes audit logs through its own logging product, often switched off by default.

Admission as governance

Admission is where a platform team's rules become enforceable. Everything that reaches etcd passes through it, in two stages: mutating admission (which may change the object, for instance to add a default label or a sidecar) and then validating admission (which may only accept or reject). Built-in plugins such as Pod Security Admission, ResourceQuota and LimitRanger run here, followed by whatever policies and webhooks you install.

There are now three ways to write your own rules, and choosing between them is an architectural decision.

For rules that can be expressed as a function of the object alone, in-process CEL policies are now the default choice. They cannot take the cluster down, which is the single most important property an admission mechanism can have. Webhooks remain the right tool for anything that needs outside data, such as verifying an image signature against a registry, and policy engines like Kyverno bring a library of ready-made policies that is worth a great deal.

A ValidatingAdmissionPolicy has two parts: the policy (what to check) and a binding (where, and what to do on failure). This one requires production workloads to reference images by digest:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: require-image-digest
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
    - apiGroups: ["apps"]
      apiVersions: ["v1"]
      operations: ["CREATE", "UPDATE"]
      resources: ["deployments", "statefulsets", "daemonsets"]
  validations:
  - expression: >-
      object.spec.template.spec.containers.all(c, c.image.contains('@sha256:'))
      && object.spec.template.spec.?initContainers.orValue([]).all(c, c.image.contains('@sha256:'))
    message: "production images must be pinned by digest (image@sha256:...)"
    reason: Invalid
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: require-image-digest-prod
spec:
  policyName: require-image-digest
  validationActions: ["Warn", "Audit"]     # start here; move to ["Deny"] later
  matchResources:
    namespaceSelector:
      matchLabels:
        platform.example.com/environment: production

The binding's validationActions is the rollout lever. Warn returns a warning to the client (it appears in kubectl output and in CI logs) without blocking; Audit records the violation in the audit log. Run a new rule in Warn and Audit for a couple of weeks, count the violations per team from the audit log, fix or grant exceptions, and only then switch to Deny. A rule that goes straight to Deny on a shared cluster breaks somebody's deploy on day one, and the political cost of that incident outlives the rule.

Webhook failure modes

Webhooks deserve special care because they are the most common way a platform team takes down its own cluster. Each webhook configuration has a failurePolicy: Fail means that if the webhook cannot be reached, the request is rejected; Ignore means it is let through. Neither is safe by default.

• Fail on a webhook that matches pods, when the webhook's own pods are down, means no pods can be created, including the webhook's replacement pods. That is a deadlock the cluster cannot resolve on its own.
• Ignore on a security webhook means an attacker who can make it slow can bypass it.

The mitigations are routine once you know them: exclude kube-system and the webhook's own namespace with a namespaceSelector; set timeoutSeconds low (a few seconds, not the thirty-second maximum); run the webhook with several replicas, a PodDisruptionBudget and a high priorityClassName; and monitor its latency as a first-class SLO. In 1.37 there is also a new safety net worth knowing: manifest-based admission configuration (Beta, via staticManifestsDir in AdmissionConfiguration) lets the API server load policies from files on disk, so a critical baseline policy exists even before any object has been created and cannot be deleted through the API.

Pod Security Admission remains the baseline for workload hardening: label each namespace with pod-security.kubernetes.io/enforce: restricted (or baseline for the few that genuinely need more), and pin the version with pod-security.kubernetes.io/enforce-version if you want upgrades not to tighten the rules unannounced. The senior addition is that the labels themselves are now a governance question: whoever can edit a namespace's labels can switch its Pod Security level off. Either keep update on namespaces away from tenants, or write a ValidatingAdmissionPolicy that refuses changes to those labels by anyone but the platform group.

Secrets and encryption at rest

The earlier levels established that a Kubernetes Secret is base64-encoded, not encrypted. The senior questions are: where does the plaintext exist, who can reach each of those places, and how do we rotate when something leaks?

The plaintext exists in four places. In etcd, unless encryption at rest is configured. In etcd backups, for the same reason. In the API server's responses to anyone authorised to read the Secret (or to create a pod that mounts it). And on the node, in a tmpfs mount inside the pod that uses it. Encryption at rest addresses the first two; RBAC and namespace design address the third; the fourth is accepted, and is why node compromise is treated as a namespace-level breach.

Encryption at rest with KMS v2

The API server takes an EncryptionConfiguration file via --encryption-provider-config. Providers are tried in order: the first provider encrypts new writes, and all listed providers are tried when reading. The recommended provider is KMS v2, where a key in an external key-management service (a cloud KMS, or Vault through a plugin) wraps the data-encryption keys, so the key that protects your Secrets never sits on the control-plane disk.

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
  - secrets
  providers:
  - kms:
      apiVersion: v2
      name: platform-kms
      endpoint: unix:///var/run/kms-provider/kms.sock
      timeout: 3s
  - identity: {}          # still able to READ data written before encryption was enabled

identity means "no encryption". It belongs at the end of the list, as a way to read old unencrypted data during the transition. If identity is listed first, every new write is stored in plain text and the configuration is decorative. The local providers aescbc, aesgcm and secretbox exist, but they keep the key in the same file on the same disk as the API server, which protects against a stolen etcd backup and not much else.

Enabling encryption only affects new writes. Existing Secrets stay as they were until something rewrites them, and the same is true after every key rotation. The old approach was to read and replace every Secret:

kubectl get secrets -A -o json | kubectl replace -f -

In 1.37 the in-tree storage version migrator is GA (storagemigration.k8s.io/v1), which does the rewrite as a tracked, resumable API operation instead of a client loop:

apiVersion: storagemigration.k8s.io/v1
kind: StorageVersionMigration
metadata:
  name: secrets-migration
spec:
  resource:
    group: ""
    resource: secrets

kubectl apply -f migrate-secrets.yaml
kubectl wait --for=condition=Succeeded \
  storageversionmigration.storagemigration.k8s.io/secrets-migration --timeout=30m

The same object migrates custom resources after a CRD's storage version changes, which is the other place teams used to write fragile one-off scripts.

On managed clusters, envelope encryption of Secrets with a customer-managed key is usually a cluster setting rather than a file you write. It is worth switching on for every production cluster, and in regulated environments (banks and government work in the Gulf commonly require customer-managed keys held in-country) it is often a precondition rather than a choice.

Keeping the secret out of Kubernetes entirely

For many teams the stronger answer is that the source of truth for a secret is not a Kubernetes Secret at all.

• The External Secrets Operator reads from a secret manager (Vault, AWS Secrets Manager, Google Secret Manager, Azure Key Vault) and writes a Kubernetes Secret, refreshing it on a schedule. Rotation happens in one place, and the Git repository only contains a reference.
• The Secrets Store CSI Driver mounts secrets from the external store directly into the pod as files, optionally without creating a Kubernetes Secret at all.
• Sealed Secrets and SOPS encrypt secrets so that the ciphertext can live in Git for GitOps; decryption happens in the cluster.
• Workload identity (EKS Pod Identity or IRSA, GKE Workload Identity, Azure Workload Identity) removes the most common secret of all, the cloud access key, by letting a ServiceAccount's short-lived projected token be exchanged for cloud credentials. A pod that reads a model from object storage should not have a static key anywhere.

The Vault guide covers the secret-manager side of this in depth.

Workload isolation and the supply chain

The Docker guide's senior level established that containers share a kernel, so root inside a container is root on the host kernel unless something remaps it. Kubernetes gives you the same controls at pod level, plus several that only make sense in a cluster.

User namespaces (spec.hostUsers: false, GA in 1.36) map the pod's users to an unprivileged range on the node, so uid 0 inside the pod is a high, unprivileged uid outside it. A container escape then lands as nobody in particular. It needs a recent kernel and a runtime that supports ID-mapped mounts, and the defaults on current distributions and containerd 2.x are fine. For most stateless workloads it is a one-line change with no visible effect, which makes it one of the best-value hardening controls available.

apiVersion: v1
kind: Pod
metadata:
  name: inference
  namespace: ml-serving
spec:
  hostUsers: false                  # user namespace: root inside is not root on the node
  automountServiceAccountToken: false
  securityContext:
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  containers:
  - name: server
    image: registry.example.com/ml/infer@sha256:4b8e...   # pinned by digest
    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      capabilities:
        drop: ["ALL"]
    resources:
      requests: { cpu: "500m", memory: "1Gi" }
      limits: { memory: "1Gi" }

Sandboxed runtimes go further for code you did not write. A RuntimeClass names a runtime handler configured on the node, such as gVisor (which intercepts system calls in user space) or Kata Containers (a lightweight VM per pod), and a pod opts in with runtimeClassName. Use them for notebooks, user-submitted evaluation code, and anything that runs a model's generated code. They cost performance and some compatibility, which is why they are a per-workload decision rather than a cluster default.

Pod identity for service-to-service traffic. Until recently, mutual TLS between pods meant a service mesh or cert-manager. In 1.37, pod certificates and ClusterTrustBundles are GA: a projected volume source called podCertificate asks the kubelet to obtain a key and certificate for the pod from a named signer (with fields such as signerName, keyType and credentialBundlePath), write them into the pod, and rotate them, while a clusterTrustBundle projection delivers the CA bundle that peers need to verify them. The signer is something you choose and run; Kubernetes provides the request-and-rotate plumbing. For platforms that want mTLS without adopting a full mesh, this is now the native building block.

NetworkPolicy at platform scale means a default-deny policy in every tenant namespace, created by the platform when the namespace is created, plus explicit allowances for DNS and for the ingress gateway. Remember that NetworkPolicy is enforced by the CNI, not by Kubernetes; on a CNI that does not enforce it, the objects are accepted and do nothing. And never allow Service externalIPs: the field is deprecated since 1.36 because of CVE-2020-8554 (any user who can create a Service can intercept traffic to an arbitrary IP) and is due for removal in 1.43. A ValidatingAdmissionPolicy that rejects it costs five lines.

Supply chain at the admission boundary

The Docker guide covered building the evidence: digest pinning, SBOMs, scanning, provenance and signatures. Kubernetes is where that evidence gets checked, at the only point every workload must pass through.

Three rules, each enforceable at admission, cover most of it: images must come from registries you control (a CEL policy on the image prefix); images must be referenced by digest (the policy in the previous section); and images must carry a valid signature from your CI identity (a webhook such as the Sigstore policy-controller or Kyverno's image verification, since checking a signature needs a registry lookup). A mirror or pull-through cache inside your cloud region is part of this picture too: it removes a public-registry outage or rate limit from your deploy path, and for Gulf and Egyptian employers with residency rules it keeps image pulls inside the region.

OCI image volumes (GA in 1.36) bring the same machinery to model weights. A volume of type image mounts the contents of an OCI artifact read-only into the pod, so a model can be versioned, signed, scanned and admitted exactly like an application image, and shipped separately from the serving code:

volumes:
- name: model
  image:
    reference: registry.example.com/models/churn-classifier@sha256:9a1f...
    pullPolicy: IfNotPresent

Multi-tenancy

Multi-tenancy is the question every growing platform faces: several teams, or several customers, share one cluster. The official documentation draws the line that matters. Soft multi-tenancy is for tenants who trust each other to a degree, typically teams in one company, where the goal is to prevent accidents and noisy neighbours. Hard multi-tenancy is for tenants who must not be able to affect or see each other at all, typically different customers, where the goal is to withstand a hostile tenant. The two need very different architectures, and the most common mistake is building the first and describing it as the second.

A namespace alone is neither. It is a scope for names and a unit to which RBAC, quotas and policy can be attached. On its own it isolates nothing: pods in different namespaces can talk to each other freely, share nodes, share the kernel, resolve each other's Services through DNS, and compete for the same API server.

Soft tenancy: the namespace template

Soft multi-tenancy is a namespace plus a fixed set of objects that the platform creates with it, every time, identically. That set is the product. Writing it down once and stamping it out (with a Helm chart, a Kustomize base, an operator, or a tool such as Capsule) is what makes it consistent.

Node isolation is optional in soft tenancy: most teams share node pools, and taints plus tolerations give dedicated pools to the few that need special hardware or stricter separation.

The leaks that remain even with all of that are worth knowing, because tenants will find them. DNS lets any pod enumerate Service names across the cluster. Cluster-scoped objects such as CRDs, StorageClasses and PriorityClasses are shared, so one team's CRD upgrade can break another's. Operators installed with cluster-wide permissions act on every namespace. And every tenant shares the API server, the scheduler, the kernel of whatever node they land on, and the cluster's upgrade calendar.

Hard tenancy: more control planes

When tenants are hostile, or are customers who need admin-like control, the answer is more control planes, not more policy.

• A virtual control plane per tenant (vcluster is the common implementation): each tenant gets its own API server, and often its own etcd or SQLite, running as pods inside a host cluster, with workloads synced down to the host's nodes. Tenants can install CRDs and have cluster-admin inside their virtual cluster without touching the host.
• A cluster per tenant, provisioned from a template with Cluster API, Terraform or a managed service's API. Strongest isolation and simplest to reason about, at the cost of per-cluster overhead and a fleet to upgrade.
• Sandboxed runtimes and user namespaces on shared nodes, where tenants run untrusted code but a separate cluster is too expensive.

The residency row deserves a note for this region. If one customer's data must stay in Saudi Arabia under the PDPL and another's in the UAE, that is not a namespace problem; the control plane, the etcd backups, the audit logs and the nodes all need to be in the right country. That means separate clusters in separate regions (AWS, Azure, Google Cloud and Oracle all run regions in the Gulf, but check that the specific managed-Kubernetes version and GPU types you need are offered in the one you pick).

Cost and capacity

A Kubernetes cluster's cost is set by the nodes it runs, and the number of nodes is set by the scheduler's bin-packing, which works on requests, not on usage. That single fact explains most Kubernetes waste. A pod that requests 2 CPUs and uses 0.2 reserves 2 CPUs of a node that you pay for, and no amount of low utilisation on the graphs frees it. The first cost lever is therefore always the same: make requests reflect reality.

The gap between requests and usage is measurable. Compare, per workload, the requested CPU and memory with a high percentile of actual usage over a couple of weeks (from Prometheus, or from the VPA's recommendations in recommend-only mode). Clusters that have never been tuned commonly show requests several times higher than usage, because every team pads its requests to be safe and nobody is rewarded for trimming them. Two tools make fixing this less risky than it used to be. The Vertical Pod Autoscaler can compute recommendations without applying them. And in-place pod resize (GA in 1.35) lets you change a running pod's CPU and memory through the resize subresource without restarting it, which removes the main operational objection to adjusting requests on stateful or slow-starting workloads.

The other levers, roughly in order of return:

Scale to zero deserves a closer look, because it is new in 1.37 and it is exactly what inference and queue-consuming workloads have wanted. With HPAScaleToZero on by default, an autoscaling/v2 HPA can set minReplicas: 0 as long as it scales on an Object or External metric, such as queue depth, and it sets a ScaledToZero condition when it does. Resource metrics cannot drive it, for a simple reason: with zero pods there is no CPU usage to measure. KEDA remains the usual source of external metrics and complements it.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: embedding-worker
  namespace: ml-serving
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: embedding-worker
  minReplicas: 0                      # Beta in 1.37, on by default
  maxReplicas: 20
  metrics:
  - type: External
    external:
      metric:
        name: queue_messages_ready
        selector:
          matchLabels:
            queue: embeddings
      target:
        type: AverageValue
        averageValue: "30"

Chargeback is organisational rather than technical, and it is the lever that changes behaviour. OpenCost (a CNCF project) allocates node cost to namespaces and labels by their requests and usage. Showing each team its monthly cost, even without billing it, typically prompts more right-sizing than any policy, because it gives teams a reason to care. Allocate by requests, not by usage, since requests are what the team reserved and what you paid for.

GPUs and ML workloads on a shared cluster

GPU workloads break assumptions the rest of Kubernetes was built on. GPUs are expensive enough that idle ones matter to the finance team, scarce enough that teams queue for them, indivisible in the old device-plugin model, and used by distributed training jobs that are useless unless every worker starts at once. A platform serving ML teams has to handle all four.

Requesting devices. The long-standing model is the device plugin: a DaemonSet advertises an extended resource such as nvidia.com/gpu, and a pod asks for a whole number of them in limits. It works, but it can only express "N identical devices". Dynamic Resource Allocation (DRA, resource.k8s.io, core structured parameters GA since 1.34) replaces it for anything richer. A driver publishes the devices on each node as ResourceSlices with attributes (model, memory, NUMA node, and so on); the platform defines DeviceClasses; workloads request devices through ResourceClaims or ResourceClaimTemplates with CEL selectors on those attributes; and the scheduler allocates matching devices. In 1.37, DRA device taints and tolerations, extended-resource compatibility and the standard resource.kubernetes.io/numaNode attribute are GA, and a claim's status.devices reports device health.

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: one-large-gpu
  namespace: ml-training
spec:
  spec:
    devices:
      requests:
      - name: gpu
        exactly:
          deviceClassName: gpu.example.com
          selectors:
          - cel:
              expression: device.capacity['gpu.example.com'].memory.compareTo(quantity('40Gi')) >= 0
---
apiVersion: v1
kind: Pod
metadata:
  name: finetune
  namespace: ml-training
spec:
  resourceClaims:
  - name: gpu
    resourceClaimTemplateName: one-large-gpu
  containers:
  - name: trainer
    image: registry.example.com/ml/trainer@sha256:77c1...
    resources:
      claims:
      - name: gpu

The device class name and attribute names depend on the vendor's DRA driver; the shape of the request is what is standard.

Queueing. Quota answers "how much may this team hold", but GPU teams need "wait your turn". Kueue adds job-level queueing on top of Kubernetes: teams submit Jobs to a queue in their namespace, Jobs are created suspended, and Kueue admits them (unsuspends them) when the team's share of GPU capacity allows, borrowing idle capacity from other teams when there is some. It understands Jobs, JobSets, Kubeflow training jobs and Ray jobs.

Gang scheduling. A distributed training job with eight workers is useless if six are scheduled and two wait for GPUs, while the six hold GPUs doing nothing. Gang scheduling means "all or none". It has long been provided by add-ons; in 1.37 the native Workload and PodGroup APIs (scheduling.k8s.io) and workload-aware preemption are Beta, so the default scheduler can place a group of pods together. Expect the ecosystem (Kueue, JobSet, training operators) to build on these APIs rather than on their own mechanisms over the next releases.

Serving. Inference is where scale to zero, OCI image volumes for weights, and in-place resize combine. The KServe guide and Ray guide take this further, and the Kubeflow guide covers training pipelines on top of all of the above.

Upgrades and migrations

Kubernetes ships three minor releases a year, and each minor gets roughly a year of patch support. The arithmetic is unforgiving: to stay on a supported version, a cluster has to take a minor upgrade about every four months, indefinitely. As of this writing the supported minors are 1.37, 1.36 and 1.35, and 1.34 reaches end of life on 27 October 2026. Upgrades are therefore not a project; they are a recurring operation, and a platform that treats them as a project falls behind within a year.

The rules that shape the plan

You cannot skip minors. 1.35 to 1.37 is two upgrades, through 1.36. Before each, move to the latest patch of your current minor, then to the latest patch of the target.

The version skew policy decides the order. In an HA control plane, API servers must be within one minor of each other. The controller manager, scheduler and cloud controller manager may be at most one minor older than the API server, never newer. Kubelets and kube-proxy may be up to three minors older, never newer. kubectl is supported within one minor either way. So the order is always: API servers, then the other control-plane components, then nodes. The three-minor kubelet window is what lets you upgrade the control plane several times while node pools are replaced on their own schedule, but it is a buffer, not a target.

Deprecated APIs are removed on a published schedule. GA APIs are not removed within a major version, but beta APIs are, three releases after deprecation, and alpha APIs can vanish at any time. The API server tells you which deprecated APIs are still being called:

kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis

The metric is labelled by group, version, resource and the release in which it will be removed. Anything non-zero for an API removed in your target version is an upgrade blocker, and the audit log (look for the k8s.io/deprecated annotation) tells you which client made the call. Third-party scanners such as pluto and kubent check manifests and Helm releases as well.

What to check before each recent upgrade

The last few releases carry specific breaking changes that a pre-flight checklist must include. These are the ones platform teams trip on:

On top of Kubernetes itself, every add-on has its own compatibility matrix: the CNI, CoreDNS, the CSI drivers, the ingress or Gateway implementation, the policy engine, the autoscaler, the monitoring stack. kubeadm upgrades none of them. A missing CNI upgrade is the classic cause of a successful control-plane upgrade followed by a networking incident.

Doing it

On a kubeadm cluster, one control-plane node at a time and then the workers:

# First control-plane node
sudo apt-mark unhold kubeadm
sudo apt-get update && sudo apt-get install -y kubeadm='1.37.1-*'
sudo apt-mark hold kubeadm
sudo kubeadm upgrade plan
sudo kubeadm upgrade apply v1.37.1

# Every other control-plane node
sudo kubeadm upgrade node

# Every node, one at a time (or in small batches)
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data
sudo apt-mark unhold kubelet kubectl
sudo apt-get install -y kubelet='1.37.1-*' kubectl='1.37.1-*'
sudo apt-mark hold kubelet kubectl
sudo systemctl daemon-reload && sudo systemctl restart kubelet
kubectl uncordon <node>

The package repository is per minor on pkgs.k8s.io, so moving from 1.36 to 1.37 starts by changing v1.36 to v1.37 in the repository definition; apt-cache madison kubeadm then lists the available patches. kubeadm upgrade renews the control-plane certificates as a side effect, which is one more reason not to skip upgrades: a kubeadm cluster that is never upgraded hits its one-year certificate expiry and stops answering.

On a managed service, the control plane is upgraded by the provider, and the node strategy is yours. The safest pattern is blue/green node pools: create a new pool at the target version, cordon the old pool, drain it gradually so PDBs are respected, and delete it once empty. It costs temporary double capacity and gives you a trivial rollback for the nodes (uncordon the old pool) until the moment you delete them. The control plane itself cannot be rolled back on any managed service, which is why the pre-flight checks happen before, not after.

Large migrations: leaving ingress-nginx

The biggest migration many platforms face right now is not a Kubernetes version at all. The ingress-nginx project was retired on 24 March 2026: existing installations keep running, but there are no more releases, bug fixes or security patches. The Ingress API itself is frozen, not removed, and the project's recommendation is the Gateway API (latest v1.6.2).

A migration of this size has the same shape as any platform migration, and it is worth practising the shape:

1. InventoryList every Ingress, its annotations and its owners. The annotations are the real work: ingress-nginx behaviours configured through annotations (rewrites, timeouts, rate limits, auth) each need an equivalent in the new implementation.
2. Choose the implementationPick a Gateway API implementation (Envoy Gateway, Istio, Cilium, NGINX Gateway Fabric, Traefik, Kong and others) and install it alongside the old controller, behind a new load balancer.
3. Split ownershipThe platform team owns the GatewayClass and the shared Gateway with its listeners and certificates; application teams own their HTTPRoutes. This separation of roles is the main design improvement over Ingress.
4. Run bothTranslate routes team by team, test them on the new load balancer's address, then move DNS per hostname, with a short TTL, so that each move can be reverted by DNS alone.
5. Close the doorOnce traffic is gone, remove the old controller and add an admission policy that rejects new Ingress objects, so the migration cannot quietly reverse.

Incident playbooks for the cluster itself

Mid's debugging routine starts from one pod. A platform incident starts from the opposite end: many things are wrong at once, and the first job is to find the single cause they share. Ordering matters more than knowledge under pressure, so this sequence puts the cheapest, highest-information checks first.

The playbooks below are the failures that recur across organisations. Each one names the symptom as you will first see it.

The API server refuses connections. The connection to the server <host>:6443 was refused. On a managed service, check the provider's status page and your own network path (a changed firewall rule or private endpoint is common). On kubeadm, the API server is a static pod run by the kubelet, so debug it from the node, not through the API:

sudo systemctl status kubelet
sudo journalctl -u kubelet --since "15 min ago" | tail -50
sudo crictl ps -a | grep -E 'kube-apiserver|etcd'
sudo crictl logs <container-id>
sudo kubeadm certs check-expiration

The most common causes are an expired certificate (the kubeadm defaults are one year, and the cluster was never upgraded), etcd unavailable, a malformed flag or config file after a manual edit, or, since 1.35, a node that was moved to cgroup v1 and a kubelet that refuses to start there.

Certificates expired. x509: certificate has expired or is not yet valid. Renew with kubeadm certs renew all, then restart the control-plane static pods (move their manifests out of /etc/kubernetes/manifests and back, one at a time), then refresh admin.conf for your own kubeconfig. Kubelet client certificates rotate on their own when rotation is enabled; if they did not, the node shows NotReady with authentication errors in the kubelet log. Put certificate expiry on a dashboard with a thirty-day warning and this playbook never runs.

Everything is slow; API calls time out. Look at API server request latency and APF rejections by priority level, then at etcd disk latency and database size. The usual culprits, in order: a client listing large collections in a loop, a slow admission webhook, a CRD conversion webhook during cache initialisation, and etcd disk latency. kubectl get --raw /metrics | grep apiserver_flowcontrol_rejected_requests_total and the audit log's user agent field usually find the client within minutes.

Nothing can be created. Creates fail with failed calling webhook ... connection refused or context deadline exceeded. A webhook with failurePolicy: Fail is down, often because its own pods cannot be created. The immediate fix is to take the webhook out of the path:

kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations
kubectl get validatingwebhookconfiguration <name> -o yaml > /tmp/webhook-backup.yaml
kubectl delete validatingwebhookconfiguration <name>
# fix the webhook deployment, then restore
kubectl apply -f /tmp/webhook-backup.yaml

Deleting a security webhook is a real decision with a real cost, and whoever makes it should say so in the incident channel. The permanent fix is the webhook hygiene from the admission section.

Name resolution fails cluster-wide. Applications log unknown-host errors across many namespaces. Check CoreDNS pods with kubectl -n kube-system get pods -l k8s-app=kube-dns, their logs, and whether they are overloaded (CPU throttled, or OOM-killed at a limit set years ago). Then resolve from a debug pod, and check whether a new default-deny NetworkPolicy forgot to allow DNS. If CoreDNS is fine but a subset of nodes fails, look at the CNI on those nodes.

Many nodes NotReady at once. Correlated node failure is never many independent problems. Candidates: the CNI DaemonSet rolled out a bad version; the container runtime was upgraded; a kernel or OS image update landed; the nodes lost their route to the API server (a load balancer or security group change); or the kubelet serving certificates expired. kubectl describe node shows the condition messages, and kubectl debug node/<name> gives a shell with the host filesystem at /host when SSH is not available. With enableSystemLogQuery: true in the kubelet configuration, the kubelet's journal is readable through the API: kubectl get --raw "/api/v1/nodes/<node>/proxy/logs/?query=kubelet".

A namespace is stuck Terminating. Some resource in it still has a finalizer, or an aggregated API is unavailable so the namespace controller cannot list everything to delete. kubectl get apiservice showing False (MissingEndpoints) is the tell for the second case; fix or remove that APIService. Removing finalizers by hand is a last resort, because the finalizer usually exists to clean something up outside the cluster (a cloud load balancer, a disk) that will then leak.

Where Kubernetes stops

Naming the boundary is a senior skill. Kubernetes is extraordinarily general, which makes it tempting to use for everything, and every use carries a fixed cost: a control plane to run or pay for, an upgrade every four months, a security model to understand, and a team that must know all of this document.

Three boundaries come up again and again in this community's work.

Stateful data. Kubernetes runs databases well with a good operator, but "well" still means you own backup, restore, failover, upgrades and performance tuning. A managed database (in a region that satisfies your residency rules) removes all of that for a price that is usually less than the engineering time. Run databases on Kubernetes when you have a reason: cost at scale, a platform that must be identical on-premises, or a database the cloud does not offer.

Managed ML platforms. SageMaker, Vertex AI and Azure ML hide the cluster entirely and charge for it. For a small team serving a handful of models, they are often cheaper in total than a Kubernetes platform, once the platform's staff time is counted. Kubernetes wins when the GPU spend is large enough that bin-packing and queueing across teams pays for the team running it, or when models must run somewhere the managed platforms do not.

The platform team itself. A Kubernetes platform needs people who can do everything in this guide, on call. If the organisation cannot staff that, a managed service with the smallest possible amount of cluster-level customisation is a better answer than a self-managed cluster run on goodwill. That is not a failure of ambition; it is the right reading of the cost.

The Docker guide covers where plain containers are enough, and the Terraform guide covers provisioning whichever option you choose.

Running Kubernetes as a platform

Once more than a couple of teams deploy to a cluster, the highest-leverage work stops being about the cluster and starts being about what teams experience. A platform is a product whose users are engineers, and it succeeds or fails on the same terms as any product: whether the easy path is the right path.

Make the safe path the default path. The namespace template, the hardened pod defaults, the digest policy, the default-deny network policy and the chargeback labels should all be things teams get without asking. A team that runs one command (or opens one pull request) and receives a namespace with all of it will follow the standard. A team that is given a wiki page will not.

GitOps as the only write path. When every change to the cluster arrives from Git through a controller such as Argo CD or Flux, the Git history becomes the change log, review happens before the change instead of after, and rebuilding a lost cluster becomes a matter of pointing the controller at the repository. Humans keep read access and a break-glass path; write access through kubectl becomes the exception that pages someone. Packaging with Helm or Kustomize sits underneath.

Own the fleet, not the cluster. Past a certain size, the right unit is several clusters (per environment, per region, per tenant class) created from the same template with Terraform or Cluster API. Upgrades then roll across the fleet in waves: a sandbox cluster first, then non-production, then one production cluster, then the rest. A fleet also makes regional residency straightforward, since a new region is a new instance of the same template.

Publish SLOs and a support model. Tell teams what the platform promises (API availability, time to provision a namespace, upgrade notice period, response time for a request) and measure it. The Docker guide's warning about forked base images applies directly: if a reasonable request takes two weeks, teams build around the platform, and you lose the standard.

Measure the platform. Without numbers, "the platform is healthy" is an opinion.

Communicate change like a product. Each upgrade gets a notice with the date, the breaking changes that affect tenants (the table in the upgrades section, filtered to what they use), and the scanner output for their namespaces. Each new policy goes out in Warn first, with a date for Deny. Teams forgive a lot from a platform that tells them early.

The review checklist

Take this checklist away from the series. Run it against a manifest, a Helm chart, a pull request to the platform repository, or a design document, and it catches most of what this guide covers before it reaches a cluster.

The complete picture

The series' final artefact is a tenant as the platform delivers it: every control from this guide, stamped out for a new team in one file. Nothing in it is new, and you should be able to justify every line.

apiVersion: v1
kind: Namespace
metadata:
  name: ml-serving
  labels:
    platform.example.com/owner: ml-platform
    platform.example.com/cost-centre: cc-4410
    platform.example.com/environment: production
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: v1.37
    pod-security.kubernetes.io/warn: restricted
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: team-edit
  namespace: ml-serving
subjects:
- kind: Group
  name: "oidc:ml-serving-engineers"      # from the identity provider, never certificates
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: edit
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute
  namespace: ml-serving
spec:
  hard:
    requests.cpu: "40"
    requests.memory: 160Gi
    limits.memory: 160Gi
    requests.nvidia.com/gpu: "4"
    persistentvolumeclaims: "20"
    services.loadbalancers: "0"            # traffic enters through the shared Gateway
---
apiVersion: v1
kind: LimitRange
metadata:
  name: defaults
  namespace: ml-serving
spec:
  limits:
  - type: Container
    defaultRequest: { cpu: 100m, memory: 256Mi }
    default: { memory: 512Mi }
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: ml-serving
spec:
  podSelector: {}
  policyTypes: ["Ingress", "Egress"]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-and-gateway
  namespace: ml-serving
spec:
  podSelector: {}
  policyTypes: ["Ingress", "Egress"]
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
      podSelector:
        matchLabels:
          k8s-app: kube-dns
    ports:
    - { protocol: UDP, port: 53 }
    - { protocol: TCP, port: 53 }
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: gateway-system
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: inference
  namespace: ml-serving
spec:
  parentRefs:
  - name: shared-gateway                    # owned by the platform team
    namespace: gateway-system
  hostnames: ["inference.example.com"]
  rules:
  - backendRefs:
    - name: inference
      port: 8080

The cluster-wide half lives in the platform repository and applies to every tenant at once: the digest ValidatingAdmissionPolicy bound to environment: production namespaces, a policy rejecting externalIPs, the KMS v2 encryption configuration, the audit policy, the APF levels for known heavy clients, and the structured authentication configuration that makes oidc:ml-serving-engineers mean something.

# Verify, rather than trust, that the tenant is what it claims to be
kubectl apply --dry-run=server -f tenant-ml-serving.yaml
kubectl auth can-i --list -n ml-serving --as=jane@example.com --as-group=oidc:ml-serving-engineers
kubectl auth can-i get secrets -n payments --as=jane@example.com --as-group=oidc:ml-serving-engineers
kubectl run probe -n ml-serving --image=busybox:1.36 --rm -it --restart=Never -- nslookup kubernetes.default
kubectl get resourcequota,limitrange,networkpolicy -n ml-serving
kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis

Where the series leaves you

Across the three levels you have gone from a first kubectl apply to owning Kubernetes as a platform. The same topics carried all the way through, each time one layer deeper:

You should now be able to look at a cluster and see not just what runs on it but what it permits and what it risks: which failure would stop changes and which would stop traffic, which client could saturate the API server, which identities can effectively read which Secrets, whether the data in etcd and its backups is encrypted and where it lives, whether one tenant can affect another, what the cluster costs and who is spending it, and how many months remain before it falls out of support.

Carry that review instinct out of here. Most Kubernetes incidents are designed in, one reasonable-looking manifest or grant at a time, and the cheapest place to stop them is the pull request.

Sources

• Kubernetes documentation home
• Kubernetes releases
• Version skew policy
• Kubernetes v1.37 release announcement
• Kubernetes v1.37 sneak peek
• Kubernetes v1.36 release announcement
• Kubernetes v1.35 release announcement
• Kubernetes v1.34 release announcement
• Deprecation and removal of Service externalIPs
• Gateway API v1.6 release
• Deprecated API migration guide
• Kubernetes deprecation policy
• Kubernetes components
• Considerations for large clusters
• API Priority and Fairness
• Upgrading kubeadm clusters
• Creating a cluster with kubeadm
• Container runtimes
• About cgroup v2
• Encrypting confidential data at rest
• Migrate Kubernetes objects using storage version migration
• Secrets
• Good practices for Kubernetes Secrets
• Role-based access control good practices
• Pod Security Standards
• Projected volumes
• Multi-tenancy
• Resource management for pods and containers
• Node-pressure eviction
• Autoscaling workloads
• Horizontal Pod Autoscaling
• Gateway API
• Ingress
• Troubleshooting clusters
• System logs

Part three of three. Senior Kubernetes interviews rarely ask what a Deployment is. They hand you a situation, such as a platform to design, an outage in progress, or a decision to defend in front of a director, and they watch how you reason. This file is organised around those three formats: system design, incident response, and arguing a trade-off to leadership. It ends with rapid-fire questions and a self-test. Everything is current for Kubernetes 1.37.

How a senior loop differs

At junior and mid level the interviewer checks whether you know the mechanism. At senior level they assume you do, and they are checking four other things.

Do you start from requirements rather than tools? A candidate who answers "design a platform for ML teams" with "I'd use Karpenter and Kueue" before asking how many teams, which GPUs, what budget and which regions has told the interviewer they pattern-match rather than design.

Do you name failure modes unprompted? Every component you propose should come with a sentence about how it fails and what happens then. "The admission webhook is in the path of every pod creation, so it runs three replicas with a PDB, excludes kube-system, and has a two-second timeout" is the shape they want.

Do you know where the boundary is? Saying when Kubernetes is the wrong answer, or when a managed service beats self-hosting, is a strong signal, because it shows you are weighing cost rather than defending a tool.

Can you be precise under pressure? In incident questions, the order of your first three commands says more than your eventual diagnosis.

A useful skeleton for design answers, which you can say out loud so the interviewer can follow:

1. ClarifyUsers, scale, tenancy (teams or customers), regions and residency, budget, what already exists.
2. ShapeHow many clusters, managed or self-managed, and why. Draw the control plane and the tenancy boundary.
3. DefaultsWhat every tenant gets without asking: namespace template, policies, identity, observability.
4. FailureWhat breaks first, how you would know, and what the blast radius is.
5. LifecycleUpgrades every four months, cost reporting, and how the design changes at ten times the scale.

System design: a shared GPU platform for ML teams

The prompt. "Five ML teams share a budget for 64 GPUs. Today each team has its own VMs and they sit idle half the time. Design a Kubernetes platform for training and serving."

Clarify first. Ask whether the GPUs are one model or several, whether training is single-node or distributed, what the serving latency targets are, whether any data has residency constraints, and whether the teams trust each other (they are colleagues, so soft tenancy is likely acceptable).

A strong answer proposes one production cluster on a managed service, with separate node pools: a CPU pool for system components and serving front ends, and one or more GPU pools, each tainted NoSchedule so that only pods that tolerate the taint land there. Each team gets a namespace from a template with quota, LimitRange, default-deny NetworkPolicy, restricted Pod Security and owner labels.

The core of the answer is sharing GPUs fairly without static partitioning. Quota alone gives each team a hard ceiling, which recreates the idle-VM problem inside the cluster. A queueing layer such as Kueue fixes that: each team has a guaranteed share, jobs are admitted when capacity is available, and a team can borrow idle GPUs from others, with preemption when the owner needs them back. Distributed training needs gang scheduling, so an eight-worker job never holds six GPUs while waiting for two; 1.37 has native Workload and PodGroup APIs in Beta for this, and queueing tools handle it today. Devices are requested through DRA (resource.k8s.io, GA core since 1.34) where the driver supports it, so jobs can ask for "a GPU with at least 40 GiB" rather than a bare count.

For serving, the answer should mention HPA scale to zero (Beta and on by default in 1.37, driven by Object or External metrics such as queue depth, never CPU), model weights shipped as OCI image volumes (GA in 1.36) so they are versioned and signed like code, and in-place resize for tuning requests without restarts.

Failure modes to name: a stray CPU pod landing on a GPU node and pinning it (the taint prevents it); a training job preempted halfway (checkpointing is the teams' job, and the platform should document preemption rules); node autoscaler unable to scale down because of a pod with local storage; GPU driver upgrades as a separate, careful rollout.

What a weak answer misses: fairness (it gives each team a fixed quota and stops), gang scheduling, and any measurement. A strong answer closes with the metric that proves success: GPU allocation and utilisation per team, compared with the VM baseline.

System design: a multi-region platform with data residency

The prompt. "A fintech serves customers in Saudi Arabia, the UAE and Egypt. Regulators require that each country's customer data stays in-country. Design the Kubernetes estate."

The key move is recognising that residency is not a namespace property. The control plane's etcd, its backups, the audit logs, the persistent volumes and the log pipeline all contain customer data or metadata, so each country needs its own cluster (or clusters) in an in-country region, or on-premises where no suitable cloud region exists. Namespaces inside one cluster cannot satisfy this, no matter how good the policy.

A strong answer describes a fleet: identical clusters created from one template with Terraform or Cluster API, one per country and environment, with GitOps (Argo CD or Flux) deploying the same application manifests to each and per-country overlays for configuration. Shared services that are not customer data, such as the CI system and the container registry, can be central, but images should be mirrored into each region so a cross-border outage does not stop deploys. Secrets come from an in-country secret manager, and encryption at rest uses a customer-managed key held in-country (a common regulatory expectation for banks in the Gulf). Backups (Velero, volume snapshots) target in-region storage only.

Failure modes and lifecycle: a fleet multiplies upgrade work, so the answer needs upgrade waves (sandbox, non-production, one production country, then the rest) and a strict "no hand edits" rule so the clusters stay identical. It should also mention that the managed Kubernetes version and GPU availability can differ by region, so the lowest common denominator constrains the template.

What a weak answer misses: backups and logs. Many candidates put the workloads in the right country and ship every audit log and etcd snapshot to a central bucket in Europe.

System design: an upgrade programme for forty clusters

The prompt. "We have forty clusters between 1.33 and 1.36. Half are out of support. Design the programme that gets them current and keeps them there."

A strong answer starts with the arithmetic: three minors a year, roughly a year of support each, so each cluster needs a minor upgrade about every four months forever, and minors cannot be skipped. A cluster at 1.33 is four sequential upgrades away from 1.37.

Then the mechanism. Inventory every cluster's version, add-ons and their versions. Pre-flight automation per target version: removed APIs from the apiserver_requested_deprecated_apis metric and a manifest scanner; containerd 2.x before 1.36; cgroup v2 before 1.35; static pods referencing Secrets or ConfigMaps before 1.37; PDBs with zero allowed disruptions; add-on compatibility. Waves: a canary cluster per class, then the rest. Blue/green node pools on managed services, so node rollback is an uncordon. Tenant communication: each team gets a notice listing the findings in their own namespaces, weeks before their wave.

To keep them current, the answer should turn the programme into a routine: a fixed quarterly calendar, the pre-flight checks in CI against every cluster continuously, and "minors behind" as a published platform metric. The strongest candidates also suggest reducing the number of clusters, or at least the number of snowflakes, since every unique cluster multiplies the cost.

Incident: every kubectl command times out

The prompt. "It is 10:00. Engineers report that kubectl hangs or times out. Applications seem fine. Walk me through it."

A strong answer first says what "applications seem fine" means: the data plane is up, so this is a control-plane or client-path problem and there is time to diagnose rather than restart things. Then, in order:

kubectl get --raw='/readyz?verbose'           # which readiness check fails, if any
kubectl get --raw /metrics | grep apiserver_flowcontrol_rejected_requests_total
kubectl get --raw /metrics | grep apiserver_request_duration_seconds_count | sort -t' ' -k2 -n | tail

If /readyz shows etcd failing, move to etcd: member health, leader changes, disk fsync latency, database size and NOSPACE alarms. If APF rejections are high for one priority level, find the flow: the audit log's user agent and the metric labels identify the client, very often a newly deployed controller or agent listing large collections in a loop. If requests are slow but not rejected, check admission webhook latency, since a slow webhook makes every write slow. If the cluster was upgraded or the API servers restarted recently, remember that in 1.37 they answer with 429 plus Retry-After while the watch cache initialises, and CRD conversion webhooks are called concurrently during that time.

Mitigation is targeted: scale down or fix the offending client, or put it in its own priority level; do not raise all APF shares, which moves the overload into etcd. Afterwards, add an alert on APF rejections per priority level and a review rule that new controllers must use watches and paginated lists.

Incident: nothing can be created

The prompt. "Deploys fail across every namespace with failed calling webhook ... context deadline exceeded. What do you do?"

A strong answer recognises the admission deadlock immediately. A webhook with failurePolicy: Fail is unavailable, and if its own pods need re-creating, they are blocked by it too. The sequence is: list webhook configurations, find the one named in the error, check whether its Service has endpoints, and if it cannot be fixed within minutes, back up the configuration and delete it, stating clearly in the incident channel what protection is now off. Fix the webhook, then restore the configuration.

The follow-up question is usually "how do you prevent it?", and the answer has five parts: exclude kube-system and the webhook's own namespace with a namespaceSelector; short timeoutSeconds; several replicas with a PDB and a high priority class; latency and error SLOs with alerts; and moving any rule that only inspects the object into an in-process ValidatingAdmissionPolicy or MutatingAdmissionPolicy, which cannot be down.

Incident: a leaked kubeconfig

The prompt. "A developer's laptop was stolen. It had a kubeconfig for production. What now?"

A strong answer asks first what credential the kubeconfig held, because the answer changes completely.

• If it used an OIDC exec plugin, disable the user in the identity provider and revoke their sessions. Tokens expire in minutes. Check the audit log for activity by that user since the theft.
• If it held a ServiceAccount token created with kubectl create token, it expires at its set duration; a long-lived legacy token Secret can be revoked by deleting the Secret.
• If it held a client certificate, there is no revocation in Kubernetes. The certificate is valid until it expires. The options are rotating the cluster CA (disruptive: every kubelet and client credential must be reissued), or, if the certificate's identity is bound through RBAC rather than system:masters, removing that user's and group's bindings so the certificate authenticates but can do nothing. If the certificate was in system:masters, RBAC cannot restrict it, and the CA rotation is the only answer.

That last point is the reason for the follow-up recommendation: humans authenticate through OIDC only, super-admin.conf stays offline, and no everyday identity is in system:masters.

Incident: nodes NotReady after an upgrade

The prompt. "After upgrading a node pool to 1.37, a third of the new nodes are NotReady and some pods that share volumes will not start."

A strong answer separates the two symptoms. For NotReady nodes, kubectl describe node for the condition message, then the kubelet journal through kubectl debug node/... or node log query. Causes specific to recent releases: a node image still on cgroup v1 (the kubelet refuses to start since 1.35), containerd 1.x (unsupported since 1.36), or a CNI version incompatible with the new kubelet. For the pods, SELinux is the clue: in 1.37 SELinuxMount is GA, so on SELinux-enforcing hosts, pods with different SELinux labels sharing a volume can fail to start; the per-pod fix is seLinuxChangePolicy: Recursive, or aligning the labels. Rollback on a blue/green node pool is uncordoning the old pool.

Arguing trade-offs to leadership

Leadership questions test whether you can turn engineering judgement into a decision someone else can make. The format that works: state the recommendation, give the two or three reasons in business terms, state the cost and the main risk honestly, and name the condition under which you would change your mind.

"Should we run our own clusters or use a managed service?" Recommend managed for almost everyone: the provider runs etcd, the API servers and control-plane upgrades, and the team spends its time on tenancy, policy and cost instead. The honest costs are provider-specific version windows, less control over API server flags, and a per-cluster fee. Self-managed wins when you must run on-premises or in a country without a suitable cloud region, or at a scale where the fee and the team's expertise both justify it. The condition to change your mind: a residency requirement that no provider can meet in-country.

"One big cluster or many small ones?" Recommend a few clusters by class (environment, region, tenant type) rather than one giant cluster or one per team. One big cluster maximises bin-packing and minimises upgrade work, but concentrates blast radius and shares one upgrade calendar. One per team isolates perfectly and multiplies cost and upgrade effort. The deciding factors are blast radius, residency, and whether tenants are internal or external. Put numbers on it: clusters times upgrades a year times hours per upgrade.

"Why should we spend a quarter on upgrades instead of features?" Frame it as risk and cost. Out-of-support minors receive no security patches; the longer the gap, the more sequential upgrades are needed and the more removed APIs pile up, so the cost grows with every quarter of delay. Offer the alternative of a routine that costs a fixed small slice of each quarter, and show the "minors behind" metric trending down.

"Can we cut the Kubernetes bill by 30 percent?" The honest answer usually starts with requests: the scheduler packs on requests, and untuned clusters commonly request several times what they use. Show the requested-versus-used table by team, propose right-sizing with VPA recommendations and in-place resize, consolidation in the node autoscaler, spot capacity for tolerant workloads, scale to zero for idle consumers, and chargeback so teams see their own spend. Give a range rather than a promise, because the saving depends on how much slack exists.

"Should the ML team use Kubernetes or SageMaker / Vertex AI?" Compare total cost, including staff time. For a small team with a few models, a managed ML platform is often cheaper and faster. Kubernetes wins when GPU spend is large enough that sharing and queueing across teams pays for the platform team, when portability or on-premises matters, or when the same platform already runs the rest of the company's workloads.

Rapid-fire questions

What strong answers share

Across every format, the answers that land have the same five habits. They clarify before designing, especially tenancy, residency and scale. They pair every component with its failure mode and its blast radius. They quote real mechanisms precisely, such as the version skew rules, the 429 behaviour, or the fact that the scheduler packs on requests, rather than gesturing at "best practices". They measure: every design ends with the metric that proves it works. And they know where Kubernetes stops, which makes their recommendations to use it believable.

The most common weak pattern is the reverse: a list of tools with no requirements, no failure modes and no numbers. If you notice yourself naming a third tool in a row, stop and say what problem each one solves.

Final self-test

Answer each aloud in under a minute, then check against the senior detailed guide.

1. What stops and what keeps working when all API servers are down?
2. Walk through what etcdutl snapshot restore does to the cluster's objects and controllers.
3. A new DaemonSet agent was deployed and kubectl became slow. What do you check, and how do you contain it without an outage?
4. Name five RBAC grants that are more powerful than they look, and why.
5. How would you roll out a new admission rule to forty teams without breaking anyone's deploy?
6. Describe the steps to enable KMS v2 encryption at rest and be certain every Secret is encrypted.
7. Design the namespace template for a new team, object by object, and say what each one prevents.
8. When is a namespace not enough, and what do you use instead?
9. Give the three largest cost levers for a cluster and how you would measure each.
10. How do you give five teams fair access to 64 GPUs, including distributed training jobs?
11. Plan an upgrade from 1.34 to 1.37 and list the pre-flight checks for each step.
12. A laptop with a production kubeconfig was stolen. What you do depends on one fact. Which fact, and what are the branches?
13. Argue for or against moving a small ML team's serving from Kubernetes to a managed ML endpoint.
14. What would you put on a one-page platform contract?

If any answer took more than a minute or needed notes, reread that section. The goal is not memorising the guide but being able to reason from the architecture to the answer, out loud, under a little pressure.

Part three of three. At this level the problems are the ones that only appear when a cluster is shared, busy and old enough to have history: a controller that saturates the API server, a webhook that blocks every deploy, a certificate nobody knew was expiring, a GPU node pinned by a stray web pod, a cluster three minors behind. Start with the trap table, then the ownership habits, the practice cards, the operational recipes, and the two review checklists. Everything is current for Kubernetes 1.37.

Production traps that only appear at scale

Cumulative: every trap from Beginner and Mid still applies. These are the ones that cause incidents for many teams at once rather than failed deploys for one.

Platform-ownership habits

These are the habits that separate a platform that teams trust from one they route around, roughly in order of leverage.

A few habits do not fit on a card but matter as much.

Keep a list of every cluster-wide grant. Every ClusterRoleBinding to a non-platform identity, every webhook, every operator with cluster-wide permissions. Review it quarterly. Grants accumulate silently, and each one is an exception to your tenancy model.

Publish your numbers. Minors behind per cluster, API server p99 latency, requested versus used resources per team, webhook error rates, time to provision a namespace. A platform that publishes its own metrics gets asked better questions.

Treat tenant requests as a product backlog with a service level. If a reasonable request (a new StorageClass, an exception to a policy, a GPU type) takes weeks, teams will build their own clusters, and your standard dies quietly. Fast turnaround is what keeps the platform the default choice.

Write the reversal condition into design documents. Every significant platform decision should state what would make you revisit it: "we run one production cluster per region until a tenant needs its own CRDs". It stops decisions becoming dogma.

Practice cards

1. Kill the scheduler, then the controller managerOn a kind cluster, move each static pod manifest out of /etc/kubernetes/manifests in turn. Scale a Deployment and delete a pod. Note which one produces no events at all, and confirm existing pods serve traffic throughout.
2. Restore etcd and watch time rewindSnapshot, create a Namespace, restore into a new data directory. The Namespace is gone. Then search the snapshot file for a Secret's value with and without encryption at rest.
3. Starve a client with APFGive a ServiceAccount its own priority level with a tiny share, hammer the API with lists as that identity, and watch it get 429s while administrators stay fast.
4. Read a Secret RBAC says you cannotGrant only create pods and read a Secret through a mounting pod. Then write the one-sentence explanation you would give a team lead.
5. Deadlock admissionInstall a webhook with failurePolicy: Fail matching pods, scale it to zero, and recover using backup, delete, fix, restore. Time yourself.
6. Encrypt, then prove itEnable encryption at rest, show an old Secret is still plain text in etcd, run a StorageVersionMigration, and show it is now encrypted.
7. Roll out a policy the polite wayDeploy a CEL digest policy in Warn, count warnings from CI logs or audit, then move to Deny and confirm the message is clear enough that nobody needs to ask you.
8. Upgrade across two minorsTake a kubeadm lab from 1.35 to 1.37 through 1.36, with a PDB that blocks drain at one point, and write the runbook as you go.
9. Find the wasteProduce the requested-versus-used table for a real namespace. Right-size the worst offender in place and confirm the restart count did not change.

Operational recipes

Short, runnable answers to the questions an owner asks most often.

Is the control plane healthy, component by component?

kubectl get --raw='/readyz?verbose'
kubectl get --raw='/livez?verbose'
kubectl -n kube-system get lease                 # who leads scheduler and controller-manager
kubectl get --raw /metrics | grep -E 'apiserver_flowcontrol_rejected_requests_total|apiserver_requested_deprecated_apis'

Is etcd healthy, and how big is it?

export ETCDCTL_API=3
etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  endpoint status --cluster -w table

When do the certificates expire (kubeadm)?

sudo kubeadm certs check-expiration

What can this identity actually do?

kubectl auth can-i --list --as=system:serviceaccount:ci:deployer
kubectl auth can-i create pods/exec -n payments --as=jane@example.com
kubectl auth can-i impersonate users --as=system:serviceaccount:ci:deployer
kubectl auth whoami

Which webhooks could block writes right now, and are they reachable?

kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations \
  -o 'custom-columns=NAME:.metadata.name,POLICY:.webhooks[*].failurePolicy,TIMEOUT:.webhooks[*].timeoutSeconds'
kubectl get apiservice | grep -v True             # unavailable aggregated APIs

Which PDBs will block the next upgrade?

kubectl get pdb -A -o 'custom-columns=NS:.metadata.namespace,NAME:.metadata.name,ALLOWED:.status.disruptionsAllowed' \
  | awk 'NR==1 || $3==0'

Which nodes run which kubelet and runtime version?

kubectl get nodes -o 'custom-columns=NAME:.metadata.name,KUBELET:.status.nodeInfo.kubeletVersion,RUNTIME:.status.nodeInfo.containerRuntimeVersion,KERNEL:.status.nodeInfo.kernelVersion'

Which kube-proxy mode is the cluster using (ipvs is deprecated)?

kubectl -n kube-system get configmap kube-proxy -o jsonpath='{.data.config\.conf}' | grep 'mode:'

Which Services still use externalIPs, and which Ingresses still need migrating?

kubectl get svc -A -o jsonpath='{range .items[?(@.spec.externalIPs)]}{.metadata.namespace}/{.metadata.name}{"\n"}{end}'
kubectl get ingress -A -o 'custom-columns=NS:.metadata.namespace,NAME:.metadata.name,CLASS:.spec.ingressClassName'

Which PVCs are unused (1.37)?

kubectl get pvc -A -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,UNUSED:.status.conditions[?(@.type=="Unused")].reason'

Re-encrypt Secrets after a key change.