The Complete Kustomize Guide

This is part one of three. It covers everything you need to do real work with Kustomize, not a teaser. By the end you can take a plain set of Kubernetes manifests, turn them into a reusable base, and produce a development version and a production version of the same application without copying a single file. You will know how to change images, replica counts, names, namespaces and configuration from one small file, how to patch anything the built-in fields do not cover, how to read the errors the tool prints, and how to preview and apply the result to a cluster. Mid-level and Senior take the same topics further; nothing here is thrown away.

Each section ends with a Try it task. Do them as you go. They take a few minutes each, and this tool only makes sense once you have watched your own YAML go in one end and come out different at the other. You do not need a cluster for most of this guide. Kustomize only reads and writes text, so a laptop and a terminal are enough until the section on applying.

The guide assumes you have seen a Kubernetes manifest before: a YAML file with apiVersion, kind, metadata and spec. If not, read the Kubernetes guide first, or at least its sections on Deployments and Services. You do not need to know Helm. Where the two tools meet, the text explains both sides.

What Kustomize is, and the problem it solves

Kustomize takes Kubernetes YAML files you already have, applies a list of edits you describe in a small file, and prints the resulting YAML. That is the whole tool. It never contacts a cluster, it never stores anything, and it never changes the files you wrote. Files go in, edited text comes out.

The problem it solves appears the first time you deploy the same application to two places. Imagine a web service. In development you want one replica, the namespace web-dev, and whatever image tag the last build produced. In production you want three replicas, the namespace web-prod, a pinned and tested image tag, and a slightly different configuration. The two deployments are 95 percent identical. The question is what to do about the other 5 percent.

The obvious answer is to copy the folder and edit the copy. It works for a week. Then someone adds a health probe to the development copy and forgets production. Someone else fixes a typo in a label in production only. Within a month the two folders have drifted, and nobody can say with confidence what differs between them or why. Kubernetes manifests have no notion of inheritance, so every copy is a fork, and forks diverge.

Kustomize's answer is to keep one copy of the shared part, called the base, and describe each environment as a small set of differences on top of it, called an overlay. The base is ordinary Kubernetes YAML that you could kubectl apply directly. The overlay is a short file that says "take the base, then set the namespace, then change the replica count, then pin the image tag". Because the differences are explicit and short, you can read them in one screen, review them in a pull request, and be sure that a fix to the base reaches every environment at once.

The official documentation describes the idea as template-free customisation, and the phrase matters. Many tools solve the same problem by turning your YAML into a template with placeholders such as {{ .Values.replicas }}. Once you do that, your manifest is no longer valid Kubernetes YAML; it is a program that has to be run before it becomes YAML. Kustomize keeps your files as real, valid, readable manifests. The edits live next to them, not inside them. You can open deployment.yaml in any editor, validate it with any tool, and apply it with plain kubectl at any time.

A useful way to hold both ideas is to compare it with two older tools. Like make, a kustomization is a declaration: you write down what you want and the tool works out the steps. Like sed, the output is your input text with edits applied, and the input is never modified. But unlike sed, Kustomize understands Kubernetes. It knows that a Deployment's name appears in a Service selector, that a ConfigMap name is referenced from a Pod, and that labels belong in particular fields. You describe the intent, and it edits the right places.

Four things that Kustomize does, in the order you will use them in this guide:

It is also built into kubectl. Since Kubernetes 1.14, kubectl apply -k and kubectl kustomize contain a copy of Kustomize, which is why so many teams meet it without ever installing anything. A later section explains the one catch with that copy: it is often older than the standalone tool.

What came before: copies, sed and templates

Understanding the alternatives makes the design choices obvious, and it is also a common interview opener. There are four approaches you will meet in real repositories.

Copy and edit. One folder per environment, each containing full manifests. It needs no tooling and is easy to understand, which is why everybody starts here. Its weakness is drift, described above. The cost grows with every file and every environment, and a review of "what is different in production" turns into a manual diff of two directory trees.

Shell scripting with sed or envsubst. Keep one set of manifests with placeholders like $IMAGE_TAG, then run a script that substitutes values before kubectl apply. This removes the copies, but the manifests are no longer valid YAML, the script is one more thing to maintain, and mistakes surface only at apply time. A typo in a sed pattern can silently change the wrong line.

Helm. The most popular Kubernetes packaging tool. A Helm chart contains templates written in Go templating language and a values.yaml file of defaults; you override values per environment. Helm is powerful and is the standard way to distribute software to other people, including most third-party projects such as ingress controllers and databases. Its cost is that templates are a programming language embedded in YAML, which gets hard to read as charts grow. Helm also installs things as numbered "releases" with their own stored state in the cluster. The Helm guide covers it in depth.

Kustomize. No templates, no parameters, no state. The base is real YAML; the edits are declared in a separate file; the output is text.

The two main alternatives make different trade-offs, and it helps to see them side by side:

Neither side is "better". Teams often use both: Helm for third-party software they install, Kustomize for their own services, and sometimes Kustomize on top of Helm output to patch a chart without forking it. Kustomize can even run Helm for you through a field called helmCharts, which the mid-level guide covers. For this guide, remember only that Kustomize is the tool for the manifests you own.

There is one more honest point in Kustomize's design that surprises newcomers. It deliberately does not have variables. If you want to inject a value such as an image tag from your build system, Kustomize's answer is to run a command that edits the kustomization.yaml file first, then build. You will see that command later in this guide. The authors made this choice on purpose: a build that depends on hidden environment variables is hard to reproduce, whereas a build that depends only on files in git gives the same output every time.

The mental model: five words

Kustomize has a small vocabulary. Learn these five words and every error message and documentation page becomes readable.

Two other terms you will see are transformer and generator. A transformer changes resources that already exist; setting the namespace, adding a label and applying a patch are all transformers. A generator creates new resources; the ConfigMap and Secret generators are the ones you will use. Every convenient field in kustomization.yaml is shorthand for a built-in transformer or generator, so the two words describe the entire set of things Kustomize does.

A kustomization deserves a moment, because the word is used in two ways. The file is called kustomization.yaml (you may also see kustomization.yml or Kustomization). The directory containing it is also called a kustomization, and in the documentation it is also called the root of that kustomization. When a later guide says "the kustomization root", it means the folder that holds the file.

The file itself is a Kubernetes-style object, with apiVersion: kustomize.config.k8s.io/v1beta1 and kind: Kustomization. The version is v1beta1 and, as of Kustomize v5.8.2, there is no v1 yet. It does not describe something that runs in a cluster; it only describes how to build YAML. You will almost always include those two lines at the top for clarity and editor support, but the tool assumes them if you leave them out.

The layering works like this:

Notice that the arrow points one way. An overlay names its base, but the base has no idea that overlays exist. That one-way dependency is what lets you add a third environment without touching anything that already works. It also means you can build the base on its own at any time, which is how you will test it.

Finally, the most important sentence to remember about how the tool runs: kustomize build reads a directory and prints YAML to standard output. It does not apply anything. Applying is a separate step, performed by kubectl or by a GitOps tool. This separation is why you can run builds freely, in any environment, with no fear of changing a cluster.

Installing Kustomize and checking the setup

You have two routes: use the copy inside kubectl, or install the standalone kustomize binary. The standalone binary is better for learning, because it is newer and its error messages and flags match this guide. The current release at the time of writing is Kustomize v5.8.2, published on 30 September 2026.

Check whether you already have it through kubectl. If you have kubectl installed, run:

kubectl version --client

The output includes a line such as Kustomize Version: v5.7.1. That is the embedded copy. It is fine for the exercises here, but it lags behind the standalone release: kubectl 1.34 and 1.35 contain v5.7.1, and kubectl 1.36 and 1.37 contain v5.8.1. None contains v5.8.2 yet. Check the version on your machine rather than assuming.

Install the standalone binary. Pick the route for your system.

On macOS with Homebrew:

brew install kustomize

On Windows with Chocolatey (Scoop also has a kustomize package, and you can download the .zip from the GitHub releases page and put kustomize.exe on your PATH):

choco install kustomize

On Linux or macOS with the official install script, which detects your operating system and processor, downloads the binary into the current directory, and refuses to overwrite an existing file:

curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh" | bash
sudo mv kustomize /usr/local/bin/

The script accepts a version and a target directory, so you can pin what you install:

curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh" | bash -s -- 5.8.2 /usr/local/bin

If you have Go installed, you can build from source. The /v5 at the end of the path is mandatory:

go install sigs.k8s.io/kustomize/kustomize/v5@v5.8.2

If you would rather not install anything, the project publishes a container image, and the entrypoint is the kustomize command itself. Image tags lag the release by a little, so check the registry for the newest one before quoting it:

docker run --rm -v "$PWD":/work -w /work registry.k8s.io/kustomize/kustomize:v5.8.1 build overlays/prod

Verify the installation:

kustomize version

v5.8.2

For more detail, ask for structured output:

kustomize version -o yaml

version: v5.8.2
gitCommit: unknown
buildDate: "2026-09-30T12:53:01Z"
goOs: darwin
goArch: arm64
goVersion: go1.26.5

Your gitCommit, goOs and goArch will differ. The line to check is version. Older releases printed a longer string such as kustomize/v5.0.0; current ones print v5.8.2. If you see a --short flag in an older tutorial, ignore it: that flag is deprecated and prints a warning.

Shell completion saves typing. For zsh, for example, kustomize completion zsh prints a script you can load from your shell profile; bash, fish and powershell work the same way.

If you ever want to know which version is running when kubectl apply -k is used, kubectl version --client answers it. That matters more than it sounds, because if your laptop and your pipeline use different versions, the same directory can build to slightly different YAML. Section "Applying to a cluster" returns to this.

Your first project, step by step

Build a tiny application from nothing. You need only a terminal. Make a working folder and a base directory inside it:

mkdir -p kustomize-demo/base
cd kustomize-demo

Step 1: write two plain manifests. These are ordinary Kubernetes objects, exactly what you would write without Kustomize. The Deployment runs the web server nginx, and the Service gives it a stable address inside the cluster.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - name: web
        image: nginx:1.25
        ports:
        - containerPort: 80

apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  selector:
    app: web
  ports:
  - port: 80
    targetPort: 80

Step 2: write the kustomization. It lives in the same directory and must be named kustomization.yaml exactly. At its simplest it lists the files to include:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deployment.yaml
- service.yaml

The resources field is the list of things to put in the output. Each entry is a path, relative to the directory holding the kustomization file. An entry can be a file, another directory that has its own kustomization (which is how overlays will refer to bases), or a remote git address. Kustomize handles the entries in order.

Step 3: build it. From the kustomize-demo folder:

kustomize build base

The argument is a directory, not a file. If you leave it out, Kustomize builds the current directory (.). The output, which goes to your terminal, is:

apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  ports:
  - port: 80
    targetPort: 80
  selector:
    app: web
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - image: nginx:1.25
        name: web
        ports:
        - containerPort: 80

Read this carefully, because it teaches four things before you have used a single feature.

First, the output is one stream of YAML, with the documents separated by ---. You could pipe it into kubectl apply -f - and get the same result as applying the two original files.

Second, the order changed. You listed the Deployment first, but the Service appears first. Kustomize sorts the output into a fixed, conventional order meant to suit kubectl apply (namespaces, then configuration, then services, then workloads, and so on). You do not control that order by default, and you rarely need to.

Third, the keys inside each object were re-sorted alphabetically. Your file had name before image; the output has image before name. Kustomize parses the YAML and writes it back out in a normalised form. The meaning is identical, but the text will not match your input character for character, and any comments are dropped. This is why people compare builds with diff rather than by eye, and why you should never expect the build output to look like your source files.

Fourth, nothing was changed in your source files. Run git status or ls and confirm it: the base directory holds the same three files. The tool is read-only with respect to your manifests.

With no edits listed, the build is an identity operation: the same objects in, the same objects out. The value appears as soon as you add edits, which is what the next sections do.

Step 4: give the output a label. Add one section to the base kustomization so you can see a transformer at work. The labels field adds labels to every resource:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deployment.yaml
- service.yaml
labels:
- pairs:
    app.kubernetes.io/name: web

Build again and look at the metadata of each object:

kustomize build base

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: web
  name: web
spec:
  ports:
  - port: 80
    targetPort: 80
  selector:
    app: web
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/name: web
  name: web
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - image: nginx:1.25
        name: web
        ports:
        - containerPort: 80

Both the Service and the Deployment now carry app.kubernetes.io/name: web, and you touched neither file. Look closely at what did not get the label: the Service's selector and the Deployment's selector and pod template are unchanged. That is deliberate, and it is the subject of an important warning in the section on common changes. For now, notice that the field name is pairs, a map of label names to values, and that labels is a list, so the dash in front of pairs: matters.

Step 5: start a second, misspelled build on purpose. Errors are part of learning. Rename the field labels to lables, then build:

kustomize build base

Error: invalid Kustomization: json: unknown field "lables"

Kustomize checks the field names strictly. A misspelling produces an immediate error that names the exact word it did not recognise, rather than silently ignoring it. Fix the spelling and continue.

Reading what the build does, and what it does not

Because every later section asks you to run kustomize build and read the output, it helps to fix a few facts about the command now.

kustomize build DIR does four things in this order: it finds kustomization.yaml inside DIR; it loads every entry under resources; it runs the generators and transformers you listed; and it prints the result to standard output. If anything fails, it prints an error to standard error and exits with a non-zero status, which is what makes it usable in scripts and pipelines.

The destination of the output is the part beginners mix up:

kustomize build base                 # print to the terminal
kustomize build base > rendered.yaml # save to one file
mkdir -p out && kustomize build base -o out/   # one file per resource, into an existing directory
kustomize build base | kubectl apply -f -   # send straight to a cluster

The -o (or --output) flag writes to a file, or to a directory if you give it one, in which case each resource gets its own file. Redirecting with > does the same as -o with a file path. Piping into kubectl apply -f - is the classic two-step deployment: build, then apply. The - means "read from standard input".

There are three things the build does not do, and they are the source of half the confusion in the first week:

1. It does not talk to a cluster. It cannot tell you whether the resources already exist, whether a namespace is present, or whether an image can be pulled. A build that succeeds only proves that the YAML could be assembled, not that Kubernetes will accept it.
2. It does not validate against Kubernetes. Kustomize knows the shape of common Kubernetes objects well enough to edit them, but it will happily print a Deployment with replicas: "three" or an unknown field. Kubernetes rejects that at apply time. If you want validation in a pipeline, add a separate tool such as kubeconform to check the output.
3. It does not remember anything. There is no release history, no rollback, no record of what you built last Tuesday. Git is your history. That is a feature when you want reproducibility, and a gap when you expect Helm-style rollbacks. Rolling back a Kustomize deployment means checking out the old commit and applying again.

There is one more piece of output you may see: lines beginning with # Warning: printed before the YAML. They are deprecation notices, and they are sent to the error stream so they do not corrupt the YAML on standard output. A later section lists the fields they refer to.

Common changes with built-in fields

The fields in this section change every resource in a kustomization. They are the first things you reach for in an overlay, and most overlays need nothing else. Each one is a one-line, declarative version of an edit you would otherwise make by hand in several files.

Start by creating the production overlay. Make the directory and add a kustomization that refers to the base:

mkdir -p overlays/prod

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base

The entry ../../base is a path from overlays/prod back up two levels and into base. It points at a directory, so Kustomize looks for a kustomization.yaml there and uses everything it produces. Building overlays/prod at this point gives output identical to the base. Now add edits one at a time.

Namespace

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
namespace: web-prod

Every namespaced resource in the output gets metadata.namespace: web-prod. You did not have to put a namespace in the base manifests, which is a good habit: a base without hardcoded namespaces can be reused anywhere. Two cautions: Kustomize does not create the namespace (add a Namespace manifest as a resource, or create it separately), and cluster-scoped resources such as ClusterRoles are not namespaced, so this field leaves them alone.

Name prefix and suffix

namePrefix: prod-

Every resource name is prefixed, so web becomes prod-web for both the Deployment and the Service. Use nameSuffix for the end of the name. This is how you run two copies of the same application in the same namespace, and it is handy for telling objects apart in logs. You do not need to fix references by hand. If a Pod refers to a ConfigMap by name, or an Ingress refers to a Service, Kustomize follows the rename and updates the reference. The documentation calls this name reference tracking, and it is one of the reasons the tool is more than sed.

Look at one thing it does not rewrite: a label selector. The Service's selector: app: web refers to labels, not to a resource name, so it stays app: web. Only real name references are rewritten.

Images

The images field changes container images without touching the Deployment file:

images:
- name: nginx
  newTag: "1.27.3"

The name is the image name as written in the base, without the tag. Here the base says nginx:1.25, so the name is nginx. The new tag replaces the old one wherever that image appears, in any kind of workload resource. You can also change the repository with newName (handy for switching to a private registry mirror) or pin to a content digest with digest. Quote tags that look like numbers ("1.27") so YAML does not read them as a floating-point value and drop a trailing zero.

This is the field that continuous-delivery pipelines edit most often, since it is the one thing that changes on every release. There is a command for it, described in the everyday commands section.

Replicas

replicas:
- name: web
  count: 3

Sets the replica count of the named Deployment, StatefulSet or other scalable resource. You could do the same with a patch, but this is shorter and says what it means. The name here is the resource's name in the base (web), not the prefixed name.

Labels and annotations

labels:
- pairs:
    environment: production
commonAnnotations:
  owner: platform-team

labels adds labels to the metadata of every resource. commonAnnotations adds annotations. Labels are used by Kubernetes to select objects, and annotations are free-form notes for people and tools, so the distinction is worth remembering: labels can change behaviour, annotations never do.

All together

Here is the overlay with every change combined, followed by its build output:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
namespace: web-prod
namePrefix: prod-
images:
- name: nginx
  newTag: "1.27.3"
replicas:
- name: web
  count: 3
labels:
- pairs:
    environment: production
commonAnnotations:
  owner: platform-team

kustomize build overlays/prod

apiVersion: v1
kind: Service
metadata:
  annotations:
    owner: platform-team
  labels:
    app.kubernetes.io/name: web
    environment: production
  name: prod-web
  namespace: web-prod
spec:
  ports:
  - port: 80
    targetPort: 80
  selector:
    app: web
---
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    owner: platform-team
  labels:
    app.kubernetes.io/name: web
    environment: production
  name: prod-web
  namespace: web-prod
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - image: nginx:1.27.3
        name: web
        ports:
        - containerPort: 80

Check each line against the overlay. The names gained the prefix, both objects moved into web-prod, the image tag changed, the replicas became 3, and the base's label plus the overlay's label both appear. Labels from the base and from the overlay accumulate; they do not replace each other.

One subtlety is worth saying aloud: the overlay never mentions the Service or the Deployment by file. It describes what should be true of the whole output, and the tool finds every place to change. That is the declarative style the official documentation calls its central idea.

Patches: changing anything else

The built-in fields cover the common cases. Sooner or later you need to change something they do not reach: a memory limit, an environment variable, a health probe, a node selector. For that there are patches. A patch is a small piece of YAML that describes a change to one resource, and Kustomize merges it into the matching resource during the build. Your original file stays untouched.

There are two kinds of patch. Both go under the same field, patches, and as a beginner you will use the first one nearly all the time.

Strategic merge patches

A strategic merge patch looks like a partial copy of the resource, containing only the fields you want to change, plus enough identity to say which resource you mean. Identity means apiVersion, kind and metadata.name. Suppose production needs three replicas and a memory limit. Create a file next to the overlay's kustomization:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: web
        resources:
          requests:
            memory: 128Mi
            cpu: 100m
          limits:
            memory: 256Mi

Reference it from the overlay:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
patches:
- path: resources-patch.yaml

Kustomize reads the patch header, finds the Deployment named web from the base, and merges the patch into it. Three rules describe the merge:

• Maps are merged. Fields in the patch are added to or replace the matching fields in the original. Fields you did not mention stay as they were. The image, the ports and the labels survive, even though the patch never mentions them.
• Lists of named items are merged by name. The containers list is matched by each entry's name. The patch entry with name: web merges into the existing container called web. You do not have to repeat its image or ports.
• Most other lists are replaced completely. If a list has no natural key, your version of the list wins and the original is discarded. Keep that in mind for something like args or command: patching it means writing the whole list.

The name in the patch header is what matters. The name must be the name as it is in the base, here web, even if the overlay also sets namePrefix: prod-. Kustomize remembers the original names while it builds, so you patch what the base calls the thing. This is the most common source of a puzzling error, which the errors section shows.

Removing a field with a patch

Patches normally add or change. To delete a field, use the special value null:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  template:
    spec:
      containers:
      - name: web
        resources: null

There is also a whole-resource directive, $patch: delete, which removes an entire resource from the output. It is the one place Kustomize lets you take something away, because the project's design otherwise avoids removal: the official guidance is that if you want less than the base provides, put less in the base.

apiVersion: v1
kind: Service
metadata:
  name: web
$patch: delete

JSON6902 patches

The second kind is a precise list of operations, written as a standard called JSON Patch (RFC 6902). Each operation has an op (add, replace or remove), a path into the document, and usually a value. Use it when you want to change one exact field, especially inside a list with no merge key, where a strategic merge patch would replace the whole list.

patches:
- target:
    group: apps
    version: v1
    kind: Deployment
    name: web
  patch: |-
    - op: replace
      path: /spec/template/spec/containers/0/image
      value: nginx:1.27.3
    - op: add
      path: /spec/template/spec/containers/0/env
      value:
      - name: LOG_LEVEL
        value: info

Three details distinguish this form. First, target is required, because the patch body carries no identity of its own; it tells Kustomize which resource to edit. Second, the body is written inline with patch: |-, a YAML block scalar, though you can also point to a file with path: and add a target. Third, paths use / between steps and numeric positions for lists, so containers/0 means the first container. If a key contains a /, as Kubernetes annotation keys often do, write it as ~1.

An add operation on a list can also append with - as the last index, as in /spec/template/spec/containers/-.

Choosing between them, and picking targets

The choice is simple. If the change reads naturally as "this resource, with these fields different", use a strategic merge patch. If it is one surgical edit deep in a list, use JSON6902. Both are listed in the same patches field and applied in the order written.

For strategic merge patches the target is inferred from the header, but you can supply a target anyway, and it can match by kind, name, namespace, labelSelector or annotationSelector. The name and namespace values are regular expressions anchored at both ends, so name: web means exactly web and name: web-.* matches everything starting with web-. One patch body can thus be applied to several resources, which is useful for something like a node selector on every Deployment:

patches:
- target:
    kind: Deployment
  patch: |-
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: ignored
    spec:
      template:
        spec:
          nodeSelector:
            pool: general

When a target is given for a strategic merge patch, the name inside the body is ignored, which is why ignored is a fine placeholder. It just has to be present.

Older spellings you will meet

Two older fields, patchesStrategicMerge and patchesJson6902, were the way to write patches in tutorials from 2019 to 2022. Both are deprecated in favour of the single patches field, and Kustomize prints a warning if you use them. kustomize edit fix converts them automatically. When you read old examples, translate in your head: same idea, different field.

Generating ConfigMaps and Secrets

Applications need configuration, and in Kubernetes that means ConfigMaps (for ordinary settings) and Secrets (for sensitive values). Writing these as YAML by hand is tedious, and worse, changing one does not restart the pods that use it. Kustomize solves both problems with generators: fields that build a ConfigMap or Secret from literals or files.

The configMapGenerator

configMapGenerator:
- name: web-config
  literals:
  - LOG_LEVEL=info
  - FEATURE_X=on

A generator creates a new resource. The build prints a ConfigMap you never wrote. Combined with the namespace and namePrefix fields from the earlier sections, here is what it emits:

apiVersion: v1
data:
  LOG_LEVEL: info
kind: ConfigMap
metadata:
  name: prod-web-config-hf678c7m2b
  namespace: web-prod

The name is not web-config. It is prod-web-config-hf678c7m2b: the prefix from namePrefix, and a hash suffix computed from the content. The hash is the feature that makes generators worth using.

Why the hash matters

Kubernetes has an irritating property: if you change a ConfigMap, pods that already use it do not notice. They keep the old values until something restarts them. Teams work around it with manual kubectl rollout restart commands and comments that say "remember to restart after editing config".

Kustomize handles it differently. When the content changes, the hash changes, so the ConfigMap gets a new name. Kustomize also rewrites every reference to the ConfigMap, in envFrom, env, volumes and so on, to the new name. A changed reference in a Deployment's pod template is a changed pod template, and Kubernetes responds to that by doing a normal rolling update. Editing the config triggers the rollout on its own.

You can watch it happen. Give the Deployment a reference to the ConfigMap by its original name, using a patch:

patches:
- target:
    kind: Deployment
    name: web
  patch: |-
    - op: add
      path: /spec/template/spec/containers/0/envFrom
      value:
      - configMapRef:
          name: web-config

Build, and the Deployment's reference is rewritten to the hashed, prefixed name:

      containers:
      - envFrom:
        - configMapRef:
            name: prod-web-config-hf678c7m2b
        image: nginx:1.25
        name: web

You wrote web-config; the output says prod-web-config-hf678c7m2b. Kustomize found the reference and updated it. Change LOG_LEVEL=info to LOG_LEVEL=debug, build again, and the suffix changes, and so does the reference. The hash you see will differ from the one above if your content differs; it depends only on the data.

There is one important consequence. Always refer to a generated ConfigMap by its original name in your manifests and patches. Never type the hashed name; it changes whenever the data does.

Files and environment files

Literals are fine for a few values. For a whole file, such as an nginx.conf, use files, which stores the file's content under a key that defaults to the file name:

configMapGenerator:
- name: nginx-conf
  files:
  - nginx.conf

To choose the key, write key=path, such as default.conf=nginx.conf. For many simple settings, envs reads a file of KEY=value lines, the same format as a .env file:

configMapGenerator:
- name: web-env
  envs:
  - settings.env

Generator files have to be inside or below the kustomization directory, because of a safety rule explained in the errors section. Use envs (plural). An older singular env key exists but is legacy.

The secretGenerator

Secrets work the same way, with a different field name:

secretGenerator:
- name: web-credentials
  literals:
  - API_TOKEN=change-me
  type: Opaque

The output is a Secret whose values are base64-encoded, with a hash suffix just like a ConfigMap. The type defaults to Opaque; use kubernetes.io/tls for certificates, with tls.crt and tls.key as files.

Options: hash names and labels

Sometimes you do not want a hash, for example when something outside Kubernetes refers to the ConfigMap by a fixed name. Switch it off for one generator:

configMapGenerator:
- name: web-config
  literals:
  - LOG_LEVEL=info
  options:
    disableNameSuffixHash: true

Or for every generator in the kustomization at once with generatorOptions:

generatorOptions:
  disableNameSuffixHash: true

Turning the hash off gives up the automatic rollout on change. If you switch it off, you have to restart the pods yourself when the data changes. options can also add labels and annotations to the generated resource, and mark it immutable.

Merging with a generator in the base

An overlay can extend a ConfigMap that the base generated. Give the overlay's generator the same name and set behavior: merge (or replace to discard the old data completely):

configMapGenerator:
- name: web-config
  behavior: merge
  literals:
  - LOG_LEVEL=warn

The default behaviour is create, which fails if the name already exists. With merge, keys in the overlay override keys in the base and new keys are added. The name, and any namespace, must match the base's ConfigMap as the base had it, before the overlay's own namespace or namePrefix changes are applied. Getting that wrong produces does not exist; cannot merge or replace, covered in the errors section.

Bases and overlays in practice

You have already used a base and an overlay. This section makes the structure explicit and covers how to grow it without making a mess.

The standard layout is one base and one overlay per environment:

kustomize-demo/
├── base/
│   ├── deployment.yaml
│   ├── service.yaml
│   └── kustomization.yaml
└── overlays/
    ├── dev/
    │   └── kustomization.yaml
    └── prod/
        ├── kustomization.yaml
        └── resources-patch.yaml

Nothing forces the names base and overlays. They are a convention every Kustomize user recognises, so following it makes your repository instantly readable to new colleagues. The directory names dev and prod are likewise only names.

What belongs in the base? Whatever is true in every environment: the Deployment, the Service, the container ports, the probes, the labels that identify the app. What belongs in an overlay? Whatever differs: replica counts, image tags, resource sizes, hostnames, namespaces, environment-specific configuration. A reliable test for any setting: "If I changed this in the base, would every environment want the change?" If yes, it belongs in the base.

Build each environment with one command, so the path is the only thing that changes:

kustomize build overlays/dev
kustomize build overlays/prod

Because all the environments share the base, a fix such as adding a readiness probe goes in one file and appears in every environment at the next build. That is the practical payoff of the whole design.

What an overlay may refer to

An overlay lists its base under resources, pointing at a directory. There are rules about what else it may load, and breaking them produces the most instructive errors in this guide.

• Referencing a directory (a base) can point anywhere, including ../../base. That is what the ../../base line does.
• Loading individual files is restricted to the overlay's own directory and below. A patch file, a generator file or a resource file living outside the overlay is rejected with a security; file ... is not in or below ... error. This is the load restriction, and it exists so a kustomization cannot quietly read arbitrary files from your disk, such as ~/.ssh/id_rsa, into a ConfigMap.
• Do not reference a single file from another kustomization's directory. If you want its contents, turn that directory into a base and reference the directory.

An overlay can also reference more than one base, or other overlays, and Kustomize will merge them. Multi-base setups go beyond a first project, but you should know the rule that applies: two resources in one build may not have the same kind, name and namespace. Include the same base twice, and you will see may not add resource with an already registered id.

Remote bases

A base does not have to be on your disk. A resources entry can be a git URL, which is how many open-source projects publish a deployable configuration:

resources:
- https://github.com/example-org/example-repo//deploy/base?ref=v1.2.3

The double slash separates the repository from the subdirectory inside it, and ref picks a tag, branch or full commit hash. Kustomize clones the repository on every build, and the default time limit is 27 seconds. For anything important, pin ref to a tag or full commit hash so the build is repeatable, and remember that a remote base is code you did not write, so read it before you trust it. Short commit hashes are not supported; use the full 40 characters.

The everyday commands, grouped by what you want to do

You have used build. Here is the whole beginner toolbox, organised by intent rather than alphabet.

See the result

kubectl kustomize DIR does the same thing as kustomize build DIR using the version embedded in kubectl. The two accept the same flags.

Start a kustomization

If you already have YAML files in a folder, kustomize create writes a starting kustomization.yaml:

kustomize create --autodetect

With --autodetect, it scans the directory for Kubernetes resource files and lists them. The generated file looks like this:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deployment.yaml
- service.yaml

Without --autodetect, pass the files yourself: kustomize create --resources deployment.yaml,service.yaml. The command also accepts --namespace, --nameprefix, --namesuffix, --labels and --annotations so you can set those fields at creation time.

Change the file with commands instead of an editor

kustomize edit edits kustomization.yaml for you. It is ideal in scripts, because editing YAML with sed is fragile, while edit understands the structure. It operates on the kustomization in the current directory, so cd into the overlay first.

cd overlays/prod
kustomize edit set image nginx=nginx:1.27.3
kustomize edit set replicas web=3
kustomize edit set namespace web-prod
kustomize edit set nameprefix prod-
kustomize edit add resource ../../base
kustomize edit add configmap web-config --from-literal=LOG_LEVEL=info

Each command changes exactly one thing. After running the first two in a dev overlay, the file contains:

images:
- name: nginx
  newName: nginx
  newTag: 1.27.3
replicas:
- count: 2
  name: web

The command normalised the entry: it filled in newName: nginx (the same as the name) and wrote the tag without quotes. That is harmless, though it is why a hand-written file and an edited file sometimes differ in style.

The set image command accepts several forms:

Tags may only contain letters, digits, ., _ and -. If you pass * as a new name, tag or digest, the existing value is kept.

This command solves the "no variables" problem mentioned earlier. In a CI pipeline, you inject the version of the image you just built like this:

cd overlays/prod
kustomize edit set image nginx=nginx:"$IMAGE_TAG"
kustomize build . | kubectl apply -f -

The shell substitutes $IMAGE_TAG before Kustomize runs, and Kustomize itself never reads your environment. The edit is recorded in the file, so you can commit it and the change is visible in git. That matches the project's own advice for passing a value into a build.

Fix deprecated fields

kustomize edit fix

It rewrites deprecated fields in the current kustomization to their modern replacements, such as patchesStrategicMerge to patches and bases to resources. Commit your work first so you can see what it did. There is a section on deprecated fields below.

Preview and apply

The next section covers these in detail.

Applying to a cluster

So far nothing has left your laptop. To deploy, you hand the built YAML to Kubernetes. You need a cluster (a local one such as kind or minikube is perfect for practice) and a working kubectl pointed at it.

One step with kubectl apply -k.

kubectl apply -k overlays/prod

The -k flag (short for --kustomize) tells kubectl that the argument is a kustomization directory, not a file. It builds with its embedded Kustomize and applies the result. You cannot combine -k with -f or -R in one command. The same flag works on kubectl diff -k, kubectl delete -k and kubectl get -k.

Two steps with a pipe.

kustomize build overlays/prod | kubectl apply -f -

The pipe form uses your standalone Kustomize instead of the one inside kubectl. Choose it when you want the newer version's features, or when you need to pass build flags, because kubectl apply -k cannot take them.

Preview first.

kubectl diff -k overlays/prod

It prints what would change between the live cluster and what you are about to apply, as a diff, without changing anything. On the first deployment everything is an addition; on later ones, the lines that differ are the ones that matter. Make it a habit before any production apply.

Create the namespace first. Setting namespace: web-prod changes objects' namespace field, but does not create the namespace. If it does not exist, the apply fails with namespaces "web-prod" not found. Create it once with kubectl create namespace web-prod, or add a Namespace manifest to the base or to the overlay's resources.

Remove what you created.

kubectl delete -k overlays/prod

This deletes every object in the build output, which is a tidy way to tear down an environment.

Where GitOps tools fit. You will meet Argo CD and Flux, which watch a git repository and apply it for you. Both understand Kustomize natively: point them at an overlay directory, and they build and apply it on every change. All the knowledge in this guide carries over unchanged, because the overlay directory is the interface. The Mid-level guide covers the details, and the catalogue has a guide for Argo CD if you want to go there next.

Configuration details and deprecated fields

A few facts about the kustomization file itself help you read other people's files and avoid outdated advice.

File name and header. The file must be called kustomization.yaml (kustomization.yml and Kustomization also work). The first two lines should be apiVersion: kustomize.config.k8s.io/v1beta1 and kind: Kustomization. They may be omitted, but if you write one you must write both. The version is v1beta1, and there is no v1 at the time of writing.

Ordering of effects. Inside one kustomization, Kustomize loads resources first, then runs generators, then transformers. You do not control the order of the convenient fields (namespace, namePrefix, images and so on) by moving lines around; they run in a fixed internal order. Patches run in the order written in the patches list. The only place order is yours to choose is the patches list.

No globs. resources: ["*.yaml"] does not work. Kustomize does not expand wildcards in the file, so list files explicitly. (kustomize edit add resource *.yaml works because your shell expands the glob before Kustomize sees it.)

Output order is fixed. Resources are sorted into a conventional order on output. A sortOptions field can change it at the top level, which a beginner rarely needs.

Deprecated fields. A large share of the tutorials online use fields that have since been deprecated. They still work, but Kustomize prints a warning and they are excluded from any future v1 version. Learn the modern names:

The warning looks like this:

# Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically.

It names the field, the replacement and the command that converts it. Treat it as a to-do item, not as an error: the build still works.

Removed features. A few behaviours from the past are gone entirely as of the v5 series and will break old examples. Generators no longer read your environment variables. The gh: shorthand for GitHub URLs was removed. Support for Starlark functions was removed in v5.5.0. If a tutorial depends on one of these, it is too old to follow.

The standalone versus embedded difference again. The fields above exist in both, but newer features, such as sourceValue in replacements (v5.7.0), need a recent version. If something from the docs fails with unknown field, check kustomize version and kubectl version --client before assuming your YAML is wrong.

Common errors and how to read them

Kustomize error messages look long and alarming, but each has a pattern: a prefix telling you what stage failed, then a detail. Read the last part first, because it usually holds the actual reason. Here are the errors you will meet in your first month.

The kustomization file is not found.

Error: unable to find one of 'kustomization.yaml', 'kustomization.yml' or 'Kustomization' in directory '/home/me/demo/overlays/qa'

You pointed build at a directory that has no kustomization file. Check the path you gave, and check the file name; kustomize.yaml and kustomisation.yaml do not count. If the directory itself does not exist you get a similar message saying must build at directory: not a valid directory.

A resource file is missing.

Error: accumulating resources: accumulation err='accumulating resources from 'service.yaml': evalsymlink failure on '/home/me/demo/base/service.yaml' : lstat /home/me/demo/base/service.yaml: no such file or directory': must build at directory: not a valid directory: ...

The phrase accumulating resources means it was loading your resources list. no such file or directory names the missing path. Kustomize tried the entry as a file first, then as a directory, and the trailing must build at directory is the second failure. The real cause is the first: the file does not exist at that path. Paths are relative to the kustomization file, not to where you ran the command.

A field is misspelled or invalid.

Error: invalid Kustomization: json: unknown field "namespce"

A key is unknown. Kustomize checks field names strictly, so a typo or a field from a newer version shows up here. A different message, yaml: line 2: did not find expected key, means the file is not valid YAML at all: check the indentation near the line it names.

A patch matches nothing.

Error: no resource matches strategic merge patch "Deployment.v1.apps/nope.[noNs]": no matches for Id Deployment.v1.apps/nope.[noNs]; failed to find unique target for patch Deployment.v1.apps/nope.[noNs]

The quoted identifier has the shape Kind.version.group/name.namespace. Here it asks for a Deployment named nope with no namespace. Compare it to the resource you meant: a typo in the name, a wrong kind, or a wrong apiVersion all cause this. Use the name the base gives the resource.

A file is outside the allowed directory.

Error: loading KV pairs: file sources: [../outside/settings.env]: security; file '/home/me/demo/outside/settings.env' is not in or below '/home/me/demo/overlays/dev'

This is the load restriction. A generator (or patch, or resource) points to a file outside the overlay's own directory tree. Move the file into the overlay, or turn the other directory into a base and refer to it as a directory. You can disable the check with --load-restrictor LoadRestrictionsNone, but as a beginner you should not. The error exists to protect you.

The same resource twice.

Error: accumulating resources: accumulation err='merging resources from 'b.yaml': may not add resource with an already registered id: Deployment.v1.apps/web.[noNs]' ...

Two resources share kind, name and namespace. The usual cause is that a base appears twice, or two files define the same object. Remove the duplicate. A namePrefix in one of the two places can also separate them.

Merging into a generated ConfigMap fails.

Error: merging from generator ... does not exist; cannot merge or replace

You used behavior: merge or replace, but no ConfigMap with that name and namespace exists in the base. The usual reasons are a typo in the name, or a namespace mismatch: generators run before the namespace field takes effect, so use the base's own namespace, not the overlay's new one.

A build that works but kubectl apply fails.

The Deployment "web" is invalid: spec.selector: Invalid value: ...: field is immutable

This is a Kubernetes error, not a Kustomize one. The build added a label to a selector (through commonLabels or includeSelectors: true) on a Deployment that already exists. Selectors cannot change. Remove that selector change, or delete the Deployment and let the apply recreate it.

Putting it all together

Here is a small end-to-end project that uses everything so far. You will build a web application with a base and two overlays, a generated ConfigMap, a patch, a pinned image and a deployment preview. It takes about fifteen minutes. The finished layout is:

shop/
├── base/
│   ├── deployment.yaml
│   ├── service.yaml
│   └── kustomization.yaml
└── overlays/
    ├── dev/
    │   └── kustomization.yaml
    └── prod/
        ├── kustomization.yaml
        └── resources-patch.yaml

The base holds what is true everywhere. The Deployment and Service are the ones from the first project. The base kustomization adds a label and a default ConfigMap:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deployment.yaml
- service.yaml
labels:
- pairs:
    app.kubernetes.io/name: shop
configMapGenerator:
- name: shop-config
  literals:
  - LOG_LEVEL=debug

Wire the ConfigMap into the Deployment by adding an envFrom block in base/deployment.yaml, using the original name:

      containers:
      - name: web
        image: nginx:1.25
        ports:
        - containerPort: 80
        envFrom:
        - configMapRef:
            name: shop-config

The dev overlay is deliberately tiny: its own namespace, a name prefix, and nothing else, so it keeps the base's debug logging and single replica.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
namespace: shop-dev
namePrefix: dev-

The prod overlay pins a tested image, scales up, sets resource limits by patch, and overrides the log level by merging into the base's ConfigMap:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
namespace: shop-prod
namePrefix: prod-
images:
- name: nginx
  newTag: "1.27.3"
replicas:
- name: web
  count: 3
patches:
- path: resources-patch.yaml
configMapGenerator:
- name: shop-config
  behavior: merge
  literals:
  - LOG_LEVEL=warn

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  template:
    spec:
      containers:
      - name: web
        resources:
          requests:
            memory: 128Mi
            cpu: 100m
          limits:
            memory: 256Mi

Build both and compare.

kustomize build overlays/dev  > dev.yaml
kustomize build overlays/prod > prod.yaml
diff dev.yaml prod.yaml

The diff should show exactly the differences you declared: the names, the namespace, the tag, the replica count, the resource limits and the log level. The ConfigMap's hash suffix differs too, because its content (the log level) differs. It should show nothing else. If something unexpected appears, you found it before Kubernetes did, which is the point.

Check that the ConfigMap wiring followed the hash. In each output, find the configMapRef under the Deployment. It should name the generated, prefixed, hashed ConfigMap, and that name should match the ConfigMap in the same file. Change a literal, rebuild, and watch both change together.

Preview and apply to a local cluster, one environment at a time:

kubectl create namespace shop-dev
kustomize build overlays/dev | kubectl diff -f -
kustomize build overlays/dev | kubectl apply -f -
kubectl get all,configmap -n shop-dev

Notice that this sequence uses the standalone build and the pipe form, as recommended earlier. For production the commands are the same with overlays/prod and shop-prod. When you are done, kustomize build overlays/dev | kubectl delete -f - removes everything.

What have you just done? You wrote the application once, described two environments in a few lines each, had the tool rewrite every reference that depended on a generated name, and verified the result before it touched a cluster. You can repeat that for any application.

What you can now do, and what comes next

You started without knowing what a kustomization was. You can now do the following, unaided:

• Explain the problem Kustomize solves and why it keeps YAML valid instead of templating it.
• Name the five core words (resource, kustomization, base, overlay, patch) and say which way the dependency arrows point.
• Install the tool, check its version, and know why the copy in kubectl may differ.
• Write a kustomization.yaml, build it, and read the output, including its normalised ordering.
• Change namespace, name prefix, images, replicas, labels and annotations with one-line fields.
• Write strategic merge and JSON6902 patches, and choose between them.
• Generate ConfigMaps and Secrets, explain what the hash suffix does, and avoid committing real secrets.
• Build a base with dev and prod overlays, and preview and apply them with kubectl.
• Recognise deprecated fields and read the most common error messages.

Practical next steps, in order of usefulness: take one of your real applications and convert it, using kustomize create --autodetect to get started; add kustomize build for each overlay to your continuous-integration checks, so a broken overlay fails a pull request before it merges; and read the output of your builds until it feels ordinary.

The Mid-level guide picks up where this ends. It explains how the build pipeline works internally, so you can predict behaviour rather than discover it, and then covers components, replacements, remote bases, Helm integration, multi-cluster layouts, secrets handling and testing. The Senior guide treats Kustomize as a platform other teams depend on. For the tools around it, go to the Kubernetes guide if the cluster side still feels shaky, the Helm guide to see the other major approach, and the Terraform guide for creating the clusters themselves.

A last piece of advice. Kustomize rewards small steps. Change one thing, build, read the output, and only then change the next. The tool never touches your files or your cluster when you build, so there is no cost to experimenting, and every confusing result is one command away from an explanation.

Sources

• Kustomize reference documentation
• Kustomization file reference
• Glossary
• Built-in plugins
• kustomize build
• kustomize create
• kustomize edit
• kustomize version
• patches
• labels
• commonLabels
• images
• replicas
• namespace
• configMapGenerator
• secretGenerator
• generatorOptions
• Eschewed features
• Versioning policy
• Installing Kustomize
• Kustomize releases on GitHub
• Declarative management of Kubernetes objects using Kustomize

Part one of three. A twenty-five-minute review of the Beginner Detailed track, then the questions interviewers ask about Kustomize. Mid-level reviews this plus its own material; Senior reviews all three.

The thirty-second answer

Kustomize customises Kubernetes manifests without templates. You keep plain, valid YAML as a base, and describe per-environment differences (namespace, image tag, replicas, patches, generated ConfigMaps) in a kustomization.yaml overlay. kustomize build reads the directory and prints the final YAML to standard output; it never contacts a cluster, and something else, such as kubectl apply, Argo CD or Flux, applies the result.

Then the line that shows you have used it: "it only ever adds or edits, the build is a pure function from files to YAML, so the same git commit always renders the same output, and there are no variables by design."

The model

The API object is kustomize.config.k8s.io/v1beta1, kind Kustomization. There is no v1 yet, and saying so is a small sign that you have read the docs.

Kustomize versus Helm

This is the most common opener. Do not answer "one is better". Answer with the trade-off.

The follow-up: "can you use both?" Yes. Helm for vendor software, Kustomize for your own services, and Kustomize on top of Helm output to patch a chart without forking it. Kustomize can also inflate a chart itself through helmCharts with --enable-helm, which is a Mid-level topic.

Say what Kustomize lacks without apology: no variables, no conditionals, no loops, no release history. The official design documents list these as deliberate non-goals, because they keep builds reproducible.

The commands to have ready

kustomize version                                   # v5.8.2; check which version you run
kustomize build overlays/prod                       # render to stdout
kustomize build overlays/prod > rendered.yaml       # save the output
kustomize build overlays/prod | kubectl apply -f -  # build, then apply with the standalone tool
kubectl kustomize overlays/prod                     # same build, using kubectl's embedded copy
kubectl apply -k overlays/prod                      # build and apply in one step
kubectl diff -k overlays/prod                       # preview the change against the cluster
kubectl delete -k overlays/prod                     # remove everything the build produced
kustomize create --autodetect                       # write a kustomization.yaml for the YAML in this folder
kustomize edit set image nginx=nginx:1.27.3         # change the images field
kustomize edit set replicas web=3                   # change the replicas field
kustomize edit add configmap app-cfg --from-literal=A=b   # add a generator
kustomize edit fix                                  # migrate deprecated fields

Two habits separate users from people who read about it: they build before they apply, and they know which version of Kustomize is running. kubectl version --client prints an embedded Kustomize Version: line, and it often lags the standalone release.

What a kustomization looks like

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
namespace: web-prod
namePrefix: prod-
images:
- name: nginx
  newTag: "1.27.3"
replicas:
- name: web
  count: 3
patches:
- path: replicas-patch.yaml
configMapGenerator:
- name: web-config
  literals:
  - LOG_LEVEL=info

Be ready to walk through it line by line. resources names the base, as a directory. namespace sets the namespace on every namespaced resource. namePrefix prefixes names and follows references. images changes tags by the name the base uses. replicas sets counts by the base's resource name. patches applies a patch file. configMapGenerator creates a ConfigMap with a hash-suffixed name.

Patches

Both live under patches:, and they apply in the order listed. Name the base's original resource name in the patch header, not the prefixed one.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 3

patches:
- target:
    kind: Deployment
    name: web
  patch: |-
    - op: replace
      path: /spec/template/spec/containers/0/image
      value: nginx:1.27.3

target.name and target.namespace are regular expressions anchored at both ends, so web matches only web and web-.* matches a prefix. In a JSON6902 path, a / inside a key is written ~1.

Generators and the hash

A generator creates a ConfigMap or Secret. The generated name carries a hash of the content, such as prod-web-config-hf678c7m2b. Kustomize rewrites every reference to the name in the same build, so when the data changes the name changes, the pod template changes, and Kubernetes performs a rolling update. The consequence to say in an interview: config changes roll out pods without a manual restart.

The traps:

• Refer to the generated object by its original name in your manifests. Never type the hash.
• disableNameSuffixHash: true removes the rollout-on-change behaviour, so it is a trade-off.
• behavior: merge or replace needs a matching name and namespace in the base, before the overlay's namespace is applied, because generators run before the namespace transformer.
• secretGenerator output is base64, not encrypted.

Labels, selectors and the immutability trap

The labels field has an option, includeSelectors, default false. With it off, labels go on object metadata only. With it on, they also go into selectors and pod template labels. The older commonLabels field behaves like includeSelectors: true, and it is deprecated for that reason.

A Deployment's spec.selector is immutable. Add a label to selectors on a live Deployment and kubectl apply fails with field is immutable. The fix is to stop changing selectors, or to recreate the Deployment. This is a favourite question, because it connects a Kustomize option to a Kubernetes rule.

The traps, and why they share a cause

Most of these share one root cause: Kustomize never talks to the cluster, so it cannot catch anything the cluster would catch. It checks that the YAML can be assembled. Kubernetes checks that the YAML is valid and allowed. When the build passes and the apply fails, the problem is on the Kubernetes side of that line. When the build fails, the problem is in the files.

Common interview questions

What problem does Kustomize solve? It removes copy-and-edit drift between environments. You keep one base of ordinary manifests and describe each environment as a small overlay of differences, so a fix to the base reaches every environment and "what differs in prod" is a short file rather than a directory diff.

What is the difference between a base and an overlay? A base is a kustomization that other kustomizations build upon; it is self-contained and can be built on its own. An overlay is a kustomization that lists one or more bases under resources and adds its own edits. The dependency points one way: overlays know their base, bases do not know overlays.

Why is it called template-free? Your manifests contain no placeholders. They stay valid Kubernetes YAML that kubectl can apply directly. Customisation lives in a separate file, and the tool produces the output by structured edits, not by text substitution.

Does kustomize build change the cluster? No. It reads files and writes YAML to standard output. It has no cluster access and keeps no state. Applying is a separate step.

How do you deploy an overlay? kubectl apply -k overlays/prod for a one-step deploy, or kustomize build overlays/prod | kubectl apply -f - when I want the standalone version and its flags. Before either, I run kubectl diff -k or read the build output.

Why might kubectl apply -k and kustomize build give different output? They can be different versions. kubectl embeds its own copy, which lags the standalone releases: kubectl 1.34 and 1.35 embed v5.7.1, and 1.36 and 1.37 embed v5.8.1. I check kubectl version --client and pin one version in CI.

How do you change an image tag per environment? With the images field in the overlay, which matches the image name as written in the base and sets newTag, newName or digest. In a pipeline, kustomize edit set image name=repo:tag edits the field before the build, so there are no variables.

How do you pass a value from CI into a build? Kustomize deliberately has no build-time variables or environment reads. You run a command such as kustomize edit set image first, which writes the value into the file, then build. The value lands in git, which keeps the build reproducible.

What are the two kinds of patch, and when do you use each? Strategic merge patches look like a partial resource and merge by Kubernetes semantics, merging named lists like containers by name. JSON6902 patches are lists of operations with paths and need a target. I use strategic merge for "change these fields on this object", and JSON6902 for one exact edit inside a list.

What does the hash suffix on a generated ConfigMap do? It derives a name from the content. When the content changes the name changes, Kustomize updates every reference, and the changed pod template triggers a rolling update. That solves the problem that pods do not restart on ConfigMap edits.

How do you keep secrets out of git? The secretGenerator only base64-encodes, so real values should not be committed. Real options are encrypted files decrypted by the deployer, Sealed Secrets, the External Secrets Operator, or creating the Secret outside git. At this level it is enough to name the problem and two options.

What is namePrefix and does it break references? It prefixes the name of every resource. Kustomize tracks renames and rewrites references that point at names, such as a Pod referring to a ConfigMap. It does not change label selectors, because those refer to labels, not names.

What is wrong with commonLabels? It adds labels to selectors and pod templates as well as metadata. Selectors are immutable after creation, so changing them on a live Deployment fails. It is deprecated in favour of labels, where includeSelectors defaults to false.

What does kustomize edit fix do? It migrates deprecated fields to current ones: patchesStrategicMerge and patchesJson6902 to patches, bases to resources, commonLabels to labels with includeSelectors: true. I run it on a clean git tree and review the diff.

What is the load restrictor? A default safety check (LoadRestrictionsRootOnly) that stops a kustomization reading files outside its own directory tree, so it cannot pull ~/.ssh into a ConfigMap. Referencing another directory as a base is allowed. Disabling it with --load-restrictor LoadRestrictionsNone breaks relocatability and should be a last resort.

How would you test a change to a base? Build every overlay before and after, and diff the outputs. The diff should contain only the intended change. Then run a schema validator such as kubeconform on the output, because Kustomize does not validate against Kubernetes.

Where does Kustomize stop and another tool start? Kustomize renders YAML; it does not apply, prune, order rollouts, track releases or roll back. kubectl, Argo CD or Flux apply it. Helm is better for distributing configurable software to other people, and a template-heavy application with real logic may be simpler in Helm.

Explain an overlay that uses a remote base. A resources entry can be a git URL such as https://github.com/org/repo//deploy/base?ref=v1.2.3. The // separates the repository from the path, and ref should be a tag or a full commit hash, since short hashes are not supported. It clones on every build with a 27-second default timeout, and it is supply-chain input I should pin and review.

Questions you should ask back

Interviews go better when you ask about the real setup. Good questions for the interviewer, which also show you understand where Kustomize sits:

• Who applies the output: kubectl in CI, Argo CD, or Flux?
• Which Kustomize version do pipelines use, and is it the same as developers' laptops?
• Are rendered manifests committed, or only built at deploy time?
• How are secrets handled, since the generator only encodes?
• Do overlays use components or deep inheritance, and how many levels?

Sixty-second self-test

Answer out loud, without looking back. If you hesitate on more than two, reread the matching section.

1. In two sentences, what is Kustomize, and what does it not do?
2. Which way does the dependency arrow point between a base and an overlay?
3. Name the five core words and define a transformer and a generator.
4. What does kustomize build print, and where does it send it?
5. What is the difference between kubectl apply -k and kustomize build | kubectl apply -f -?
6. Which field changes an image tag, and what name do you give it?
7. Why does a generated ConfigMap have a hash in its name, and what does that cause?
8. Which name do you use in a patch header when the overlay sets namePrefix?
9. What is the trap in commonLabels and includeSelectors: true?
10. A build passes but kubectl apply fails. Which side of the line is the problem on?
11. Why does Kustomize have no variables, and how do you inject an image tag instead?
12. Which fields are deprecated, and which command converts them?

If you can answer twelve of twelve, you are ready for a beginner-level Kustomize interview. Next, the Tips & Tricks track for this level covers the mistakes that cost beginners the most time, and the Mid-level guide takes up components, replacements, Helm and testing.

Part one of three. Three mistakes cause most beginner trouble with Kustomize: treating the build as if it talked to the cluster, forgetting which name a patch or generator must use, and leaning on advice written for an older version. Start with the table, then the habits, the recipes and the checklist.

Common errors at this level

The habits that pay off most

Name things in the base, change them in the overlay. Put shared facts in the base and only differences in overlays. If an overlay grows to the size of the base, the split is wrong.

Keep overlays short. A good overlay fits on one screen. Use the built-in fields first (namespace, namePrefix, images, replicas, labels) and reach for a patch only when those cannot express the change. Short files are easier to review, and reviewers are what stand between you and a bad deploy.

Prefer a patch file for anything bigger than a few lines. Inline patches with patch: |- are fine for a one-line JSON6902 edit. A long inline block inside a kustomization is hard to read and hard to highlight in an editor. Move it to its own file and use path:.

Commit the kustomization changes together with the manifest changes. They describe one system. A pull request that touches the base and its overlays in one commit is easier to review and to revert.

Practice cards

Each card is a five-minute exercise. Do them on a scratch copy of the demo project.

Reading the output: a short tour

The fastest way to stop being surprised by Kustomize is to know what a build does to text. Four rules cover nearly everything you will see.

Output order is fixed, not yours. Resources come out in a conventional order: namespaces first, then configuration objects, then services, then workloads. Your order in resources does not change it. So if you expected your Deployment first, it is not broken.

Keys are sorted alphabetically and comments are dropped. The build parses and re-emits YAML. That is why you compare builds with diff, and why a source file and its output never look identical even when nothing has been edited.

Names change in three ways. A prefix or suffix, a hash on a generated object, or both. prod-web-config-hf678c7m2b is prefix, original name and hash. Learn to split such names mentally into those three parts.

References follow names but not labels. A rename updates the things that point at a resource by name. It does not touch selectors, because selectors match labels. If a Service stops finding its pods after a change, suspect a label, not a name.

Debugging a build that fails

When a build stops, work through the same four checks in order. They resolve almost every beginner error in under two minutes.

1. Read the last sentence of the error first. The front of an error names the stage (accumulating resources, loading KV pairs, merging from generator). The end names the reason (no such file or directory, unknown field, is not in or below).
2. Build the base alone. kustomize build base. If it fails, the problem is in the base. If it succeeds, the overlay is at fault. This halves the search instantly.
3. Remove things until it builds. Comment out patches and generators one at a time, from the bottom up. The last one you removed is the culprit.
4. Check the version. kustomize version and kubectl version --client. A field from a newer guide can be unknown on an older binary.

If the output builds but looks wrong, turn the question around: which field produced the line you do not like? Search your kustomization for the key. Usually exactly one field is responsible, and you can test by removing it.

kustomize build overlays/prod | grep -n -B2 -A2 "image:"

This finds every image line with its neighbours, handy for checking a tag change. Replace image: with namespace:, replicas: or name: for the other fields. For anything longer, save the output to a file and open it in an editor with YAML highlighting.

For a stack trace on an unexpected crash, add --stack-trace to the build command. You will seldom need it, but it is the flag to mention when you report a bug.

The working folder: a layout to copy

Start every project with the same shape, and teammates will recognise it in seconds:

my-app/
├── base/
│   ├── deployment.yaml
│   ├── service.yaml
│   └── kustomization.yaml
└── overlays/
    ├── dev/
    │   └── kustomization.yaml
    ├── staging/
    │   └── kustomization.yaml
    └── prod/
        ├── kustomization.yaml
        └── resources-patch.yaml

Three conventions that save time later. Name patch files after what they change (resources-patch.yaml, probes-patch.yaml), not after the environment. Keep one resource per file, named by its kind and role. Leave namespace out of the base manifests entirely, and set it in each overlay, so the base can be deployed anywhere.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
namespace: my-app-dev
namePrefix: dev-
labels:
- pairs:
    environment: dev

Because every overlay starts from this template, adding a fourth environment means copying one short file and changing three values.

Build-time versus apply-time

Half of all beginner confusion comes from mixing up the two moments. Keep them in separate boxes.

A passing build says "the YAML could be assembled". It does not say "the cluster will accept it". Add a validator after the build if you want an early check on Kubernetes rules, and use kubectl diff -k for the final look.

Labels and selectors, without the pain

Labels produce more beginner outages than any other feature, so it is worth a short, separate list of rules.

• Labels on metadata are safe. Adding environment: prod to every object's metadata.labels is always fine.
• Labels in selectors are permanent. A Deployment's spec.selector and a Service's spec.selector tie objects together. For a Deployment, the selector cannot change after creation.
• labels keeps selectors alone unless you set includeSelectors: true. Leave it off for anything that already runs.
• commonLabels is the old, risky version. If a tutorial uses it, translate it to labels.
• For a brand-new application, includeSelectors: true is acceptable, because nothing is live yet. Decide before the first apply, and then never change it.

If you ever hit field is immutable, the recovery is to delete the Deployment and apply again. That interrupts the service for a moment, so in production prefer to avoid the situation rather than cure it.

Generators: small rules that prevent surprises

• Use configMapGenerator for anything that changes per environment, so config edits cause rollouts.
• Refer to generated objects by their original name only.
• If something outside the build needs a fixed name, disable the hash for that generator and accept that you must restart pods yourself.
• For behavior: merge, use the name and namespace the base generated, before any prefix or overlay namespace.
• Keep generator source files inside the kustomization directory, because of the load restriction.
• Use envs: (plural) for files of KEY=value lines; the singular env: is legacy.
• Never put real credentials in literals. Base64 is reversible by anyone.

Old tutorials and what to translate

Much of the Kustomize content on the internet predates version 5. A quick translation table saves hours:

If an example uses a feature in a way that produces an unknown field error on your version, suspect its age before you suspect your typing.

Keeping your machine and repository clean

• Do not commit build output by accident. If you redirect builds to rendered.yaml or out/, add those names to .gitignore, unless your team deliberately commits rendered manifests.
• Do not commit the downloaded binary. The install script writes kustomize into the current directory; move it to a directory on your PATH and delete it from the repository folder.
• Delete local test clusters when you are finished (kind delete cluster), so they do not consume memory.
• Do not leave edit commands unexplained in history. If a script runs kustomize edit set image, commit the resulting change with a message that names the image and tag.
• Check git diff after kustomize edit fix. It rewrites files. Review it as you would review any other automated change.

Small things to do from day one

# 1. Enable shell completion (zsh example; bash, fish and powershell also work)
source <(kustomize completion zsh)

# 2. Save a baseline render, change something, and compare
kustomize build overlays/prod > /tmp/before.yaml
# ...edit...
kustomize build overlays/prod > /tmp/after.yaml
diff /tmp/before.yaml /tmp/after.yaml

# 3. Preview against the cluster before applying
kubectl diff -k overlays/prod

# 4. Validate every overlay in a loop
for d in overlays/*; do kustomize build "$d" > /dev/null || exit 1; done

And a few habits that cost nothing:

• Quote image tags that look like numbers ("1.20").
• Start every kustomization.yaml with apiVersion and kind, for editor support.
• Write a one-line comment above any patch that explains why it exists. Comments in source files survive; it is only the build output that loses them.
• Put kustomize version output in your bug reports and pull request descriptions when behaviour differs between machines.
• When something surprises you, reduce it to the smallest kustomization that shows the problem. Most of the time the mistake becomes obvious while you are cutting the example down.

Where beginners lose the most time

Looking back over the table above, the mistakes cluster into four groups, and knowing the group tells you where to look.

Path mistakes. Every no such file or not a valid directory error. Cause: paths in a kustomization are relative to that file. Remedy: stand in the directory with the file and check each path with ls.

Name mistakes. Every no resource matches and cannot merge or replace error. Cause: a patch or a merge names something as the overlay sees it, not as the base defines it. Remedy: open the base, copy the name from there, and remember that generator merges use the base's namespace too.

Version mistakes. Every unknown field error that seems impossible, and every case where two machines disagree. Cause: two Kustomize versions, or docs newer than your binary. Remedy: print both versions and align them.

Cluster mistakes. Every problem that appears only after a good build: immutable selectors, missing namespaces, rejected fields. Cause: the build never contacted the cluster. Remedy: kubectl diff -k, then read Kubernetes' own message, which is usually precise.

A short checklist

Before you call a Kustomize change finished:

• The base builds on its own: kustomize build base.
• Every overlay builds with no errors: a loop over overlays/*.
• No deprecation warnings remain; commonLabels and the old patch fields are gone.
• Images are pinned to a specific tag, with quoted values.
• Patch headers use the base's original resource names.
• Generated ConfigMaps and Secrets are referenced by original name, with the hash left on.
• No real secret values in literals or committed files.
• Labels in selectors have not changed on anything already live.
• A saved before-and-after diff shows only the intended changes.
• kubectl diff -k was read before apply, and the namespace exists.
• The Kustomize version is the same on your laptop and in the pipeline.

When these habits feel automatic, move on to the Mid-level guide, where components, replacements, Helm integration and testing take the same tool further.

This is part two of three. Beginner showed you how to write a base, add an overlay and run kustomize build. At work that is where the trouble starts: a patch silently does nothing, a label change breaks a live Deployment, a ConfigMap edit does not roll the pods, a Helm chart renders without a namespace, and the CI job renders different YAML from the laptop. This level gives you the mental model precise enough to predict each of those before it happens, and the production patterns that keep a repository of overlays understandable two years later. It is written against kustomize v5.8.2, the current release at the time of writing.

Where this picks up

Each section starts with the problem it solves and ends with a Try it you can do on a scratch directory in a few minutes.

How a build actually works

Beginner taught the rule of thumb: the base holds what is common, the overlay holds what differs. The mechanism underneath lets you predict output instead of reading it after the fact. Kustomize is a client-side, stateless renderer: a pure function from a directory of files to a stream of YAML. It has no server, never talks to a cluster, stores no release state and cannot roll anything back. Everything it knows comes from the files it reads during that one invocation, which is precisely why the same commit produces the same output on any machine that runs the same version.

A build proceeds in a fixed order, and the order matters because each step sees only what earlier steps produced.

First, resources: is processed depth first and in order. If an entry is a file, it is parsed into Kubernetes objects. If it is a directory, kustomize recurses: that directory must contain its own kustomization, which is built completely (its own generators, transformers and all) before its output joins the accumulated pile of the parent. A git URL is cloned and then treated as a directory. The consequence is that a base never sees its overlays. By the time the overlay's patches run, the base has already been rendered in full, and the overlay operates on the finished objects.

Second, components: are applied to that accumulated pile, and then generators run and add new objects. Third, transformers run over everything accumulated so far, including what the generators just produced. Last, validators run. A validator is a transformer that is not allowed to change anything, only to fail the build.

The fields you write every day (namespace, namePrefix, labels, images, replicas, patches) are not separate mechanisms. Each is shorthand for a builtin transformer, and "everything is a transformer" is the official framing. namePrefix: prod- is equivalent to a PrefixSuffixTransformer listed under transformers:. You rarely need that equivalence, but it explains why the fields run in a fixed internal order rather than in the order you typed them, and it is the doorway to custom transformers later.

One consequence of the ordering surprises people constantly: generators run before the namespace transformer. When an overlay uses behavior: merge to extend a ConfigMap that a base generated, the generator is looking for that ConfigMap as it was before any namespace: was applied. We return to this in the generators section, because it is behind one of the most common confusing errors.

Seeing the pipeline with your own eyes

Two features let you watch the build instead of imagining it. The first is to build a single directory at a time, from the innermost base outward, and diff successive renderings. The second is buildMetadata, which stamps provenance into the output:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
buildMetadata:
- originAnnotations
- transformerAnnotations
- managedByLabel

With originAnnotations, each object gains a config.kubernetes.io/origin annotation recording the file (and, for remote bases, the repository and ref) it came from. With transformerAnnotations, each object records which transformers touched it. With managedByLabel, objects get app.kubernetes.io/managed-by: kustomize-<version>. Turn these on in a scratch build when you need to answer "where did this label come from?" in an overlay stack three levels deep. Do not leave them on in committed output you diff against a cluster, because the annotations are noise that will not exist on live objects unless you applied them.

Identity, names and references

Before patches and generators make sense you need to know how kustomize decides that two objects are the same object. Every resource has a resource ID: group, version, kind, name and namespace. Kustomize prints it in error messages in a compact form such as Deployment.v1.apps/web.[noNs], where [noNs] means no namespace was set in the source file. Two resources with the same ID in one build is an error, and it is the error you will meet most often when composing bases:

Error: accumulating resources: accumulation err='merging resources from 'b.yaml': may not add resource with an already registered id: Deployment.v1.apps/web.[noNs]'...

The usual causes are a file listed twice, two bases that both include the same shared resource (a diamond import), or an overlay that lists a base and also lists one of the files inside it. A diamond looks innocent: overlay prod includes ../base-a and ../base-b, and both of those include ../common. Kustomize builds common twice, produces two identical IDs and refuses. The fixes are to restructure so the shared piece is included exactly once, to use a namePrefix or nameSuffix in intermediate overlays so the two copies no longer collide, or, for shared opt-in pieces, to use a component (see the overlays section).

Name references

Renaming is where kustomize does real work beyond text editing. When you set namePrefix: prod-, a Deployment called web becomes prod-web, and a Service's name follows suit. But a Deployment that mounts a ConfigMap by name, a Pod that uses a ServiceAccount, an Ingress that points at a Service, and a RoleBinding that references a Role all carry the old name inside a field. Kustomize ships a built-in table of which fields in which kinds refer to which other kinds (the nameReference configuration) and rewrites those fields when it renames the target. The same machinery keeps references correct when a generator appends a hash.

This bookkeeping is also the reason that patches can address a resource by several names. A patch may refer to a resource by its current name, its original name, or any intermediate name along the way. In practice this means a patch in an overlay can say name: web even though an inner overlay already prefixed it to dev-web; the original name still resolves. It also means you must be careful when two different resources could be matched by an original name, which is one more reason to keep names distinctive across bases.

The table covers built-in kinds. For a custom resource whose fields hold names of other objects (a CRD that has spec.secretName, for instance), the table knows nothing, and kustomize will rename the Secret without updating your CRD. The fix is a configurations: entry that teaches the nameReference transformer about the new field; we meet that in the extension section. For now, remember the rule: references update only where kustomize has been told to look.

Namespaces, and a collision you can cause

The namespace: field sets metadata.namespace on every namespaced resource in the build and, as a transformer, also updates fields that embed a namespace (RoleBinding subjects, for example). Cluster-scoped objects such as ClusterRoles, Namespaces and CRDs are not given a namespace. A subtle failure appears when two resources with the same kind and name live in different namespaces in your inputs and you then apply namespace: on top, collapsing them into the same place:

Error: namespace transformation produces ID conflict: [...]

The cure is to keep them in separate kustomizations, or rename one of them. The general habit worth forming is that a kustomization's namespace: should describe the place the whole variant lives, and anything that legitimately spans namespaces (a monitoring agent that installs into kube-system, for instance) belongs in its own kustomization, applied separately.

Patches in depth

A patch changes an object that a base already defined. Beginner used patches with a file. The field is one mechanism for two different patch languages, and choosing the right one is the skill.

patches:
# Strategic merge, from a file. Target is inferred from apiVersion, kind and name in the patch.
- path: replicas-patch.yaml

# JSON6902, inline. A target is REQUIRED.
- patch: |-
    - op: replace
      path: /spec/template/spec/containers/0/image
      value: nginx:1.27.3
  target:
    group: apps
    version: v1
    kind: Deployment
    name: web

Strategic merge patches

A strategic merge patch (SMP) is a partial Kubernetes object. You write only the fields you want to change, plus enough identity (apiVersion, kind, name) to find the target. Kustomize merges it into the matching object using Kubernetes merge semantics. Maps merge key by key. Scalars are replaced. Lists are the interesting part: for many well-known lists, Kubernetes declares a merge key, and elements with the same key value are merged rather than the whole list being replaced. containers merge by name. env merges by name. ports merge by containerPort. volumes and volumeMounts merge by name and mountPath respectively.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 5
  template:
    spec:
      containers:
      - name: web            # merge key: matches the existing container
        resources:
          limits:
            memory: 512Mi
        env:
        - name: LOG_LEVEL    # merge key: adds or overrides this one variable
          value: debug

Everything in web that the patch does not mention survives, including the image, the ports and the other env variables. That is the main benefit of a strategic merge: the patch states intent ("the web container gets this memory limit") rather than a full restatement.

The exact semantics become visible through two directives that you place inside the patch:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  template:
    spec:
      containers:
      - name: metrics-sidecar
        $patch: delete        # removes this list element

$patch: delete removes the matching list element (or, placed at the top level of the patch, the whole object). $patch: replace swaps the whole element or list for what you wrote instead of merging. Setting a map field to null removes that field, which is the way to delete a key from a base that you do not control.

This is the one loophole in kustomize's "only adds" philosophy, and it is a patch directive, not a kustomization field. If you find yourself deleting large parts of an upstream base, stop and ask whether the right move is a fork or a smaller base instead.

JSON6902 patches

A JSON6902 patch is an RFC 6902 list of operations (add, remove, replace, move, copy, test), each with a JSON Pointer path. It is exact where an SMP is declarative. You choose it when you need to address a list element by position, when you need to append (/- as the final path segment), or when a merge key does not exist.

patches:
- target:
    kind: Deployment
    name: web
  patch: |-
    - op: add
      path: /spec/template/spec/containers/0/args/-
      value: --enable-tracing
    - op: replace
      path: /metadata/labels/app.kubernetes.io~1version
      value: "2.4.0"

Two details bite. First, in a JSON Pointer the character / inside a key is written ~1, so the label key app.kubernetes.io/version becomes app.kubernetes.io~1version, and a ~ is written ~0. Second, a positional path such as containers/0 silently points at the wrong container if someone inserts a sidecar ahead of it in the base. A JSON6902 patch that is correct today is a latent bug tomorrow; prefer a strategic merge whenever a merge key can identify the element.

Targets and selectors

A target: selects which resources receive the patch, and it is how one patch reaches many objects. The target fields are group, version, kind, name, namespace, labelSelector and annotationSelector. The important detail is that name and namespace are anchored regular expressions, so name: web means ^web$, and name: web-.* matches any name beginning web-.

patches:
- patch: |-
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: not-used
    spec:
      template:
        spec:
          nodeSelector:
            pool: general
  target:
    kind: Deployment
    labelSelector: "tier=web"

When a target selects by fields like this, the metadata.name inside the patch body is ignored. It exists only to make the patch a valid object. This is how you apply one scheduling policy, one set of tolerations or one security context to every Deployment with a given label, without listing each one. Patches apply in list order, so a later patch sees the result of an earlier one.

The options block has two settings, allowNameChange and allowKindChange, both default false. A patch that changes metadata.name or kind is refused unless you opt in, because a patch that renames its own target defeats the identity tracking that later transformers rely on.

When a patch does nothing, or fails

Patches fail loudly when nothing matches:

Error: no resource matches strategic merge patch "Deployment.v1.apps/nope.[noNs]": no matches for Id Deployment.v1.apps/nope.[noNs]; failed to find unique target for patch Deployment.v1.apps/nope.[noNs]

Read this message as: the header of the patch (kind, apiVersion, name) did not match any accumulated resource. The usual causes are a typo, the wrong apiVersion, or a name that an inner overlay already prefixed. Because the original name resolves too, the last of these is rarer than people fear. When a target-based patch matches nothing, it fails the same way.

The dangerous case is the opposite: a patch that does match and does something other than you intended. Two recurring shapes:

• A list replaced instead of merged. For built-in kinds, the merge keys come from the OpenAPI schema kustomize embeds. For a custom resource, kustomize has no schema, so a list in an SMP is replaced wholesale, and your one-element patch deletes the other elements. The fix is the openapi: field, pointing at a schema that declares the merge keys, or switching to JSON6902.
• An environment variable lost. An SMP that lists env: for a container merges by name, so it is safe. A JSON6902 replace on /spec/template/spec/containers/0/env replaces the entire array, which is almost never what you meant.

The embedded schema is an additional trap worth knowing: it is still built from Kubernetes v1.21.2. A field introduced after that release is unknown to it. For a native Kubernetes kind this rarely matters, but it does mean kustomize is less helpful than you would expect when a patch relies on list semantics for newer fields.

Generators and the hash suffix

A generator produces objects rather than changing existing ones. The two you use constantly are configMapGenerator and secretGenerator, and they solve a problem a hand-written ConfigMap has: Kubernetes does not restart Pods when a ConfigMap they consume changes. Edit the ConfigMap, apply, and the running containers keep the old environment (and mounted files update eventually, but the process rarely re-reads them).

The generator's answer is the name-suffix hash. Kustomize computes a hash of the generated object's content and appends it to the name, so app-config becomes something like app-config-89g4tffbfk. Then, using the name-reference table, it rewrites every reference to that ConfigMap in the build: the envFrom, the volume, the valueFrom. A content change therefore changes the name, which changes the Pod template, which makes the Deployment roll out. Configuration changes become deployments, reviewed in the same diff as the code, and the rollout is a normal, observable rolling update.

configMapGenerator:
- name: app-config
  files:
  - application.properties          # key is the file's basename
  - custom-key=config/app.yaml      # explicit key name
  envs:
  - config.env                      # KEY=VALUE lines
  literals:
  - LOG_LEVEL=info
  options:
    labels:
      team: payments
    immutable: false

The envs: key is the current spelling for environment-style files. The older singular env: is legacy; use envs:. Note that in v5 a generator no longer reads from the local environment, so nothing in your shell leaks into a ConfigMap.

Overlays and behavior

An overlay often wants to extend a ConfigMap the base generated instead of defining a new one. behavior controls that:

resources:
- ../../base
configMapGenerator:
- name: app-config
  behavior: merge       # create (default) | merge | replace
  literals:
  - LOG_LEVEL=warn
  - FEATURE_X=on

create is the default and errors if the name already exists. merge adds keys and overrides existing ones, keeping the rest. replace throws away the base's data and uses yours. Both merge and replace require a base object to exist with the same name and namespace, matched before any transformation. That last clause is the trap, and it follows directly from the build order: generators run before the namespace transformer.

Error: merging from generator &{...}: id resid.ResId{Gvk:resid.Gvk{Group:"", Version:"v1", Kind:"ConfigMap", ...}, Name:"nope", Namespace:""} does not exist; cannot merge or replace

If the base generates app-config with no namespace and the overlay sets namespace: web-prod and declares the generator with namespace: web-prod, the lookup looks for a base ConfigMap in web-prod, finds none (the namespace has not been applied yet) and fails. The cure is to declare the generator with the namespace the base object has at that moment (usually none), and let namespace: apply afterwards.

Options and their sharp edges

generatorOptions applies to every generator in a kustomization, and each generator can carry its own options. The fields are disableNameSuffixHash, immutable, labels and annotations. Two sharp edges deserve attention.

The global setting wins over a per-generator false. If generatorOptions.disableNameSuffixHash: true is set globally, a per-generator disableNameSuffixHash: false cannot turn the hash back on. This follows the usual boolean zero-value semantics in Go, where "false" and "unset" are indistinguishable, and the official docs call it out. If you need a mix, do not set the global option at all.

Turning the hash off gives up the rollout. Some reasons are legitimate: an operator expects a fixed ConfigMap name, or a mutating webhook cannot cope with hashes. But you then own the restart problem. A common substitute is an annotation on the pod template carrying a checksum, updated by your pipeline, which brings back the roll-on-change behaviour at the cost of a manual step.

immutable: true sets immutable: true on the generated object, which is a natural pairing with hashing: a hashed ConfigMap never needs to change, and making it immutable lets the API server stop watching it.

Secrets

secretGenerator has the same shape as configMapGenerator and adds a type (default Opaque, or for example kubernetes.io/tls). The caution is at the level of the whole design: a generated Secret is base64-encoded, not encrypted. Committing the source .env file next to the kustomization commits the plaintext. The safe patterns commit only encrypted or referenced material, and we cover them in the production section. The generator is appropriate for deriving a Secret from files that a pipeline places at build time and that never enter git.

Replacements: copying a value from one place to another

Some values appear in several places and must agree: a Service name that is also an environment variable for a client, a hostname that is baked into an Ingress and also into a ConfigMap. Without templating, kustomize solved this first with vars, which used $(VAR) syntax and had several sharp edges, and is now deprecated. The replacement is replacements: it reads a value from a field of one resource, and writes it into fields of others.

replacements:
- source:
    kind: Service
    name: web
    fieldPath: metadata.name          # default is metadata.name
  targets:
  - select:
      kind: Deployment
      name: client
    fieldPaths:
    - spec.template.spec.containers.[name=client].env.[name=API_HOST].value
    options:
      create: true                    # create the field if missing

The anatomy is straightforward. source must resolve to exactly one resource, and fieldPath says which field to read. Each entry of targets has a select (which resources), an optional reject (which to exclude) and one or more fieldPaths (where to write). Field paths use dots for nesting and [key=value] to pick a list element, which is how containers.[name=client] addresses a container by name rather than by position.

Options that change behaviour

• options.delimiter with options.index on the source takes one piece of a delimited string, such as the first part of a hostname split on dots.
• options.create: true on a target creates the destination field when it does not exist. The default is false, and a missing field is an error.
• sourceValue, available since v5.7.0, supplies a static string instead of reading a field. It is mutually exclusive with source. Use it to write a constant into many places, and to keep the constant in one visible spot.
• Since v5.8.0, select and reject accept regular expressions, and a field path can descend into a YAML or JSON string value. A ConfigMap that embeds a JSON document can be edited in place with a path such as data.config\.json.config.hostname, where dots inside a key are escaped with a backslash.
• options.encoding exists in the source code but is not implemented. Do not rely on it.

replacements:
- sourceValue: "eu-south-1"
  targets:
  - select:
      kind: ConfigMap
      name: settings
    fieldPaths:
    - data.REGION

The source does not have to ship

A replacement source is often a helper object that exists only to carry configuration, and you do not want it in the cluster. Mark it with the annotation config.kubernetes.io/local-config: "true": it is available to the build, and omitted from the output.

apiVersion: v1
kind: ConfigMap
metadata:
  name: env-settings
  annotations:
    config.kubernetes.io/local-config: "true"
data:
  HOSTNAME: shop.example.com

Listed in resources: and referenced as a replacement source, this object feeds values into an Ingress, a Certificate and a ConfigMap, and never reaches the API server. It is the cleanest way to express "this overlay has one hostname, and three objects need it".

Failure modes

Two messages to recognise:

Error: nothing selected by ConfigMap.[noVer].[noGrp]/nope.[noNs]:data.x
replacements must specify a source and at least one target

The first means the source selector matched no resource. Check the name both before and after prefixes, since replacements run after earlier transformers in the same kustomization. The second is an incomplete entry, which also appears after the experimental kustomize edit fix --vars when a vars entry was declared but never referenced.

The reason to prefer replacements over anything clever is that they are explicit: the source and the destination are both written down, and a change to one is visible in review. They do not interpolate into arbitrary strings the way vars did, which is deliberately less convenient and considerably more predictable.

Labels and selectors without breaking live workloads

The label transformers deserve their own section because a mistake here is the classic kustomize outage. The history matters: the original commonLabels field adds labels to every object and injects them into selectors and pod templates, so that a Service keeps selecting the Pods its Deployment creates. That is convenient on the first deploy. On the second, when you add another label, the Deployment's spec.selector changes, and Kubernetes rejects it:

The Deployment "web" is invalid: spec.selector: Invalid value: ...: field is immutable

A Deployment's selector cannot be changed after creation. The only recovery is to delete and recreate the Deployment, with the downtime that implies. commonLabels is deprecated for this reason, and kustomize prints a build warning when it sees it:

# Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically.

The labels field

labels makes the choice explicit, per group of labels:

labels:
- pairs:
    app.kubernetes.io/part-of: shop
    team: payments
  includeSelectors: false        # default. Do not touch selectors.
  includeTemplates: true         # add to spec.template.metadata.labels

With includeSelectors: false (the default), the labels go on object metadata only. With includeTemplates: true, they additionally go on spec.template.metadata.labels so the Pods carry them (useful for cost allocation and log filtering), while the selectors are left alone. With includeSelectors: true, you get the old commonLabels behaviour, selectors and all. includeVolumeClaimTemplates does the same for a StatefulSet's claim templates.

The operational rule is short. Put identity labels in the base once, at the first deploy, and never change them. Add anything that may change later (owner, cost centre, version) through labels with includeSelectors: false. If you inherit a repository using commonLabels, run kustomize edit fix: it converts to labels with includeSelectors: true, which reproduces the same output and so does not change your live selectors.

A trap in the CLI

kustomize edit add label team:payments writes the deprecated commonLabels by default. To write the modern field, add --without-selector, and add --include-templates if you want them on the pod template (that flag requires --without-selector).

kustomize edit add label --without-selector --include-templates team:payments

Argo CD has its own copy of this trap

If you use Argo CD's spec.source.kustomize.commonLabels, it adds labels including selectors, exactly like the deprecated field. Argo CD provides labelWithoutSelector and labelIncludeTemplates to get the labels behaviour. The same immutability rule applies, so read the option before you set it.

Designing overlays and components

The base-and-overlay pattern is easy to start and easy to ruin. A repository with a good shape is boring to navigate; one with a bad shape requires a map. This section is about shapes.

The standard layout and where it strains

shop/
├── base/
│   ├── kustomization.yaml
│   ├── deployment.yaml
│   └── service.yaml
├── components/
│   ├── ldap/
│   └── external-db/
└── overlays/
    ├── dev/kustomization.yaml
    ├── staging/kustomization.yaml
    └── prod/kustomization.yaml

Each overlay lists ../../base and declares what differs: replica counts, resource limits, image tags, a hostname. That handles three environments comfortably. It strains when you multiply the dimensions. Suppose you run two clusters (a Gulf-region cluster for customers who need in-country data, and a European one) and three environments in each. Six overlays that each restate the same production-versus-dev differences is a maintenance problem, and the natural instinct is an intermediate overlay per dimension, with a final overlay that includes both parents. That is exactly the diamond shape from earlier: both intermediate overlays include the base, so the base is included twice and the build fails on duplicate IDs.

Two ways out exist. One is a strict chain, where ksa-prod lists ../cluster-ksa, which lists ../env-prod, which lists ../../base, so the base is included exactly once. The other, better for dimensions that vary independently, is to express them as components.

As a guideline, keep inheritance depth to three or four levels. Beyond that, answering "where does this value come from?" means walking the chain, and every change becomes a search exercise.

Components: opt-in features

A component is a kustomization with kind: Component and apiVersion: kustomize.config.k8s.io/v1alpha1. It bundles resources, patches and generators that represent a feature, and it is applied to the accumulated resources of whichever kustomization includes it. Where an overlay says "start from this base", a component says "whatever you already have, add this".

apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component
patches:
- path: use-external-db.yaml
configMapGenerator:
- name: app-config
  behavior: merge
  literals:
  - DB_MODE=external

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
components:
- ../../components/external-db
- ../../components/ldap

Components are the right tool when features vary independently of environments. A "tracing" component can be turned on for staging and prod; a "read-only replica" component only for one region. Because the same component can be included by several overlays without being included twice in any one of them, components also cure the diamond problem when the shared piece is a cross-cutting customisation rather than a whole base.

The rules are worth stating. A component must be listed under components:, not resources:, and a normal kustomization must not be listed under components:. Each mistake has its own message:

Error: accumulating resources: ... '<dir>/comp' must resolve to a file': expected kind != 'Component' for path '<dir>/comp'
Error: accumulating components: accumulateDirectory: "expected kind 'Component' for path '<dir>/base' but got 'Kustomization'"

The v1alpha1 version means the API could change, though components have existed since v3.7.0. Because a component patches what is already accumulated, a component's behavior: merge generator needs a ConfigMap of that name to exist already, which ties the component to the base it expects. Write that expectation in a comment at the top of each component.

When the structure is wrong

Some smells tell you the layout has outgrown itself. Overlays that are near copies with one line different want a component or an intermediate overlay. Patches that change the same field in several layers mean nobody decided which layer owns that field. A base that contains environment-specific values "as a default" is wrong: the base should be valid for nobody in particular, and the overlays supply the particulars. Deep chains where a base edit requires touching every overlay mean the base is too big, or the overlays are compensating for a base with no sensible defaults.

Remote bases, the load restrictor and vendoring

A resources: entry can be a git URL, which lets one repository build on another: a platform team publishes a base, and application teams overlay it. The syntax has one detail that trips everyone once.

resources:
- https://github.com/example-org/platform-base//deploy/base?ref=v1.2.3
- git@github.com:example-org/platform-base//deploy/monitoring?ref=3f2a9c1d8e7b6a5f4e3d2c1b0a9f8e7d6c5b4a39

The double slash // separates the repository root from the subdirectory inside it. The query parameters are ref (a branch, a tag or a full commit SHA; short SHAs are not supported), version (an alias, ignored if ref is present), timeout (default 27 seconds) and submodules (default true). Ports in URLs are not supported, and the older github.com/org/repo/path?ref= go-getter form is accepted but discouraged. In v5 the gh: host shorthand and the automatic .git stripping for AWS and Azure URLs were removed.

Pin, and know what a pin costs

A remote base is a supply-chain input: its content arrives in your manifests, with your privileges, at deploy time. Two habits follow. Pin ref to a full commit SHA, because tags can move, and review upstream changes as you would a dependency bump. And remember that every build clones the repository again, with no cache. A build with three remote bases performs three clones, each subject to the 27-second default timeout. On a slow network, through a TLS-inspecting corporate proxy, or with a large repository or submodules, you will see:

Error: hit 27s timeout running '/usr/bin/git ...'

The mitigations are ?timeout=120 for a slow but working path, ?submodules=false when the submodules are not needed, a pinned ref, and, for the real fix, vendoring the base. Some environments cannot reach GitHub at all: regulated workloads in the Gulf, where a build network is air-gapped or limited to an internal mirror, are a common case, and there the fix is structural rather than a longer timeout.

kustomize localize

kustomize localize (alpha) copies a kustomization and downloads every remote reference, rewriting them to local paths, so the result builds with no network.

kustomize localize overlays/prod /tmp/prod-local
kustomize build /tmp/prod-local

By default it verifies the copy by building both and comparing the output. It does not yet localize Helm or KRM plugin fields, and it alphabetises fields in the files it rewrites, so review the diff before committing it. Treat it as a way to produce a hermetic bundle, not as a migration tool for your source tree.

The load restrictor

The other boundary is about local files. By default, a kustomization may read files only from its own directory or below. A generator, a patch file or a resource file outside that root is refused:

Error: loading KV pairs: file sources: [../outside/x.yaml]: security; file '<dir>/outside/x.yaml' is not in or below '<dir>/ov'

This is the load restrictor, LoadRestrictionsRootOnly by default, and it exists to stop a kustomization from reading arbitrary local files (an SSH key, a cloud credentials file) into a generated ConfigMap or Secret. It applies to files, not to directories: another kustomization directory may be referenced from anywhere, including ../base. A common variation is an overlay listing a single file inside a sibling base, which produces a similar message ending must build at directory: ... file is not directory. The fix in both cases is to reference the base directory, not its file, or to make the shared directory a kustomization of its own.

The flag --load-restrictor LoadRestrictionsNone disables the check. It works, and the documentation says plainly that it breaks the relocatability of the kustomization. In a repository you share, or in CI with credentials, leave the restrictor on.

Helm charts inside Kustomize

Plenty of software ships as a Helm chart, and plenty of teams want to customise it with overlays rather than forking. Kustomize can render a chart and feed the result into the normal pipeline via the helmCharts generator. This needs the helm binary on your PATH (Helm 3, or Helm 4 with kustomize 5.8.1 or newer) and an explicit opt-in flag, because running an external program is not something a build does by default.

helmGlobals:
  chartHome: charts
helmCharts:
- name: ingress-nginx
  repo: https://kubernetes.github.io/ingress-nginx
  version: 4.11.3
  releaseName: ingress
  namespace: ingress-nginx
  includeCRDs: true
  valuesInline:
    controller:
      replicaCount: 2
patches:
- path: add-pod-annotations.yaml

kustomize build --enable-helm overlays/prod

Kustomize runs the equivalent of helm template, then treats the output like any other resources, so the patches and transformers in the same kustomization apply to it. That is the point: patch a chart's output without forking the chart. The Helm guide covers the chart side.

What to pin and what to know

Pin version. An omitted version means "latest", and a render that changes without a commit defeats the reason for using kustomize. includeCRDs defaults to false, so a chart's CRDs silently go missing unless you set it, and the failure shows up later as "no matches for kind" at apply time. valuesInline, valuesFile and additionalValuesFiles provide values, and valuesMerge (merge, override or replace) controls how they combine. If a chart already exists in chartHome/<name>, kustomize uses it as-is and does not check its version; otherwise it runs helm pull into that location.

The support policy is deliberately narrow: bug fixes, security fixes and flags analogous to helm template. There is no private repository or registry authentication, and no post-renderer. If your chart lives in a private registry, pull it in a pipeline step first and use the local form.

Where it goes wrong

Four hazards, all worth knowing before you adopt this pattern in production.

The missing flag. Without --enable-helm the build fails with must specify --enable-helm. kubectl apply -k does not accept build flags, so the working form is kubectl kustomize --enable-helm DIR | kubectl apply -f -. Argo CD sets it through kustomize.buildOptions: --enable-helm in argocd-cm.

The Helm 4 error. Kustomize up to and including 5.8.0, and therefore kubectl up to 1.35, runs helm version -c --short, and Helm 4 removed -c:

Error: unknown shorthand flag: 'c' in -c : unable to run: 'helm version -c --short' ... (is 'helm' installed?): exit status 1

Upgrade to kustomize 5.8.1 or later, or point --helm-command at a Helm 3 binary.

The namespace regression. Since v5.8.0, the kustomization's namespace: is passed to Helm as the release namespace. The side effect, still open as issue #6058 at the time of writing, is that resources from charts whose templates do not set metadata.namespace no longer receive the kustomization namespace. The output renders without error and lands in the wrong place. A JSON6902 patch adding /metadata/namespace, or setting the namespace in the chart values, are the workarounds recorded in the issue. Check rendered output for missing namespaces whenever you combine namespace: with helmCharts.

The security fix. A releaseName or name beginning with - could be used to inject Helm flags, and so to run code on the machine doing the build. Kustomize 5.8.2 rejects it with releaseName must not start with '-'. Because kubectl 1.36 and 1.37 still embed 5.8.1, anyone rendering kustomizations from a source they do not control, particularly in CI with cluster credentials, should use a standalone kustomize of 5.8.2 or later.

Two alternatives

Inflation is not the only way to combine the tools. You can run helm template once, commit the output as a base, and overlay it: transparent, at the cost of re-rendering on every chart upgrade. Or you can use Helm's --post-renderer to run kustomize over Helm's output, which patches a chart at install time while Helm still owns the release. Choose by who owns the release: if Argo CD or Flux deploys the manifests, inflation or a committed render is natural; if a person runs helm upgrade, a post-renderer fits.

Teaching Kustomize about custom resources

Built-in kinds work because kustomize ships knowledge about them: which fields hold names, which hold images, which lists have merge keys. A custom resource from an operator has none of that, and three fields fill the gap.

crds: lists CRD or OpenAPI files so name-reference handling recognises custom kinds. openapi: supplies a schema so strategic merge patches on a custom resource know their list merge keys; without it, a list in an SMP is replaced, not merged. configurations: points at transformer configuration files, each a set of field specs that tell a builtin transformer about additional paths.

images:
- path: spec/template/spec/containers[]/image
  kind: Rollout

configurations:
- configs/images.yaml
images:
- name: api
  newTag: "2.4.1"

With that file, the images transformer also updates the image field in an Argo Rollouts Rollout, where before it touched only built-in workload kinds. The same mechanism teaches nameReference that a custom kind's spec.secretName points at a Secret, so the hash suffix stays consistent. Since v5.8.2 you can register a configuration with kustomize edit add configuration.

KRM functions and plugins

When the builtin transformers are not enough, kustomize supports KRM functions: programs that read a ResourceList on standard input and write one to standard output, configured by an object carrying a config.kubernetes.io/function annotation. They run as containers or as executables, are referenced from generators:, transformers: or validators:, and are alpha. Legacy exec plugins and Go plugins are deprecated, and Starlark functions were removed in v5.5.0.

Because a function is arbitrary code running with your privileges, kustomize requires opt-in flags, and the failure without them is informative:

Error: loading generator plugins: failed to load generator: external plugins disabled; unable to load external plugin 'Gen'

Pass --enable-alpha-plugins for containerized functions, and add --enable-exec for executable ones (the help text says plainly "do not use for untrusted configs"). --network and --mount grant a container network or filesystem access. Kustomize provides no sandbox of its own; container isolation is the only protection. Use functions for the cases no builtin covers, prefer containerized over executable, and never enable them on a repository you did not write and review.

Images, promotion and the CI pipeline

Kustomize is a renderer, so the interesting questions are what you feed it and what you do with the output. The pipeline in most teams has one shape: CI builds an image, something records the new tag, a render step produces YAML, a check step validates it, and a deployer applies it.

The images field and what edit set image writes

The images transformer rewrites image references without you editing the Deployment:

images:
- name: shop/api                # the name as written in the base
  newName: registry.example.com/shop/api
  newTag: "2.4.1"
- name: shop/worker
  digest: sha256:0a1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8f9

You can write it by hand, or let CI update the file in place:

kustomize edit set image shop/api=registry.example.com/shop/api:2.4.1
kustomize edit set image shop/worker@sha256:0a1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8f9
kustomize edit set image shop/api=*:2.4.2          # keep the existing new name

The forms are name=newName:newTag, name:newTag, name=newName and name@digest, and * preserves the existing value. Tags may contain only letters, digits, ., _ and -. The command edits kustomization.yaml in the current directory, so run it in the environment's overlay, then commit. Since v5.7.0 the transformer also updates Image Volumes.

Two judgement calls matter. Prefer digests in production. A tag is a mutable pointer; a digest identifies content. The pipeline that built the image knows its digest, and writing it into images: means the overlay states exactly what runs, and a rollback is a revert of one line. And decide who writes the file. CI running edit set image and committing is one model. Argo CD Image Updater and Flux image automation are another, and both write the same images: field. Do not let two writers fight over one overlay.

Render in CI, and validate the output

Because kustomize is a pure function, you can test its output without a cluster. A typical validation job renders every overlay and runs the result through static checks:

set -euo pipefail
for overlay in overlays/*/; do
  name=$(basename "$overlay")
  kustomize build "$overlay" > "rendered/$name.yaml"
  kubeconform -strict -summary -ignore-missing-schemas "rendered/$name.yaml"
done

kubeconform validates against Kubernetes schemas, and -ignore-missing-schemas lets custom resources pass rather than fail, so supplement it with a schema source for your CRDs. Policy tools read the same YAML: conftest for Rego policies, the Kyverno CLI for Kyverno policies, kube-linter or kube-score for best-practice checks. The rendered YAML is what reaches the cluster, so it is the right artifact to test, not the inputs.

Diffing, before merge and against the cluster

The render-to-render diff compares the old and new commit and shows what a change actually does:

kustomize build overlays/prod > /tmp/before.yaml      # on the old commit
git checkout my-change
kustomize build overlays/prod | diff /tmp/before.yaml -

The cluster diff compares the render to live state:

kubectl diff -k overlays/prod

kubectl diff exits 0 with no differences, 1 when there are differences (which is not an error), and greater than one for real errors, so a CI job must treat code 1 as "changes present" and not as a failure.

Rendered manifests as a pattern

A widely used pattern commits the rendered output per environment to a separate directory or branch. A pull request then shows the real YAML change: a two-line edit in an overlay appears as the exact Deployment diff that will reach the cluster, catching the surprise where a base edit altered twelve objects. The cost is another artifact to keep in sync, which a CI check must enforce (render, compare to the committed copy, fail if they differ). The benefit is a review surface that does not require the reviewer to run kustomize in their head.

The version trap

kubectl apply -k uses the kustomize vendored into your kubectl, and that lags the standalone releases: kubectl 1.34 and 1.35 embed v5.7.1; kubectl 1.36 and 1.37 embed v5.8.1; none embeds v5.8.2 yet. A CI job running kustomize build | kubectl apply -f - and a laptop running kubectl apply -k are therefore running different renderers, and the difference shows up as a diff you cannot explain. kubectl version --client prints a Kustomize Version: line. The remedy is to pick one renderer for everything that matters, pin its version in CI, in container images and in Argo CD, and produce reviewed output with kustomize build.

Secrets in a declarative repository

A declarative repository wants everything in git; a secret must never be in git in readable form. The resolution is to commit something that is safe to read and let the cluster turn it into the real thing. Since secretGenerator produces base64, which is an encoding and not a protection, it only fits when the source files are supplied by a pipeline at build time and never committed.

Four patterns cover most needs:

All of these share a property: the kustomization treats the committed object like any other resource, so it can be patched per overlay (the vault path in prod differs from dev) and renamed by namePrefix. Mind the interaction with the hash suffix, which belongs to generated objects. A SealedSecret or ExternalSecret is an ordinary resource and gets no hash, so a change to the referenced value does not roll the Pods by itself. Pair these with a mechanism that restarts workloads on change, or use the annotation kustomize.config.k8s.io/needs-hash: "true" on an eligible non-generated ConfigMap or Secret to opt in to hashing.

Argo CD and Flux

Kustomize renders; something else applies. In a GitOps setup that something is a controller in the cluster that watches a repository, renders it and reconciles the cluster toward it. Both major controllers understand kustomize, in different ways, and the differences are where the pain lies.

Argo CD

Argo CD detects a kustomization.yaml in an application's path and renders with its own bundled kustomize. The application's spec.source.kustomize block can override many fields without touching the repository:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: shop-prod
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://git.example.com/platform/shop.git
    targetRevision: main
    path: overlays/prod
    kustomize:
      images:
      - registry.example.com/shop/api:2.4.1
      namespace: shop-prod
  destination:
    server: https://kubernetes.default.svc
    namespace: shop-prod
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

The available overrides include namePrefix, nameSuffix, images, replicas, commonLabels, labelWithoutSelector, labelIncludeTemplates, commonAnnotations, namespace, patches and components. The controller-wide setting kustomize.buildOptions in argocd-cm supplies build flags, which is where --enable-helm goes. To run a specific kustomize version, register binaries with kustomize.path.<version> and select one per application with spec.source.kustomize.version.

Two cautions. First, deciding in the Application and deciding in the repository are two places for the same decision, and the Application override is invisible to someone reading the overlay. Keep the Application minimal and keep decisions in the repository. Second, prune: true is what cleans up superseded hashed ConfigMaps; without it they accumulate.

Flux

Flux's Kustomization is a different object from the kustomization file: a custom resource in the kustomize.toolkit.fluxcd.io/v1 API that tells the controller where to find a source, which path to build, how often to reconcile and whether to prune.

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: shop-prod
  namespace: flux-system
spec:
  interval: 10m
  path: ./overlays/prod
  prune: true
  sourceRef:
    kind: GitRepository
    name: shop
  targetNamespace: shop-prod
  postBuild:
    substitute:
      CLUSTER_NAME: ksa-1

Flux adds capabilities around the build: patches, images, components, namePrefix and nameSuffix, healthChecks, dependsOn for ordering between Kustomizations, wait, SOPS decryption, and postBuild.substitute and substituteFrom for variable substitution. That last one deserves care. It is a Flux feature applied after the kustomize build, not part of kustomize, and it replaces ${VAR} strings in your rendered output. A manifest that legitimately contains ${...} (a shell snippet in a ConfigMap) needs escaping.

Flux also limits what kustomize may do: it does not support helmCharts in a kustomization file (use a HelmRelease instead) and does not allow remote bases. A repository that works with a local kubectl apply -k can therefore fail under Flux for reasons that have nothing to do with your YAML.

Debugging beyond the basics

Most kustomize failures fall into a handful of categories, and each has a first move. Learning the category from the message saves more time than any tool.

A method

When the message is not enough, reduce until the failure disappears, then add back. Build the base alone, then each overlay layer in turn. If the base renders, the fault is in a layer; build each layer with its own parent only and bisect. To locate which transformer is responsible for a value, turn on buildMetadata: [originAnnotations, transformerAnnotations], build, and read the annotations on the offending object. For a stack trace from inside kustomize add --stack-trace; for Helm output add --helm-debug.

The kustomize cfg helpers (cat, count, grep, tree) are alpha but useful for exploring the output, for example kustomize build overlays/prod | kustomize cfg grep "kind=Deployment". Do not go looking for cfg set and the setter commands: they were removed in v5.0.

Silent failures deserve a checklist

The errors above are the easy ones. The harder failures produce valid output that does something other than what you meant. Check these whenever output looks plausible but behaves wrongly:

• A chart rendered without a namespace because of the v5.8.x regression.
• A chart's CRDs absent because includeCRDs is false.
• An SMP on a custom resource that replaced a list for lack of a schema.
• A sortOptions setting ignored because it was not in the top-level kustomization.
• A per-generator disableNameSuffixHash: false overridden by a global true.
• An edit add label that wrote the deprecated commonLabels.

The last resort for any of these is to render, read the actual output, and compare it with what you expected, line by line. Because the output is plain YAML, nothing is hidden.

Ordering, performance and keeping builds fast

Kustomize has no documented hard limit on size, but builds get slow for predictable reasons: the number of resources, the number of transformers applied to each, and, most of all, network and subprocess work. The list below runs roughly from most to least expensive.

1. Remote bases. Each build clones every remote repository with no cache. Pin, vendor, or localize.
2. Helm inflation. Each build runs helm template, and possibly helm pull. Vendoring the chart into chartHome avoids the pull; a committed render avoids the run entirely.
3. Many overlays rendered in CI. Render only the overlays a change affects, using path filters, rather than all of them on every commit.
4. KRM functions. A containerized function starts a container for every build.

Kustomize 5.0 shipped significant performance improvements, and 5.8.0 introduced a regression around Helm namespace propagation that later releases worked on (issue #5971). Time a render when you upgrade: a slowdown in CI is a bug report waiting to be written.

Output order

The output order is not arbitrary. By default kustomize sorts resources in a legacy order that puts Namespaces first, then things like ResourceQuotas, StorageClasses and CRDs, through to workloads, with validating and mutating webhook configurations last. That helps a plain kubectl apply succeed, since a Namespace must exist before what it contains. You can change it in the top-level kustomization only:

sortOptions:
  order: legacy
  legacySortOptions:
    orderFirst:
    - Namespace
    - CustomResourceDefinition
    orderLast:
    - ValidatingWebhookConfiguration

order: fifo keeps the order you listed. The setting is silently ignored in any kustomization that is not the top-level one, a common source of confusion when someone adds it to a base. It replaces the deprecated --reorder flag. If Argo CD or Flux applies your manifests, it does its own ordering, and sortOptions mostly matters for kubectl apply and for stable diffs.

Upgrades and deprecation

The Kustomization API is still v1beta1, and the project's versioning policy is conservative: a field's meaning never changes, deprecation comes with a minor release and a non-fatal warning, and removal comes only with a major version and a fatal "unknown field" error. The deprecated fields remain in v1beta1; they will not exist in a future v1.

The deprecations you will meet

Run kustomize edit fix in a clean git tree and read the diff. The --vars conversion is experimental and can rewrite resource files, so never run it with uncommitted work, and review every converted entry by hand.

Upgrading safely

The behaviour changes across recent versions show why a blind upgrade is risky. v5.4.0 stopped turning null YAML values into the string "null". v5.5.0 removed Starlark functions. v5.8.0 changed how Helm receives the namespace. Each altered output or broke a build for someone. The procedure is mechanical: render every overlay with the old version and save the output; render every overlay with the new one and diff each pair; read every difference and decide whether it is intended; then pin the new version in CI, in images, in Argo CD (kustomize.path.<version>) and in developer tooling, all at once.

A golden-file test makes this routine: commit the rendered output of every overlay and have CI fail when it changes unexpectedly. That single test catches upgrade drift, an accidental base edit and a chart version change.

Also note what not to run: v5.8.0 itself, whose release notes carry a regression warning. Go straight to 5.8.1 or later, and to 5.8.2 if you render Helm charts from any source you do not fully control.

Putting it all together

Here is a compact repository that exercises most of the ideas above. The service is an API with a worker, deployed to dev, staging and prod. It uses a hashed ConfigMap, a component for tracing, a replacement for the hostname, a safe labelling scheme, an image pinned by digest in prod, and a CI job that renders, validates and diffs.

shop/
├── base/
│   ├── kustomization.yaml
│   ├── api-deployment.yaml
│   ├── worker-deployment.yaml
│   ├── service.yaml
│   └── ingress.yaml
├── components/tracing/
│   ├── kustomization.yaml
│   └── tracing-patch.yaml
├── overlays/
│   ├── dev/kustomization.yaml
│   ├── staging/kustomization.yaml
│   └── prod/
│       ├── kustomization.yaml
│       ├── env-settings.yaml
│       └── resources-patch.yaml
└── ci/render-all.sh

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- api-deployment.yaml
- worker-deployment.yaml
- service.yaml
- ingress.yaml
labels:
- pairs:
    app.kubernetes.io/part-of: shop
  includeSelectors: false
  includeTemplates: true
configMapGenerator:
- name: app-config
  literals:
  - LOG_LEVEL=info

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
- env-settings.yaml
components:
- ../../components/tracing
namespace: shop-prod
images:
- name: shop/api
  newName: registry.example.com/shop/api
  digest: sha256:0a1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8f9
replicas:
- name: api
  count: 6
configMapGenerator:
- name: app-config
  behavior: merge
  literals:
  - LOG_LEVEL=warn
patches:
- path: resources-patch.yaml
  target:
    kind: Deployment
    labelSelector: "app.kubernetes.io/part-of=shop"
replacements:
- source:
    kind: ConfigMap
    name: env-settings
    fieldPath: data.HOSTNAME
  targets:
  - select:
      kind: Ingress
      name: shop
    fieldPaths:
    - spec.rules.0.host

#!/usr/bin/env bash
set -euo pipefail
mkdir -p rendered
for overlay in overlays/*/; do
  name=$(basename "$overlay")
  kustomize build "$overlay" > "rendered/$name.yaml"
  kubeconform -strict -summary -ignore-missing-schemas "rendered/$name.yaml"
done
git diff --exit-code rendered/

Walk through what happens on a change. A developer edits the LOG_LEVEL literal. The hash suffix changes, every reference follows, and the Deployments roll. CI re-renders, kubeconform passes, and the rendered/ diff shows exactly the Deployments and the ConfigMap changing, which is the review. The pipeline merges, Argo CD or Flux applies, and pruning removes the old hashed ConfigMap. To promote a new image, CI runs kustomize edit set image in the prod overlay with the digest, commits, and the same render-validate-diff loop runs; rolling back is reverting one commit.

Where you are now

You can predict a build from its order (resources, components, generators, transformers, validators), explain why a generator's merge fails against a namespace, choose between strategic merge and JSON6902 patches and know when each silently misbehaves, use the hash suffix as a rollout mechanism, copy values with replacements instead of inventing templates, add labels to live workloads without tripping an immutable selector, organise environments with overlays and opt-in features with components, pin and vendor remote bases, inflate Helm charts while watching for their known hazards, teach kustomize about custom resources, render and validate and diff in CI, keep secrets out of git, and run the whole thing under Argo CD or Flux with an understanding of which renderer is which.

Senior takes each of these to the platform level. How the tool's failure modes behave at scale and across many teams. The trust model of remote bases, Helm inflation and KRM functions as a supply-chain question. Multi-tenant repositories with guardrails so one team's overlay cannot affect another's. Governing upgrades across a fleet with golden-file tests. Custom transformers and functions as an internal product, and the point at which to stop using kustomize and choose a different tool.

Sources

• Kustomize reference
• The kustomization file
• Glossary
• Builtin plugins
• kustomize build
• kustomize edit
• kustomize localize
• patches
• labels
• commonLabels
• components
• configMapGenerator
• secretGenerator
• generatorOptions
• replacements
• helmCharts
• sortOptions
• images
• openapi
• buildMetadata
• Extending Kustomize
• Eschewed features
• Versioning policy
• Installing kustomize
• Kustomize releases
• Remote build example
• Issue 6058: Helm namespace regression
• Issue 6241: Helm releaseName flag injection
• Kubernetes: declarative management with Kustomize
• Argo CD: Kustomize
• Flux: Kustomization

Part two of three. A cumulative review of Beginner and Mid-level material, organised by topic rather than level, in about thirty-five minutes. At this level interviewers stop asking "what is an overlay" and start asking what you do when a rollout does not happen, a patch matches nothing, or staging renders differently from your laptop. Fast review first, scenario questions in the middle, the short model answers at the end. Senior reviews all three levels.

Where this picks up

The model, in one screen

Kustomize is a template-free customisation tool. You keep plain Kubernetes YAML untouched and describe the edits in a kustomization.yaml; kustomize build emits the final YAML. It is a client-side, stateless renderer: it has no server, no cluster access, no release history and no rollback. Applying, pruning and ordering belong to whatever deploys the output.

The build order is what lets you predict surprises: resources (depth first), then components, then generators, then transformers, then validators. A base is fully rendered before its overlay's transformers see it. The convenience fields (namespace, namePrefix, labels, images, replicas, patches) are all shorthand for builtin transformers.

Kustomize versus Helm, the question you will always get: Helm is a package manager with templates, values and a release object that records what was installed and supports rollback. Kustomize has no templates and no release state; it edits valid YAML declaratively. They combine: Kustomize can inflate a chart with helmCharts, or patch Helm's output through a post-renderer. Say which one owns the release in your answer.

The deliberate non-goals are a good signal of depth. No ${VAR} substitution, no build-time reading of environment variables, no removal directives (only $patch: delete inside a patch), and no globs in the file. Injecting a value is done before the build, for example with kustomize edit set image, where the step is visible in CI.

Commands to have ready

kustomize build overlays/prod                       # render to stdout
kustomize build overlays/prod -o out/               # one file per resource, into a directory
kubectl kustomize overlays/prod                     # same, with kubectl's embedded (older) version
kubectl apply -k overlays/prod                      # build and apply
kubectl diff -k overlays/prod                       # preview against the cluster; exit 1 means "differences"
kustomize edit set image web=registry.example.com/web:1.4.2
kustomize edit fix                                  # migrate deprecated fields
kustomize version                                   # the renderer you are actually running
kubectl version --client                            # includes "Kustomize Version:"
kustomize build --enable-helm overlays/prod         # needed for helmCharts

Know the limits of each. kubectl apply -k cannot be combined with -f or -R, and it cannot take build flags, so --enable-helm needs kubectl kustomize --enable-helm DIR | kubectl apply -f -. kubectl diff exits 0 for no differences and 1 for differences, so a CI script that treats non-zero as failure breaks on a healthy change. kustomize edit add label writes the deprecated commonLabels unless you pass --without-selector.

Scenario questions

These are the questions that separate people who use the tool from people who have debugged it. Each answer starts with the reasoning, since interviewers mark the route as much as the destination.

1. I changed a ConfigMap and the pods did not restart. Why, and how would you fix it?Kubernetes does not restart pods when a ConfigMap they consume changes. If the ConfigMap is a literal resource, nothing changes in the pod template, so the Deployment sees no new revision. The fix is configMapGenerator: the hash suffix changes with content, every reference is rewritten, and the pod template changes, so a normal rolling update happens. Mention the cost: old hashed ConfigMaps linger until the deployer prunes them. And mention the exception: with disableNameSuffixHash: true you have given the rollout up.
2. My strategic merge patch builds, but the change does not appear. What do you check?First confirm it matched anything: a patch that matches nothing is an error ("no resource matches strategic merge patch"), so silence means it matched something. Then check which object. A patch with a target selector applies to every match, and the metadata.name in the patch body is ignored. Then check list semantics: on a custom resource with no schema, a list is replaced, not merged. Finally build the layers separately and use transformerAnnotations to see which transformer touched the field, because a later patch may have overwritten it.
3. A teammate added a label to commonLabels and the deploy to production failed with "field is immutable". What happened?Kustomize's commonLabels also edits spec.selector and pod template labels. A Deployment's selector cannot change after creation, so the API server rejected the update. Recovery means deleting and recreating the Deployment, with the disruption that implies. The prevention is the labels field with includeSelectors: false (and includeTemplates: true if pods need the label), identity labels set once in the base, and kubectl diff -k before applying. kustomize edit fix converts commonLabels to labels with includeSelectors: true, which reproduces the old output without changing live selectors.
4. The overlay build fails with "may not add resource with an already registered id". Where do you look?Two resources with the same kind, name and namespace entered the build. The usual causes: a file listed twice, an overlay that lists a base and also a file inside it, or a diamond, where two bases both include a shared one. Fixes depend on intent. If the shared thing should exist once, restructure so one parent owns it, or convert it to a component. If it should exist once per variant, add a namePrefix or nameSuffix in the intermediate overlays.
5. behavior: merge says the ConfigMap does not exist, but I can see it in the base. Why?Generators run before the namespace transformer, so the merge looks for the base ConfigMap as it was before namespace: was applied. If you set the overlay's namespace on the generator entry, it searches in a namespace the base object does not yet have. Match the name and the base's original namespace, usually none, and let the overlay's namespace: apply afterwards.
6. Staging and production both use one base. Staging renders fine; production fails on a tiny Deployment change. How do you catch that earlier?Render every overlay in CI on every change, validate the output with kubeconform and a policy tool, and compare rendered output to a committed copy or a previous commit. A base edit that only breaks one overlay is exactly what a matrix render finds. Adding kubectl diff -k against a non-production cluster catches the cases where the YAML is valid but the cluster rejects it.
7. The same overlay produces different YAML on my laptop and in CI. Give me your checklist.Compare renderer versions first: kubectl apply -k uses the kustomize embedded in kubectl (v5.7.1 in kubectl 1.34 and 1.35, v5.8.1 in 1.36 and 1.37) while CI may run a standalone binary. Compare flags, particularly --enable-helm. Compare the Helm version, since Helm 4 needs kustomize 5.8.1 or newer. Compare remote base refs: a branch moves, a full SHA does not. Fix by pinning one renderer version in every place, and rendering through that.
8. Kustomize renders a Helm chart but the resources have no namespace. What is going on?Since v5.8.0, the kustomization's namespace: is passed to Helm as the release namespace, and resources from charts whose templates do not set metadata.namespace no longer receive it. This is a known open issue (#6058). Workarounds are a JSON6902 patch adding /metadata/namespace or setting the namespace in the chart's values. A strong answer also says to check rendered output for namespaces whenever namespace: and helmCharts appear together, and to check includeCRDs, which defaults to false.
9. A build that worked yesterday now times out fetching a remote base. What are your options?The error is "hit 27s timeout running git". Each build clones every remote base with no cache, so slow networks, big repositories, submodules and TLS-inspecting proxies all hit it. Short-term: ?timeout=120 or ?submodules=false. Real fix: pin ref to a full SHA for reproducibility and vendor the base, either by copying it or with kustomize localize, so the build needs no network. In restricted environments, including regulated ones in the Gulf, the vendored copy is often the only option.
10. How would you deliver a new image version to production safely?CI builds the image once, records its digest, and runs kustomize edit set image in the production overlay (or an image-automation tool writes the same field), then commits. The pipeline renders, validates and shows the rendered diff for review, and the deployer applies. Rolling back is reverting one commit. The points that score: digest over a mutable tag, one writer for the file, the same artifact promoted through environments, and a reviewable rendered diff.
11. Helm printed "unknown shorthand flag: 'c'" when I ran a kustomize build. What does it mean?Kustomize up to 5.8.0 runs helm version -c --short and Helm 4 removed -c. Kubectl up to 1.35 embeds such a kustomize. Upgrade kustomize to 5.8.1 or later (the standalone binary, since the embedded one lags), or point --helm-command at a Helm 3 binary.
12. Someone proposes rendering third-party kustomizations in our CI with cluster credentials. Concerns?Helm inflation and KRM functions run programs with the privileges of the build. A releaseName beginning with a dash could inject Helm flags (issue #6241, fixed in 5.8.2, and kubectl 1.36 and 1.37 still embed 5.8.1), so use a standalone 5.8.2 or newer. Exec functions need --enable-exec, whose help text says not to use it on untrusted configs. Remote bases are supply-chain inputs, so pin full SHAs and review changes. Keep the load restrictor on. Render untrusted input in a sandbox without credentials.

Trade-off questions

Interviewers use these to see whether you choose by reasons or by habit. There is rarely a single right answer, and the strongest responses name what they would be giving up.

1. Overlay or component?An overlay describes a place: what makes production different. A component describes a feature you switch on: tracing, an external database, an LDAP sidecar. Choose a component when the variation is independent of environment, or when two overlays would otherwise both include the same customisation and create a diamond. Give up a little: components are still v1alpha1, and a component that merges into a generated ConfigMap depends on the base having it.
2. Strategic merge or JSON6902?Strategic merge states intent and survives refactoring because list elements are addressed by a merge key. JSON6902 is exact and can append or address by position, but a positional path silently changes meaning when a sidecar is inserted ahead of it. Use strategic merge by default and JSON6902 when you must append, remove a keyless element or patch a custom resource.
3. Kustomize or Helm?Kustomize for your own applications, where you control the YAML and want reviewable, template-free differences between environments. Helm when you consume third-party software packaged as charts, or need a release object with upgrade and rollback. For a third-party chart that you must patch, inflate it with helmCharts or commit a rendering, instead of forking it. The honest answer mentions that many teams use both.
4. Generated ConfigMaps with hashes, or stable names?Hashes give you automatic rollouts and immutable objects, at the cost of unpredictable names (which matter to anything that refers to a name outside kustomize's knowledge) and of orphaned objects until pruning. Stable names are simple and friendly to external references, but you must restart workloads yourself when the contents change.
5. helmCharts inflation or a committed helm template output?Inflation keeps the source small and the chart version explicit, but every build runs Helm, needs the flag, cannot authenticate to private registries, and is unsupported in Flux. A committed render is transparent and builds fast with no external program, but upgrading means re-rendering and reviewing a large diff. Pick the committed render when the environment cannot run Helm at build time.
6. Commit rendered manifests or render in the cluster controller?Rendered commits give reviewers the real YAML diff and decouple you from the controller's kustomize version. The cost is a second artifact that must stay in sync, which CI has to enforce. Rendering in the controller is simpler and has a single source of truth, but reviews show inputs, not effects, and a controller upgrade changes output silently.
7. Where should per-environment decisions live: overlay, Argo CD Application or Flux Kustomization?The repository's overlay, by default. An override in the controller object is invisible to someone reading the overlay and creates a second place for the same decision. Use the controller layer for what genuinely belongs to the deployment, such as which cluster, which namespace and sync policy.
8. Flat layout or intermediate overlays?A flat overlays/dev, staging, prod layout is easy to read until you multiply by clusters. Intermediate layers remove repetition, at the price of indirection; past three or four levels you must walk the chain to learn where a value comes from. Components often give the deduplication without the depth.

What a strong answer covers that a weak one misses

The same question often gets two very different answers. Here is what separates them on the topics that come up most.

Three habits also mark a strong candidate. They ask which version of kustomize is in use before answering anything version-sensitive. They describe how they would verify, by rendering and reading the output, rather than asserting. And they distinguish what kustomize does (render) from what the deployer does (apply, prune, order, health).

Field-by-field review

A fast pass through the fields an interviewer may ask you to write from memory. Practise writing each without looking.

images takes name, newName, newTag and digest. The name is the image as written in the base, and the transformer rewrites container and init container images in known paths. edit set image accepts name=newName:newTag, name:newTag, name=newName and name@digest, with * to preserve a value.

replicas takes name and count, and changes the replica count of the named workload without a patch.

patches takes path or patch, an optional target (required for JSON6902) and options. Target fields are group, version, kind, name, namespace, labelSelector and annotationSelector, and name and namespace are anchored regexes.

configMapGenerator and secretGenerator take name, files, envs, literals, options and behavior. generatorOptions applies to all generators, and a global disableNameSuffixHash: true cannot be overridden per generator.

replacements take source (or sourceValue, since 5.7.0) and targets, each with select, optional reject, fieldPaths and options such as create and delimiter.

labels takes pairs, includeSelectors, includeTemplates and includeVolumeClaimTemplates.

helmCharts takes name, repo, version, releaseName, namespace, the values fields, includeCRDs and skipHooks, with helmGlobals for chartHome and configHome.

sortOptions, buildMetadata, crds, openapi and configurations are the less common fields, and they are the ones a senior-leaning interviewer will probe.

Exit codes, errors and what they mean

Interviewers like to read an error aloud and ask what it means. Here are the ones to recognise on sight.

Common interview questions

1. What problem does Kustomize solve?Keeping one set of Kubernetes manifests and deriving variants per environment without templating or copying. You edit nothing in the base; overlays declare differences and kustomize build emits the result.
2. What are the stages of a build?Resources (depth first), components, generators, transformers, validators. A base is rendered completely before an overlay's transformers run.
3. What is a component and how does it differ from an overlay?A kind: Component kustomization in v1alpha1 bundling resources, patches and generators for an optional feature, applied to the including kustomization's accumulated resources. An overlay says where to start; a component says what to add.
4. How do you delete something from a base?Kustomize only adds, so the preferred route is to change the base or use a smaller one. Within a patch, $patch: delete removes an object or list element, and setting a field to null removes it.
5. What is the difference between namePrefix and labels with selectors?A prefix renames resources and rewrites references, and is safe. Labels with includeSelectors: true change selectors, which are immutable on live workloads.
6. How are two patches for the same field resolved?In list order; the later patch sees the result of the earlier one and wins on a conflicting scalar.
7. What are merge keys?Fields (name for containers and env, containerPort for ports) that identify a list element so a strategic merge patch updates it instead of replacing the list. Custom resources have none without an openapi schema.
8. What does replacements do and why not vars?It copies a value from a field of one resource into fields of others, with both ends written explicitly. vars used $(VAR) interpolation, had sharp edges, and is deprecated. edit fix --vars helps convert but is experimental.
9. How do you keep a helper object out of the output?Annotate it with config.kubernetes.io/local-config: "true"; it is available during the build and omitted from output.
10. What is the load restrictor?The default LoadRestrictionsRootOnly: files must be in or below the kustomization root. It stops generators reading arbitrary local files. Directories may be referenced from anywhere. Disabling it breaks relocatability.
11. What does the // mean in a git URL?It separates the repository from the subdirectory inside it. Combine with ?ref= pinned to a full SHA.
12. Why pin a full SHA and not a tag?Tags can move. A full commit SHA identifies content, and short SHAs are not supported.
13. What is kustomize localize?An alpha command that copies a kustomization and downloads its remote references, rewriting them as local paths, for hermetic or air-gapped builds. It does not yet cover Helm or plugin fields.
14. Why can kubectl apply -k and kustomize build disagree?Different versions. kubectl embeds an older kustomize and cannot take build flags.
15. How do you teach Kustomize about a custom resource?crds for name references, openapi for list merge keys, and configurations with field specs so builtin transformers such as images reach new paths.
16. What are KRM functions and why are they opt-in?Programs that read and write a ResourceList, run as containers or executables, configured by the config.kubernetes.io/function annotation. They are alpha and run arbitrary code, so they need --enable-alpha-plugins (plus --enable-exec for executables).
17. What changed in 5.0 that still trips people up?Generators no longer read the local environment; patches is no longer an alias for strategic merge patches; the gh: shorthand was removed; several kustomize cfg commands were removed; and several fields were deprecated.
18. How do you secure secrets in a Kustomize repo?Never commit plaintext. Use encrypted files (SOPS), sealed resources, or references to a vault via External Secrets; generate from CI only when the source never enters git. The secretGenerator output is base64, not encryption.
19. How do you check a render in CI?Build every overlay, validate with kubeconform, run policy checks with conftest or the Kyverno CLI, and compare against a committed render or the previous commit.
20. What does Flux not support?Remote bases and helmCharts in a kustomization file. Use local paths and a HelmRelease.

Final self-test

• List the build stages in order, and say why a generator's merge fails after you add namespace:.
• Explain what makes a strategic merge patch safer than a JSON6902 patch, and when you must use the latter.
• State why commonLabels can cause an outage and what to use instead.
• Explain how the hash suffix turns a config change into a rollout, and what must prune the old objects.
• Say what a diamond import does and two ways to fix it.
• Give the rule for //, ref and timeout in a remote base, and what localize adds.
• Name the four Helm hazards: the flag, the CRD default, the namespace regression and the releaseName validation.
• Say which kustomize kubectl apply -k uses and why CI should not rely on it.
• Name what Argo CD and Flux each add, and what Flux forbids.
• Describe how you would upgrade kustomize across thirty overlays without a surprise.

Answering these in under a minute each means you are ready. Senior adds the platform questions: multi-tenant repositories, supply-chain trust for bases and functions, governance for upgrades across a fleet, and when to stop using Kustomize.

Part two of three. At this level your overlays build and your pipeline deploys, so the problems change character. They are no longer "why will it not build" but "why did the rollout not happen", "why did staging render differently from my laptop", and "why did one label change take down production". Start with the error table, then the practices in order of leverage, the practice cards, and the operational recipes you will reach for at work.

Common errors at this level

Cumulative: Beginner's errors still apply, and these are the ones that appear once things work.

The practices that pay off most

Why the order matters

The first three cards are first because they prevent outages and non-reproducible builds: a selector change is an incident, and an unpinned renderer is a silent source of drift. The next three turn configuration and image changes into reviewable, reversible steps. The last two are about keeping the repository healthy, which matters in a year, not this week.

Practice cards

1. Watch a config change become a rolloutMount a generated ConfigMap with envFrom, apply, change a literal, and apply again. Run kubectl rollout status and see the Deployment roll without touching it. Then set disableNameSuffixHash: true and repeat: nothing rolls.
2. Reproduce the immutable-selector outage on purposeApply with commonLabels: {release: r1}, change it to r2, and read the error. Switch to labels with includeSelectors: false, and confirm the same change is now a rolling update.
3. Break a patch three waysMisspell the name in an SMP, use the wrong apiVersion, and use a positional JSON6902 path after reordering containers. Match each outcome to its message or silent failure.
4. Trace a value to its sourceTurn on originAnnotations and transformerAnnotations in a three-level overlay chain, and identify which layer set the replica count.
5. Fix a diamond two waysBuild a diamond, read the duplicate-ID error, then fix it once with a suffix and once by making the shared piece a component.
6. Move a hostname with a replacementPut the hostname in a local-config ConfigMap and copy it into an Ingress and a Deployment env var. Confirm the helper never appears in output.
7. Vendor a remote baseUse a pinned git base, time the build, run kustomize localize, disconnect, and build the localized copy.
8. Diff the two renderersRender the same overlay with kubectl kustomize and with a standalone kustomize build, and diff the outputs. Record both versions.
9. Build a golden-file testRender every overlay to golden/, commit, and make CI run git diff --exit-code golden/. Change a base value and watch CI fail.
10. Clean up deprecationsRun kustomize edit fix in a clean tree, diff the rendered output before and after, and confirm it is unchanged.

Recipe: render, validate and diff in CI

The core of every kustomize pipeline is three steps. A minimal script covers them and fails on any broken overlay:

#!/usr/bin/env bash
set -euo pipefail

kustomize version
mkdir -p rendered

for overlay in overlays/*/; do
  name=$(basename "$overlay")
  echo "== $name"
  kustomize build "$overlay" > "rendered/$name.yaml"
  kubeconform -strict -summary -ignore-missing-schemas "rendered/$name.yaml"
done

# Fail if committed renders are out of date
git diff --exit-code rendered/

Print the version at the top of the log so that a failing build tells you which renderer ran. -strict rejects unknown fields, and -ignore-missing-schemas avoids failing on custom resources, at the cost of not checking them; add a CRD schema source if you want those validated too. If CI needs a path filter, render only overlays whose directory, or whose base, changed.

For policy, feed the same renderings to your checker, for example conftest test rendered/ with Rego policies, or the Kyverno CLI against your cluster policies. Because the input is final YAML, a policy written once covers every overlay.

Recipe: promote an image through environments

CI builds the image once and records its digest. Promotion is an edit to one overlay at a time:

cd overlays/staging
kustomize edit set image shop/api@sha256:0a1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8f9
git add kustomization.yaml
git commit -m "staging: shop/api to build 1842"

To promote to production, the same digest is written into the production overlay by a later, reviewed step:

cd ../prod
kustomize edit set image shop/api@sha256:0a1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8f9

The property you want is that the artifact running in production is the artifact that ran in staging. Two writers for one file is how that breaks, so decide whether CI, Argo CD Image Updater or Flux image automation owns each overlay, and give each overlay exactly one owner.

Recipe: add a label to a live workload safely

The safe path in four steps. Check what you would change, add the label without selectors, preview against the cluster, and apply.

kustomize edit add label --without-selector --include-templates team:payments
kustomize build overlays/prod | grep -B2 -A6 "selector:"     # selectors must be unchanged
kubectl diff -k overlays/prod
kubectl apply -k overlays/prod

Remember that kubectl diff exits 1 when there are differences, so in a script handle the status explicitly:

set +e
kubectl diff -k overlays/prod > /tmp/diff.txt
status=$?
set -e
if [ "$status" -gt 1 ]; then echo "diff failed"; exit "$status"; fi

If the diff shows a spec.selector change, stop: the apply will fail, and the fix is to undo the selector-affecting label, not to recreate the Deployment.

Recipe: replace a deprecated field

Deprecations are warnings now and errors in a future major version, so clear them in a quiet moment:

git status                       # must be clean
kustomize build overlays/prod > /tmp/before.yaml
kustomize edit fix
kustomize build overlays/prod | diff /tmp/before.yaml - && echo "identical output"
git diff kustomization.yaml

Run it in each kustomization directory, since edit fix acts on the file in the current directory. The conversion of commonLabels yields labels with includeSelectors: true, so the output stays identical; you may then decide whether to go further and stop putting changeable labels in selectors. Converting vars needs kustomize edit fix --vars, which is experimental and can rewrite resource files: run it alone in a clean tree, read every change, and expect to finish by hand.

Recipe: make a Helm chart patchable

To customise a third-party chart without forking:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: monitoring
helmCharts:
- name: kube-state-metrics
  repo: https://prometheus-community.github.io/helm-charts
  version: 5.25.1
  releaseName: ksm
  namespace: monitoring
  includeCRDs: true
patches:
- target:
    kind: Deployment
    name: ksm-kube-state-metrics
  patch: |-
    - op: add
      path: /spec/template/spec/nodeSelector
      value:
        pool: system
- target:
    kind: ServiceAccount
    name: ksm-kube-state-metrics
  patch: |-
    - op: add
      path: /metadata/namespace
      value: monitoring

kustomize build --enable-helm . | kubectl apply -f -

Three checks make it safe. Verify the pinned version exists in the repository. Search the output for resources missing a namespace, since the v5.8.x regression may leave some without one, which is why the example patches it explicitly. And keep includeCRDs deliberate. The release name and chart name in the example are illustrative; replace them with the real ones and confirm the generated names in the output, since patch targets must match them.

Recipe: debug an overlay stack

When the wrong value appears, do not read files; ask the build:

cat >> overlays/prod/kustomization.yaml <<'EOF'
buildMetadata:
- originAnnotations
- transformerAnnotations
EOF
kustomize build overlays/prod | grep -B3 -A12 "name: api"

The config.kubernetes.io/origin annotation names the file that produced the object, and the transformer annotations list what touched it. Remove the buildMetadata block afterwards, because those annotations do not exist on live objects and will pollute a cluster diff. For a failing build, also try:

kustomize build --stack-trace overlays/prod 2>&1 | head -40
kustomize build --enable-helm --helm-debug overlays/prod 2>&1 | head -60

Then bisect: build the base alone, then each layer with only its parent. The first layer that fails contains the fault.

Recipe: upgrade kustomize without surprises

Treat the upgrade as a test with two binaries and a diff:

mkdir -p /tmp/old /tmp/new
for overlay in overlays/*/; do
  name=$(basename "$overlay")
  kustomize-old build "$overlay" > "/tmp/old/$name.yaml"
  kustomize-new build "$overlay" > "/tmp/new/$name.yaml"
done
diff -r /tmp/old /tmp/new && echo "no rendering differences"

Here kustomize-old and kustomize-new are your two pinned binaries under distinct names. Read every difference. Changes of the kind to look for include null handling (5.4.0), removed Starlark functions (5.5.0), and Helm namespace behaviour (5.8.0). Avoid v5.8.0 itself, whose release notes carry a regression warning, and move to 5.8.2 if you render any chart you do not control, for the releaseName validation fix. When the diff is clean, update the pin in CI, the container image tag (checking that the tag exists in the registry), and Argo CD's kustomize.path.<version> together.

Recipe: keep secrets out of the repository

Decide per secret, and write the decision down:

1. Static credentials a team owns (an API key for an internal service): encrypt with SOPS and let Flux or a function decrypt, or seal with Sealed Secrets.
2. Credentials that live in a vault (database passwords rotated elsewhere): commit an ExternalSecret that references the path, and patch the path per overlay.
3. One-off bootstrap material (the first admin token): create it with a pipeline step and apply it outside git.

Then audit: search history for accidental commits, and confirm that no secretGenerator reads from a tracked file.

git log --all --oneline -- '*.env' '*secret*'
grep -rn "secretGenerator" --include=kustomization.yaml .

If anything sensitive was committed, rotating it is the remedy. Removing the file from the latest commit does not remove it from history.

Reading a rendered diff like a reviewer

A rendered diff is only useful if you know what to look for in it, and most reviewers skim. Read it in a fixed order. Start with the list of object names that changed: an overlay edit that touches a dozen objects deserves a question, while a two-line change to one Deployment needs none. Next, look for anything in a selector: block, since a selector change on a live workload is the one edit that cannot be applied. Then look at names. A generated ConfigMap with a new hash is expected when its content changed and suspicious when it did not, because it means something upstream of the generator moved.

After that, scan images: every changed image should be a change you meant, and in production it should be a digest. Check namespaces next, which matters most in any overlay that inflates a Helm chart, because a resource that silently lost its namespace still renders cleanly. Finally, look at what disappeared. Kustomize never deletes anything from a cluster, but a resource vanishing from the render means the deployer may prune it, so a removal in the diff is a deletion in production if pruning is on.

If you review a large diff often, teach CI to summarise it. Even a simple count of objects added, changed and removed per overlay, posted on the pull request, makes the surprising change visible before anyone has to open a file.

Habits worth adopting now

1. Name the renderer in every bug report"Kustomize 5.8.2, standalone, with --enable-helm" narrows a problem faster than a stack trace. Print the version at the top of every CI job.
2. Review rendered YAML, not inputsA one-line overlay edit can change a dozen objects. Make the rendered diff part of the pull request.
3. Keep the base valid for nobody in particularNo environment names, hostnames or secrets. If you can apply the base unmodified to a dev cluster, it is about right.
4. One owner per field, one place per decisionWhen a value is set in the overlay, the Argo CD Application and a Flux patch, nobody knows which wins. Choose one.
5. Prefer strategic merge, and address by namePositional JSON6902 paths are latent bugs. Use them only when a merge key cannot express the change.
6. Pin everything that can moveThe renderer, the chart version, the remote base SHA, the image digest. A build should not change without a commit.
7. Treat plugins and Helm as codeEnable them only on repositories you trust, and never in a job that holds credentials and renders someone else's input.
8. Prune deliberatelyHashed objects are an advantage only when something deletes the old ones. Check that your deployer prunes, and check what it prunes.
9. Clear deprecation warnings when you see themThey are free to fix now. Run kustomize edit fix, diff, and commit.
10. Write the rollback downWith a digest in images:, rollback is a revert. If your rollback is anything more elaborate, the pipeline has a hidden step.

A checklist before you merge an overlay change

• I rendered every affected overlay, not only the one I was editing.
• I read the rendered diff and every changed object is expected.
• No selector, in any Deployment, StatefulSet or Service, changed.
• New labels went through labels with includeSelectors: false.
• Every image is pinned to a digest or an immutable tag in production.
• Helm charts have a pinned version, includeCRDs is deliberate, and I checked namespaces in the output.
• No secret value is in a tracked file, and no secretGenerator reads one.
• Remote bases are pinned to a full SHA.
• The kustomize version in CI matches the one I tested with.
• There are no new deprecation warnings.
• I know which deployer applies this and whether it prunes.

Where the series goes next

Senior takes these habits and makes them policy. It covers multi-tenant repositories where one team's overlay cannot affect another's, the trust model for remote bases, Helm inflation and KRM functions, how to govern upgrades and deprecations across many repositories, how to run the renderer as an internal platform, and when the right answer is to leave Kustomize for something else. For the neighbouring tools, see the Helm guide, the Kubernetes guide, and the Terraform guide for the infrastructure that the cluster runs on.

This is part three of three. It closes the series by taking every topic from Beginner and Mid one level further and adds the work you own when Kustomize is not your tool but your team's platform: the thing sixty engineers edit on a Friday afternoon and a pipeline trusts with production.

Kustomize looks harmless. It has no server, no database, no cluster connection and no state, so it is easy to conclude there is nothing to operate. That conclusion is how organisations end up with nine hundred overlay files, three different kustomize versions in three different places, a base that nobody dares touch, and a CI job that runs a stranger's code with cluster credentials in its environment. The tool is small; the estate built on top of it is not, and the estate is what you own now.

The questions change at this level. The earlier parts asked how to make an overlay produce the right YAML. This part asks what happens when two teams disagree about a base, which version of the binary your deployer actually runs, what a hostile pull request can make kustomize build do, and how you would prove that an upgrade changed nothing.

Where this picks up

I am starting with the architecture, because every other decision in this track follows from one fact: Kustomize holds no state.

A pure function, and what that implies

Kustomize is a client-side renderer. It reads files, applies the transformations you declared, and writes YAML to standard output. It does not talk to a cluster, it does not know what is currently deployed, and it does not remember what it produced last time. Given the same files and the same binary it gives the same output, every time, on every machine. That property is worth more than any feature, and almost every production practice in this guide exists to protect it.

Because the build is pure, three things become possible that are hard with stateful tools. You can render in CI with no credentials at all, which means rendering can happen in the least trusted place in your pipeline. You can diff two commits by rendering both and comparing text, which means a reviewer sees the real change and not the change to a template. And you can reproduce a historical deployment by checking out the commit and building, which means "what was running at 14:00 last Tuesday" has an answer that does not depend on anyone's memory.

The same property takes things away. There is no release object: Helm records each install as a release in the cluster and can roll back to the previous one; Kustomize has nothing equivalent, so rollback is a git revert followed by a re-deploy. There is no awareness of what exists: if you delete a resource from the base, Kustomize simply stops emitting it, and whether the cluster object disappears depends entirely on the deployer's pruning behaviour. There is no ordering beyond the emitted sort, so health gating, waiting for a CRD to become established, and sequencing a migration job all belong to the deployer. And there is no parameter channel: the project deliberately refuses ${VAR} substitution and refuses to read environment variables or command-line values during the build, so that the output depends only on files.

The owner's consequence is that Kustomize's failure modes cluster in two places: inside the build, where a wrong rule produces wrong YAML or no YAML, and at the boundary with the deployer, where correct YAML meets a cluster with history. The next sections take them in that order.

How the build actually runs

You know the fields. To predict behaviour in a large overlay tree you need the order in which they act, because most surprising output is an ordering effect.

Kustomize processes a kustomization depth first. When it meets a directory in resources:, it builds that directory completely, including all of its own resources, generators, transformers and validators, and only then merges the result into the accumulated set of the parent. So when an overlay's patch runs, the base has already been fully transformed: its name prefix is applied, its labels are applied, its generated ConfigMaps have hash suffixes. The overlay sees the base as a finished product, never as source.

Within one kustomization the order is: resources and components are accumulated, then generators run, then transformers, then validators. The convenience fields you use daily, such as namePrefix, labels, images and patches, are shorthand for built-in transformers and run in a fixed internal order within that transformer stage. "Everything is a transformer" is literal: namePrefix: bob- is equivalent to a PrefixSuffixTransformer configuration listed under transformers:. You can see the built-in set with kustomize edit alpha-list-builtin-plugin.

One consequence of this ordering is written into the official docs and still catches people: generators run before the namespace transformer. A configMapGenerator with behavior: merge or behavior: replace has to match the base's ConfigMap by its original name and namespace, not the namespace the overlay is about to impose. If it does not match, the build fails with does not exist; cannot merge or replace, and the message looks like a missing resource when the real cause is a namespace that has not yet been rewritten.

Identity and name references

Every object in a build is tracked by a resource ID, written like Deployment.v1.apps/web.[noNs]: group, version, kind, name and namespace. Two objects with the same ID in one build is an error, which is why including the same base twice fails with may not add resource with an already registered id.

Kustomize also tracks renames. When a prefix, a suffix or a content hash changes a name, it rewrites every reference it knows about: a Deployment's envFrom, a volume's configMap.name, an Ingress backend, a RoleBinding's roleRef. The list of known reference fields is the nameReference configuration, and it covers the core Kubernetes kinds. It does not know about your custom resources. If a CRD object refers to a ConfigMap by name, and that ConfigMap gets a hash suffix, the reference silently keeps the old name and the application fails at runtime. That is one of the most expensive classes of bug in Kustomize estates, and the fix lives in the configurations: field, covered below.

A patch may refer to a resource by its current name, its original name, or any intermediate name. That generosity is convenient and occasionally dangerous: an overlay patch that targets web keeps matching after a mid-level overlay adds a prefix, but it also means two different bases renamed into collision can make a patch ambiguous. When a patch fails with failed to find unique target, the cause is almost always a typo in the header, a wrong apiVersion, or zero matches because the target was dropped by an earlier stage.

The diamond and other accumulation failures

Accumulation is where structural mistakes surface. The one you will meet most is the diamond: overlay prod includes base-a and base-b, and both include common. The build sees common twice, and every resource in it collides with itself. There is no de-duplication by design; identical IDs are an error because silently picking one would hide a real conflict.

There are three honest ways out. Stop including common from the intermediate bases and include it once from the top. Convert common's optional behaviour into a component and opt in from the top. Or give each intermediate base a distinct namePrefix so the objects are genuinely different. What you must not do is reach for --load-restrictor LoadRestrictionsNone to reference files across directories, because that turns a structural problem into a security and portability problem.

The second structural failure is the load restrictor. By default a kustomization may read files only from its own directory or below; it may reference other directories anywhere, provided each is itself a kustomization. The error reads security; file '.../outside/x.yaml' is not in or below '.../ov'. The remedy is always structural: make the shared directory a kustomization and reference the directory. A related variant looks different but has the same cause, when an overlay references ../base/deployment.yaml (a file) instead of ../base (a directory), and Kustomize complains that the path "must build at directory".

Repository architecture for many teams

Nothing in Kustomize dictates a layout, and that freedom is what makes estates decay. The layout is a platform decision because it determines who can change what, how large a review diff is, and how many places a base change has to be tested.

The standard shape is one base/ per application and one overlay per environment under overlays/. It works well until you have several clusters, where the environment is no longer the only axis. Then the tree becomes a matrix, and the question is how to keep it from multiplying.

platform/
├── bases/
│   └── payments/                  # one application, environment-neutral
├── components/
│   ├── hpa/                       # opt-in features
│   ├── pdb/
│   ├── network-policy-strict/
│   └── istio-sidecar/
├── envs/
│   ├── staging/                   # environment-wide settings
│   └── prod/
└── clusters/
    ├── riyadh-prod/
    │   └── payments/kustomization.yaml   # envs/prod + components + cluster patches
    ├── cairo-prod/
    │   └── payments/kustomization.yaml
    └── riyadh-staging/
        └── payments/kustomization.yaml

The rules that keep this sane are few. Depth is a budget. Three levels of inheritance (base, environment, cluster) is readable; four starts requiring a diagram; five means nobody can predict the output without building it. Variation goes into components, not into deeper overlays, because a component is an opt-in flag at the top and an overlay is a permanent ancestor. Leaf directories are thin: a cluster's kustomization should list its resources, its components and a small number of patches, and nothing clever. And each directory is one buildable unit with its own kustomize build, so CI can render and validate every leaf independently.

For teams in the Gulf and Egypt this matrix is usually real rather than theoretical. A regulated workload may need to run in an in-country region for residency reasons while a sibling workload runs elsewhere, and the differences between those clusters (endpoints, node pools, storage classes, policy strictness) belong in the cluster leaf while the application stays in the shared base. Keeping residency-driven differences in a named leaf also makes them auditable: the leaf is the answer to "what is different about the Riyadh deployment".

Components, and the honest status of the feature

Components are the right tool for opt-in behaviour, but two facts temper that. First, they carry apiVersion: kustomize.config.k8s.io/v1alpha1 and kind: Component, and have done since they were added in v3.7.0. They are widely used, including by Argo CD's components support, but the API is not stable in the way v1beta1 Kustomizations are, so pin your version and read release notes before upgrading. Second, their semantics are subtle: a component's transformations apply to the accumulated resources of the kustomization that includes it, including resources that came from bases, not only resources declared inside the component.

That is powerful and dangerous in equal measure. A component that adds a podAntiAffinity patch to every Deployment will hit every Deployment in the including kustomization, including ones you did not know were there. Write component patches with explicit target selectors, prefer label selectors that your own conventions guarantee, and test each component against the smallest kustomization that includes it.

Two errors flag misuse and read as opposites. Putting a Component under resources: fails with expected kind != 'Component'; putting a plain Kustomization under components: fails with expected kind 'Component' ... but got 'Kustomization'. The fix is to use each field for its own kind, and to resist the temptation to "make it work" by relabelling the kind.

Patching at scale, and custom resources

At Mid level a patch is a way to change a field. At scale it is a liability with a maintenance cost, because every patch is a coupling between an overlay and the exact shape of the thing it patches. A base refactor that moves a container from index 0 to index 1 breaks every JSON6902 patch that said /spec/template/spec/containers/0, and breaks it silently if the path still exists but now points at the wrong container.

Some guidelines earn their keep. Prefer strategic merge patches over JSON6902 for containers, because they merge lists of containers and env vars by name rather than by position. Use JSON6902 where you need precise operations such as removing a single entry or adding to a specific index, and keep those patches near the base they depend on. Use target with a label selector for policy-shaped changes that apply to many objects, and use a specific name for one-off changes. Remember that name and namespace in a target are anchored regular expressions, so web means ^web$ and web.* matches a prefix. And beware over-matching: a selector that accidentally matches more objects than you intended raises no error at all, so review selector-based patches by examining the rendered diff, not the patch file.

Why patches on custom resources fail quietly

Kustomize merges lists using merge keys, and it learns merge keys from an OpenAPI schema. The built-in schema is based on Kubernetes v1.21.2, which is old, and it knows nothing about your CRDs. For a core kind, a strategic merge patch that supplies one container merges by the name key. For a custom resource, Kustomize has no merge key for a list, so a patch on a list in a CRD replaces the whole list. Nothing errors. The overlay author wanted to change one entry, and the rendered output has lost all the others.

There are two remedies, and they address different problems. To teach merging, give Kustomize a schema through the openapi field:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
openapi:
  path: schema/my-crd-openapi.json
patches:
- path: add-listener.yaml

To teach name references and transformers about your custom kinds, use configurations: with transformer configs, and crds: to supply CRD definitions. A configuration file can tell the images transformer where images live in a custom kind, tell nameReference that a field in your CRD refers to a ConfigMap, or tell the label transformer which paths to touch. Version 5.8.2 added kustomize edit add configuration to manage this field from the CLI.

nameReference:
- kind: ConfigMap
  version: v1
  fieldSpecs:
  - kind: MyWorkload
    path: spec/configRef/name

The discipline is the same in both cases: for every custom kind you deploy through Kustomize, write a test that renders a representative overlay and asserts that a hashed ConfigMap reference was rewritten and a list patch merged rather than replaced. Without that test you learn about the gap in production.

Replacements, and the end of variables

vars is deprecated, and replacements is its successor. The design difference matters to an owner. A var declared a global name and a $(NAME) token that could appear anywhere in a string, which made data flow invisible and build behaviour dependent on string scanning. A replacement declares a source field on exactly one resource and one or more target field paths, so the data flow is explicit and inspectable. Since 5.7.0 a source can be a static value with sourceValue, since 5.8.0 selectors accept regular expressions, and since 5.8.0 a field path can reach inside a string that contains embedded JSON or YAML.

Two cautions apply. Replacements are still a build-time mechanism that copies data between objects in the same build; they are not a general templating language, and the moment you are composing strings from five sources you are building a template engine in YAML and should reconsider the tool. And the options.encoding field exists in the type but is not implemented, so do not design around it.

A recurring pattern is the local-config source: a resource carrying the annotation config.kubernetes.io/local-config: "true" is used during the build, for example as the source of a replacement, and is omitted from the output. That lets an overlay hold a small environment descriptor, such as a hostname and a region, and propagate it into several manifests without the descriptor itself being deployed.

resources:
- ../../base
- env-descriptor.yaml          # annotated config.kubernetes.io/local-config: "true"
replacements:
- source:
    kind: ConfigMap
    name: env-descriptor
    fieldPath: data.hostname
  targets:
  - select:
      kind: Ingress
    fieldPaths:
    - spec.rules.0.host
    - spec.tls.0.hosts.0

Scaling: which part breaks first

Kustomize has no documented hard limit, and no server to saturate, so "scaling" means something specific: the time and attention it takes to build, render, review and ship as the number of overlays, resources and teams grows. In practice four things break, in this rough order.

Review breaks first. A base change that touches every environment produces a pull request whose meaningful effect is invisible in the source diff. Reviewers approve a two-line change to a base and have no idea it alters forty rendered Deployments. This is a human scaling limit and it arrives long before any performance limit. The remedy is to make the rendered change the thing under review, which the governance section shows.

Remote fetches break second. A remote base is cloned with git on every build, with no cache, under a default timeout of 27 seconds. One remote base in one overlay is a minor annoyance. Forty overlays each with a remote base means forty clones per pipeline run, each one a chance for a network blip, and on a slow or TLS-inspecting network the failure reads hit 27s timeout running '/usr/bin/git ...'. Adding ?timeout=120 or ?submodules=false treats the symptom. Pinning to a tag or full commit SHA and vendoring with kustomize localize treats the cause.

Helm inflation breaks third. Every build of a kustomization with helmCharts runs the helm binary, and when the chart is not already under chartHome, it runs helm pull too. A large chart rendered across many overlays is the usual reason a previously fast pipeline takes minutes. Commit the pulled chart (or render it once to a base) so that builds are local, and reserve live inflation for places where it earns its cost.

Build time grows last. Kustomize is fast for typical applications. It grows with resource count, transformers and patches, and the 5.0 release shipped large performance improvements, while 5.8.0 introduced and 5.8.x then fixed a slowdown in Helm namespace propagation. The practical rule is to measure rather than assume: time kustomize build on your biggest leaf, record it as a number, and watch it across upgrades.

Measure before you optimise. A simple harness builds every leaf, times it, and records resource counts:

#!/usr/bin/env bash
# Build every leaf kustomization, report time and rendered size.
set -euo pipefail
find . -name kustomization.yaml -not -path './bases/*' -not -path './components/*' \
  | sort | while read -r f; do
    d=$(dirname "$f")
    start=$(date +%s.%N)
    n=$(kustomize build "$d" --enable-helm | grep -c '^kind:' || true)
    end=$(date +%s.%N)
    printf '%-50s %4s objects %6.2fs\n' "$d" "$n" "$(echo "$end - $start" | bc)"
  done

The trust model

Kustomize is usually run by a pipeline that holds credentials, on files that many people can change, so the question "what can a kustomization make kustomize build do" is a security question. The answer is: by default, little; and the settings that widen it are all opt-in and all worth treating as privileged.

The load restrictor

By default a kustomization can read files only from its own directory tree. Without that rule, a pull request could add a configMapGenerator with files: [../../../../etc/passwd] or a path into a runner's home directory, and the generated ConfigMap, committed or applied, would carry the contents. The restrictor, LoadRestrictionsRootOnly, is what stops this. The alternative --load-restrictor LoadRestrictionsNone exists for legitimate reasons, but the official help says plainly that it breaks the relocatability of a kustomization, and it also removes the protection. Treat any pipeline that passes it as a finding, and any engineer who asks for it as a reason to fix a directory layout.

Plugins and functions are code execution

KRM functions and plugins are the extension mechanism, and they are all alpha. Kustomize runs them only when you pass --enable-alpha-plugins, and exec functions additionally need --enable-exec, which the help text marks "do not use for untrusted configs!". Containerised functions need a container runtime, and they can be given network with --network and host paths with --mount. Kustomize provides no sandbox of its own. A function that runs as an executable has your user's privileges; a containerised one has whatever the container runtime grants. The official documentation's warning is the right mental model: a plugin from the internet can transform your config and also do anything the user running the build can do to the machine.

Two policy rules follow. Builds of untrusted input never enable plugins: a pull request from a fork, a third-party base you are evaluating, a tenant's repository. And plugins on trusted builds are pinned, reviewed, and run in the smallest container that works. Note also that Starlark functions were removed in 5.5.0 and that the legacy exec and Go plugin mechanisms are deprecated, so any plugin written that way is also a migration item.

The Helm flag-injection bug

The clearest recent example is issue #6241, fixed in kustomize v5.8.2. Kustomize builds a helm template command line from the fields of a helmCharts entry. Before the fix, a releaseName (or name) value that began with - was passed through as if it were a Helm flag, so a value like --post-renderer=/some/program made Helm run that program. The result was client-side code execution by anyone who could influence a kustomization that was built with --enable-helm. The fix rejects such values:

Error: ... releaseName must not start with '-', got "--post-renderer=/bin/echo"

The relevant facts for an owner are these. Every release up to and including v5.8.1 is affected. kubectl 1.36 and 1.37 embed v5.8.1, so kubectl kustomize --enable-helm inside those is affected too, and a pipeline that runs a standalone binary of 5.8.2 is not. The exposure requires --enable-helm, so a build that never enables Helm is not reachable through this path, but your Argo CD may have enabled it globally through kustomize.buildOptions, which means every application in that instance is a reachable surface. The action is to move every CI image and every deployer's kustomize to 5.8.2 or later, and to decide separately whether Helm inflation should be on for repositories that accept outside contributions.

Remote bases are supply-chain inputs

A resources: entry that points at a git URL is a dependency on someone else's repository, evaluated at build time. If it names a branch, what you deploy changes when that branch changes. If it names a tag, what you deploy changes when someone moves the tag. Only a full commit SHA is immutable, and Kustomize does not support short SHAs. Treat a remote base as you treat a package dependency: pin it to a full SHA, update the pin by pull request with the upstream diff attached, and keep the list of remote bases discoverable by grep.

The strongest posture is to have no remote fetching at build time. kustomize localize (alpha) copies a kustomization and downloads its remote references into local paths, so a build becomes hermetic and offline-capable. It does not yet localize Helm or KRM plugin fields, and it reorders fields alphabetically, so review its output rather than assuming it is cosmetically identical. By default it verifies itself by building both trees and comparing the output. Flux goes further and refuses remote bases outright, which is a useful signal about what the GitOps community regards as safe.

# Vendor a kustomization and all of its remote bases for a hermetic build
kustomize localize overlays/prod vendored/prod
diff <(kustomize build overlays/prod) <(kustomize build vendored/prod) && echo identical

Secrets

Kustomize generates Secrets, and that is the source of a dangerous misunderstanding. secretGenerator produces a Kubernetes Secret, whose data is base64-encoded. Base64 is an encoding, not encryption. A secretGenerator fed from a file in git puts a readable secret in git, and a rendered manifest committed for review puts it in a pull request too.

The reason Kustomize cannot solve this itself is the same reason it is reproducible: it has no decryption step and will not read environment variables or call a secret store during the build. So secrets have to be handled around Kustomize, with one of four patterns:

• Encrypted files decrypted by the deployer. With SOPS, the encrypted file is committed, and Flux's kustomize-controller decrypts it (spec.decryption.provider: sops). Argo CD can achieve the same through a KRM function or a plugin such as the third-party ksops. The plaintext exists only in the cluster's control plane or the function's memory.
• Sealed Secrets. You commit a SealedSecret custom resource, which is safe in git and which an in-cluster controller turns into a Secret. To Kustomize it is just another resource.
• External references. With the External Secrets Operator or the Secrets Store CSI driver, git holds only a reference to a secret in your vault, such as Vault, and the operator fetches it at runtime. Kustomize renders the reference.
• Create the Secret outside git. A pipeline generates the Secret from a vault and applies it directly. This breaks the "git describes everything" property and needs its own drift story.

A secret in a hashed generator deserves one more note. The hash suffix makes a change to a secret trigger a rollout, which is usually what you want, but the old Secret remains in the cluster until something prunes it, and a pruned-too-eagerly deployer can delete a Secret that a still-rolling pod refers to. Decide your pruning policy with hashed objects in mind.

# Safe: a reference, not a value. The operator resolves it at runtime.
resources:
- external-secret.yaml      # kind: ExternalSecret, points at a vault path

# Unsafe: a value in git, encoded but not protected.
# secretGenerator:
# - name: db
#   files: [db-password.txt]

Helm inside Kustomize, and where it does not fit

Kustomize and Helm solve different halves of one problem: Helm parameterises, Kustomize patches. Teams end up combining them, and there are three ways to do it with different risk profiles.

Inflation with helmCharts. Kustomize runs helm template for you and feeds the result into the pipeline, so you can patch chart output without forking. It needs --enable-helm, and it inherits all the risks above. The official position on scope is narrow and worth stating to teams that ask for more: Kustomize supports bug fixes, security fixes and flags analogous to helm template; it does not support private repository or registry authentication, and it does not support post-renderers. The stated long-term direction is a KRM function.

Render once, commit the result. Run helm template in a script, write the output into a base directory, and overlay that. You lose live updates, and you gain a base that builds with no Helm, no network and no --enable-helm flag. For any chart that changes slowly this is the lower-risk option and it makes the diff of a chart upgrade a reviewable YAML diff.

Helm with Kustomize as post-renderer. The reverse direction: Helm renders, then runs Kustomize on the output. It is a common way to patch a chart that a team does not own while still using Helm's release management for install and rollback.

Whichever you choose, a few specifics matter at this version.

includeCRDs defaults to false. A chart whose CRDs live in its crds/ directory will render without them, and the failure appears at apply time as an unknown kind.

Since v5.8.0 the kustomization's namespace: is passed to Helm as the release namespace. The side effect, still open as issue #6058 at the time of writing, is that resources from charts whose templates do not set metadata.namespace no longer receive the kustomization namespace: on 5.8.1 and 5.8.2 they come out with none. Version 5.7.1 did set it. The workaround in the issue is a JSON6902 patch that adds /metadata/namespace, or setting the namespace in the chart's values. This is a silent failure, so the rendered-output assertion "every namespaced resource has a namespace" belongs in your CI.

Helm 4 support arrived in 5.8.1. With older kustomize (including the kubectl 1.34 embed at 5.7.1) and Helm 4 installed, --enable-helm fails with unknown shorthand flag: 'c' in -c, because the old version runs helm version -c --short and Helm 4 removed the flag. Either upgrade Kustomize or point --helm-command at a Helm 3 binary.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: ingress-nginx
helmGlobals:
  chartHome: charts
helmCharts:
- name: ingress-nginx
  repo: https://kubernetes.github.io/ingress-nginx
  version: 4.11.3
  releaseName: ingress
  namespace: ingress-nginx
  includeCRDs: true
  valuesInline:
    controller:
      replicaCount: 2
patches:
- target: {kind: Deployment}
  patch: |-
    - op: add
      path: /metadata/namespace
      value: ingress-nginx

Delivery: the binary you test is not the binary that deploys

An owner's most repeated incident is version skew. A platform has several kustomize binaries and assumes they agree: the one on developers' laptops, the one in the CI image, the one embedded in kubectl, and the one inside Argo CD or Flux. They do not agree, and a build can succeed in one and fail or differ in another.

As of this writing the latest stable Kustomize is v5.8.2, released on 2026-09-30. The Kubernetes releases of the same day embed older versions: kubectl 1.34 and 1.35 embed v5.7.1, and 1.36 and 1.37 embed v5.8.1. No kubectl release embeds 5.8.2 yet. Check your client with kubectl version --client, which prints a Kustomize Version: line. Because kubectl apply -k uses the embedded binary, a pipeline that tests with a standalone 5.8.2 and deploys with kubectl apply -k is testing one program and running another.

There is a second, sharper limit: kubectl apply -k takes no build flags. It cannot enable Helm. A kustomization that uses helmCharts has to be deployed through kubectl kustomize --enable-helm DIR | kubectl apply -f -, or through a GitOps controller configured to allow it.

Argo CD

Argo CD detects a kustomization.yaml and builds it with its own bundled kustomize. You control behaviour at two levels. Instance-wide build options live in argocd-cm under kustomize.buildOptions, for example --enable-helm, and they apply to every application on that instance, which is why the Helm bug above is an instance-level exposure. Per-application settings live in spec.source.kustomize and include namePrefix, nameSuffix, images, replicas, namespace, patches, components (since v2.10), commonLabels, labelWithoutSelector, labelIncludeTemplates, commonAnnotations, forceCommonLabels, forceCommonAnnotations and version. To run a specific binary, register it under kustomize.path.<version> in argocd-cm and select it with spec.source.kustomize.version; this is how you move applications to a new Kustomize one at a time.

Be careful with Argo's commonLabels field. It is a parameter override applied at render time, and like its kustomization counterpart it changes selectors unless you set labelWithoutSelector. An innocuous label added to an Application can make Argo attempt to change an immutable spec.selector.

Flux

Flux uses the kustomize.toolkit.fluxcd.io/v1 Kustomization custom resource, which is not the same object as a kustomization.yaml file and is the most common naming confusion in this space. Flux's resource points at a source and a path, and runs kustomize on that path, with fields like prune, targetNamespace, patches, images, components, namePrefix, commonMetadata, decryption, wait, healthChecks and dependsOn. It also offers postBuild.substitute and substituteFrom for variable substitution, which is a Flux feature that runs after the build and is not part of Kustomize. Flux does not support helmCharts in a kustomization file (you use a HelmRelease instead) and does not support remote bases.

Both controllers give you what Kustomize itself does not: pruning, health assessment, drift correction and ordering. Read their documentation for the pruning rules before you delete anything from a base.

Multi-tenancy

Kustomize has no concept of a tenant. What it has is a set of fields that, used consistently, let one base serve many tenants, and a set of limits that decide where that stops being safe.

The basic pattern is one base and one overlay per tenant. Each tenant overlay sets namespace, a namePrefix or nameSuffix where names must not collide, labels that identify the tenant, and any per-tenant quota or policy. Tenant features are expressed as components, so a tenant opts in to a feature by listing a component in its overlay.

The namespace: field overrides the namespace of every namespaced resource, which makes isolation easy to express and hard to forget. Its limits matter. Cluster-scoped objects are not namespaced, so two tenants that both render a ClusterRole or a CRD with the same name collide at the cluster level even though their namespaced objects are separate. Give cluster-scoped objects a per-tenant namePrefix, or better, keep them out of tenant overlays and own them centrally. And the namespace: transformer can fail when it would collapse two same-named resources from different namespaces into one, with namespace transformation produces ID conflict; that error is correct, and the answer is to split the kustomizations.

A larger question is who is allowed to edit what. Kustomize has no access control, and git is where it lives. The usual answer is repository structure plus code owners: the platform team owns bases/ and components/, each tenant team owns its leaf under clusters/ or tenants/, and required reviewers enforce it. That lets a tenant change its own overlay without being able to alter the shared base. The dangerous gap is that a tenant overlay is still a kustomization, and a tenant who can edit it can add a resources: entry that pulls in any other directory in the repo, or a remote URL, or a patch that removes a security setting the base applied. Structural separation in git does not by itself constrain what a tenant's kustomization may produce.

That constraint has to be enforced where it can be: at the deployer and the cluster. In Flux, spec.targetNamespace places a tenant's resources in its namespace and spec.serviceAccountName makes the controller apply them as that service account, so the tenant's RBAC bounds what the tenant's manifests can do even if they try to create something out of scope. In Argo CD, AppProject definitions restrict the destinations and resource kinds an application may use. Then an admission policy engine such as Kyverno enforces the properties you care about on whatever arrives, regardless of how it was rendered. The layering is deliberate: Kustomize shapes the YAML, the deployer limits the authority of the apply, and admission control is the last check that does not depend on trusting anything upstream.

Upgrades and migrations

Because Kustomize is a pure function, an upgrade is unusually testable: render everything with the old binary, render everything with the new one, and diff. If the diff is empty, the upgrade is safe for your estate; if not, every difference is a decision. Skipping that test is the only way to be surprised, and surprises have happened: 5.4.0 changed how null values were handled, 5.5.0 removed Starlark functions, and 5.8.0 changed Helm namespace propagation.

#!/usr/bin/env bash
# Upgrade gate: render every leaf with two versions and report any difference.
set -euo pipefail
OLD=${OLD:-/opt/kustomize/5.7.1/kustomize}
NEW=${NEW:-/opt/kustomize/5.8.2/kustomize}
status=0
while read -r d; do
  if ! diff -u <("$OLD" build "$d" --enable-helm 2>/dev/null) \
               <("$NEW" build "$d" --enable-helm 2>/dev/null) > /tmp/render.diff; then
    echo "DIFFERS: $d"; head -40 /tmp/render.diff; status=1
  fi
done < <(find . -name kustomization.yaml -not -path './bases/*' -not -path './components/*' -exec dirname {} \;)
exit $status

A version history worth knowing

The changes that altered output or behaviour are the ones to check against. The 5.0 release removed a batch of commands and behaviours: the kustomize cfg and kustomize fn subcommands other than cfg cat|count|grep|tree and fn run, the legacy aliasing of patches to patchesStrategicMerge, the gh: host shorthand in remote URLs, reading the local environment in generators, and automatic stripping of .git from AWS and Azure URLs. The 5.4.0 release stopped replacing null YAML values with the string "null". The 5.5.0 release removed Starlark support and added --helm-debug. The 5.7.0 release added sourceValue for replacements, image-volume support in images, and helmCharts[].devel. The 5.8.0 release added regex selectors and embedded-string targets for replacements and passed the namespace to Helm, and carried a warning against using it at all; use 5.8.1 or later, which fixed that regression and added Helm 4 support. The 5.8.2 release fixed the Helm flag injection, added edit add configuration and edit remove component, and loads its built-in schema from a precompiled bundle, which is still based on Kubernetes v1.21.2.

Deprecations as a programme

Several fields are deprecated and produce a warning on stderr during the build: commonLabels, patchesStrategicMerge, patchesJson6902, vars, bases, and imageTags. They are deprecated forever in v1beta1: the official versioning policy says a field's meaning never changes, deprecation brings a minor bump and a non-fatal message, and removal requires a major API version. There is no v1 Kustomization API yet, so nothing is scheduled to break today. The reason to migrate is not urgency; it is that the deprecated fields are excluded from a future v1, the warnings train people to ignore stderr, and some of them are actively unsafe.

commonLabels is the one to care about most. It adds labels to metadata and also rewrites selectors, and a Deployment's spec.selector is immutable. Applying a changed commonLabels to live objects fails with field is immutable, and recreating the Deployment is otherwise the only way out. Its replacement, labels, defaults to includeSelectors: false, which adds labels without touching selectors. The automatic migration from commonLabels turns it into labels with includeSelectors: true, which preserves the old behaviour, risk included. That is the right mechanical translation for existing live objects, since changing selectors would itself break them, but it means a conversion does not make a label safe; it keeps it exactly as dangerous as before. Use includeSelectors: false for every new label.

There is also a trap in the CLI. kustomize edit add label k:v writes the deprecated commonLabels by default. Pass --without-selector to get labels, and --include-templates (which requires --without-selector) to add the labels to pod templates.

The tool for the mechanical part is kustomize edit fix, which converts commonLabels, patchesStrategicMerge, patchesJson6902 and bases. The vars conversion is separate and experimental: kustomize edit fix --vars may rewrite resource files, and can leave a replacements must specify a source and at least one target error when a variable was never referenced. Run it in a clean git tree, one directory at a time, and review the diff by rendering before and after.

# Migrate one directory and prove the render did not change
git diff --quiet || { echo "dirty tree"; exit 1; }
kustomize build overlays/prod > /tmp/before.yaml
( cd overlays/prod && kustomize edit fix )
kustomize build overlays/prod > /tmp/after.yaml
diff /tmp/before.yaml /tmp/after.yaml && echo "render unchanged"

Governance and the CI gates that matter

Governance for a tool with no runtime is a set of checks that run on the rendered output, plus rules about who may change what. The principle is that every policy should be checkable by a machine against the render, so that review effort goes to intent rather than to arithmetic.

A gate that catches most of what matters has six parts, run for every leaf directory on every pull request:

1. Build. kustomize build succeeds with the deployer's pinned version. A leaf that does not build fails the change.
2. No warnings. The build prints deprecation warnings to stderr. Capture stderr and fail on # Warning: lines so the estate does not regress.
3. Schema validation. Run kubeconform (or an equivalent) over the render, with the schemas for your CRDs loaded. This catches misspelt fields, wrong types, and removed API versions before the cluster does.
4. Policy. Run conftest, the Kyverno CLI, or kube-linter over the render. Policies should encode real rules: no :latest, resource limits present, non-root, no privileged containers, a namespace on every namespaced object.
5. Secret scan. Run a scanner over the render, as argued above.
6. Render diff. Compare the render to the base branch's render and post it on the pull request. This is the highest-value step, because it turns "approve two lines" into "approve this exact change to forty objects".

name: render
on: [pull_request]
jobs:
  render:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
      with: {fetch-depth: 0}
    - name: Install pinned kustomize
      run: |
        curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh" \
          | bash -s -- 5.8.2 /usr/local/bin
        kustomize version
    - name: Render leaves (no plugins, no Helm on untrusted input)
      run: |
        mkdir -p out
        for d in $(find clusters -name kustomization.yaml -exec dirname {} \;); do
          kustomize build "$d" > "out/$(echo "$d" | tr / _).yaml" 2> "out/$(echo "$d" | tr / _).err"
          ! grep -q '^# Warning' "out/$(echo "$d" | tr / _).err"
        done
    - name: Validate
      run: kubeconform -strict -ignore-missing-schemas -summary out/*.yaml

Two governance points deserve emphasis because they are easily missed. First, the workflow above pipes the public install script into a shell. For a platform that cares about its supply chain, download the release tarball, verify it against checksums.txt, and bake the verified binary into your CI image instead. The release tags contain a slash, so the URL encodes it as kustomize%2Fv5.8.2. Second, the render-diff step needs its own caution: if it runs kustomize build on a pull request from a fork, it is building attacker-controlled files, and the rule from the trust section applies in full. No Helm, no plugins, no credentials.

Code ownership completes the picture. Protect bases/ and components/ with required reviewers from the platform team, give each tenant or team ownership of its own leaves, and require that a change touching a base also shows its rendered effect on every leaf. That last rule is what stops the quiet blast radius of a base edit.

Rendered manifests as an artefact

A popular pattern is to commit the rendered output for each environment to a separate branch or directory, so that the deployer applies plain YAML and the pull request diff shows the actual change. It removes Kustomize and Helm from the cluster-side trust path entirely, since the deployer never runs them, and it makes the history of what was deployed a literal git log. The cost is a generated tree that must never be hand-edited, a pipeline that must be trusted to produce it, and more files in the repository. For regulated environments that want an auditable record of exactly what was applied, it is often worth it.

Incident playbooks

Incidents involving Kustomize fall into a small number of shapes. Each has a recognisable symptom and a first check that is cheap. The order below is the one I would follow with a pager going off.

A few specific incidents appear often enough to be worth carrying in your head.

A rolling restart nobody asked for. A generated ConfigMap's name carries a hash of its contents, so any change to its content changes the name, which changes the Deployment's pod template, which starts a rollout. That is the feature working. A surprise rollout usually means something changed the content: a generator's literals edited in an overlay, a file touched by a formatter, or a hash input that differs between Kustomize versions. Diff the render, find the changed ConfigMap, and trace its source.

The apply fails with field is immutable. Someone added a label through commonLabels or includeSelectors: true, which rewrote a selector. Remove the selector-changing label, or plan a recreate of the affected workload. Do not ever "fix" it by force-replacing a production Deployment during an incident without a plan for traffic.

Objects appear to be missing after a migration. A deployer prunes objects that vanished from the render. If a rename changed the name of an object (a prefix added, a generator's behaviour changed), the old object is deleted and a new one created. In a stateful case that can mean losing a volume claim. Read the rendered diff for removed and added pairs that differ only in name.

The deployer cannot render, suddenly. Usually an upstream moved: a remote base branch changed, a git host timed out (hit 27s timeout), a Helm repository was unreachable. Pinning a full SHA and vendoring chart output turns each of these from an incident into a non-event.

A patch silently stopped applying to a CRD list. A schema or merge key changed, or an openapi file was dropped. Compare the render of the list field before and after.

# Provenance for debugging: where did each object come from, and what touched it?
buildMetadata:
- originAnnotations
- transformerAnnotations
- managedByLabel

Where Kustomize stops

Choosing a tool is part of owning a platform, and Kustomize has real limits. Some of them are design decisions the project documents as deliberate non-goals: it will not remove fields (you fork the base or use a patch), it will not do unstructured text edits or ${VAR} templating, it will not read the environment or command-line arguments during a build, and it does not support globs in kustomization files.

Those choices define where it fits.

Two boundaries are subtler. The first is parameterisation. When the number of variables per environment grows, overlays become a long list of patches that say "set this field to this value", which is a values file written in the most laborious possible dialect. If most of your overlay is field-by-field overrides of the same handful of settings, a chart with a values file would be shorter and clearer. The second is composition depth. Overlays compose by inheritance, and inheritance stops being readable at a few levels. When teams start asking for conditional inclusion, per-tenant feature matrices and computed names, the pressure is towards a programming language, and the honest answer is to use one, rather than to keep extending YAML.

They also combine. A common production shape is Helm for third-party applications and their defaults, Kustomize for the last mile of environment-specific patches, and a GitOps controller applying the result. Nothing in that stack is wrong, and the skill is knowing which layer a change belongs in.

And for what it does not do at all, see Kubernetes for the runtime it feeds, and Trivy for scanning the output.

Running Kustomize as a platform

Owning Kustomize for a team is mostly about making the safe path the easy path. Nearly everything above reduces to a small number of platform assets, which you build once and every team inherits.

A paved road of bases and components. Publish a small set of well-tested bases for your common workload shapes (a stateless web service, a worker, a cron job) and components for common features (an HPA, a PodDisruptionBudget, strict network policy, a metrics sidecar). Teams then write an overlay that selects from the menu, instead of copying YAML and drifting. Version them: a base referenced by a tag or a path in a monorepo that the platform team controls.

A pinned toolchain. Publish one container image containing the kustomize version, the validators, the secret scanner and the policy engine, and use it in every pipeline and in local development through a wrapper script. A pipeline that installs tools by piping a script to a shell cannot be trusted to be identical next week.

A template repository and a generator. A new service should appear with a working layout, CI and code owners in one command. kustomize create --autodetect --recursive is for bootstrapping a kustomization from existing YAML, and is useful for migrations, but a platform should offer a scaffold that encodes your conventions.

A compatibility contract. Decide and publish what a base promises: which labels are stable, which names are stable, what a component may change. A base that changes a label renames a selector, and that is a breaking change for every overlay and every live workload built on it. Treat base changes like library changes, with versioning, a changelog, and deprecation periods.

Defaults that fail closed. The platform's pipelines should refuse plugins, Helm inflation and the load restrictor override unless a repository has been explicitly allowed. Tenants who need an exception ask for one, and the exception is recorded.

Metrics. Track a short list: the time to build the slowest leaf, the number of leaves on deprecated fields, the count of unpinned remote bases, the number of distinct kustomize versions in use, and the fraction of leaves passing the full gate. These are the numbers that show an estate decaying before it fails.

The adoption path for a team that is new

For a team that is starting from plain manifests or from copy-pasted YAML, an approach that has worked is to go in stages. First, get every environment building with one pinned version and no warnings. Second, introduce the render gate and the render-diff comment so the team sees rendered changes. Third, consolidate duplicated YAML into a base. Fourth, express environment differences as small overlays. Only then introduce components, and only for features that exist in more than one place. Skipping to components first produces a clever structure that nobody understands.

Cost

The tool costs nothing to run: it is Apache-2.0, part of a Kubernetes SIG, and has no runtime footprint in the cluster. The costs are CI minutes for rendering, network time for remote bases and Helm, and, above all, engineering time spent on overlay sprawl and on incidents from skew. A budget for the platform is a budget for the gate and the paved road, not for Kustomize itself.

The review checklist

Use this for pull requests that touch Kustomize and for design documents that propose a new layout.

For a pull request

• Does every affected leaf still build, with the deployer's version, and with no deprecation warnings?
• Is there a rendered diff, and have I read it rather than the source diff?
• If a base or component changed, which leaves change, and do the owners of those leaves know?
• Do new labels use labels with includeSelectors: false, not commonLabels?
• Does any patch target a custom resource list, and is the openapi schema or a test in place so it merges rather than replaces?
• Does any generated name flow into a custom resource, and is there a nameReference configuration for it?
• Are there new remote bases, and is each pinned to a full commit SHA?
• Are there new --enable-helm, plugin or --load-restrictor uses, and were they approved for this repository?
• Does any secretGenerator read a file from git, and could the value be a reference instead?
• Does every namespaced object in the render have a namespace, including Helm output?
• Does a rename or prefix change remove and recreate any stateful object?

For a design document

• Which tool owns install, upgrade, rollback and pruning, and is it Kustomize? If so, what is the rollback procedure?
• How many inheritance levels does the layout have, and where do optional features live?
• Which versions of kustomize run in laptops, CI, kubectl and the deployer, and how is skew prevented?
• What is the trust boundary for builds: who can change a kustomization that runs with credentials?
• How are secrets handled, and is any plaintext reachable from the render?
• How are tenants isolated beyond namespace:, and what enforces it at the cluster?
• What is the upgrade procedure, and what is the render-diff evidence it requires?
• What is the exit plan if the estate outgrows overlays?

The complete picture

Step back and the whole track fits on one page. Kustomize is a stateless function that turns plain manifests and declared edits into YAML. At Beginner level you learned the nouns: base, overlay, patch, generator. At Mid level you learned the mechanics: how it accumulates, how patches and replacements behave, how it composes with Helm and with the deployers. At Senior level the subject became the estate around the function.

The argument of this part can be put in a few sentences. The build is pure, so you can test it completely; take advantage by rendering everything, always, with the deployer's version. The tool has no state, so rollback, pruning and ordering live elsewhere; know where. The tool runs code on files many people edit, so untrusted input gets no Helm, no plugins and no credentials. Custom resources and renames fail silently, so assert on the render. Versions skew by default, so pin them and upgrade by diffing. And a platform is a paved road plus a contract, not a repository of clever YAML.

Where the series leaves you

You started by learning that Kustomize changes YAML without templates. You finish able to run it as the configuration layer of a platform: to design a layout that stays readable, to keep builds trustworthy when inputs are not, to upgrade without surprises, and to tell your leadership, with evidence, when a piece of the estate belongs in another tool.

What to do next depends on what your platform is missing. If you deploy with GitOps, read the Argo CD and Flux guides for the pruning, health and identity features that Kustomize deliberately lacks. If you distribute applications, read the Helm guide and decide which layer owns parameters. If you run the clusters themselves, continue with Kubernetes and Terraform for what sits under and around all of this, and Vault for where secrets should come from. To check the output rather than trust it, pair the gate with Trivy.

Then test yourself with the Senior interview and tips files. They take the same material and ask it the way a hiring panel and an incident channel do.

Sources

• Kustomize reference index
• The kustomization file reference
• Kustomize glossary
• Built-in plugins reference
• kustomize build
• kustomize edit
• kustomize localize
• Patches
• Labels
• commonLabels (deprecated)
• Components
• configMapGenerator
• secretGenerator
• Replacements
• Vars (deprecated)
• helmCharts
• sortOptions
• buildMetadata
• openapi
• Extending Kustomize
• Containerized KRM functions
• Exec KRM functions
• Eschewed features
• Versioning policy
• Installing Kustomize
• Kustomize releases
• Release v5.8.2
• Issue 6241: Helm releaseName flag injection
• Issue 6058: Helm namespace regression
• Kubernetes: declarative management with Kustomize
• Argo CD: Kustomize
• Flux: Kustomizations

Part three of three. The senior interview does not ask what a patch is. It hands you a platform problem, a failing pipeline or a budget conversation, and watches how you reason. This file works through the three kinds of question a senior Kustomize candidate meets: system-design prompts, incident-response questions, and arguing a trade-off to leadership. About fifty minutes. Read the framing first, then answer each prompt aloud before reading the model answer.

How senior answers are judged

A mid-level answer names the right field. A senior answer names the right field, says what breaks if you are wrong, says how you would notice, and says what you would put in place so nobody else has to rediscover it. Interviewers listen for four habits.

You start from the properties of the tool. Kustomize is a stateless, client-side, pure function from files to YAML. It holds no release state, it does not talk to a cluster, and it refuses templating and build-time environment input on purpose. Most good answers are a consequence of one of those facts. "Rollback is a git revert and a re-deploy, because there is no release object" is a senior sentence. "You can just roll back" is not.

You separate the renderer from the deployer. Pruning, ordering, health, drift and rollback belong to kubectl, Argo CD or Flux. A candidate who credits Kustomize with things the deployer does has told the panel they have not operated it.

You name the trust boundary. A build reads files that many people edit and often runs with credentials in reach. Helm inflation, plugins and the load restrictor override are the places that boundary is widened. Say so unprompted.

You ask for evidence before you change anything. The natural evidence for this tool is a rendered diff, so a strong answer keeps reaching for "render old, render new, compare".

System-design prompts

Prompt 1: a layout for forty services across four clusters

"We have forty services, two environments and four clusters in two regions. Design the Kustomize repository."

Start by stating the axes of variation, because the layout is the answer to "what varies, and how often". Services differ from each other. Environments differ in scale and policy. Clusters differ in endpoints, storage classes and residency rules. Those are three axes, and a naive layout crosses them into a matrix of forty times eight leaves that all need editing when a base changes.

The design I would give is a per-service environment-neutral base, a small set of shared components for opt-in behaviour, environment overlays that carry the settings common to an environment, and thin cluster leaves that select a base, an environment, some components and a few cluster-specific patches. Three levels of inheritance, no more. Optional behaviour such as an HPA, a PodDisruptionBudget or strict network policy lives in components so a service opts in at the top and does not inherit it permanently from an ancestor. Each leaf is a directory with its own kustomization.yaml, buildable on its own.

Then the part that makes it a platform design and not a diagram. Every leaf is rendered in CI with one pinned version, the render is validated and policy-checked, and the rendered diff is posted on every pull request, so a base change shows the forty leaves it alters. Code owners give the platform team the bases and components, and each service team its own leaves. Residency differences, for example a regulated workload that must run in an in-country region, sit in the named cluster leaf, so they are auditable.

What I would flag as risks: component semantics (they act on everything the including kustomization has accumulated, so every component patch needs an explicit target), the alpha v1alpha1 API of Component, and the load restrictor, because the temptation in a matrix layout is to share files across directories and the answer is to make the shared directory a kustomization instead.

A strong answer also says when it would stop: if most leaves are just field-by-field overrides of the same dozen values, a Helm chart with values files is a better fit for that service.

Prompt 2: adopting Kustomize in a company that uses Helm everywhere

"Every team ships a Helm chart. Leadership wants Kustomize for environment configuration. How do you introduce it without a rewrite?"

The senior move is to refuse the framing that one replaces the other. Helm parameterises; Kustomize patches. Charts stay, and Kustomize takes the last mile.

There are three integration options, and naming them with their costs is the substance of the answer. First, helmCharts inflation: Kustomize runs helm template and patches the output. It needs --enable-helm, it widens the trust boundary because it executes the helm binary with arguments from the kustomization, and its official scope is narrow: it supports flags analogous to helm template, but no private repository authentication and no post-renderers. Second, render once and commit: run helm template in a script, put the output in a base, overlay that. This builds with no Helm and no network, and a chart upgrade becomes a reviewable YAML diff. Third, Helm with Kustomize as a post-renderer, which keeps Helm's release history and rollback and lets you patch a chart you do not own.

I would recommend the second for slow-moving third-party charts and the third where Helm's release management is already valued, and keep the first for places that genuinely need live inflation, since it is the one with the Helm flag-injection history. The issue fixed in v5.8.2 let a releaseName beginning with - inject a Helm flag such as --post-renderer, so anything rendering inflation on input that strangers can change must be on 5.8.2 or later.

Rollout plan: pick one service, reproduce its current manifests exactly (render the old and the new and prove the diff is empty), then move environments one at a time. Do not change charts and overlays in the same pull request.

Prompt 3: multi-tenant platform on one cluster

"Twenty teams deploy to one shared cluster. Each team writes its own Kustomize overlay. How do you stop one team breaking or attacking another?"

The first thing to say is that Kustomize does not enforce tenancy, and never will: it shapes YAML, and a tenant who controls a kustomization controls what that YAML contains, including patches that strip a base's security settings or resources: entries that pull in other directories. So the answer is layered, with Kustomize in the least-trusted position.

Git: code owners restrict who can change bases and components, and each team owns only its own leaf. Deployer: in Flux, spec.targetNamespace and spec.serviceAccountName make the controller apply each tenant's manifests as that tenant's service account, so RBAC bounds the tenant even if the manifests overreach; in Argo CD, AppProject limits destinations and resource kinds. Cluster: admission policy, for example Kyverno, enforces the properties that matter whatever was rendered (non-root, no privileged, resource limits, allowed registries).

On the Kustomize side, tenant overlays set namespace, and use namePrefix on cluster-scoped objects, because namespace: does not apply to a ClusterRole or a CRD and two tenants rendering the same name would collide. Better still, tenants do not ship cluster-scoped objects at all. Builds of tenant repositories run without plugins, Helm or credentials.

A candidate who says "use namespaces and you are done" has answered a different question.

Prompt 4: a hermetic, auditable supply chain

"A regulator wants to know exactly what was deployed and that no build pulled anything from the internet at deploy time. What do you build?"

Two goals, and Kustomize supports both if you set it up for them. For "exactly what was deployed": commit the rendered manifests for each environment as an artefact, produced by a trusted pipeline, so that the deployer applies plain YAML and the history of deployments is a git log. For "nothing from the internet": no remote bases, or vendor them. kustomize localize is the alpha tool that copies a kustomization and downloads its remote references to local paths, verifying by default that both trees build to identical output. Its limits are real: it does not yet localize Helm or KRM plugin fields, and it reorders fields alphabetically. Helm charts get vendored by committing them under chartHome or rendering them to a base.

Then pin everything else. The kustomize binary is downloaded from the release page with the checksum from checksums.txt verified, baked into an image, and referenced by digest. Remote references that remain are pinned to a full commit SHA, since tags can move and short SHAs are not supported. The pipeline that renders runs with no network and no cluster credentials, which the stateless design permits. Flux's kustomize-controller refuses remote bases altogether, which fits this model.

Close with the gap: signing the rendered artefact and verifying it at admission is a separate control that Kustomize does not provide, and you would add it.

Incident-response questions

Interviewers use incidents to see whether you work from evidence. The pattern they reward is: contain, compare, isolate, then fix and prevent.

Incident 1: production rolled out at 03:00 and nobody merged anything

"Every pod in a service restarted overnight. No one deployed. What happened?"

A rolling restart with no human change has a short list of causes. The first to check is the deployer, which may have synced because something in git changed that nobody thought of as a deploy: a base edit that altered a leaf, a bumped remote base, a Helm chart version that floated because version was omitted and the chart was pulled again. The second is Kustomize's own mechanism: generated ConfigMaps and Secrets carry a content hash, so a changed input changes the name, which changes the pod template, which starts a rollout. That is the feature working, and the question becomes which input changed. Diff the rendered output of the last-known-good commit and the current one using the deployer's version. If the render is identical, Kustomize is innocent and the cause is in the cluster or in an operator. If it differs, find the changed ConfigMap and follow the hash.

The preventive answer: pin remote bases to full SHAs and Helm charts to explicit versions so nothing floats, and run the render-diff gate so a change to a base shows up in a pull request rather than at night.

Incident 2: an apply fails with field is immutable

"A Deployment failed to apply with spec.selector: field is immutable. The diff was 'just a label'."

The label was added through commonLabels, or labels with includeSelectors: true, which also rewrote the selector of every Deployment and Service in scope. A Deployment's selector is immutable after creation. The immediate options are to remove the selector-touching label so the apply succeeds, or to plan a controlled recreate of the workload, and on a production Deployment the second needs a traffic plan.

The deeper answer is about policy. labels defaults to includeSelectors: false, which adds labels without touching selectors, so every new label should use that form. kustomize edit add label writes the deprecated commonLabels by default, and needs --without-selector to write labels. The automatic migration kustomize edit fix converts commonLabels to labels with includeSelectors: true, which preserves the old behaviour, risk included, which is correct for live objects but means the migration does not make a label safe. Add a gate that fails the build if the rendered selector of an existing workload changes.

Incident 3: the deploy succeeded and a whole class of objects vanished

"After a refactor, a set of objects disappeared from production and recreated empty."

Kustomize does not delete anything; the deployer pruned. A refactor changed names, for example by adding a namePrefix in an intermediate overlay, or by moving a resource between kustomizations so that a namespace: now applied. To the deployer, the old name disappeared from the render and a new one appeared, which is a delete and a create. For a stateful object that is data loss.

First, stop further syncing if pruning is automatic, and recover from backups. Then the analysis: a rendered diff between the two commits would have shown removed and added objects differing only in name, which is the signature. The prevention is to make that signature a failing check, to treat renames of stateful objects as migrations with a plan, and to consider disabling automated pruning, or protecting objects with the deployer's prune-protection mechanism, for stateful resources.

Incident 4: a pipeline on a pull request was turned against the company

"A security researcher reports that opening a pull request on your configuration repository ran code on your CI runner. The pipeline only renders YAML."

The pipeline did not only render YAML if it passed --enable-helm or --enable-alpha-plugins. With Helm, a helmCharts releaseName beginning with - was passed to Helm as a flag in releases up to v5.8.1, so --post-renderer=... ran a program of the attacker's choosing. With plugins, a KRM function or exec plugin is arbitrary code with the runner's privileges, and Kustomize provides no sandbox. The help text for --enable-exec itself says not to use it for untrusted configs.

Response: stop the affected pipeline, rotate any credentials the runner held, check the runner's logs for what executed, and upgrade to kustomize 5.8.2 or later everywhere, including any tool that embeds an older copy (kubectl 1.36 and 1.37 embed 5.8.1). Structural fix: pipelines that run on untrusted changes get no Helm, no plugins, no credentials, and no network beyond what the job needs. The render job needs no secrets; if yours has them, that is the finding.

Incident 5: "It builds on my laptop"

"A change passes locally and in CI, but the GitOps controller shows a render error."

Version skew. The laptop has one binary, CI another, and Argo CD or Flux embeds a third. A feature present in the newer version, such as sourceValue in replacements (5.7.0), regex selectors (5.8.0), Helm 4 support (5.8.1) or the Helm release-name validation (5.8.2), fails or behaves differently on an older one. The diagnostic is to ask which kustomize version the deployer runs and render with that exact version. The fix is process: one pinned version used in CI and in the deployer, registered in Argo CD under kustomize.path.<version> if multiple must coexist, and a CI job that renders with the deployer's binary.

Arguing a trade-off to leadership

Leadership questions test whether you can translate a technical position into risk, cost and time without losing accuracy. The format is usually: here is a proposal, argue for or against, and say what you would do.

"Why not just use Helm for everything?"

Helm and Kustomize are not competitors. Helm is a package format with a release lifecycle: it suits distributing an application to other people with a values file, and it gives install, upgrade and rollback history. Kustomize has no release object, no templating and no parameters, so it suits configuration of software you own or patching software you do not. The honest position is that you would use Helm where you distribute or consume packaged software and want release management, and Kustomize where you want plain manifests as the source of truth with reviewable environment differences. Often both: a chart, then Kustomize for the last-mile patches, then a GitOps controller. The cost of a single-tool mandate is a team writing templates in a patch language or patches in a template language.

"Can we skip the render-diff gate? It slows pull requests."

Quantify what it buys. A base change with a two-line source diff may alter dozens of rendered objects, and without the gate the reviewer approves what they can see. The gate's cost is a few seconds of CI and a comment to read. Its benefit is that every production change is reviewed at the level it takes effect. Compare with the cost of one incident like a pruned stateful object or an immutable selector failure during a release. Offer to make it fast: render only the leaves whose inputs changed, and cache the pinned binary.

"Why are we spending a sprint on deprecated fields? Nothing is broken."

The honest position is that nothing is broken and nothing is scheduled to break: there is no v1 Kustomization API yet, deprecated fields stay in v1beta1, and removal would require a major version with at least two releases of warning. So this is not an emergency. The reasons are smaller but real. Deprecated fields are excluded from a future v1. The warnings print on every build, which teaches engineers to ignore stderr, and that habit hides real warnings. And commonLabels specifically is a live hazard because it rewrites immutable selectors. The proposal is to migrate directory by directory with kustomize edit fix, proving each render is unchanged, spread across normal work rather than a sprint, and to add a gate so the count of deprecated uses can only fall.

"Should we upgrade to the newest release the day it ships?"

Not automatically, and not never. Version 5.8.2 is a security release for anyone who renders Helm inflation on inputs others control, so that case is a priority. For everyone else the procedure matters more than the date: render every leaf with old and new, diff, and proceed when the diff is empty or understood. Some releases carry warnings: 5.8.0 carried a regression note and 5.8.1 fixed it, so the day-one release of a minor version deserves a short soak. Also be specific about where the new version needs to land: the deployer's binary, the CI image and the laptops, not only one of them, and remember that the kubectl-embedded copy lags.

"What is the exit strategy if this does not scale?"

A good answer has signals and options. The signals: most overlays are field overrides of the same settings, inheritance deeper than three or four levels, components growing conditional logic, build times climbing, or teams maintaining computed names by convention. The options: move parameterised applications to Helm charts; move logic-heavy generation to a configuration language or a program that emits manifests; or push behaviour into an operator. Because Kustomize works on plain manifests, the exit is cheap: the rendered output is already valid Kubernetes YAML and can become the input to whatever replaces it.

Follow-up questions and model answers

These are the short, sharp questions that come between the big prompts. Answer each in two or three sentences.

How do you know a Kustomize upgrade is safe? Render every leaf with the old and new versions and diff. An empty diff is the evidence. Anything else is a decision, and each difference needs an owner.

What does namespace: not do? It does not apply to cluster-scoped objects, and it can fail with an ID conflict if two same-named objects from different namespaces would collapse into one. Since 5.8.0 it is also passed to Helm as the release namespace, and Helm-inflated resources whose templates set no namespace do not receive it on 5.8.1 and 5.8.2, which is issue #6058.

Why does a generator's behavior: merge fail with "does not exist"? Generators run before the namespace transformer, so the generator must name the base ConfigMap's original name and namespace, not the ones the overlay is about to apply.

What is wrong with secretGenerator from a file in git? The output is base64, not encrypted, so the file is a plaintext secret in git. Use encrypted files decrypted by the deployer, Sealed Secrets, or references resolved at runtime by an operator.

Why do patches on a custom resource lose list entries? Kustomize merges lists using keys from an OpenAPI schema based on Kubernetes 1.21.2, and has none for your CRD, so list patches replace. Supply an openapi schema.

Why does a hashed ConfigMap name not reach my custom resource? The built-in name-reference rules cover core kinds. For your kind, add a nameReference entry via configurations:.

What does kubectl apply -k not support that kustomize build does? Build flags. It cannot enable Helm or plugins and it uses the embedded, older kustomize.

How would you keep a remote base safe? Pin a full commit SHA, review upstream changes when bumping, or vendor it with kustomize localize, and keep the list discoverable.

What is the difference between Flux's Kustomization and a kustomization.yaml? The Flux one is a custom resource that tells a controller what to build and apply and with what identity. The file describes how to transform manifests. Flux's postBuild.substitute is Flux-side substitution and not a Kustomize feature.

What does a strong answer cover that a weak one misses? It names the deployer, the trust boundary and the evidence, and it states a failure mode. Weak answers recite fields.

Sixty-second self-test

Answer each aloud in one sentence. If you cannot, reread the matching section of the Senior detailed guide.

1. What three properties follow from Kustomize being a pure, stateless function, and what three things does it take away?
2. In what order do resources, generators, transformers and validators run, and why does behavior: merge depend on it?
3. What are the four ways to handle secrets around Kustomize, and why is secretGenerator alone not one of them?
4. What did v5.8.2 fix, who was exposed, and why are kubectl 1.36 and 1.37 still exposed?
5. Why does a list patch on a custom resource lose entries, and what are the two fixes?
6. Why is commonLabels dangerous on live objects, and what does kustomize edit fix do with it?
7. Which binary renders production, and how do you prove it matches CI?
8. What three layers enforce tenancy, and which one is Kustomize?
9. What makes a build hermetic, and what does kustomize localize not cover?
10. When would you stop using Kustomize for a service?

If you answered nine or ten without hesitation, you can carry a senior interview on this tool. If you stumbled on the third, fourth or eighth, those are the topics panels tend to probe, because they are where real estates have real incidents.

Part three of three. At this level the problems are not failed builds. They are the things that appear only when dozens of people edit hundreds of overlays: a base change that silently alters forty workloads, a binary that differs between CI and the cluster, a pipeline that runs a stranger's pull request with credentials in reach. Start with the error table, then the practices, the platform habits, and the review checklists.

Production traps that only appear at scale

Everything from Beginner and Mid still applies. These are the causes of incidents and of slow, expensive confusion, not of red build output.

Three of these deserve a closer look because they recur in almost every estate.

The silent ones are the expensive ones. Loud failures are cheap: the build stops, someone fixes it. The costly incidents produce no error at all. A patch that replaced a list on a CRD, a namespace missing from Helm output, a generated name that did not reach a custom resource, and a selector that changed quietly on a new label all build successfully. The defence is to assert on the render, because a check that only confirms "it built" cannot see any of them.

Version skew is the default, not an accident. There are at least four places kustomize runs, and they update on different schedules. As of this writing the current release is v5.8.2, kubectl 1.34 and 1.35 embed v5.7.1, kubectl 1.36 and 1.37 embed v5.8.1, and no kubectl release embeds 5.8.2 yet. If you do not actively converge them, they diverge.

Review scales worse than compute. You will run out of reviewer attention long before you run out of CI minutes. Every practice in this file that helps a reviewer, such as rendered diffs, small leaves, explicit targets and thin overlays, is worth more than a performance tweak.

The practices that pay off most

The order matters. If you can do only three things this quarter, do the first three: a rendered-diff gate, one pinned version, and a hard rule that untrusted builds run with nothing. Those three remove the largest share of incidents and the largest share of security exposure.

Practice cards

1. Make a base change visibleEdit a label in a shared base and count the lines in the source diff. Then render every leaf before and after and count the changed objects. The ratio is the argument for a render-diff gate.
2. Reproduce version skewInstall two kustomize versions. Render one overlay that uses a newer feature with both and read the failure. Write down where each version lives in your estate.
3. Break a selector on purposeIn a scratch cluster, apply a Deployment, add a label through commonLabels, and apply again. Read field is immutable in full. Redo it with includeSelectors: false.
4. Lose a list, then keep itPatch one entry of a list in a custom resource and watch the others disappear. Add an openapi schema with the right merge key and compare.
5. Follow a hashChange one literal in a configMapGenerator. Diff the render and follow the new name into the pod template. Explain in one sentence why a rollout begins.
6. Build a hostile pull requestIn a disposable repository, add a helmCharts entry whose releaseName starts with a dash. On a version before 5.8.2 observe the behaviour; on 5.8.2 read the rejection. Never run this on a machine with real credentials.
7. Vendor and compareTake an overlay with a remote base, run kustomize localize, and diff the two builds. Note which fields the tool does not cover.
8. Migrate and proveRun kustomize edit fix on one directory with commonLabels. Diff the render before and after. It should be empty.
9. Write the tenant boundaryAdd a patch that removes a base's securityContext. Then write the admission rule that refuses the result and run it against the render.
10. Time your slowest leafBuild your biggest overlay with and without --enable-helm and with a remote base replaced by a local copy. Decide which of the three is your actual problem.

Platform-ownership habits

The habits below are what separate a team that has Kustomize from a team that runs it as a platform.

Write down who owns each base and what it promises. A base is a library with consumers. A label, a container name, a port name and a resource name are all interfaces, and changing them breaks overlays that patch by name and workloads whose selectors matched. Publish a short contract: which names and labels are stable, what components may change, and how deprecations are announced.

Treat the toolchain as a product. One image with the pinned kustomize, the validator, the policy engine and the secret scanner. One wrapper script for developers that uses the same image. Upgrades to it go through the render-diff procedure like any other change, and its version appears in every pipeline log.

Default to closed. The platform's pipelines refuse Helm inflation, plugins and the load-restrictor override unless a repository is on an allowlist. Exceptions are recorded with a reason and an owner, and reviewed on a schedule. The cheapest vulnerability fix is the feature that was never enabled.

Keep a deprecation burn-down. Count leaves using each deprecated field and show the number falling. Migrate a directory at a time with kustomize edit fix, proving an unchanged render each time. Never combine a version bump, a migration and a manifest change in one pull request.

Measure the estate. A small set of numbers, tracked over time, warns you before things fail: build time of the slowest leaf, number of distinct kustomize versions in use, count of unpinned remote bases, count of deprecated-field uses, and the fraction of leaves passing the full gate.

Keep an exit plan alive. Kustomize's output is plain Kubernetes YAML, so leaving is cheap, but only if you notice the signals in time: most overlays being field-by-field overrides, inheritance getting deeper, components accumulating conditions. Review those signals every quarter and write down the decision.

Separate the renderer's runbook from the deployer's. When an incident starts, the first question is whether the render changed or the cluster did. Keep two short runbooks: one for render failures (nested errors, provenance, bisecting the overlay chain) and one for apply failures (immutability, pruning, health), so the responder does not start by working out which they have.

Review checklists

For a pull request touching Kustomize

• Every affected leaf builds with the deployer's kustomize version and prints no deprecation warnings.
• A rendered diff is attached, and the reviewer has read it, not only the source diff.
• If a base or a component changed, the affected leaves are listed and their owners are aware.
• New labels use labels with includeSelectors: false.
• No existing workload's rendered spec.selector changed, and no object's name changed where the object is stateful.
• Patches that target a custom resource list have an openapi schema or a render test.
• Any generated name referenced by a custom resource has a nameReference configuration.
• Every namespaced object in the render has a metadata.namespace, including Helm output.
• New remote bases are pinned to full commit SHAs; new charts have explicit versions.
• No new --enable-helm, --enable-alpha-plugins, --enable-exec or --load-restrictor LoadRestrictionsNone without a recorded exception.
• No secretGenerator fed from a plaintext file in git.
• The render passes schema validation, policy checks and the secret scan.

For a design document

• Who deploys the output, and who owns pruning, ordering, health and rollback? If the answer includes Kustomize, say how.
• What is the layout, how many inheritance levels does it have, and where do optional features live?
• Which kustomize binary runs where, and how is skew prevented and detected?
• Who can change a kustomization that runs with credentials, and what do those builds refuse to do?
• How are secrets supplied, and can plaintext reach the render, the repository or a pull request?
• How are tenants separated beyond namespace:, and what enforces it at the deployer and the cluster?
• How are custom resources handled: schema, name references, and tests?
• What is the upgrade procedure, and what evidence does it require?
• How are remote inputs pinned, inventoried and, where required, vendored?
• What are the exit signals, and who reviews them?

For an upgrade

• Render every leaf with old and new versions and review the diff.
• Check whether the release changes Helm behaviour (namespace propagation in 5.8.0, Helm 4 in 5.8.1, release-name validation in 5.8.2) and test every Helm-inflated leaf.
• Decide separately for each place kustomize runs, since the kubectl-embedded copy lags.
• Register the new version next to the old in Argo CD (kustomize.path.<version>) and move applications one at a time.
• Roll the change in three steps: binary, field migrations, manifest changes, with a render diff for each.

Operational recipes

A render diff for a pull request

#!/usr/bin/env bash
# Compare rendered output of every leaf between the base branch and HEAD.
set -euo pipefail
BASE=${1:-origin/main}
work=$(mktemp -d)
git worktree add -q "$work/base" "$BASE"
changed=0
while read -r d; do
  a=$(cd "$work/base" && kustomize build "$d" 2>/dev/null || echo "BUILD FAILED")
  b=$(kustomize build "$d" 2>/dev/null || echo "BUILD FAILED")
  if [ "$a" != "$b" ]; then
    changed=$((changed+1))
    echo "=== $d"
    diff <(echo "$a") <(echo "$b") | head -60 || true
  fi
done < <(find clusters -name kustomization.yaml -exec dirname {} \;)
git worktree remove --force "$work/base"
echo "$changed leaves changed"

Assert that the render is well-formed

#!/usr/bin/env bash
# Fail if any namespaced object lacks a namespace, or a workload selector changed.
set -euo pipefail
render=$(kustomize build "$1" --enable-helm=false)
missing=$(echo "$render" | yq -r 'select(.kind == "Deployment" or .kind == "StatefulSet" or .kind == "Service") | select(.metadata.namespace == null) | .kind + "/" + .metadata.name')
if [ -n "$missing" ]; then
  echo "objects with no namespace:"; echo "$missing"; exit 1
fi

Find every remote base and whether it is pinned

grep -rn --include=kustomization.yaml -E '(github\.com|gitlab\.com|git@|https://)' . \
  | grep -v -E 'ref=[0-9a-f]{40}' \
  || echo "all remote references are pinned to a full SHA"

Provenance for a debugging session

buildMetadata:
- originAnnotations
- transformerAnnotations
- managedByLabel

Render with these on a scratch copy only. They add annotations to every object, so do not commit them into rendered artefacts unless you want them in production.

Install a verified binary

VER=v5.8.2; OS=linux; ARCH=amd64
curl -LO "https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2F${VER}/kustomize_${VER}_${OS}_${ARCH}.tar.gz"
curl -LO "https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2F${VER}/checksums.txt"
sha256sum --ignore-missing -c checksums.txt
tar xzf "kustomize_${VER}_${OS}_${ARCH}.tar.gz" && sudo install kustomize /usr/local/bin/
kustomize version

Bake the result into a CI image rather than fetching on every run, and print kustomize version in the pipeline log so the version in use is always visible.

Run the checks inside a container

The project publishes registry.k8s.io/kustomize/kustomize. Check the tags first, since a brand-new release may not have an image yet.

docker run --rm -v "$PWD":/work -w /work registry.k8s.io/kustomize/kustomize:v5.8.1 build overlays/prod

Migrate one directory safely

git diff --quiet || { echo "commit or stash first"; exit 1; }
kustomize build overlays/prod > /tmp/before.yaml
( cd overlays/prod && kustomize edit fix )
kustomize build overlays/prod > /tmp/after.yaml
diff /tmp/before.yaml /tmp/after.yaml && echo "render unchanged, safe to commit"

A short checklist to keep

• One kustomize version everywhere, verified, and printed in every pipeline.
• A rendered diff on every pull request, read by a human.
• Untrusted builds run with no Helm, no plugins and no credentials.
• New labels never touch selectors.
• Custom resources have a schema, name-reference rules and a test.
• Remote inputs are pinned to full SHAs or vendored.
• Secrets are references or encrypted, and the render is scanned.
• Tenants are bounded at the deployer and at admission, not only in git.
• Upgrades are a render diff, done one change at a time.
• Someone owns each base and has written down what it promises.

If you take one sentence from this series, let it be this one: Kustomize is a function you can test completely, so test it completely, with the deployer's version, and review what it renders rather than what you typed.

Continue with the Argo CD and Flux guides for the deployers that carry what Kustomize deliberately does not, and the Helm guide for the other half of the configuration problem.