The Complete Helm Guide

This is part one of three. It covers everything you need to do real work with Helm, not a teaser. By the end you can install a packaged application into a Kubernetes cluster with one command, change its configuration without editing YAML by hand, upgrade it, roll it back when the upgrade goes wrong, and remove it cleanly. You can also build your own chart from nothing, preview what it will do before it touches a cluster, test it, package it, and push it to a registry. Mid-level and Senior take the same topics further; nothing here is thrown away.

The guide targets Helm 4.3, the current release at the time of writing. Helm 4 shipped in November 2025, the first new major version in six years. If you have seen older tutorials, most of what they teach still works, and this guide points out the few places where the new version behaves differently.

Each section ends with a Try it task. Do them as you go. They take a few minutes each, and Helm only makes sense once you have watched a release install, break, and recover on a cluster of your own. You need a terminal, a little comfort with YAML, and a rough idea of what a Kubernetes Deployment and Service are. If those two words are new, read the Kubernetes guide's beginner part first at /student-guides/kubernetes, then come back.

What Helm is, and the problem it solves

Helm is the package manager for Kubernetes. It takes a bundle of Kubernetes configuration files, fills in the values you choose, sends the result to your cluster, and remembers exactly what it sent so it can upgrade or undo it later.

The diagram is the whole idea, and the rest of the guide fills it in. To see why it exists, start with what deploying to Kubernetes looks like without it.

A real application is never one object. A small web service needs a Deployment to run the containers, a Service to give them a stable address, a ConfigMap for settings, probably an Ingress for outside traffic, a ServiceAccount, maybe a HorizontalPodAutoscaler. That is five to ten YAML files, and every one of them contains values that change between environments: the image tag, the number of replicas, the hostname, the memory limit. On your laptop you want one replica and a tiny memory limit. In production you want three replicas and a real limit.

Without a package manager, teams handled this in three painful ways. They copied the folder of YAML per environment and edited each copy, so the copies drifted apart and nobody knew which differences were deliberate. They ran sed or envsubst over the files before kubectl apply, which works until a value contains a slash or a quote. Or they kept a wiki page of steps that slowly went out of date. None of these answered the questions that matter during an incident: what exactly did we deploy last Tuesday, what did it replace, and how do we get back?

Helm solves that with three ideas, each of which gets its own section below.

Packaging. The files are bundled into a chart, a folder (or a single versioned archive) with a fixed layout. The chart is the unit you version, share, and install. Somebody else's chart for PostgreSQL, Prometheus, or an ingress controller installs with the same command as your own.

Templating. Inside the chart, the parts that change are replaced with placeholders, and a file of values fills them in. One chart serves every environment; only the values differ.

Release tracking. Every install is recorded as a release with numbered revisions. helm history shows the past, helm rollback returns to it, and helm uninstall removes everything the release created and nothing it didn't.

It is worth being honest about what Helm is not. It is not a deployment platform that watches your cluster. Helm acts only when you run it. If someone edits a live object by hand afterwards, Helm does not notice until the next upgrade. Continuous reconciliation is the job of GitOps tools such as Argo CD and Flux, which the Senior part of this series discusses. And it is not a container builder; building images is Docker's job, covered in /student-guides/docker. Helm picks up after that: the image exists in a registry, and now you need to run it on Kubernetes repeatably.

Helm 3, Helm 4, and which one you are learning

You will meet both version numbers in job descriptions, blog posts and Stack Overflow answers, so get the picture straight once.

Helm 2 (2016 to 2019) had a server-side component called Tiller that ran inside the cluster with broad permissions. It was a security problem and was removed. Helm 3 (November 2019) made Helm a pure command-line client. Helm 4 reached general availability on 12 November 2025, announced at KubeCon North America in Atlanta. The 4.x line releases a minor version roughly every four months and a patch about monthly; this guide was checked against 4.3.0, released on 9 September 2026.

Helm 3 has been closed to new features. Its last minor release, 3.22.0, arrived a day after 4.3.0, and Helm 3 receives security fixes only until 10 February 2027. After that date it gets no patches at all. So if you are learning Helm today, learn Helm 4. Employers will still have Helm 3 in places, and the commands you learn transfer almost entirely.

The old flag names still work in Helm 4; they print a deprecation warning and then do what you asked. That is why a Helm 3 tutorial rarely breaks outright. This guide always uses the current names, and a callout flags each place where older material differs. The mid-level guide covers the new wait strategies and server-side apply in depth; as a beginner you only need to know they exist.

The four nouns: chart, values, release, revision

Helm's vocabulary is small. Learn these four words precisely and every command reads as a sentence. Two more, repository and registry, appear in the next section.

A chart is a package. It is a folder of files (or that folder compressed into a .tgz archive) containing Kubernetes manifest templates, a Chart.yaml describing the package, and a values.yaml of default settings. A chart on its own does nothing, just as a recipe on its own does not feed anyone. It has a version, which follows semantic versioning (three numbers such as 1.4.0) and is required. It also has an informational appVersion, the version of the application inside, which can be any string.

Values are the inputs. The chart's own values.yaml holds defaults. You override some of them at install time, either in a file of your own (-f my-values.yaml) or one at a time with --set. Helm merges everything into one structure that the templates read as .Values. When two sources disagree, the later one wins, and --set beats -f. We work through the exact order in the values section.

A release is one installed copy of a chart. It has a name you choose and lives in a namespace. The name is unique inside its namespace, which means you can install the same chart twice as two separate releases, blog-staging and blog-prod, and they will not interfere. Release names must be valid DNS labels (lowercase letters, digits and hyphens) and at most 53 characters long. The chart is the recipe; the release is the dish on the table.

A revision is one numbered step in a release's life. Installing creates revision 1. Every upgrade creates the next number. Rolling back does not rewind the counter: it creates a new revision that copies the content of an older one. So if you install, upgrade twice, and roll back to revision 1, you are at revision 4, whose content matches revision 1. By default Helm keeps the last ten revisions per release.

Two more terms are worth adding now, because people use them casually and beginners get lost. The application (or app) is the software inside the chart, for example nginx. The cluster is the Kubernetes installation Helm talks to. A sentence like "I installed the nginx chart as a release called web in the staging namespace, and I'm on revision 3" now makes complete sense: the package, the name, the place, and the point in its history.

How Helm talks to your cluster

Helm is a client program and nothing else. There is no Helm server running in your cluster, no daemon, no controller. When you run helm install, the helm binary on your machine renders the templates, then calls the Kubernetes API server directly, exactly as kubectl does, and exits.

Three practical consequences follow, and they explain behaviour that otherwise looks mysterious.

Helm uses your credentials and your context. It reads the same kubeconfig file as kubectl (by default ~/.kube/config, or whatever the KUBECONFIG variable points at) and uses the current context, meaning the currently selected cluster and user. If kubectl talks to the wrong cluster, so does Helm. Always check kubectl config current-context before installing anything. Helm also has no more permission than you do; if your account cannot create Deployments in a namespace, helm install there fails with a normal Kubernetes permission error.

Helm stores its memory inside the cluster. After each install or upgrade, Helm writes a record of that revision into the release's namespace, by default as a Secret named like sh.helm.release.v1.web.v3 (release web, revision 3). The record contains the chart, the values you supplied, and the exact manifest that was sent, compressed and base64-encoded. That is how helm history and helm rollback work from any machine: the state is in the cluster, not on your laptop. You can see these Secrets with kubectl get secrets -n <namespace>, and it is worth doing once so they stop being magic.

Helm does nothing between commands. Nothing watches the release, nothing repairs it. A pod crashing is handled by Kubernetes; a hand edit to the Deployment stays until the next helm upgrade. This is simple and predictable, and it is also the reason teams add a GitOps controller on top for continuous correction.

Since the release record is a Secret that contains the values you passed in, anyone who can read Secrets in that namespace can read your values. Remember this when you are tempted to pass a database password as a value; the later sections and the tips file return to it.

Installing Helm and checking your setup

Helm is a single static binary, so installation is mostly about choosing the most convenient place to get it. The project publishes signed release archives at get.helm.sh, and most package managers carry it.

On macOS the quickest route is Homebrew:

brew install helm

On Windows use one of the three common package managers:

choco install kubernetes-helm
scoop install helm
winget install Helm.Helm

On Linux you can use the project's install script. Note the script name: the Helm 4 script is get-helm-4. The older get-helm-3 script installs Helm 3, which is a common way to end up on the old version by accident.

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-4
chmod 700 get_helm.sh
./get_helm.sh

The script picks the latest release. To pin a version, set DESIRED_VERSION=v4.3.0 before running it. If you prefer to do it by hand, download the archive and verify its checksum before unpacking, which is the careful way to install any binary:

curl -LO https://get.helm.sh/helm-v4.3.0-linux-amd64.tar.gz
curl -LO https://get.helm.sh/helm-v4.3.0-linux-amd64.tar.gz.sha256sum
sha256sum -c helm-v4.3.0-linux-amd64.tar.gz.sha256sum
tar -zxvf helm-v4.3.0-linux-amd64.tar.gz
sudo mv linux-amd64/helm /usr/local/bin/helm

Replace linux-amd64 with darwin-arm64, windows-amd64 and so on to match your machine. sha256sum -c should print OK; anything else means the download is corrupt or has been tampered with.

Other Linux routes exist: sudo dnf install helm on Fedora and sudo snap install helm --classic via Snap. Distribution packages can lag behind the official release, so check the version afterwards.

Now verify the installation. Three commands tell you everything:

helm version --short
helm env
kubectl config current-context

The first prints something like v4.3.0+g<commit>. The full helm version prints a longer structure with the Go version and the Kubernetes client library version it was built against, which matters because Helm 4 is supported against the four newest Kubernetes minor versions around the one it was compiled with (for 4.3 that is 1.34 through 1.37). Using Helm against a Kubernetes release newer than the one it was built for is not recommended; there is no compatibility guarantee.

helm env prints the paths and settings Helm will use. You do not need to memorise it, but read it once. Notice where your repository list lives (HELM_REPOSITORY_CONFIG, a file called repositories.yaml), where downloaded charts are cached (HELM_CACHE_HOME), and which storage driver is selected (HELM_DRIVER, normally secret).

The last check proves Helm can reach a cluster, which is the most common thing a first-time setup gets wrong:

helm list -A

-A means all namespaces. On a fresh cluster you get an empty table with only the column headers. That is a success: it proves your kubeconfig is valid, the API server is reachable, and your user may list Secrets across namespaces. If you see Error: kubernetes cluster unreachable, the problem is your kubeconfig or network, not Helm; fix it with kubectl first.

Finally, enable shell completion, which saves real typing because Helm has many subcommands and flags. For bash:

source <(helm completion bash)

Use zsh, fish or powershell in place of bash as appropriate, and add the line to your shell startup file to keep it.

A cluster to practise on

Helm needs a Kubernetes cluster to install into. For learning, use a disposable local one so a mistake costs you nothing. Pick whichever you already have or find easiest; all of them give you a kubeconfig context that Helm uses without any further setup.

• Docker Desktop includes a single-node Kubernetes cluster you switch on in its settings.
• minikube creates a local cluster with minikube start.
• kind ("Kubernetes in Docker") runs a cluster as containers; kind create cluster --name helm-lab creates one and selects the context kind-helm-lab.
• Any small cluster from a cloud provider works too, but watch the bill and remember to delete it.

Whichever you choose, finish with the same two checks:

kubectl config current-context
kubectl get nodes

The first prints your context name; the second lists at least one node with status Ready. Then create a namespace for practising so your experiments stay in one place and are easy to wipe:

kubectl create namespace helm-lab

From here on, the guide installs things into namespaces we name explicitly. That is a habit worth having from day one. If you omit -n, Helm uses the default namespace, where things pile up and get forgotten.

Finding charts: repositories, Artifact Hub, and OCI registries

Before you can install someone else's chart you have to find it, and before Helm can fetch it, it needs to know where to look. Charts are distributed in two ways, and you will meet both.

A chart repository is an ordinary web server that hosts an index.yaml file, a catalogue of every chart and version it offers, plus the chart archives themselves. You tell Helm about a repository once with helm repo add, giving it a short local nickname. Helm downloads the catalogue and remembers it in your repositories.yaml.

An OCI registry is the same kind of registry that stores container images, used here to store charts. There is no helm repo add step; you refer to a chart by its full address, which starts with oci://. This is the direction the ecosystem is moving, and many projects now publish charts only this way. It is also how your own team will most likely publish charts, using the container registry it already has.

Artifact Hub (artifacthub.io) is the search engine for both. It is a CNCF-hosted catalogue where you type "postgres" or "prometheus" and get the charts, their documentation, their versions and, importantly, the exact helm repo add command or oci:// address to use. You can also search it from the terminal with helm search hub.

Start with the classic repository route. The Helm project itself publishes a tiny example repository, which is ideal for learning because the chart is small enough to read:

helm repo add examples https://helm.github.io/examples
helm repo update
helm search repo hello

The first command adds the repository under the nickname examples. The second refreshes the catalogues of every repository you have added; run it before installing anything, because Helm works from the cached catalogue, not the live server. The third searches your local catalogues. You should see something like:

NAME                    CHART VERSION   APP VERSION   DESCRIPTION
examples/hello-world    0.1.0           1.16.0        A Helm chart for Kubernetes

Read each column. The chart is named hello-world in the repository examples, so its full reference is examples/hello-world. The chart version (0.1.0) is the version of the packaging. The app version (1.16.0) is the version of the software inside, here a label for the nginx image tag. The two are independent: a chart can be repackaged many times around the same application version.

Other search and listing commands you will use constantly:

helm repo list                       # the repositories you have added
helm search repo nginx --versions    # every version of every chart matching "nginx"
helm search hub wordpress            # search Artifact Hub (needs internet access)
helm repo remove examples            # forget a repository

helm search repo only shows stable versions by default. Add --devel to include pre-releases such as 1.0.0-rc.1.

For an OCI chart you skip adding anything and use the address directly. Charts in an OCI registry are commonly pulled like this:

helm show chart oci://registry-1.docker.io/bitnamicharts/nginx

There is no catalogue to search in that case; you know the address, or you found it on Artifact Hub or in the project's documentation.

Region matters a little here. If your employer has data-residency rules, which is common for banks, telecoms and government work in the Gulf and Egypt, check where your chart registry and the images it points to are hosted before you rely on them. Public registries serve from wherever their provider decides; many teams mirror charts and images into a registry inside their own cloud region for exactly this reason. Section "Packaging and sharing" shows how you would push your own chart to one.

Your first install

Never install a chart blind. Look at what it is and what it lets you configure. Two show commands do that:

helm show chart examples/hello-world
helm show values examples/hello-world

show chart prints the Chart.yaml (name, versions, description). show values prints the default values.yaml, which is the complete list of knobs the chart author chose to give you. For hello-world it begins like this:

replicaCount: 1

image:
  repository: nginx
  pullPolicy: IfNotPresent
  tag: ""

service:
  type: ClusterIP
  port: 80

Read it as a menu. You can change replicaCount, pick a different image, or change the service type and port. Anything not on the menu you cannot change without editing the chart. There are also helm show readme and helm show all, which print the chart's documentation and everything together.

Now install it:

helm install web examples/hello-world -n helm-lab --create-namespace

Take the command apart, because every install you ever run has this shape: helm install <release-name> <chart> [flags]. Here web is the name you are giving this copy, examples/hello-world is the chart to install, -n helm-lab says which namespace to put it in, and --create-namespace creates that namespace if it does not exist yet.

Helm prints a summary:

NAME: web
LAST DEPLOYED: Thu Oct  1 10:15:32 2026
NAMESPACE: helm-lab
STATUS: deployed
REVISION: 1
DESCRIPTION: Install complete
NOTES:
1. Get the application URL by running these commands:
  export POD_NAME=$(kubectl get pods --namespace helm-lab -l "app.kubernetes.io/name=hello-world,app.kubernetes.io/instance=web" -o jsonpath="{.items[0].metadata.name}")
  ...

Read it line by line. NAME and NAMESPACE are the release's identity. STATUS: deployed means Helm successfully sent everything to the cluster. REVISION: 1 is the first step of its history. NOTES is text the chart author wrote, rendered with your values, telling you what to do next; it comes from the chart's templates/NOTES.txt. Always read it, since it usually contains the exact command to reach the application.

There is one important subtlety about STATUS: deployed. By default in Helm 4, helm install returns as soon as the objects are accepted by Kubernetes. It does not wait for your pods to be running. "Deployed" means "sent", not "healthy". If the image name was wrong, you would still see deployed, and then kubectl get pods would show ImagePullBackOff. To make Helm wait until things are actually ready, add the wait flag:

helm install web examples/hello-world -n helm-lab --wait

Plain --wait uses the watcher strategy, which watches the objects until Kubernetes reports them ready, or fails after the timeout (five minutes by default, changeable with --timeout 2m). The mid-level guide explains the other strategies. As a beginner, adding --wait to installs and upgrades is a good habit, because it turns a silent failure into a visible one.

Confirm the pods from Kubernetes' side. Helm only created them; Kubernetes runs them:

kubectl get pods -n helm-lab
kubectl get all -n helm-lab

You should see one pod whose name starts with web-hello-world- and whose status becomes Running, plus a Deployment and a Service of the same base name. The exact names come from the chart's helper templates, so another chart will name things differently.

Looking at a release: list, status and get

Once a release exists, a family of read-only commands tells you everything about it. Learn these before any command that changes things, because they are how you find out what state you are actually in.

helm list -n helm-lab

prints one row per release in that namespace:

NAME  NAMESPACE  REVISION  UPDATED                               STATUS    CHART              APP VERSION
web   helm-lab   1         2026-10-01 10:15:32 +0000 UTC         deployed  hello-world-0.1.0  1.16.0

The CHART column combines the chart name and its version; APP VERSION is the application's. Useful variations: helm list -A shows every namespace, helm list -a includes releases in every state (by default, failed and uninstalled ones are hidden), and -o json or -o yaml gives machine-readable output for scripts.

helm status web -n helm-lab

prints the same summary you saw at install time, including the notes, for the current revision. It is the command to run when you come back to a release after a week and want to remember how to reach it.

Three get subcommands dig into what Helm stored:

helm get values web -n helm-lab       # only the values YOU supplied
helm get values web -n helm-lab -a    # the merged result, defaults included
helm get manifest web -n helm-lab     # the exact YAML Helm sent to the cluster
helm get notes web -n helm-lab        # the NOTES text again
helm get all web -n helm-lab          # everything in one go

helm get values with no flags shows only your overrides, so for a plain install it prints USER-SUPPLIED VALUES: null. That is not an error; you supplied nothing. -a (or --all) shows the computed values, defaults merged with your overrides, which is what the templates actually saw.

helm get manifest deserves special attention. It is the rendered, final YAML, after every placeholder was filled in. When a deployed application behaves strangely, reading the manifest shows you precisely what Kubernetes was told, with no templating in the way. Compare it with kubectl get deployment web-hello-world -n helm-lab -o yaml and you can see what Kubernetes added on top.

Finally, the release's memory, which you met earlier:

kubectl get secrets -n helm-lab

You will see a Secret named sh.helm.release.v1.web.v1. That single object is revision 1 of release web.

Changing configuration with values

The installed release uses defaults. To change them you supply values, and Helm offers two ways that you will use daily.

--set overrides a single value on the command line. It uses dots to reach nested keys:

helm upgrade web examples/hello-world -n helm-lab --set replicaCount=3 --set service.port=8080

-f (or --values) passes a YAML file whose structure mirrors the chart's values.yaml. This is the right tool as soon as you have more than one or two overrides, because the file can be reviewed, committed to git, and reused. Create values-staging.yaml:

replicaCount: 2

image:
  tag: "1.27-alpine"

service:
  port: 8080

and apply it:

helm upgrade web examples/hello-world -n helm-lab -f values-staging.yaml

The file contains only what differs from the defaults. You do not copy the whole values.yaml in; Helm merges your file on top of the defaults, key by key. That is the point of the design: a small, meaningful file per environment.

Note that these examples use helm upgrade. You have not learned it formally yet, and the next section does, but values only take effect when a release is installed or upgraded, so both commands accept exactly the same values flags. helm install ... -f values-staging.yaml works identically for a first install.

The --set syntax, and its sharp edges

--set has a compact language. The forms you need are:

The reason --set-string exists catches everyone once. --set version=1.10 is parsed as the number 1.1, because YAML treats an unquoted 1.10 as a float and drops the trailing zero. Likewise --set enabled=true gives a boolean and --set id=007 may lose its zeros. When a value must stay text, use --set-string, or quote it in a values file.

Precedence: who wins

When the same key is set in several places, the highest source wins. From lowest to highest:

1. The chart's own values.yaml (the defaults).
2. The parent chart's values, if this chart is a dependency of another (covered briefly later).
3. Files passed with -f, in the order given: a later file overrides an earlier one.
4. --set, --set-string, --set-file and --set-json flags, which beat every file.

That gives a clean working pattern. Keep a base file, layer an environment file on top, and use --set only for values that vary per run, such as an image tag supplied by your CI pipeline:

helm upgrade --install web ./hello-web -n helm-lab \
  -f values.yaml -f values-prod.yaml \
  --set image.tag="$GIT_SHA"

Seeing exactly what was applied

After any change, helm get values web -n helm-lab shows your overrides for the current revision and helm get values web -n helm-lab -a shows the merged result. When a value "did not take effect", compare these two first; nine times out of ten the key was misspelled, so Helm happily stored a value that no template reads. Helm does not warn about unknown keys unless the chart ships a schema that forbids them.

Upgrading a release

helm upgrade takes an existing release and moves it to a new chart version, new values, or both. Its shape mirrors install:

helm upgrade web examples/hello-world -n helm-lab -f values-staging.yaml

Helm renders the chart with the new inputs, compares the result with what it deployed last time, sends only the differences to Kubernetes, and records a new revision. You can confirm the revision number went up with helm list.

There are four different things you might be changing, and it helps to name which one you are doing:

• Changing values of the same chart version: helm upgrade web <chart> -f new-values.yaml.
• Moving to a newer chart version: helm upgrade web examples/hello-world --version 0.2.0. Run helm repo update first so the newer version is in your catalogue.
• Changing your own chart and re-deploying it from a local folder: helm upgrade web ./hello-web.
• Changing nothing but re-running, for example in a pipeline. This still creates a new revision.

The idempotent install-or-upgrade pattern

Scripts and pipelines should not have to know whether a release already exists. helm upgrade --install (short form -i) does an install if the release is absent and an upgrade if it is present:

helm upgrade --install web examples/hello-world -n helm-lab --create-namespace -f values-staging.yaml --wait

This is the command that CI/CD pipelines run. The same command is safe to run once or fifty times.

A trap that catches everyone: values do not persist the way you expect

Consider this sequence. You install with --set replicaCount=3. A week later you run helm upgrade web examples/hello-world --set image.tag=1.27-alpine. How many replicas do you get?

In Helm 4, as in Helm 3, each upgrade starts from the chart's defaults plus only the values you pass that time. The earlier replicaCount=3 is gone, and you are back to the default of 1. (The one exception: if you pass no values flags at all, Helm reuses the previous release's values. The moment you pass any override, it starts from the defaults.) The safe approach is to pass the same complete set of values on every upgrade, from files in git. Then the command is the full description of the desired state, and nothing depends on invisible history.

Two flags change this behaviour, and both are traps for beginners. --reuse-values merges the previous release's values with your new overrides but ignores any new defaults added in a newer chart version, which leads to templates reading keys that do not exist. --reset-then-reuse-values is a safer middle path, but the robust rule stays the same: keep your values in files and pass them every time.

Making upgrades fail safely

Two flags earn their place on upgrades you care about. --wait (discussed earlier) makes Helm wait for readiness. --rollback-on-failure goes further: if the upgrade fails, Helm automatically rolls back to the previous good revision.

helm upgrade --install web examples/hello-world -n helm-lab -f values-staging.yaml \
  --rollback-on-failure --timeout 5m

History and rollback

Every install and upgrade left a numbered revision. helm history lists them:

helm history web -n helm-lab

REVISION  UPDATED                   STATUS      CHART              APP VERSION  DESCRIPTION
1         Thu Oct  1 10:15:32 2026  superseded  hello-world-0.1.0  1.16.0       Install complete
2         Thu Oct  1 10:22:10 2026  superseded  hello-world-0.1.0  1.16.0       Upgrade complete
3         Thu Oct  1 10:24:41 2026  deployed    hello-world-0.1.0  1.16.0       Upgrade complete

Only one revision is deployed at a time, the live one. Earlier successes are superseded. A revision whose upgrade failed shows failed. The DESCRIPTION column records what happened, and you can write your own description on upgrades with --description "raise replicas for demo day", which is useful context for your future self.

To see what a past revision contained, use the --revision flag on get:

helm get values web -n helm-lab --revision 2
helm get manifest web -n helm-lab --revision 2

To undo a change:

helm rollback web 2 -n helm-lab

This rolls release web back to the state of revision 2. Omit the number (helm rollback web -n helm-lab) and Helm goes back to the immediately previous revision. Afterwards helm history shows a new revision 4 whose description says "Rollback to 2". History is never rewritten; it only grows, which is why you can always see what happened.

Rollback restores Kubernetes objects and the values of that revision. It cannot restore things outside Helm's control. If revision 3 ran a database migration, rolling back the chart does not undo the migration. If someone manually deleted a PersistentVolumeClaim, it is gone. Treat rollback as "redeploy the old description", not as a time machine.

Add --wait to rollbacks, too, so you know when the old version is actually serving again. In Helm 4.3 you can also attach an explanation, for example --description "rolled back: bad image tag", up to 256 characters.

Uninstalling a release

When you no longer want a release:

helm uninstall web -n helm-lab

Helm deletes the objects it created for that release (the Deployment, the Service, and so on) and then removes the release records. Afterwards helm list -n helm-lab no longer shows it and kubectl get all -n helm-lab is empty of those objects. Because Helm knows what it created, it removes exactly that and nothing else; it does not sweep the namespace.

Some details worth knowing:

• The namespace stays. Even if --create-namespace made it, helm uninstall does not delete it. Remove it yourself with kubectl delete namespace helm-lab when you are done.
• --keep-history removes the objects but keeps the release records, so helm list -a shows the release as uninstalled and helm rollback can bring it back. Without this flag, history is deleted with the release, and you cannot roll back.
• Names stay reserved when history is kept. Installing a new release with the same name after --keep-history fails with cannot reuse a name that is still in use. Use helm upgrade --install (which revives it), uninstall without history, or choose another name.
• Persistent data. A PersistentVolumeClaim created by a chart's StatefulSet is deliberately not always deleted, because deleting data silently would be dangerous. Check with kubectl get pvc -n <namespace> after uninstalling a database chart.
• Newer safety check. Since Helm 4.3, uninstall checks that each object actually carries this release's ownership labels before deleting it, and skips others with the warning skipping delete of resource not owned by this release. That protects you from deleting something another release or person created.

Preview an uninstall with helm uninstall web -n helm-lab --dry-run if you are unsure what would go.

Anatomy of a chart

So far you have used somebody else's chart. The real leverage comes from writing your own, and that starts with knowing what is inside one. Helm can scaffold a complete example:

helm create mychart

This makes a folder called mychart with a working, installable sample application (an nginx Deployment with a Service, and optional Ingress and autoscaler). The exact files vary slightly between Helm versions, but the layout is stable:

mychart/
  Chart.yaml          metadata: name, version, description
  values.yaml         the default values (the "menu" of settings)
  .helmignore         files to leave out when packaging
  charts/             dependencies live here (subcharts)
  templates/
    deployment.yaml   Kubernetes manifests with {{ placeholders }}
    service.yaml
    ingress.yaml
    _helpers.tpl      reusable named snippets (not rendered on their own)
    NOTES.txt         the message printed after install
    tests/
      test-connection.yaml   a Pod that checks the app works

Go through the pieces in the order Helm cares about them.

Chart.yaml is the chart's identity card. Its required fields are apiVersion (always v2 for current charts), name, and version. Everything else is optional but useful: description, type (application by default, or library for charts that only share templates), and appVersion. The key line to remember is that version is the chart's own version and must change whenever the chart changes, because registries and helm upgrade --version rely on it being unique.

values.yaml holds the defaults. A good one is commented, small, and grouped: image: has its repository, tag and pull policy together, service: has its type and port together. This file is your chart's public interface. Names you put here become promises to everyone who overrides them.

templates/ is the heart. Every file here (except those starting with an underscore and NOTES.txt) is treated as a Kubernetes manifest template. Helm runs each through the template engine, joins the results, and sends them to the cluster. Files starting with _ are not turned into objects; they hold named snippets other templates reuse, which is why the conventional _helpers.tpl has that name.

templates/NOTES.txt is plain text, also templated, printed after install and upgrade. templates/tests/ holds manifests annotated as tests, which helm test runs. charts/ holds other charts your chart depends on. .helmignore works like .gitignore: patterns for files that should not end up in the packaged archive, such as editor backups.

The generated scaffold is a good reference but is larger than a beginner needs. To understand every line, we will now write a smaller chart by hand.

Writing your first chart, step by step

We will build a chart called hello-web that serves a web page with nginx. The page's headline comes from a value, so one chart can greet different audiences. It has a ConfigMap (holding the page), a Deployment (running nginx and mounting the page) and a Service (giving it an address). Build it a file at a time so each piece is understood.

Start with the folder and the two files every chart needs:

mkdir -p hello-web/templates
cd hello-web

apiVersion: v2
name: hello-web
description: A tiny web page served by nginx
type: application
version: 0.1.0
appVersion: "1.27"

The appVersion is quoted on purpose. YAML would read an unquoted 1.27 as a number, and a value such as 1.10 would silently become 1.1. Quote version strings everywhere.

Now the defaults. Decide what a user might reasonably want to change, and expose exactly that:

# Number of pod copies to run
replicaCount: 1

image:
  repository: nginx
  tag: "1.27-alpine"
  pullPolicy: IfNotPresent

# The headline shown on the web page
message: "Hello from Helm"

service:
  type: ClusterIP
  port: 80

resources:
  requests:
    cpu: 50m
    memory: 32Mi
  limits:
    memory: 64Mi

Next, the ConfigMap template. It is ordinary Kubernetes YAML with a few placeholders in double curly braces:

apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ include "hello-web.fullname" . }}
  labels:
    {{- include "hello-web.labels" . | nindent 4 }}
data:
  index.html: |
    <!doctype html>
    <h1>{{ .Values.message }}</h1>
    <p>Release {{ .Release.Name }}, namespace {{ .Release.Namespace }}, revision {{ .Release.Revision }}.</p>

Three kinds of placeholder appear, and they are the ones you will write most. {{ .Values.message }} reads a value you defined. {{ .Release.Name }} reads a fact Helm supplies about the release itself (its name, namespace and revision number). {{ include "hello-web.fullname" . }} calls a named snippet that we define in a moment. The trailing dot passes the current context to the snippet.

The Deployment is the longest template, but every line in it is ordinary Kubernetes except the placeholders:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "hello-web.fullname" . }}
  labels:
    {{- include "hello-web.labels" . | nindent 4 }}
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      {{- include "hello-web.selectorLabels" . | nindent 6 }}
  template:
    metadata:
      labels:
        {{- include "hello-web.selectorLabels" . | nindent 8 }}
      annotations:
        checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}
    spec:
      containers:
        - name: web
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          ports:
            - name: http
              containerPort: 80
          readinessProbe:
            httpGet:
              path: /
              port: http
          resources:
            {{- toYaml .Values.resources | nindent 12 }}
          volumeMounts:
            - name: page
              mountPath: /usr/share/nginx/html
      volumes:
        - name: page
          configMap:
            name: {{ include "hello-web.fullname" . }}

Two lines deserve an explanation because they are idioms you will reuse in nearly every chart. The resources: block uses toYaml, which converts a whole structure from your values into YAML text, and nindent 12, which starts a new line and indents that text by twelve spaces so it lines up under resources:. Without the correct indentation, YAML breaks. The checksum/config annotation hashes the rendered ConfigMap into the pod template. Kubernetes only restarts pods when the pod template changes; editing a ConfigMap alone does not change it. By embedding the ConfigMap's hash, any change to the page changes the annotation, which triggers a rolling restart. It is a small trick with a large payoff.

The Service gives the pods a stable address:

apiVersion: v1
kind: Service
metadata:
  name: {{ include "hello-web.fullname" . }}
  labels:
    {{- include "hello-web.labels" . | nindent 4 }}
spec:
  type: {{ .Values.service.type }}
  ports:
    - port: {{ .Values.service.port }}
      targetPort: http
      name: http
  selector:
    {{- include "hello-web.selectorLabels" . | nindent 4 }}

The selector must match the pods' labels; that is how a Service finds its pods. Because both use the same named snippet, hello-web.selectorLabels, they can never drift apart. This is the main reason helpers exist.

Finally, define those snippets. Files beginning with an underscore are not rendered as manifests, so helper definitions live there:

{{/* Name shared by every object in this release, e.g. "demo-hello-web" */}}
{{- define "hello-web.fullname" -}}
{{- printf "%s-%s" .Release.Name .Chart.Name | trunc 63 | trimSuffix "-" -}}
{{- end }}

{{/* Labels that identify which pods belong to this release */}}
{{- define "hello-web.selectorLabels" -}}
app.kubernetes.io/name: {{ .Chart.Name }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end }}

{{/* Labels put on every object */}}
{{- define "hello-web.labels" -}}
{{ include "hello-web.selectorLabels" . }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
helm.sh/chart: {{ printf "%s-%s" .Chart.Name .Chart.Version }}
{{- end }}

A define block gives a name to a snippet. The first, hello-web.fullname, builds a name from the release name and the chart name, cuts it to 63 characters (the Kubernetes limit for many names) and removes a trailing hyphen. Prefixing snippet names with the chart name, as here, avoids collisions when charts are combined. Release demo of this chart produces objects named demo-hello-web, so two releases in one namespace never fight over names.

Add a notes file so users know what to do next:

Release {{ .Release.Name }} is installed in namespace {{ .Release.Namespace }}.

To view the page from your laptop, run:

  kubectl port-forward -n {{ .Release.Namespace }} service/{{ include "hello-web.fullname" . }} 8080:{{ .Values.service.port }}

then open http://localhost:8080

That is the whole chart: two metadata files and five templates. Before it goes near a cluster, the next section shows how to preview and check it. First, see the tree you just built:

hello-web/
  Chart.yaml
  values.yaml
  templates/
    _helpers.tpl
    configmap.yaml
    deployment.yaml
    service.yaml
    NOTES.txt

The template language in ten minutes

You have now seen the language in use. Here it is systematically, because nearly all chart-writing is a handful of ideas.

Helm templates use Go's text/template engine with a large library of extra functions called Sprig, plus Helm's own additions. Anything inside {{ }} is an action; everything outside is copied as-is. There are only a few kinds of action.

Reading data. The dot at the start of a name means "the current data". At the top of a template, the dot holds everything Helm gives you:

Pipelines. A vertical bar passes a value to a function, like a Unix pipe. {{ .Values.image.tag | quote }} wraps the tag in quotes. Functions chain left to right: {{ .Values.name | upper | trunc 10 }}.

Defaults and requirements. {{ .Values.image.pullPolicy | default "IfNotPresent" }} supplies a fallback when a value is empty. {{ required "image.tag is required" .Values.image.tag }} does the opposite: it aborts the render with your message if the value is missing, which is a kind way to tell a user what they forgot.

Conditionals. Include a block only when something is true:

{{- if .Values.ingress.enabled }}
apiVersion: networking.k8s.io/v1
kind: Ingress
# ...
{{- end }}

A value is "false" if it is false, 0, empty, or missing. That is how optional objects work: the whole file is wrapped in an if, and when the flag is off it renders to nothing.

Loops. range repeats a block for each item in a list or map:

env:
  {{- range $key, $val := .Values.env }}
  - name: {{ $key }}
    value: {{ $val | quote }}
  {{- end }}

Scope changes, and the dollar sign. Inside a range or with block, the dot changes to mean the current item, so .Values and .Release are no longer reachable through it. The variable $ always means the root. Write {{ $.Release.Name }} inside loops. Forgetting this is the most common source of "nil pointer" errors in beginner charts.

Whitespace. A hyphen inside the braces trims whitespace: {{- removes whitespace before, -}} removes it after. Without trimming, a line holding only an {{ if }} leaves a blank line in the output. YAML tolerates blank lines but not wrong indentation, so the trimming you need most is the nindent function from earlier, which produces a newline followed by the indented text.

Named templates and include. define names a snippet, and include "name" . inserts it. There is also a keyword template that does similar work, but it cannot be piped into other functions, so use include. That is why {{ include "x" . | nindent 4 }} is everywhere.

Comments. {{/* like this */}} is a template comment, removed from the output. A YAML comment with # is passed through to the output and visible in helm get manifest.

Previewing and checking a chart before it touches a cluster

The most valuable habit in Helm is to look at what will be sent before sending it. Three commands form a ladder, from fastest and least certain to slowest and most certain.

helm lint checks that the chart is well formed: valid Chart.yaml, parseable templates, sensible structure.

helm lint hello-web

==> Linting hello-web
[INFO] Chart.yaml: icon is recommended

1 chart(s) linted, 0 chart(s) failed

[INFO] lines are suggestions; [WARNING] lines deserve a look; [ERROR] lines fail the lint. Add --strict to treat warnings as errors in CI, and --set key=value or -f to lint with particular values. Lint does not contact a cluster.

helm template renders the chart locally and prints the YAML. It is the single most useful debugging command you have, because it shows exactly what Helm would send:

helm template demo hello-web -n helm-lab
helm template demo hello-web -n helm-lab --set replicaCount=3 --show-only templates/deployment.yaml

The first argument is a release name (a stand-in; nothing is installed). --show-only limits the output to one file, and --debug shows the output even when it is invalid YAML, so you can diagnose syntax errors. helm template needs no cluster at all, which is why pipelines use it for quick checks.

There is an important limit: helm template has no cluster to ask, so anything that depends on the cluster (such as whether a resource kind exists) is not validated. That is what the third rung is for.

helm install --dry-run=server renders the chart and sends it to the real API server for validation, but does not save anything:

helm install demo hello-web -n helm-lab --dry-run=server --debug

The API server checks the objects against the actual schema of your cluster, including custom resource definitions. It catches errors that local rendering cannot, such as a wrong field name, a missing CRD or a forbidden action. A plain --dry-run (or --dry-run=client) renders locally without contacting the cluster.

When a render fails, the error points at a file, a line and a column. Here are two you will meet, with real wording.

A missing parent key produces a Go template error that ends like this:

nil pointer evaluating interface {}.text

It happens when a template reads .Values.msg.text but msg was never defined, so there is nothing to take .text from. Fix it by defining msg in values.yaml, or guard the access with {{ if .Values.msg }}.

Bad indentation produces a YAML error:

Error: YAML parse error on hello-web/templates/deployment.yaml: error converting YAML to JSON: yaml: line 35: mapping values are not allowed in this context

Use --debug flag to render out invalid YAML

Follow its advice: run helm template --debug, find the reported line in the printed output, and check the indentation around the last inserted value. The cause is nearly always toYaml without nindent, or the wrong number passed to nindent.

Installing your chart, testing it, and iterating

Now put the chart into the cluster. The command is the same as before, except that the chart argument is a folder path:

helm install demo ./hello-web -n helm-lab --create-namespace --wait
kubectl get pods -n helm-lab

When the pod is Running, open the page as the notes told you:

kubectl port-forward -n helm-lab service/demo-hello-web 8080:80

Visit http://localhost:8080 in a browser (or curl http://localhost:8080 from another terminal). You should see "Hello from Helm" and a line naming the release, namespace and revision. port-forward stays in the foreground; press Ctrl-C to stop it.

Now experience the development loop. Change the headline from the command line and upgrade:

helm upgrade demo ./hello-web -n helm-lab --set message="Hello, MLOps MENA" --wait

Refresh the page. The headline changed and the page now says revision 2. The pods restarted because of the checksum/config annotation: the rendered ConfigMap changed, so its hash changed, so the pod template changed. Remove that annotation and repeat the experiment to see what it prevents: the ConfigMap would update but the pods would keep serving stale content until something else restarted them.

Chart tests

A deployed release that looks fine may not work. Helm has a built-in way to check: a manifest in templates/tests/ annotated "helm.sh/hook": test, run by helm test. Create one that fetches the page from inside the cluster:

apiVersion: v1
kind: Pod
metadata:
  name: {{ include "hello-web.fullname" . }}-test
  labels:
    {{- include "hello-web.labels" . | nindent 4 }}
  annotations:
    "helm.sh/hook": test
spec:
  restartPolicy: Never
  containers:
    - name: wget
      image: busybox:1.36
      command: ["wget"]
      args: ["-qO-", "http://{{ include "hello-web.fullname" . }}:{{ .Values.service.port }}/"]

The test is a Pod that runs to completion. If it exits with status 0 the test passes; any other exit code fails it. Upgrade so the chart includes it, then run:

helm upgrade demo ./hello-web -n helm-lab --wait
helm test demo -n helm-lab --logs

--logs prints the pod's output, so you see the fetched HTML. Test pods are not part of the release's normal resources; they exist only when you run helm test and stay afterwards for inspection. Helm's delete policy for hooks defaults to removing a previous hook pod of the same name before creating a new one, so running the test repeatedly works.

Hooks in one paragraph

A test is one kind of hook: a manifest that Helm runs at a particular moment rather than deploying as part of the release. The annotation "helm.sh/hook" names the moment: pre-install, post-install, pre-upgrade, post-upgrade, pre-rollback, post-rollback, pre-delete, post-delete, or test. The classic use is a database migration Job that must finish before an upgrade proceeds. You can order several hooks with "helm.sh/hook-weight" (a number in quotes; lower runs first) and control cleanup with "helm.sh/hook-delete-policy". Hooks are powerful and easy to misuse; the mid-level guide treats them properly. For now know that a hook's resources are not tracked as part of the release, so helm uninstall does not automatically delete them unless a delete policy says so.

Dependencies: charts that use other charts

Real applications need other things: a web app needs a database, a cache, perhaps a message queue. Rather than copying their YAML into your chart, you declare them as dependencies. Each dependency is itself a chart, called a subchart, installed together with yours as part of the same release.

You declare dependencies in Chart.yaml:

apiVersion: v2
name: hello-web
version: 0.2.0
appVersion: "1.27"
dependencies:
  - name: postgresql
    version: "~16.0.0"
    repository: oci://registry-1.docker.io/bitnamicharts
    condition: postgresql.enabled

Each entry names the chart, a version or version range, and where to find it. Possible repositories are an https:// repository URL, an oci:// address (without the chart name, which comes from name:), or file://../path for a chart in a neighbouring folder. The condition line makes the dependency optional: it is installed only if postgresql.enabled is true in your values.

Then download the subcharts into your chart's charts/ folder:

helm dependency update ./hello-web

This fetches the charts, stores them in charts/ as archives, and writes a Chart.lock file recording the exact resolved versions, like a lock file in npm or pip. Commit Chart.lock; use helm dependency build when you want to rebuild from it exactly, and helm dependency list to see the current state.

If you forget this step, Helm tells you clearly:

Error: found in Chart.yaml, but missing in charts/ directory: postgresql

The fix is helm dependency update, or add --dependency-update to the install command.

You configure a subchart through your own values, under a key named after the subchart:

postgresql:
  enabled: true
  auth:
    database: hello

Everything under postgresql: is passed to the subchart's own values. A special top-level key, global:, is visible to the parent and every subchart, which is how you push one setting (for example a registry mirror) into all of them.

Packaging and sharing a chart

A chart folder is fine for yourself. To hand it to a colleague, a pipeline, or a cluster elsewhere, you package it into a single versioned archive and publish that somewhere they can reach.

helm package ./hello-web

Successfully packaged chart and saved it to: /home/you/hello-web-0.1.0.tgz

The filename is built from the chart name and the version in Chart.yaml. That is why bumping version on every change matters: two different charts must never share a version, because anything that cached 0.1.0 would never see your fix. helm package also accepts --version and --app-version to override those fields, which CI pipelines use to stamp a build.

You can install the archive directly, which proves it is self-contained:

helm install demo2 ./hello-web-0.1.0.tgz -n helm-lab

Now publish it. The modern route is an OCI registry. First log in; in Helm 4 the login argument is only a host name, with an optional port, not a full URL:

helm registry login registry.example.com -u myuser --password-stdin

The --password-stdin flag reads the password from standard input instead of the command line, so it does not land in your shell history. Typical use in a pipeline is echo "$REGISTRY_TOKEN" | helm registry login ghcr.io -u myuser --password-stdin.

Then push the archive. The destination is the registry and a path; Helm adds the chart name and uses the chart version as the tag:

helm push hello-web-0.1.0.tgz oci://registry.example.com/charts

The chart now lives at oci://registry.example.com/charts/hello-web with tag 0.1.0. Anyone with access installs it by that address:

helm install demo oci://registry.example.com/charts/hello-web --version 0.1.0 -n helm-lab

Always pass --version, so what you deploy is what you tested. Helm 4 can also install by digest, the content hash of the artifact (oci://.../hello-web@sha256:<digest>), which is immutable and cannot be changed after the fact the way a tag can.

To try the whole flow without an account anywhere, run a throwaway registry on your machine:

docker run -d -p 5000:5000 --name registry registry:2
helm push hello-web-0.1.0.tgz oci://localhost:5000/charts --plain-http
helm show chart oci://localhost:5000/charts/hello-web --version 0.1.0 --plain-http

--plain-http is needed because this local registry has no TLS. Never use it, or --insecure-skip-tls-verify, against a real registry.

For a team, the registry you choose is usually the one your cloud provider offers (ECR, Artifact Registry, ACR, or Oracle's OCIR), the one built into your code host (GHCR) or a self-hosted one such as Harbor. If your organisation has regional or sovereignty constraints, choose a registry hosted in an approved region, and mirror public charts into it rather than pulling from the internet at deploy time.

The older method, a classic repository, is still common: you generate an index.yaml with helm repo index ./dir --url https://charts.example.com and host the folder on any web server, GitHub Pages or an object-storage bucket. You do not need it to start; the OCI route is simpler.

Configuration, files and environment variables

Helm has very little configuration of its own, which is a feature. Three places matter.

Files on your machine. Helm keeps three directories: a cache (downloaded charts and repository catalogues), a config directory (your repository list and registry logins) and a data directory (plugins). On Linux they are ~/.cache/helm, ~/.config/helm and ~/.local/share/helm; on macOS they live under ~/Library; on Windows under %APPDATA% and %TEMP%. helm env prints the real paths on your system. The two files you might edit or inspect are repositories.yaml (the repositories from helm repo add) and registry/config.json (registry credentials, in the same format as Docker's). Treat the latter like a password file.

Environment variables. Most settings can be supplied as HELM_* variables instead of flags, which is convenient in pipelines. The ones a beginner meets:

Global flags that work on every command include -n/--namespace, --kube-context, --kubeconfig, and --debug. --debug is a close friend: it prints extra detail about what Helm is doing, which is often the fastest way to see why something fails.

One more flag worth knowing is --timeout, the maximum time Helm waits for each Kubernetes operation (five minutes by default). If your images are large or your cluster is slow, raise it rather than retrying blindly.

Common errors and how to read them

Helm errors look alarming but follow a pattern: a prefix naming the operation (INSTALLATION FAILED, UPGRADE FAILED), then the cause, often quoting a Kubernetes message. Read from the end; the last clause is usually the real reason. The table lists the ones beginners meet, with real wording.

Two of these deserve a longer explanation.

"Another operation is in progress." Helm marks a release pending-install or pending-upgrade while working, and back to deployed or failed when done. If the process is killed midway (a pipeline timeout, Ctrl-C, a closed laptop) the marker stays, and Helm refuses to start another operation so two runs cannot collide. Look at helm history <release>; if the top revision is pending and nothing is running, roll back to the last good revision. Avoid the situation by making your pipeline's job timeout longer than Helm's --timeout.

Existing resources and ownership. Helm labels and annotates everything it creates with the release name, namespace and app.kubernetes.io/managed-by: Helm. If you try to install a chart that wants to create an object that already exists without those markers, Helm refuses rather than overwriting it, with a long message containing invalid ownership metadata. This commonly happens when you created something with kubectl apply and later put it in a chart. Your choices are to delete the old object, add the missing labels and annotations with kubectl, or tell Helm to adopt it with --take-ownership. Choose deliberately; adoption means the release now controls that object.

A debugging routine

When something fails, work down this list instead of guessing:

1. Read the error from the end.
2. helm list -a -n <ns> and helm history <release> -n <ns>: what state is the release in?
3. helm template with the same values: does it render, and is the output what you expected?
4. helm install --dry-run=server --debug: does the cluster accept it?
5. kubectl get pods, kubectl describe pod <name> and kubectl logs <name>: if Helm succeeded but the app is broken, it is now a Kubernetes question, and the Kubernetes guide at /student-guides/kubernetes picks up there.
6. helm get manifest <release> and helm get values <release> -a: compare what was deployed with what you intended.

Putting it all together

Here is one small end-to-end project that uses everything above: take the hello-web chart and run it for two audiences, a "staging" and a "production" release, each with its own values, then upgrade, break, recover and clean up. It takes about fifteen minutes.

Step 1. Prepare two values files, each only the differences from the defaults.

replicaCount: 1
message: "Hello from STAGING"

replicaCount: 2
message: "Hello from PRODUCTION"
resources:
  requests:
    cpu: 100m
    memory: 64Mi
  limits:
    memory: 128Mi

Step 2. Check before you deploy. Lint, render both, and dry-run against the cluster.

helm lint ./hello-web -f values-production.yaml
helm template prod ./hello-web -f values-production.yaml | grep -E "replicas|memory"
helm install prod ./hello-web -n prod --create-namespace -f values-production.yaml --dry-run=server

Step 3. Install both environments. Same chart, two namespaces, two releases.

helm upgrade --install staging ./hello-web -n staging --create-namespace -f values-staging.yaml --wait
helm upgrade --install prod ./hello-web -n prod --create-namespace -f values-production.yaml --wait
helm list -A

helm list -A shows two releases of one chart, which is the whole point of separating the chart from its values.

Step 4. Verify with the built-in test. (This needs the templates/tests/test-connection.yaml file from the testing section; add it and upgrade both releases first.)

helm test staging -n staging --logs
helm test prod -n prod --logs

Step 5. Ship a change safely. Change the production headline and upgrade with automatic rollback. Because you pass the same values file again, nothing else reverts.

helm upgrade prod ./hello-web -n prod -f values-production.yaml \
  --set message="Hello, launch day" --rollback-on-failure --wait --description "launch day banner"
helm history prod -n prod

Step 6. Break it on purpose, then recover. Point production at an image tag that does not exist, without the rollback flag, and watch it fail.

helm upgrade prod ./hello-web -n prod -f values-production.yaml \
  --set image.tag=does-not-exist --wait --timeout 60s

After a minute Helm reports a timeout and the new revision shows failed in helm history prod -n prod, while kubectl get pods -n prod shows a pod in ImagePullBackOff. The old pods kept serving because the Deployment rolls out gradually and the new pod never became ready. Recover:

helm rollback prod -n prod --wait
helm history prod -n prod
kubectl get pods -n prod

History shows a new revision labelled as a rollback, and the pods are healthy again. Repeat step 6 with --rollback-on-failure and notice that Helm performs the recovery itself.

Step 7. Package and publish. Bump version in Chart.yaml to 0.1.1 if you changed the chart, then package and push to a local registry as in the packaging section.

Step 8. Clean up.

helm uninstall staging -n staging
helm uninstall prod -n prod
kubectl delete namespace staging prod helm-lab

If you did all eight steps, you have used every verb Helm has for everyday work: repo, search, show, install, upgrade --install, list, status, get, history, rollback, test, lint, template, package, push, uninstall.

What you can now do, and what comes next

You can now do real work with Helm:

• Explain Helm's model: a chart plus values makes a release, and every change is a revision.
• Install Helm 4, verify it, and point it at the right cluster and namespace.
• Find charts on Artifact Hub and in repositories or OCI registries, inspect them with show, and install, upgrade, roll back and uninstall them.
• Configure releases with values files and --set, and predict which value wins.
• Build your own chart with templates, helpers and a test, and check it with lint, template and --dry-run=server.
• Package a chart and publish it to a registry.
• Read Helm's common errors and debug methodically.

Keep these habits from day one: check kubectl config current-context before you act, always pass -n, keep values in files in git, pass the full values on every upgrade, add --wait (and --rollback-on-failure for anything important), and run helm template before you install a chart you wrote.

Mid-level picks up where this stops. It explains how Helm renders and applies a release under the hood, the Helm 4 wait strategies and server-side apply, hooks and dependencies in depth, schemas that validate values, chart testing in CI, and deploying with pipelines. Senior covers running Helm as a platform for a team: security and provenance, supply chain, multi-tenancy, upgrades across Helm and Kubernetes versions, and where Helm stops and GitOps tools take over.

Natural next reading in this catalogue: /student-guides/kubernetes for what Helm actually creates and how to debug it, /student-guides/terraform for creating the clusters Helm deploys into, and /student-guides/docker for building the images your charts run.

Sources

• Helm documentation home
• Helm 4 overview
• Installing Helm
• Quickstart guide
• Using Helm
• Charts
• Chart hooks
• Chart tests
• Registries (OCI)
• Chart template guide
• Chart best practices
• Version skew policy
• Helm command reference
• helm install
• helm upgrade
• Helm 4 release announcement
• Helm 3 end of life
• Helm releases on GitHub
• Artifact Hub

Part one of three. A beginner Helm interview is rarely about templating trivia. It checks that you understand what Helm is (and is not), that you can name its four nouns without hesitating, that you know what happens on an upgrade and a rollback, and that you can debug a failed release in a sensible order. Read the model first, then the commands, then the traps, then rehearse the questions out loud.

The thirty-second answer

Helm is the package manager for Kubernetes. A chart is a versioned package of templated Kubernetes manifests with default values; helm install renders the chart with your values and applies the result to a cluster as a named release, and every change to that release is a numbered revision that you can inspect or roll back.

If you get thirty more seconds, add the architecture: Helm is a client-side tool only, with no server in the cluster since Helm 3. It uses your kubeconfig and your permissions, and it stores each revision as a Secret in the release's namespace. The current major version is Helm 4, released in November 2025, and charts written for Helm 3 keep working.

The model

The interviewer wants four nouns, used precisely. Say them as a chain and the rest follows.

A chart is a directory (or a .tgz archive of one) with a Chart.yaml, a values.yaml and a templates/ folder. Values are the configuration: the chart's defaults, overridden by your files and flags. A release is one installed instance, identified by a name and a namespace, so one chart can be installed many times. A revision is a numbered snapshot: install is revision 1, each upgrade or rollback adds one.

Two supporting terms come up straight away. A repository is a web server hosting an index.yaml and chart archives, which you add with helm repo add. An OCI registry stores charts the way a container registry stores images, addressed as oci://host/path/chart, and there is no repo add step for it. Artifact Hub is the public search site for both.

Helm's division of labour with Kubernetes is a frequent probe. Helm renders templates and sends manifests to the API server. Kubernetes does everything after that: scheduling pods, pulling images, restarting crashes. Helm acts only when you run it, so it does not watch the cluster or correct drift. That is the job of a GitOps controller such as Argo CD or Flux.

What a release is made of, and where it lives

State is the subtle part, and a good answer here separates strong candidates. Each revision is stored as a Kubernetes Secret in the release's namespace, of type helm.sh/release.v1, named sh.helm.release.v1.<release>.v<revision>. Inside is the chart, the values you supplied, and the rendered manifest, compressed and base64-encoded. Consequences worth stating:

• Anyone with read access to Secrets in that namespace can read the values you passed, so values are not a safe place for passwords.
• Helm works from any machine, because state lives in the cluster and not on your laptop.
• By default only the last ten revisions are kept (--history-max, or HELM_MAX_HISTORY).
• The storage driver can be changed with HELM_DRIVER (secret is the default; configmap and memory exist; sql is beta).

Helm 3 versus Helm 4

Expect one question on this because it shows whether your knowledge is current.

The safe one-line answer: "Helm 4 keeps chart and release compatibility, renames a few flags, makes waiting smarter, and uses server-side apply for new releases."

The commands to have ready

You should be able to type these without looking. Group them by intent so you can recite them as a workflow.

# Find and inspect
helm repo add examples https://helm.github.io/examples
helm repo update
helm search repo hello
helm show values examples/hello-world

# Install, observe, change, undo, remove
helm install web examples/hello-world -n helm-lab --create-namespace --wait
helm list -n helm-lab
helm status web -n helm-lab
helm upgrade web examples/hello-world -n helm-lab -f values.yaml
helm history web -n helm-lab
helm rollback web 1 -n helm-lab
helm uninstall web -n helm-lab

# See what was deployed
helm get values web -n helm-lab
helm get manifest web -n helm-lab

# Write and check a chart
helm create mychart
helm lint mychart
helm template demo ./mychart -f values.yaml
helm install demo ./mychart --dry-run=server --debug
helm package ./mychart
helm push mychart-0.1.0.tgz oci://registry.example.com/charts

The one to name if asked about CI is the idempotent pattern: helm upgrade --install <release> <chart> -n <ns> -f values.yaml --rollback-on-failure --wait. It installs when the release is new and upgrades when it exists, so the pipeline needs no branching.

Values and precedence

Interviewers love the question "which value wins?". The answer is an ordered list, lowest to highest: the chart's values.yaml; the parent chart's values if it is a subchart; each -f file in the order given (later files override earlier); then --set and its variants (--set-string, --set-file, --set-json), which override every file. The merged result is what templates read as .Values.

The follow-up is usually about --set syntax or typing. Know these: dots for nesting (a.b=c), braces for lists (name={a,b}), an escaped comma (\,) for a literal comma, and --set-string when a value such as 1.10 must not become the number 1.1. --set key=null deletes a default key.

A second favourite is "what does upgrade do with my previous values?". The accurate answer: if you pass any overrides, Helm starts from the new chart's defaults plus what you passed this time, so earlier --set values are not remembered. If you pass no values flags at all, it reuses the previous release's values. --reuse-values merges old values with new overrides but ignores defaults added in a newer chart, which can leave templates reading missing keys. The robust practice is to keep values in files in git and pass them every time.

Upgrade, rollback and history

Explain the lifecycle as numbers. Install creates revision 1. An upgrade renders the chart with the new inputs, sends the differences to Kubernetes, and stores revision 2. A rollback to revision 1 does not delete revision 2; it creates revision 3 with revision 1's content. History only grows (up to the cap), so you can always see what happened.

What rollback cannot do is also a good question. It restores the objects and values Helm tracked. It does not undo a database migration that a hook ran, it does not recreate a volume someone deleted, and it cannot roll back to a revision older than the history cap. Say that rollback is "redeploy an old description", not a time machine.

The classic traps, and why they share one cause

Most beginner failures come from forgetting that Helm is a thin layer: it renders and sends, and everything else is Kubernetes or your own inputs. The traps, in the order interviewers ask about them:

1. "Deployed" is not "healthy". By default Helm 4 returns once objects are accepted. A wrong image tag still shows STATUS: deployed and the pod sits in ImagePullBackOff. Use --wait (or --rollback-on-failure, which implies it).
2. Wrong cluster or namespace. Helm follows your current kubeconfig context and, without -n, the default namespace. helm uninstall web without -n answers release: not found. Check kubectl config current-context.
3. Stale repository catalogue. A new chart version is invisible until helm repo update.
4. Upgrade reverts settings. Passing only some values on an upgrade resets the rest to defaults.
5. YAML indentation in templates. The engine substitutes text. A multi-line value without nindent produces YAML parse error ... mapping values are not allowed in this context.
6. Secrets in values. They end up in shell history and in the readable release Secret.
7. A killed run leaves the release pending. The next command says another operation (install/upgrade/rollback) is in progress. Check helm history, then roll back to the last good revision.
8. A name already in use. cannot reuse a name that is still in use means the release exists, often failed or uninstalled with history kept.
9. An object Helm did not create. invalid ownership metadata appears when a chart wants to create something that already exists without Helm's labels. Delete it, label it, or adopt it deliberately with --take-ownership.

What unites them: Helm only knows what you told it, and Helm only checks what it can check locally. Everything the cluster decides afterwards (readiness, scheduling, image pulls) is invisible to a bare helm install, and everything you leave implicit (context, namespace, previous values) is a place for state to hide.

Template questions you should be able to answer

Why include rather than template? include returns text you can pipe into other functions, such as nindent; template cannot be piped.

What does nindent do? It writes a newline and then indents the text by the given number of spaces. It is how a multi-line value from toYaml lands at the right depth in YAML.

What is the $ for? Inside range and with, the dot changes to the current item. $ always refers to the root context, so $.Values and $.Release.Name keep working in loops.

How does a chart roll pods when a ConfigMap changes? A pod template annotation that holds a hash of the rendered ConfigMap, for example checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}. Kubernetes only restarts pods when the pod template changes, and the hash makes it change.

What is _helpers.tpl? A file whose name starts with an underscore, so it is not rendered as a manifest, conventionally holding define blocks such as names and labels.

helm template versus --dry-run=server? template renders locally with no cluster, so it cannot know which APIs exist. --dry-run=server renders and sends to the API server for validation without saving anything.

Debugging order

Give this order when asked what you do when a release fails. It shows you think in layers and start with the cheapest evidence.

1. Read the error from the end; the last clause is usually the cause.
2. helm list -a -n <ns> and helm history <release> -n <ns> for the release's state.
3. helm template with the same values: does it render, and does it look right?
4. helm install --dry-run=server --debug: does the API server accept it?
5. kubectl get pods, kubectl describe pod, kubectl logs: if Helm succeeded but the app is unhealthy, this is a Kubernetes problem.
6. helm get manifest and helm get values -a: compare what was deployed with what you meant to deploy.

Common interview questions

1. What is Helm and why use it?Helm is the package manager for Kubernetes. It bundles manifests into versioned charts, lets one chart serve many environments through values, and tracks each installation as a release with revisions, so upgrades, rollbacks and removal are single commands.
2. Chart, release and revision: define each.A chart is the package. A release is one installed instance of a chart, named and namespaced. A revision is one numbered step in a release's history, created by every install, upgrade and rollback.
3. Is there a server component?No. Helm 3 removed Tiller, so Helm is a client that talks to the Kubernetes API with your kubeconfig and your RBAC permissions, and stores its state as Secrets in the cluster.
4. Where does Helm keep release history?In Secrets named sh.helm.release.v1.<release>.v<n> in the release namespace, containing the chart, supplied values and rendered manifest. The last ten revisions are kept by default.
5. Install versus upgrade?install creates a new release and fails if the name exists; upgrade changes an existing one. upgrade --install does whichever is needed and is the standard pipeline command.
6. What do -f and --set do, and which wins?Both supply values. -f reads a YAML file and --set sets a single key. The order from lowest to highest is chart defaults, parent values, -f files left to right, then --set flags, so --set wins.
7. I set --set version=1.10 and got 1.1. Why?The value was parsed as a number, and the trailing zero disappears. Use --set-string, or quote the value in a values file.
8. How do you preview what Helm will deploy?helm template renders locally. helm install --dry-run=server additionally has the API server validate the result without persisting it. helm lint checks the chart's structure.
9. How does rollback work?helm rollback <release> [revision] creates a new revision whose content matches the chosen one. History is preserved, and the counter keeps increasing. It does not undo external side effects such as a database migration.
10. An upgrade reverted my earlier --set values. Why?Passing any override makes Helm start from chart defaults plus only the values given this time. Keep all values in files in git and pass them on every upgrade, or consider --reset-then-reuse-values; --reuse-values ignores new chart defaults.
11. helm install says deployed but the app is down. Why?Helm 4 does not wait for readiness by default; "deployed" means the objects were accepted. Add --wait so Helm watches until resources are ready, or --rollback-on-failure to also recover automatically.
12. What changed from --atomic in Helm 4?It was renamed --rollback-on-failure. The old name still works with a deprecation warning. The new flag also makes --wait default to the watcher strategy.
13. What does STATUS: pending-upgrade with "another operation is in progress" mean?A previous run was interrupted and left the release marked busy. Check helm history, then roll back to the last good revision. Prevent it by giving CI a job timeout longer than --timeout.
14. What does helm uninstall remove?Everything the release created, plus its records. It keeps the namespace. --keep-history retains the records so you can roll back, at the cost of reserving the name. Persistent volume claims from StatefulSets may survive, so check.
15. What is a helper template?A named snippet created with define, usually in _helpers.tpl, inserted with include. Charts use them for names and labels so every object agrees, for example so a Service's selector always matches the pod labels.
16. How do you make pods restart when a ConfigMap changes?Add a pod-template annotation containing the hash of the rendered ConfigMap, using sha256sum. The pod template changes, so Kubernetes rolls the pods.
17. What is values.yaml, and what makes a good one?It is the chart's default configuration and its public interface. A good one is small, grouped by concept, commented, and safe by default, because every key is a promise to users.
18. What are hooks and tests?Hooks are manifests annotated helm.sh/hook that run at lifecycle moments such as pre-upgrade; they are not tracked as normal release resources. A test is a hook of type test, usually a Pod, run by helm test; exit code 0 means pass.
19. Repository versus OCI registry?A repository is a web server with an index.yaml that you add with helm repo add. An OCI registry stores charts like images and is addressed by oci:// with --version, with no add step. OCI is the modern direction and reuses the registry a team already has.
20. How do you publish your own chart?helm package builds name-version.tgz from the chart and its version; helm registry login <host> authenticates; helm push name-version.tgz oci://host/path uploads it. Bump the chart version on every change.
21. Does Helm detect if someone edits a live object?Not between runs. It acts only when invoked. A GitOps controller provides continuous reconciliation on top of Helm.
22. What is a subchart and how is it configured?A dependency declared in Chart.yaml and stored in charts/ after helm dependency update. The parent sets its values under a key named after the subchart, and global: values reach every subchart.
23. Is it safe to put a password in --set?No. It lands in shell history and in the release Secret, readable by anyone who can read Secrets in the namespace. Use a secret manager or reference a Secret created separately.
24. Should you use Helm 3 or Helm 4 today?Helm 4. Helm 3 has a final minor release (3.22) and receives only security fixes until 10 February 2027. Charts and existing releases carry over without migration.
25. What if a release already contains an object with the same name?If an object exists without Helm's ownership labels, install fails with invalid ownership metadata. Delete it, add the labels and annotations, or adopt it on purpose with --take-ownership.

Sixty-second self-test

• Say the thirty-second answer, then add the no-server, Secret-per-revision detail.
• Define chart, values, release and revision, and explain why one chart gives many releases.
• State the values precedence order from lowest to highest.
• Explain what rollback does to the revision counter and what it cannot undo.
• Say why STATUS: deployed does not mean the app works, and which flag changes that.
• Name the Helm 4 renames: --atomic and --force, and the new meaning of --wait.
• Explain why an upgrade can "forget" earlier --set values.
• Give the debugging order in six steps.
• Explain include, nindent and $ in one sentence each.
• Name the two ways to distribute charts and show the push command.

Part one of three. Most beginner Helm trouble traces back to four causes: acting on the wrong cluster or namespace, trusting "deployed" as if it meant "healthy", passing only part of your values, and getting YAML indentation wrong inside templates. Start with the table, then the habits and practice cards, then the sections that explain the causes.

Common errors at this level

The habits that pay off most

Practice cards

Short, self-contained exercises. Each takes a few minutes and ends in a result you can check on screen.

1. Find Helm's memoryInstall any chart, then run kubectl get secrets -n <ns>. Find sh.helm.release.v1.<release>.v1. Upgrade once and list again: two Secrets, one per revision.
2. Make "deployed" lieInstall with --set image.tag=does-not-exist and no --wait. Note STATUS: deployed, then kubectl get pods showing ImagePullBackOff. Repeat with --wait --timeout 60s.
3. Lose a setting on purposeInstall with --set replicaCount=3. Upgrade with only --set image.tag=1.27-alpine. Count the pods. Then redo it with both values in one file.
4. Prove precedencePut replicaCount: 2 in a file and pass it with -f plus --set replicaCount=4. Check helm get values <rel> -a.
5. Break a template, read the errorChange nindent 12 to indent 2 in a Deployment template. Run helm template, then again with --debug, and find the line it blames.
6. Roll back to a past revisionMake three upgrades with different headlines. Run helm history, roll back to revision 1, and note the new revision number and its description.
7. Trigger the ownership checkCreate a ConfigMap with kubectl create configmap demo-hello-web, then install a chart that wants that name. Read invalid ownership metadata and fix it without --take-ownership.
8. Interrupt an upgradeStart an upgrade with --wait to a broken image, press Ctrl-C, and run helm history. Observe the state, then recover with a rollback.

Reading a failed release, step by step

When a command fails, resist the urge to re-run it with random changes. Work down from the cheapest evidence to the most specific. The order matters because each step narrows which layer the fault lives in: your command, your chart, the API server, or the running application.

First, read the error message from the end. Helm wraps errors: a prefix such as UPGRADE FAILED: names the operation and the tail holds the cause, often copied from Kubernetes. context deadline exceeded is about time; invalid ownership metadata is about who owns an object; YAML parse error is about your template.

Second, ask what state the release is in:

helm list -a -n web
helm history my-app -n web

-a includes releases in every state, so you can see a failed or pending-upgrade one that a plain helm list hides. helm list --failed and helm list --pending narrow it down. The history tells you which revision failed and what its description says.

Third, render the chart with the same inputs:

helm template my-app ./chart -n web -f values.yaml --debug

If this fails, the problem is in the chart or values and the cluster is innocent. If it succeeds, read the output for the thing you changed.

Fourth, let the cluster judge it:

helm upgrade my-app ./chart -n web -f values.yaml --dry-run=server --debug

The API server validates the objects against the real schema. This catches a wrong field name, a missing CRD or a forbidden action without changing anything.

Fifth, if Helm succeeded but the app misbehaves, stop using Helm and use Kubernetes tools:

kubectl get pods -n web
kubectl describe pod <pod-name> -n web
kubectl logs <pod-name> -n web
kubectl get events -n web --sort-by=.lastTimestamp

A pod stuck in ImagePullBackOff has a wrong image or missing registry credentials. CrashLoopBackOff means the container starts and dies, so read its logs. Pending usually means nothing has room for it or a volume cannot be bound; describe says which.

Sixth, compare intention with reality:

helm get values my-app -n web -a
helm get manifest my-app -n web

If the merged values are not what you meant, the bug is in how you passed them. If the manifest is what you meant but the app differs, the difference is in the cluster.

Values hygiene

Values are your chart's public interface, and most day-to-day mistakes are about them. A few rules keep them boring, which is what you want.

Give every environment a small file, and keep the chart's defaults safe. Defaults should be something you would not mind running by accident: one replica, modest resources, no public exposure. Environment files then state only what differs:

replicaCount: 3
image:
  tag: "1.27-alpine"
resources:
  requests:
    cpu: 200m
    memory: 128Mi
  limits:
    memory: 256Mi

Pass files in a fixed order, base first, then the environment, and use --set only for values that genuinely change per run, such as an image tag stamped by your pipeline:

helm upgrade --install my-app ./chart -n web \
  -f values.yaml -f values-production.yaml \
  --set image.tag="$GIT_SHA" \
  --wait --rollback-on-failure

Look at what is actually in effect. helm get values <release> -a prints the merged result. When a setting "does not work", put it next to helm show values <chart>; a misspelled key is the usual culprit, and Helm will not warn you. The defence is a values.schema.json in your chart, a file describing allowed keys and types. Helm validates against it on install, upgrade, lint and template and fails with values don't meet the specifications of the schema(s). A tiny one already catches typos:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "required": ["image"],
  "properties": {
    "replicaCount": { "type": "integer", "minimum": 1 },
    "image": {
      "type": "object",
      "required": ["repository", "tag"],
      "properties": {
        "repository": { "type": "string" },
        "tag": { "type": "string" }
      }
    }
  }
}

Quote strings that look like something else. "1.10", "007", "yes", "on" and "123456" can all be read as numbers or booleans by YAML. Quote version numbers, IDs and anything you want to stay text, and use --set-string on the command line.

Do not edit a chart you did not write just to change a value. If the value is not on the chart's menu (helm show values), the answer is a different chart version, a post-processing tool (a mid-level topic) or a fork, not a manual edit of a downloaded archive that the next upgrade will overwrite.

Secrets: what not to put in values

The release record is a Kubernetes Secret containing the values you passed, and helm get values prints them in plain text. So any password, token or key that goes into --set or a committed values file is readable by anyone who can read Secrets in that namespace, and by everyone with access to your git history. Do not use values as a secret store.

The pattern that works from day one is to create the Secret separately and have the chart reference it by name. Add a value for the name:

database:
  existingSecret: ""   # name of a Secret you created yourself
  secretKey: password

and use it in the Deployment with a guard, so a missing name fails loudly and kindly at render time:

          env:
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: {{ required "database.existingSecret is required" .Values.database.existingSecret }}
                  key: {{ .Values.database.secretKey }}

Then create the Secret out of band, for example with kubectl create secret generic db-credentials --from-literal=password=... -n web, or better with a tool that syncs from a proper secret manager. The mid-level and senior guides cover those options (External Secrets, Sealed Secrets, encrypted values files). What the beginner needs to hold on to: the chart carries a reference; the secret is delivered separately.

If you must see what a dry run contains, add --hide-secret so Secret contents are not printed into your terminal or CI logs.

Namespaces, contexts and the wrong-cluster problem

Every Helm command depends on two pieces of ambient state: the kubeconfig's current context and the namespace. Both are easy to forget, and both have caused real outages.

Get into the habit of making them explicit. In interactive work, a quick kubectl config current-context is enough. In scripts and pipelines, pass them:

helm upgrade --install my-app ./chart \
  --kube-context prod-cluster -n web \
  -f values-production.yaml --wait

A few related points:

• One namespace per environment or team is a good default. Release names are unique per namespace, so my-app can exist in staging and production without clashing.
• helm list only shows the namespace you ask for, or default. If a release seems to have vanished, run helm list -A.
• Shell prompts help. Many people show the current context in their prompt, and some tools add a coloured warning for production contexts. It costs nothing and prevents expensive mistakes.
• Do not share one kubeconfig between environments casually. Separate files selected with KUBECONFIG make a mistake harder.

Making your own charts better

Once you are writing charts, a handful of conventions separate a chart that is pleasant to use from one that is painful.

Bump version on every change. A registry, a cache and helm upgrade --version all assume a version identifies one exact chart. Changing templates without changing version produces confusing results later. appVersion is informational and tracks the application; version tracks the packaging.

Use helpers for names and labels, and use them everywhere. The Service's selector and the Deployment's pod labels must match; generating both from one define block guarantees it.

Expose a few good values, not every field. A value for each Kubernetes field turns the chart into a worse copy of the Kubernetes API. Expose what real users change: image, replicas, resources, service type, optional features behind an enabled flag.

Set resource requests and limits by default. A chart without them can be scheduled anywhere and can starve its neighbours. Sensible defaults in values.yaml protect cost and stability.

Add a readiness probe. It is what makes --wait meaningful. Without one, Kubernetes considers a pod ready the moment its process starts, and Helm reports success before the app can serve.

Write NOTES.txt. Tell the next person how to reach the app. It costs ten lines and saves a support thread.

Add at least one test under templates/tests/, and run helm test after deploys. A test that fetches the app's health endpoint from inside the cluster catches broken Services and wrong ports that a pod listing misses.

Lint in CI. helm lint --strict plus helm template with each environment's values file catches most mistakes before anyone deploys. Keep the rendered output for review if your team reads manifests.

Comment values.yaml. Every key should say what it does and what units it uses. helm show values is the documentation most users will ever read.

Moving from Helm 3 habits to Helm 4

If you learned Helm from older material, or your team still has Helm 3 scripts, these are the changes that will actually affect you. The good news is that the old flags keep working for now, so nothing breaks suddenly; you will see deprecation warnings pointing to the new names.

Two behaviours are worth knowing even if you never change a flag. New releases are applied with server-side apply by default, while releases created by Helm 3 keep their old apply method until you choose otherwise. And waiting is event-driven with watcher, which needs permission to list and watch the resources it waits on. If you ever see a permission error about watch in CI, that is why; the mid-level guide explains the options.

Also plan the calendar. Helm 3 gets only security fixes until 10 February 2027 and nothing after. If you maintain pipelines that pin a Helm 3 image, schedule the change to Helm 4 now.

Keeping a cluster tidy while you practise

Practice clusters fill up with half-forgotten releases. A few commands keep them clean:

helm list -A                                  # everything Helm knows about
helm list -A --failed                         # the broken ones
helm uninstall <release> -n <namespace>       # remove a release
kubectl get pvc -A                            # volumes that may outlive a release
kubectl delete namespace <namespace>          # remove the namespace itself

Uninstall does not delete a namespace, even one created by --create-namespace. Persistent volume claims from database charts often survive an uninstall on purpose, so check them and delete what you no longer need; they can keep costing money in a cloud cluster.

If you use a managed cloud cluster for practice, set a reminder to delete it. A forgotten load-balancer Service created by a chart keeps a real cloud load balancer billing. For the same reason, prefer a local cluster for learning.

Small things to do from day one

Name releases meaningfully. blog-staging beats hello-world-1712345678. Reserve --generate-name for experiments.

Add --description to upgrades in pipelines, for example with the commit hash. It appears in helm history and answers "what was that change?" months later.

Keep history modest. The default of ten revisions is fine. Set --history-max explicitly in pipelines so the behaviour is visible.

Turn on shell completion with source <(helm completion bash) (or zsh, fish, PowerShell). It teaches you the flags as you type.

Use -o yaml or -o json with helm list and helm status when scripting, instead of parsing the table.

Prefer OCI registries for your own charts, reusing the registry you already have, and always install with --version.

Read helm show readme before installing someone else's chart. The configuration table in it is usually better than the raw values.yaml.

A starter setup to copy

Copy this skeleton into a new project and delete what you do not need. Each line comes from something above.

my-app/
  chart/
    Chart.yaml
    values.yaml
    values.schema.json
    templates/
      _helpers.tpl
      deployment.yaml
      service.yaml
      NOTES.txt
      tests/test-connection.yaml
  values-staging.yaml
  values-production.yaml
  deploy.sh

#!/usr/bin/env bash
set -euo pipefail

ENV="${1:?usage: deploy.sh staging|production}"
NAMESPACE="my-app-${ENV}"
VERSION="${GIT_SHA:-dev}"

helm lint ./chart --strict -f "values-${ENV}.yaml"
helm template my-app ./chart -n "${NAMESPACE}" -f "values-${ENV}.yaml" > /dev/null

helm upgrade --install my-app ./chart \
  -n "${NAMESPACE}" --create-namespace \
  -f ./chart/values.yaml -f "values-${ENV}.yaml" \
  --set image.tag="${VERSION}" \
  --rollback-on-failure --wait --timeout 5m \
  --description "deploy ${VERSION}"

helm test my-app -n "${NAMESPACE}" --logs

Read it as a checklist made executable: it stops on any error, validates before it deploys, names the namespace, passes the full values every time, waits, rolls back on failure, records why, and finishes with a test.

Checklist

Before you run a Helm command that changes a cluster:

• I ran kubectl config current-context and it is the cluster I meant.
• The command has -n <namespace>.
• I read the rendered YAML, or at least ran helm template or --dry-run=server.
• All values come from files in git, plus at most an image tag from --set.
• No secret appears in a values file or a --set flag.
• The chart version is pinned with --version, or the chart is one I own and have bumped.
• The command includes --wait, and --rollback-on-failure if the release matters.
• I know which revision is live now (helm history), so I know what to roll back to.

After it finishes:

• helm status shows deployed, and kubectl get pods shows Running and ready.
• helm test passes, or I opened the app myself.
• I wrote a meaningful --description, or the pipeline did.

This is part two of three. Beginner got you to the point where you can install a chart, override a few values, upgrade and roll back. This level is for the day Helm stops being a command you run and becomes something your team depends on: the chart that five services share, the pipeline that deploys on every merge, the release that is stuck in pending-upgrade at eleven at night. By the end you should be able to predict what Helm will do before you press enter, and explain afterwards why it did it.

Everything here is written against Helm 4.3.0 (September 2026). Helm 4 changed several defaults that matter at this level, including how it applies manifests, what --wait means and how post-renderers work, so where a Helm 3 habit now does something different, the section says so.

Where this picks up

Each section starts with the problem it solves and ends with a Try it you can run against a local cluster (kind, k3d or minikube all work) in a few minutes. You will need a chart of your own: helm create web is enough, and the examples below assume that scaffold.

What one helm upgrade actually does

When a deploy fails, the first useful question is which step failed, and you can only ask it if you know the steps. Helm is a client-side program. There is no server component in the cluster (Tiller went away with Helm 3), so everything below happens inside the helm process on your laptop or CI runner, using your kubeconfig and your RBAC permissions.

For helm upgrade --install web ./web -n web -f values-prod.yaml, Helm works through this sequence:

1. Load the chartRead Chart.yaml, the templates and the vendored subcharts in charts/. A dependency declared in Chart.yaml but missing from charts/ fails here, before any cluster contact.
2. Read the release historyList the release records for web in namespace web. This decides whether this is an install or an upgrade, and fails early if another operation is still marked as in progress.
3. Merge valuesChart defaults, then parent-chart overrides for subcharts, then each -f file left to right, then every --set flag. The result becomes .Values.
4. Validate valuesIf the chart ships values.schema.json, the merged values are checked against it. A mismatch stops everything here.
5. RenderRun every template through Go's template engine with the merged values and built-in objects. required and fail fire here, as do nil-pointer and YAML errors.
6. Build objectsParse the rendered YAML into Kubernetes objects and map each kind to an API the cluster serves. A removed API version or a missing CRD fails here.
7. Run pre-hookspre-install or pre-upgrade hooks are created, in weight order, and Helm waits for them to finish.
8. ApplyCreate, update and delete objects so the cluster matches the new manifest. For new Helm 4 releases this is server-side apply.
9. Wait, then run post-hooksDepending on the wait strategy, block until resources are ready, then run post-* hooks.
10. RecordWrite a new revision record as a Secret, and mark the previous deployed revision as superseded.

Two properties fall straight out of that list. First, steps one to six never touch your workloads, so every failure in that range is safe: nothing changed in the cluster. That is why helm template and helm upgrade --dry-run=server are such good debugging tools, because they run exactly those steps and stop. Second, failures after step eight leave the cluster partially changed. Some objects are updated, some are not, and the release is recorded as failed. What happens next depends on the flags you passed, which is the subject of the section on failing safely.

The pending state also comes from this sequence. Before applying, Helm writes the new revision with status pending-upgrade (or pending-install, pending-rollback), and only updates it to deployed or failed at the end. If the process is killed in between, for example by a CI job timeout or a Ctrl-C, the record stays pending-* forever, and the next run refuses with another operation (install/upgrade/rollback) is in progress. Helm is not holding a lock anywhere; it is reading that status from the Secret.

The release record, and where state lives

Helm has no database of its own. The whole memory of a release, every revision you can roll back to, lives in the cluster as ordinary Kubernetes objects in the release's namespace. Understanding those objects is what lets you debug Helm when helm itself is confused.

By default each revision is a Secret of type helm.sh/release.v1, named sh.helm.release.v1.<release>.v<revision>. The name has not changed in Helm 4, which is why Helm 4 can manage Helm 3 releases in place with no migration. Each Secret carries labels you can select on:

kubectl get secret -n web -l owner=helm

NAME                        TYPE                 DATA   AGE
sh.helm.release.v1.web.v1   helm.sh/release.v1   1      2d
sh.helm.release.v1.web.v2   helm.sh/release.v1   1      1d
sh.helm.release.v1.web.v3   helm.sh/release.v1   1      4m

kubectl get secret -n web -l owner=helm,name=web --show-labels
kubectl get secret -n web -l owner=helm,status=failed     # every failed revision

The release key inside the Secret is the full release as JSON, gzipped and base64-encoded by Helm, then base64-encoded again by Kubernetes because that is how Secret data is stored. It contains the chart itself, the values the user supplied, the rendered manifest, the hooks and, in Helm 4, the apply method used. You can open it by hand:

kubectl get secret sh.helm.release.v1.web.v3 -n web -o jsonpath='{.data.release}' \
  | base64 -d | base64 -d | gunzip | jq 'keys'

kubectl get secret sh.helm.release.v1.web.v3 -n web -o jsonpath='{.data.release}' \
  | base64 -d | base64 -d | gunzip | jq -r '.info.status'

You rarely need to, because helm get manifest, helm get values and helm get hooks read the same payload. The reason to know the format is the day you have to repair a record by hand, and to understand three consequences.

Anyone who can read Secrets in the namespace can read your values. If you pass a database password with --set, it sits in every revision's record in plain text once decoded. helm get values web -n web prints it. Treat the ability to read Secrets in a release namespace as equivalent to reading everything ever passed to that release.

Records have a size ceiling. A Kubernetes object must fit in etcd's limit of roughly 1 MiB. A chart that embeds large files through .Files.Get, or an umbrella chart with dozens of subcharts, can produce a record too big to store, and the upgrade fails with Secret "sh.helm.release.v1…" is invalid: data: Too long. The fixes are to split the chart, stop embedding large files, or, for the rare genuine case, switch storage driver.

History costs space and is capped. Every upgrade adds a record. --history-max (or HELM_MAX_HISTORY) caps the number kept, and the default is 10. Setting it to 0 means unlimited, which in a busy namespace eventually means thousands of large Secrets in etcd. Leave it at 10 or set it explicitly in CI.

The storage driver is chosen with HELM_DRIVER: secret (the default), configmap, memory (for tests), or sql, a beta PostgreSQL backend configured through HELM_DRIVER_SQL_CONNECTION_STRING. The configmap driver stores the same payload without Secret semantics, so more people can typically read it and it is excluded from Secret encryption at rest. There is almost no reason to choose it today.

Values: precedence, upgrades and type traps

Most real Helm bugs are values bugs: a setting that did not take effect, a default that silently changed, a number that became a string. The rules are simple once stated precisely.

Precedence, lowest to highest

The chart's values.yaml is the base. For a subchart, the parent chart's values under the subchart's name (or alias) override the subchart's own defaults. Then each -f file is merged in order, left to right, so the rightmost file wins on conflicts. Then the --set family of flags is applied on top of all files. Maps are merged key by key, recursively. Lists are not merged: a list in a later source replaces the earlier list entirely. That last rule surprises people who override one item of an env: list and lose the rest.

helm upgrade --install web ./web -n web \
  -f values.yaml \           # repository-wide defaults for this service
  -f values-prod.yaml \      # environment layer, wins over the first file
  --set-string image.tag="$GIT_SHA"   # one per-deploy value, wins over both

To delete a key that a lower layer set, give it null, either in a file or as --set key=null. This is how you remove a default nodeSelector or resources.limits block rather than trying to override it with an empty value.

The --set family and type coercion

--set parses values: true becomes a boolean, 3 becomes an integer, and so on. That is usually what you want, until it is not. The classic incident is an image tag. A short Git SHA such as 1234e56 looks like scientific notation to the parser, becomes the number 1.234e+61, and renders as an unreadable float in the image reference. Tags like 2.10 become 2.1.

The template side matters too: quote strings where they are emitted, for example image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}" or value: {{ .Values.region | quote }}, so a value that happens to look like a number or a boolean still renders as a YAML string.

What an upgrade does with the previous values

This is where teams lose settings. On helm upgrade, Helm has to decide what to do with the values used last time. There are four behaviours:

--reuse-values is the trap. It looks like the safe choice ("keep what was there"), but it merges against the previous values rather than the new chart's defaults. If the new chart version adds probes.readiness.enabled: true as a default, a --reuse-values upgrade never sees it, and the template reads a missing key. The fix in CI is to never rely on the previous release as a source of configuration: pass the full, versioned set of values files on every deploy. Then the deploy is reproducible from Git alone, and nobody's ad-hoc --set from last Tuesday survives by accident.

Helm 4 also accepts multi-document values files: several YAML documents separated by --- in one -f file. It is convenient, but check the result with helm template before relying on it in a pipeline, because a stray separator in a file that previously worked in Helm 3 now means something.

Templates that stay readable

A chart is code that produces YAML, and it deserves the same care as any other code. Most unreadable charts come from ignoring four mechanisms: named templates, scope, whitespace control and guard functions.

Named templates and include

A named template is a reusable fragment defined with define and conventionally kept in templates/_helpers.tpl. Files whose names start with an underscore are not rendered as manifests, so helpers can live there without producing output. The scaffold already defines web.fullname, web.labels and web.selectorLabels. Use them everywhere so that every object carries the same labels:

{{- define "web.labels" -}}
helm.sh/chart: {{ printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
{{ include "web.selectorLabels" . }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end }}

Call named templates with include, not template. Both render the fragment, but include returns a string you can pipe, so you can indent it correctly:

metadata:
  name: {{ include "web.fullname" . }}
  labels:
    {{- include "web.labels" . | nindent 4 }}

nindent 4 adds a newline and indents every line by four spaces. Its sibling indent does not add the leading newline. Most YAML parse errors in charts are an include or a toYaml with the wrong indentation, and the error message points at a line number in the rendered output, not your template, so reproduce it with helm template --debug, which prints the broken YAML.

Scope and the dollar sign

Inside range and with, the dot (.) is rebound to the current item. That is the point of those blocks, and also why .Release.Name suddenly fails inside a loop. The root context is always available as $:

data:
  {{- range $key, $value := .Values.settings }}
  {{ $key }}: {{ $value | quote }}
  {{- end }}
  release: {{ $.Release.Name | quote }}

When you pass context into a named template from inside a loop, pass $ or build a dict explicitly (dict "root" $ "item" .), rather than the loop item, or the helper will not find .Chart and .Release.

Guards: required, default, with, fail

A template that reads .Values.database.host when database is absent fails with nil pointer evaluating interface {}.host. Choose deliberately between three responses. If the value has a sensible default, put the default in values.yaml, where it is documented, rather than scattering default calls through templates. If the value is mandatory and has no safe default, make the failure explicit and readable with required:

image: "{{ .Values.image.repository }}:{{ required "image.tag is required: pass --set-string image.tag=<sha>" .Values.image.tag }}"

If a whole block is optional, wrap it in with, which both tests for presence and rescopes:

{{- with .Values.nodeSelector }}
nodeSelector:
  {{- toYaml . | nindent 8 }}
{{- end }}

fail "message" is the general form, for invariants such as "autoscaling and a fixed replica count cannot both be set".

tpl, config checksums and lookup

tpl renders a string as a template. It lets users put template expressions inside values, such as host: "{{ .Release.Name }}-db", and have them evaluated: {{ tpl .Values.extraConfig . }}. Use it sparingly, because it makes values executable and harder to reason about.

A Deployment does not restart when only its ConfigMap changes, because the Pod template did not change. The standard fix is to put a hash of the rendered ConfigMap into a Pod annotation, so any config change changes the Pod template and triggers a rollout:

spec:
  template:
    metadata:
      annotations:
        checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}

lookup reads a live object from the cluster during rendering, for example to reuse an existing generated password instead of creating a new one on every upgrade. Understand its limits before using it: helm template and client-side dry runs have no cluster, so lookup returns an empty map there, and your template must handle that case. Only --dry-run=server and real installs query the cluster. Helm 4.3 logs at debug level when a lookup returns empty, which helps when you are unsure whether it ran. Every lookup also makes rendering depend on cluster state, so the same chart and values can produce different manifests in two clusters. That is sometimes the point, and always a cost.

Failing safely: waiting, rollback and stuck releases

The problem this section solves is the half-finished deploy. Helm applies objects in one pass and, by default, returns as soon as the API server has accepted them. "Accepted" is not "running": the Deployment exists, the image pull may be failing, and Helm has already printed STATUS: deployed. Pipelines that trust that message ship broken releases with a green tick.

What --wait means in Helm 4

In Helm 3, --wait was a boolean that polled resources until they looked ready. In Helm 4 it is a strategy, and it applies to install, upgrade, rollback and uninstall:

watcher is the one to use in pipelines. It opens watches rather than polling, so it reacts quickly and understands full reconciliation, including custom resources that publish kstatus-style conditions. Its cost is RBAC: the identity running Helm needs list and watch on the kinds the chart deploys, in that namespace. Since Helm 4.1 the watching is namespace-scoped, so no cluster-wide list permission is needed. If your CI role only has get and list, or a proxy in front of the API server breaks long-lived watches, fall back to --wait=legacy, which needs only list access.

Two companions matter. --wait-for-jobs makes Helm also wait for Jobs to complete, and only has an effect when a wait is enabled. --timeout bounds each Kubernetes operation, defaulting to 5m0s; under watcher a zero timeout is not allowed because there would be nothing to wait for. A chart whose Deployment needs eight minutes to pull a large ML image needs --timeout 15m, not a retry loop.

Rolling back automatically

--rollback-on-failure (Helm 3's --atomic, renamed in Helm 4) does two things. It turns on waiting, defaulting to watcher, and if the upgrade fails or times out it rolls the release back to the previous revision. On a first install there is nothing to roll back to, so Helm uninstalls the partial release instead. That is the behaviour you want: a failed first install that leaves a failed record behind is what produces "web" has no deployed releases on the next attempt.

helm upgrade --install web oci://ghcr.io/acme/charts/web --version 1.4.0 \
  -n web --create-namespace -f values.yaml -f values-prod.yaml \
  --rollback-on-failure --wait-for-jobs --timeout 10m \
  --description "deploy ${GIT_SHA}"

The old names still work in Helm 4 but print Flag --atomic has been deprecated, use --rollback-on-failure instead. Rename them in your pipelines now, because deprecated flags are the ones that disappear in a future major version.

A related flag, --cleanup-on-fail, deletes resources that this upgrade newly created if it fails, without rolling the rest back. Use it when you want a failed upgrade to stay visible for inspection but not leave new orphaned objects behind.

Reading helm history and rolling back by hand

Rollback creates a new revision that copies the old one, so the history only ever grows:

helm history web -n web

REVISION  UPDATED                   STATUS      CHART      APP VERSION  DESCRIPTION
1         Mon Sep 28 09:12:44 2026  superseded  web-1.3.0  2.3.0        Install complete
2         Tue Sep 29 14:03:10 2026  superseded  web-1.4.0  2.3.1        deploy 4f2a9c1
3         Tue Sep 29 14:41:55 2026  failed      web-1.5.0  2.4.0        deploy 88d0e7b

helm rollback web 2 -n web --wait --timeout 5m --description "rollback: 88d0e7b failed readiness"
helm history web -n web --show-rollback-revision

After the rollback the history gains a revision 4 whose description says it is a rollback to revision 2. Helm 4.3 lets you set --description on rollback (up to 256 characters), so the reason travels with the record. Get into the habit of writing the commit or ticket into --description on every deploy; it is the cheapest audit trail you will ever set up.

Clearing a stuck pending-* release

The message is Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress. The cause, as explained in the first section, is a revision record left in pending-install, pending-upgrade or pending-rollback because the Helm process died mid-operation. Work through these in order and stop at the first that works:

1. Confirm nobody is actually running a deploy. Check the CI system and your colleagues, because forcing a second operation over a live one is how records get corrupted.
2. Run helm history web -n web and find the last revision with status deployed or superseded that you trust.
3. Run helm rollback web <that-revision> -n web. A rollback is allowed from a pending state and writes a fresh revision, which resets the status.
4. If even that is refused, delete only the stuck record: kubectl delete secret sh.helm.release.v1.web.v<N> -n web, where <N> is the pending revision. Helm then sees the previous revision as the latest and the release is usable again. This removes Helm's memory of one attempt, not any Kubernetes objects.

The prevention is mundane and effective: make the CI job's own timeout longer than Helm's --timeout, so the job is never killed in the middle of a Helm run, and use --rollback-on-failure so a timeout ends in a clean state.

Server-side apply and field ownership

Helm 3 applied manifests with a three-way merge computed on your machine, the same idea as kubectl apply in its client-side mode. Helm 4 changes the default for new releases to server-side apply (SSA, specified in HIP-0023). This is the largest behavioural change in the release for anyone running Helm next to other controllers, so it deserves its own section.

The difference in one paragraph

With client-side apply, Helm diffed the old chart manifest, the new chart manifest and the live object, and sent a patch. Conflicts were resolved by whatever the patch happened to overwrite. With server-side apply, Helm sends the full desired manifest and the API server tracks, per field, which manager owns it. Helm becomes one field manager among several; kubectl, an HPA controller, Argo CD and a mutating webhook may each own other fields on the same object. If two managers try to own the same field with different values, the server returns a conflict instead of silently picking a winner.

The flags, and the latching rule

• helm install --server-side is a boolean and defaults to true.
• helm upgrade and helm rollback take --server-side as a string, true, false or auto, and the default is auto.
• auto means "reuse the method the previous revision used". The method is stored in the release record as apply_method, either ssa or csa.
• --force-conflicts tells the server to take ownership of conflicting fields anyway.

The consequence is latching: a release installed by Helm 3 has no apply_method, so it is treated as client-side and keeps using client-side apply after you upgrade your CLI to Helm 4. New releases use SSA. The two populations behave differently, and a cluster migrated to Helm 4 will contain both for months. Check which one a release uses rather than guessing:

kubectl get secret sh.helm.release.v1.web.v3 -n web -o jsonpath='{.data.release}' \
  | base64 -d | base64 -d | gunzip | jq -r '.apply_method'

To move an existing release to SSA, pass --server-side=true on an upgrade. Do it deliberately, in a quiet window, on one release first, because field ownership changes: fields that kubectl or another controller previously owned may now conflict.

The conflict you will actually meet

The classic case is replicas and an HPA. The chart sets spec.replicas: 3 and a HorizontalPodAutoscaler scales the Deployment to 7. Under client-side apply, the next helm upgrade quietly reset the replicas to 3, fighting the autoscaler on every deploy. Under SSA, Helm and the HPA controller are different managers, and the upgrade fails with a message from the API server:

Error: UPGRADE FAILED: failed to create typed patch object (web/web; apps/v1, Kind=Deployment):
Apply failed with 1 conflict: conflict with "kube-controller-manager" using apps/v1: .spec.replicas

The exact manager name in the message varies (you may see kubectl-client-side-apply or argocd-controller), but the shape is always "conflict with manager X: .field.path". There are two correct responses, and the order matters:

1. Stop setting the field. If an HPA owns replicas, the chart should not render replicas at all when autoscaling is enabled. {{- if not .Values.autoscaling.enabled }} replicas: {{ .Values.replicaCount }} {{- end }} is exactly what helm create scaffolds, for this reason.
2. Force it, knowingly. --force-conflicts makes Helm take ownership of the contested fields. Use it for a one-off migration, not as a default, because it hides the real question of who should own the field.

Combining --force-conflicts with --server-side=false, or with a release that latched to client-side apply, fails with forceConflicts enabled when serverSideApply disabled. The flag is meaningless without SSA.

Hooks and chart tests

Some work does not belong in the release's ordinary manifest: a database migration that must finish before the new Pods start, a smoke test that must run after they do. Hooks let a chart attach Kubernetes objects to lifecycle events, and chart tests are the same mechanism pointed at verification.

How a hook is declared

A hook is a normal manifest, usually a Job or a Pod, with annotations that say when to run it:

apiVersion: batch/v1
kind: Job
metadata:
  name: {{ include "web.fullname" . }}-migrate
  annotations:
    "helm.sh/hook": pre-install,pre-upgrade
    "helm.sh/hook-weight": "-5"
    "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
spec:
  backoffLimit: 0
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: migrate
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          command: ["./manage", "migrate", "--no-input"]

The events are pre-install, post-install, pre-upgrade, post-upgrade, pre-rollback, post-rollback, pre-delete, post-delete and test. Three details decide whether hooks are pleasant or painful.

Weights order hooks. helm.sh/hook-weight is a string containing an integer, and hooks run in ascending order. The quotes matter, because YAML would otherwise make it a number and Kubernetes annotations must be strings. Hooks with equal weights run in no promised order.

Helm blocks on hooks. For each hook Helm creates the object and waits for it to complete: a Job until it succeeds, a Pod until it reaches Succeeded. This wait happens even under the default hookOnly strategy, because hooks are exactly what that strategy waits for. If a pre-upgrade hook fails, the upgrade stops before any ordinary resource changes, which is the point of putting migrations there.

Hooks are not part of the release. Helm creates them but does not manage them afterwards: helm uninstall does not delete them, and they are not in the release's manifest (helm get hooks shows them separately). That is why the delete policy exists.

Delete policies and log retention

helm.sh/hook-delete-policy controls cleanup and accepts before-hook-creation, hook-succeeded and hook-failed. If you set none, before-hook-creation is the default. That default already solves the most common hook failure: a Job named web-migrate from the last release still exists, so creating the new one fails with jobs.batch "web-migrate" already exists. Deleting the old object before creating the new one avoids it.

Choose the rest on purpose. hook-succeeded tidies up after success. Omitting hook-failed means a failed Job and its Pod remain, so you can read the logs with kubectl logs job/web-migrate -n web. Combine both, as in the example, and you get a clean namespace when things work and evidence when they do not. The related helm.sh/hook-output-log-policy annotation, with values hook-succeeded and hook-failed, asks Helm itself to print the hook's logs on those outcomes, which is convenient in CI where the cluster is not at hand.

Chart tests

A test is a hook with the event test. It is not run during install or upgrade; you run it explicitly:

apiVersion: v1
kind: Pod
metadata:
  name: "{{ include "web.fullname" . }}-test-connection"
  annotations:
    "helm.sh/hook": test
spec:
  restartPolicy: Never
  containers:
    - name: wget
      image: busybox:1.36
      command: ['wget']
      args: ['--spider', '-T', '5', '{{ include "web.fullname" . }}:{{ .Values.service.port }}']

helm test web -n web --logs --timeout 3m
helm test web -n web --filter name=web-test-connection

--logs prints each test Pod's logs, which is what you want when a test fails in CI. --filter runs a subset by name. A test passes when its Pod exits zero. The scaffold's wget --spider only proves that the Service answers a TCP connection and an HTTP request; it says nothing about whether the application works. A useful test calls a real health or smoke endpoint, and in an ML service might send one fixed request through the model and check the response shape.

In a pipeline the sequence is helm upgrade --install … --rollback-on-failure, then helm test, and a failing test fails the job. Because tests are hooks, add a delete policy so the test Pods do not pile up: "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded. Helm 4 removed the old helm test --create-pods flag entirely, so if a script still passes it, remove it.

CRDs: the part Helm deliberately does not manage

A CustomResourceDefinition (CRD) teaches the API server a new kind, such as a ServiceMonitor or a KServe InferenceService. CRDs are the one place where Helm's upgrade story is intentionally incomplete, and operators of ML platforms meet them constantly, because serving, monitoring and pipeline tools are mostly operators.

Helm offers one built-in mechanism. Plain, non-templated CRD files placed in the chart's crds/ directory are installed before the rest of the chart, only on first install, and are skipped if a CRD with that name already exists. Helm never upgrades or deletes them. The reasoning is caution: a CRD is cluster-scoped, deleting one deletes every custom resource of that kind in every namespace, and a chart upgrade in one namespace should not be able to do that. The cost is that a new chart version which changes a CRD's schema does not apply the change.

Real charts therefore choose one of three strategies, and you should know which one a chart you depend on uses:

Useful flags: --skip-crds on install or upgrade skips the crds/ directory, helm template --include-crds renders them (they are omitted by default), and helm show crds <chart> prints them, with documents separated by --- in Helm 4.

The other CRD problem is ordering within one release. If a chart ships both a CRD and a custom resource of that kind in templates/, the first install fails with no matches for kind "X" in version "y" and a hint to ensure CRDs are installed first, because Helm's render-and-map step needs the API to know the kind. This is why operator charts put the CRD in crds/ or a separate chart and the custom resource in a later release.

Dependencies, subcharts and library charts

A chart rarely stands alone. Your ML service wants a database, a cache and perhaps a metrics exporter, and somebody else already packaged them. Helm's dependency mechanism composes charts, and the most useful thing to learn about it is where it stops being worth the trouble.

Declaring and vendoring dependencies

Dependencies are listed in Chart.yaml:

apiVersion: v2
name: web
version: 1.4.0
appVersion: "2.3.1"
kubeVersion: ">=1.31.0-0"
dependencies:
  - name: postgresql
    version: "~16.0.0"
    repository: oci://registry-1.docker.io/bitnamicharts
    condition: postgresql.enabled
    alias: db
  - name: common
    version: 1.x.x
    repository: oci://ghcr.io/acme/charts

helm dependency update ./web resolves the version ranges, downloads the chart archives into web/charts/ and writes Chart.lock with the exact versions and a digest. helm dependency build reads the lock and fetches exactly what it says, which is what CI should run, because it is reproducible. --skip-refresh avoids refreshing the repository cache, and helm install --dependency-update does the update as part of the install. A missing vendored chart produces found in Chart.yaml, but missing in charts/ directory: postgresql. Helm 4.1 builds dependencies concurrently, which is noticeable with a long list.

The repository field accepts an https:// repository URL, an @alias for a repository you have added, an oci:// path without the chart name (Helm appends name), or file://../path for a chart in the same monorepo. Commit Chart.lock, and decide deliberately whether to commit charts/*.tgz. Committing them makes builds independent of the registry; not committing them keeps the repository small. Either choice beats discovering during an incident that an upstream chart disappeared.

Passing values to subcharts

A subchart is configured from the parent, under a key that matches the subchart's name, or its alias when one is set:

postgresql:          # matches the dependency name; with alias "db" this key would be db:
  enabled: true
  auth:
    database: web
global:
  imagePullSecrets:
    - name: registry-creds

Keys under global: are visible to the parent and to every subchart, which makes them the right place for shared settings such as image pull secrets or a region label. Everything else stays private to the subchart. The condition field points at a boolean value, so postgresql.enabled: false removes the subchart entirely, and tags does the same for groups of subcharts. If both are present, the condition wins. import-values lets a parent pull values up from a subchart's exports, which is rarely worth the indirection.

To reach a subchart's values from the parent's templates, write the path as a normal values lookup, for example .Values.postgresql.auth.database. To reach a subchart's named templates, the names are global: a subchart defining common.labels makes it callable from anywhere, which is both the feature and the source of name collisions. Namespace every named template with the chart name, as the scaffold does with web.fullname.

Library charts

A chart with type: library contains only named templates and cannot be installed. It exists so that an organisation can write its labels, probes, security contexts and resource conventions once and have fifty service charts depend on it. The service chart's Deployment becomes a one-liner that calls the library's common.deployment template.

The trade-off is real. A library gives consistency and lets a platform team fix something everywhere by releasing one version. It also creates a second layer of indirection that readers must learn, and a bug in the library is a bug in every consumer the moment they bump it. Start without one. Extract a library when you have three charts that have drifted apart in the same way.

When not to use an umbrella chart

An umbrella chart is a chart that is only dependencies, deploying a whole stack as one release. It is tempting, and it has a price. Every subchart renders on every upgrade, one failing subchart fails the release, all history lives in one record that grows with the stack (and approaches the 1 MiB ceiling), and rolling back one service rolls back all of them. Separate releases per service, deployed by a pipeline or a tool such as Helmfile, are usually easier to operate. Reserve umbrella charts for things that genuinely version and deploy together.

Testing a chart before it reaches a cluster

A chart is code, so it needs the same layers of checking as any other code, from cheap and fast to expensive and realistic. The goal is for most mistakes to die on a laptop or in the first thirty seconds of a pull request.

Lint and JSON Schema for values

helm lint checks a chart's structure and renders it with default values. --strict turns warnings into failures, and --with-subcharts includes dependencies. Helm 4 lint also checks the crds/ directory and no longer ignores *.yml files, so a chart that passed under Helm 3 may report new findings. Lint only exercises the defaults, though, and most bugs live in the other value combinations, which is where a schema helps.

A values.schema.json file in the chart root validates the merged values on install, upgrade, lint and template. Helm 4 supports JSON Schema 2020-12. A small one catches the mistakes that otherwise surface as a confusing render error ten steps later:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "required": ["image"],
  "properties": {
    "replicaCount": { "type": "integer", "minimum": 1, "maximum": 50 },
    "image": {
      "type": "object",
      "required": ["repository", "tag"],
      "properties": {
        "repository": { "type": "string", "minLength": 1 },
        "tag": { "type": "string", "pattern": "^[A-Za-z0-9_.-]+$" }
      }
    },
    "resources": {
      "type": "object",
      "required": ["requests", "limits"]
    }
  }
}

Note what this does for the type trap from earlier. tag declared as string turns --set image.tag=1234e56 from a silent float into a failure at the start of the run: values don't meet the specifications of the schema(s) in the following chart(s), followed by the path and the reason. A schema is also the place to enforce organisation rules, such as "every workload declares resource requests and limits", which matters in cloud accounts where defaults drive the bill. --skip-schema-validation exists as an escape hatch for install, upgrade, template and, in Helm 4, lint. Treat each use as a code smell.

Rendering is the unit test

Because Helm is a function from values to YAML, the most valuable test is to render and look. helm template runs steps one to six of the upgrade sequence without a cluster:

helm template web ./web -f values-prod.yaml --show-only templates/deployment.yaml
helm template web ./web -f values-prod.yaml --kube-version 1.34.0 \
  --api-versions monitoring.coreos.com/v1 --is-upgrade > rendered.yaml

--kube-version and --api-versions let you render as if against a particular cluster, which is how you test the .Capabilities branches of a chart. --is-upgrade flips .Release.IsUpgrade. A cheap and strong pattern is a golden-file test: render each supported values file, commit the output into tests/golden/, and have CI fail when the rendered YAML differs from the committed copy. Every change then shows up in review as a diff of what will actually be deployed, not just a diff of templates.

The community helm-unittest plugin asserts on individual rendered fields (equal, matchRegex, hasDocuments) from YAML test files, which reads better than golden files when you want to say "with autoscaling on, there is no replicas field". Install it with helm plugin install; since Helm 4 verifies plugin signatures by default, read the plugin section later in this guide before you do.

Policy checks on the rendered output

Templates can render valid YAML that is still a bad idea: a container running as root, no resource limits, a latest tag. Scanners operate on the rendered manifest, so pipe helm template into them: kube-linter, checkov and trivy all read rendered YAML (see the Kubernetes guide for what good looks like, and the Trivy guide for scanning). Enforcing the same rules again at admission time with Kyverno or Gatekeeper means a chart that slips past CI still cannot reach production unnoticed.

Server-side dry run

Everything above is offline. The last cheap check asks the real API server:

helm upgrade --install web ./web -n web -f values-prod.yaml --dry-run=server --debug

--dry-run takes none (the default), client and server. A bare --dry-run means client, which renders without contacting the cluster and so cannot run lookup or validate against admission webhooks and CRD schemas. server does both and persists nothing. Use it in pull-request pipelines against a disposable or shared non-production cluster. Helm 4.2 fixed --dry-run=server to respect generateName:. The older helm template --validate still works but is deprecated with the message use '--dry-run=server' instead, and the two flags are mutually exclusive.

Real installs in CI

The top of the pyramid is installing into a throwaway cluster (kind or k3d in the pipeline) and running helm test. The community chart-testing tool (ct lint, ct install) automates this for a repository of charts: it detects which charts changed in a pull request, checks that the version was bumped, lints and installs only those. A failing install in a disposable cluster is cheap; the same failure in production is an incident.

OCI registries, packaging and promotion

The problem: a chart that exists only as a directory on someone's laptop is not deployable, and a chart fetched from latest is not reproducible. You need an immutable, versioned, access-controlled artifact, and Helm 4 wants that artifact in an OCI registry, the same kind of registry that holds your container images.

Packaging

helm package ./web                       # produces web-1.4.0.tgz
helm package ./web --version 1.4.1 --app-version "2.3.2" -d dist/
helm package ./web --dependency-update  # vendors charts/ first

The archive name comes from name and version in Chart.yaml. --version and --app-version override them at package time, which CI uses to stamp a build number. Helm 4 produces reproducible archives: the same inputs give the same bytes, and since 4.3 SOURCE_DATE_EPOCH is honoured, so you can pin the timestamp and get an identical digest from two builds. Reproducibility is what makes a digest meaningful as a record of exactly what was reviewed.

The .helmignore file, written like .gitignore, excludes files from the archive. Add your tests/golden/ directory and CI scripts to it, so they neither bloat the package nor count toward the release-record size ceiling.

Pushing and pulling

echo "$REGISTRY_TOKEN" | helm registry login ghcr.io -u "$REGISTRY_USER" --password-stdin
helm push web-1.4.0.tgz oci://ghcr.io/acme/charts
helm pull oci://ghcr.io/acme/charts/web --version 1.4.0 --untar
helm install web oci://ghcr.io/acme/charts/web --version 1.4.0 -n web
helm registry logout ghcr.io

Three OCI-specific rules trip people up:

• helm registry login takes a host only, optionally with a port, as in registry.example.com:5000. Passing oci://ghcr.io or https://ghcr.io fails in Helm 4. Older scripts often included the scheme.
• helm push takes a .tgz and a reference without the chart name. Helm builds the repository path from the chart's name and the tag from its version. Push web-1.4.0.tgz to oci://ghcr.io/acme/charts and it lands at ghcr.io/acme/charts/web:1.4.0.
• There is no helm repo add for OCI. The full oci:// reference goes into install, pull and Chart.yaml dependencies. There is also no helm search repo over an OCI registry, so discoverability comes from your registry's UI or Artifact Hub.

Credentials are stored in ~/.config/helm/registry/config.json on Linux, in the Docker config format. --plain-http and --insecure-skip-tls-verify are for local registries used in tests; do not put them in a production pipeline. The major registries (ECR, GAR, ACR, GHCR, Harbor, Artifactory, Docker Hub) all accept Helm charts, so for Gulf and Egyptian employers with data-residency rules the usual answer is a registry in the same regional cloud account as the cluster, which also keeps pull latency and egress charges down.

Tags versus digests

A chart version such as 1.4.0 is a tag, and a tag can in principle be overwritten by a later push. Helm 4 adds installation by digest, which cannot:

helm install web oci://ghcr.io/acme/charts/web@sha256:3d2f…a91c -n web

It is the same argument as for container images. When a chart is promoted from staging to production, promote the digest that staging proved, not a tag that might now point at something else. Many registries can be configured to make tags immutable, which is the simpler guard.

Promotion, not rebuilding

The release discipline mirrors images. Build and package once, in CI, on merge. Push with an immutable version. Deploy that same artifact to staging, run helm test, and then deploy the same version or digest to production with only the values file changing:

VERSION=1.4.0
helm upgrade --install web oci://ghcr.io/acme/charts/web --version "$VERSION" -n web-staging -f values-staging.yaml --rollback-on-failure
helm test web -n web-staging
helm upgrade --install web oci://ghcr.io/acme/charts/web --version "$VERSION" -n web -f values-prod.yaml --rollback-on-failure

The chart is the code, the values file is the environment, and only the second changes between stages. Keep chart versions honest: bump version for any change to the chart, and treat appVersion as informational. It records the application release the chart defaults to and does not affect Helm's behaviour. Image tags should come from values, set per deploy with --set-string image.tag=…, not from appVersion, unless you deliberately release chart and app together.

Classic HTTP repositories

Some charts still live in classic repositories, which are just a web server with an index.yaml and tarballs. You build an index with helm repo index ./dir --url https://charts.example.com, and the --merge flag adds new entries to an existing index. GitHub Pages with the chart-releaser tool, S3 and GCS all work. For a new internal chart, prefer OCI: one registry, one set of credentials, one place for retention.

Helm in CI/CD and GitOps

Helm only acts when somebody runs it. That sentence explains every production pattern around it. Either a pipeline runs Helm on each merge, or a controller in the cluster runs it (or something like it) continuously.

The pipeline pattern

A dependable deploy job has a fixed shape: authenticate, render or dry-run for the record, upgrade with a rollback safety net, test, and report.

set -euo pipefail
helm upgrade --install web "oci://ghcr.io/acme/charts/web" --version "$CHART_VERSION" \
  -n web --create-namespace \
  -f values.yaml -f "values-${ENV}.yaml" \
  --set-string image.tag="$IMAGE_TAG" \
  --rollback-on-failure --wait=watcher --wait-for-jobs --timeout 10m \
  --history-max 10 --labels "team=ml-platform" \
  --description "deploy ${GIT_SHA} by ${CI_JOB_URL}"
helm test web -n web --logs

Each flag earns its place. upgrade --install (-i) makes the command idempotent, so the same job works for the first deploy and the hundredth. --set-string keeps the tag a string. --history-max caps stored revisions. --labels attaches metadata to the release record that you can later filter on with helm list -l team=ml-platform. --description links the revision to its cause.

Give the pipeline the least privilege that works. The CI identity needs namespace-scoped permissions to manage the kinds the chart deploys, to read and write Secrets there (the release records are Secrets), and watch for the watcher strategy. It should not be cluster-admin. Anything cluster-scoped, such as CRDs, ClusterRoles and webhooks, belongs to a platform-owned pipeline with broader rights, which is another reason to keep those out of application charts.

Serialise deploys per release. Two jobs deploying the same release at once is how you get another operation … is in progress or, worse, interleaved changes. Use your CI system's concurrency groups so one release has one deploy at a time.

Pin everything that moves. The Helm binary version in the CI image, the chart version or digest, and every plugin version. A pipeline that installs latest Helm will one day pick up a major release the way Helm 3 users met Helm 4.

Helm under GitOps controllers

A GitOps controller watches a Git repository and keeps the cluster equal to it. The two common ones treat Helm very differently, and the difference affects how you debug.

If you run Argo CD, never run helm upgrade by hand against a release it manages, because Helm will not see its records and the controller will undo or fight your change. If you run Flux, helm history and helm get values work as usual, and the HelmRelease status is the first place to look. See the Argo CD and Flux guides for the controller side. Helmfile sits in between: a declarative file listing releases and values that you apply from CI with helmfile apply, a good step for a team that has outgrown shell scripts but not yet adopted a controller.

Whichever you choose, the same hygiene applies: chart versions pinned, values in Git, environments as values files, and image tags bumped by an automated pull request (Renovate and Dependabot both understand chart dependencies) so that every change is reviewed.

Helm and Terraform

Terraform and OpenTofu have a helm_release resource that calls Helm for you. It suits the platform layer, where the thing you install (an ingress controller, an operator, a monitoring stack) is part of the cluster's own definition and lives in the same state as the cluster. It suits applications much less well, because every application deploy becomes a Terraform plan against a shared state. A common split is Terraform for cluster add-ons and Helm or GitOps for everything teams ship. The Terraform guide covers the provider side.

Secrets and configuration you can live with

The release record problem from earlier has a direct corollary: anything you pass to Helm ends up in a Secret in the cluster, readable by whoever can read Secrets in that namespace, and in your CI logs if you are careless. Plain --set db.password=… is therefore the wrong default for real credentials.

The patterns that work share one idea: the credential does not pass through Helm as a value. Instead, the chart renders a reference, and something else resolves it.

• External Secrets Operator. The chart renders an ExternalSecret that names a path in Vault, AWS Secrets Manager, Google Secret Manager or Azure Key Vault, and the operator creates the real Kubernetes Secret. Helm's record holds only the path. This is usually the best answer for a team with a secret store.
• Sealed Secrets. You encrypt the secret with a cluster's public key and commit the ciphertext, which a controller decrypts in-cluster. Good for small teams with no secret store, but tied to one cluster's key.
• helm-secrets with SOPS. A plugin that decrypts encrypted values files on the fly, as in -f secrets://values.enc.yaml. The values are decrypted on your machine and then passed to Helm, so they do end up in the release record.
• Vault Agent or the CSI driver. Pods fetch secrets at start-up, so neither the chart nor the record contains them. See the Vault guide.

The first and last options keep the secret out of the record altogether; the middle two protect it in Git but not in the cluster. Know which property you are buying.

Two smaller habits help. Use --hide-secret with any dry run whose output reaches a log. And never rely on lookup to preserve a generated password as your only copy of it. If the release is uninstalled and reinstalled, the lookup finds nothing and you have a new password that the database does not know. helm get values prints whatever was passed, unmasked, so treat that command as a privileged operation.

For non-secret configuration, keep one values file per environment next to the chart's deploy configuration, and keep the differences small: if values-prod.yaml and values-staging.yaml differ in forty lines, your staging is not testing your production.

Debugging beyond --debug

When a Helm operation misbehaves, the cause is in one of four places: your values, your templates, the cluster's response, or Helm's own record of the release. A reliable order of inspection saves the time spent staring at the wrong one.

Start with what Helm believes. helm status web -n web shows the last revision's status and notes, helm history shows the sequence, and helm get values web -n web -a (all values, including chart defaults) and helm get manifest web -n web show the inputs and the rendered output of the current revision. Add --revision N to see an earlier one. Together they answer "what did we last ask for?", which is the half of every incident that is usually quick to settle.

Then compare with reality. Helm's manifest is what it sent; the cluster may differ because a controller edited fields, a webhook mutated the object, or somebody ran kubectl edit. The community helm-diff plugin compares the last manifest with what an upgrade would produce, and the honest version of the same check is:

helm get manifest web -n web | kubectl diff -f -

Reproduce the render. helm template --debug prints the YAML even when it does not parse, which is the only way to see a bad indentation. Add --show-only templates/deployment.yaml to narrow it, and --set or -f the same values the pipeline used. Most "works on my machine" chart bugs are a different values file.

Use the verbose output. --debug (or HELM_DEBUG=1) prints structured logging in Helm 4, which now uses Go's slog throughout. It shows the release lifecycle, including waits and retries. Client-side throttling shows up as slow operations with messages about rate limits when a chart creates many objects; raise --burst-limit (default 100) and --qps or set HELM_BURST_LIMIT if you deploy operators with hundreds of CRDs.

Read the cluster's side. For a wait that timed out, the Helm error only says resource not ready or context deadline exceeded. The reason is in the cluster: kubectl describe pod, kubectl get events -n web --sort-by=.lastTimestamp, kubectl logs job/<hook> for hooks. Pair that with kubectl rollout status deployment/web -n web, which gives a second opinion on readiness.

Plugins and post-renderers

Helm 4 redesigned plugins (HIP-0026). A plugin's plugin.yaml now has apiVersion: v1, a type (cli/v1 for new subcommands, getter/v1 for new download protocols such as s3://, postrenderer/v1 for post-renderers) and a runtime (subprocess, or extism/v1 for sandboxed WebAssembly). Helm 3 style plugins without apiVersion still load. The most important behaviour change is that helm plugin install --verify defaults to true: an unsigned plugin installed from a URL or VCS repository fails. For a plugin you have reviewed and trust, --verify=false is the explicit override:

helm plugin install https://github.com/databus23/helm-diff --verify=false
helm plugin list

A post-renderer receives the rendered manifests on standard input and returns modified ones. The standard use is applying a Kustomize overlay to a chart you do not control. In Helm 4 --post-renderer takes the name of an installed postrenderer/v1 plugin, not a path to an executable, so a Helm 3 pipeline with --post-renderer ./kustomize.sh breaks until the script is wrapped as a plugin:

apiVersion: v1
name: kustomize-pr
version: 0.1.0
type: postrenderer/v1
runtime: subprocess
runtimeConfig:
  platformCommand:
    - command: "${HELM_PLUGIN_DIR}/kustomize.sh"

helm plugin install ./kustomize-pr --verify=false
helm upgrade --install web ./web -n web --post-renderer kustomize-pr --post-renderer-args=--overlay=prod

Every person and pipeline that modifies the release must use the same post-renderer, or the next upgrade silently reverts its changes. That rule is a strong argument for using a post-renderer only when you cannot change the chart or its values, and for contributing the missing value upstream otherwise.

Moving from Helm 3 to Helm 4

Helm 3 stops receiving security fixes on 10 February 2027, three months later than first announced, and v3.22.0 was the final minor release. Anyone still on Helm 3 at work is on a clock, and the good news is that the move is much gentler than Helm 2 to 3 was. Charts with apiVersion: v2 keep working unchanged, and Helm 4 manages existing Helm 3 releases in place, because the record format is still sh.helm.release.v1. There is no migration tool, because there is nothing to migrate.

What changes is the CLI and its defaults. Work through this list on your pipelines:

A sensible sequence is: upgrade the CLI in CI images and in a staging pipeline first; run every job once and collect the deprecation warnings; rename the flags; convert post-renderers to plugins; fix registry login calls; install plugins in the v4 format. Then, separately and deliberately, decide whether to move long-lived releases to server-side apply with --server-side=true. New releases will use it automatically. Old ones continue with client-side apply until you choose.

The Kubernetes compatibility policy is also stricter than you may expect. Helm 4 supports the Kubernetes release it was built against and the three before it (n-3), so Helm 4.3 covers Kubernetes 1.34 to 1.37. Do not run a Helm much older than your cluster, and do not run one newer, since there is no forward-compatibility promise. Minor releases arrive roughly every four months, so put the Helm version in the same upgrade calendar as your cluster versions.

If you use Helm's Go SDK in your own tools, the import path is now helm.sh/helm/v4/..., charts moved to pkg/chart/v2 and releases to pkg/release/v1, and logging uses slog, so SDK code needs real edits rather than flag renames.

Putting it all together

Here is one service taken through everything in this guide. The team owns web, an inference API. Its chart lives in charts/web in the application repository.

1. Chart structure. Chart.yaml declares apiVersion: v2, kubeVersion: ">=1.34.0-0" and an optional condition-guarded Redis dependency. values.yaml holds commented defaults, including resource requests and limits. values.schema.json requires image.tag as a string and bounds replicaCount.
2. Templates. Labels and names come from _helpers.tpl. The Deployment guards replicas behind autoscaling.enabled, carries a checksum/config annotation, and reads its API key from a Secret referenced by name. A pre-upgrade Job runs the migration with before-hook-creation,hook-succeeded, and tests/smoke.yaml sends one fixed request to the service.
3. Pull request checks. helm lint --strict, helm template for each environment's values into golden files (the diff is reviewed), a scan of the rendered YAML, and helm upgrade --install --dry-run=server --hide-secret against a shared non-production cluster.
4. Release. On merge, CI bumps the chart version, runs helm package, and pushes to oci://ghcr.io/acme/charts, recording the digest.
5. Deploy. The job upgrades staging with --rollback-on-failure --wait=watcher --wait-for-jobs --timeout 10m --description "$GIT_SHA", runs helm test --logs, and, if green, repeats with values-prod.yaml and the same chart version. Only one deploy per release runs at a time.
6. Operate. helm history web -n web is the audit log, helm get values -a and helm get manifest answer "what did we ask for", and kubectl diff answers "what changed underneath us". A stuck pending-upgrade is cleared by rolling back to the last good revision. CRDs are applied by the platform pipeline before charts change.

The whole process reduces to a few rules you can now apply to any chart: values are inputs you pass completely every time, the chart is an immutable versioned artifact, Helm only acts when it is run, and the cluster is the truth.

Where you are now

You can read a Helm failure and say which step of the upgrade sequence produced it, and therefore whether the cluster was touched. You can decode a release record, explain what lives in it and why it is sensitive, and clear a stuck release without guessing. You know the values precedence rules, why --reuse-values misleads and why a lone --set resets your settings. You can write templates that stay readable, and protect them with a schema, golden-file renders and policy scans.

You understand Helm 4's behaviour on its own terms: waiting as a strategy, --rollback-on-failure, server-side apply with field ownership and latching, plugin-based post-renderers, and verified plugins. You can package, push and promote charts through an OCI registry by digest, choose how CRDs are managed, decide when a dependency or an umbrella chart is worth it, and place Helm correctly beside CI, Flux, Argo CD and Terraform.

The senior level treats Helm as a platform for other teams: where it breaks at scale, how to secure the supply chain with provenance and signing, how to design library charts and upgrade paths for dozens of services, and when to reach for a different tool. Before that, pair this guide with the Kubernetes guide for the objects your charts create, and with Kustomize for the post-rendering patterns mentioned above.

Sources

• Helm documentation
• Helm 4 overview
• Helm 4 changelog
• Helm 4 released (blog)
• Helm 3 end of life (blog)
• Helm releases on GitHub, including v4.3.0 and v3.22.0 notes
• Installing Helm
• Using Helm
• Helm version skew policy
• helm install reference
• helm upgrade reference
• helm plugin reference
• Charts
• Chart hooks
• Chart tests
• Library charts
• Registries and OCI
• Provenance and integrity
• Plugins
• Advanced Helm techniques: post rendering, storage backends
• Role-based access control
• Chart template guide
• Chart best practices
• Chart tips and tricks
• Troubleshooting FAQ
• HIP-0023: server-side apply
• HIP-0026: plugin system

Part two of three. At this level an interviewer is no longer checking that you can install a chart. They want to hear how you reason when a deploy goes wrong, what you would change in a pipeline, and whether you understand the machinery well enough to predict it. This guide is organised by topic, with scenario questions and trade-offs first and a rapid-fire round at the end. Everything is written against Helm 4.3 and calls out where Helm 3 habits now behave differently.

Where this picks up

The model you must be able to draw

Interviewers often open with "explain how Helm works" and watch how far down you go. A strong answer fits in a minute.

Helm is a client-side tool. It merges values, renders a chart's templates into Kubernetes manifests, sends them to the API server using the caller's kubeconfig and RBAC, and stores a record of each revision as a Secret in the release namespace. There is no server component, and nothing reconciles the cluster between runs.

Three follow-ups come almost every time, so have the answers ready.

"Where is release state stored?" In the release's namespace as Secrets named sh.helm.release.v1.<release>.v<revision>, labelled owner=helm. The payload is the release JSON, compressed and encoded, holding the chart, the supplied values, the rendered manifest and, in Helm 4, the apply method. That is why Helm 4 can manage Helm 3 releases without migration.

"What is the difference between a chart, a release and a revision?" A chart is the package. A release is one named installation of it in a namespace. A revision is one numbered state of that release, created by every install, upgrade or rollback.

"What happened to Tiller?" It was removed in Helm 3. Helm uses your credentials directly, so its power is exactly your RBAC, which is also how you limit a pipeline.

Scenario: the upgrade fails at eleven at night

This is the most likely scenario question. The interviewer says: "A deploy failed and the release is broken. Walk me through it."

A weak answer jumps to helm rollback. A strong answer first establishes which step failed, because that tells you whether the cluster was touched.

1. Read the error and place it in the sequence. values don't meet the specifications of the schema is value validation. execution error at (web/templates/…) is rendering. resource mapping not found is object building against the API. All of those happen before anything is applied, so the cluster is unchanged and nothing needs cleaning up. context deadline exceeded or resource not ready comes from the wait after applying, so the cluster is partly changed.
2. Check the record. helm history web -n web shows the last statuses. A failed revision is finished. A pending-upgrade means the process died mid-operation and the next run will refuse with another operation (install/upgrade/rollback) is in progress.
3. Look at the workload. kubectl get pods -n web, kubectl describe, kubectl get events --sort-by=.lastTimestamp. A wait timeout is almost always an image pull error, a failing readiness probe, a quota problem or a slow hook Job.
4. Restore service first. helm rollback web <last-good-revision> -n web --wait. Give the revision explicitly rather than relying on the default of "previous", because the previous revision might be the failed one.
5. Then diagnose offline. Reproduce with helm template --debug using the same values files, and helm upgrade --dry-run=server.

What earns extra credit: mentioning that rollback re-applies manifests but cannot undo a migration a pre-upgrade hook ran, and that with --rollback-on-failure most of this is automatic. What loses points: suggesting kubectl delete on the Deployment as the first move, or forcing with --force-replace without understanding that it deletes and recreates resources.

Scenario: "another operation is in progress"

The cause is a revision record stuck in a pending-* status, because the client was killed mid-run, often by a CI timeout shorter than Helm's own --timeout. There is no lock to release. A strong answer lists the escalation: confirm nobody is deploying, find the last good revision in helm history, run helm rollback to it, and only as a last resort delete the stuck sh.helm.release.v1.<release>.v<N> Secret with kubectl. Then it prevents recurrence: a CI job timeout longer than --timeout, --rollback-on-failure, and concurrency control so two jobs never deploy the same release.

Scenario: a setting silently disappeared

"We ran a hotfix with --set image.tag=abc and production lost its ingress hosts. Why?"

On helm upgrade, passing any values flag means Helm starts from the chart defaults plus what you passed now. Only when you pass no values flags at all are the previous release's values reused. The hotfix --set therefore dropped every value that had come from a previous -f file. The options, and the trade-offs between them:

The principle to state is that a pipeline should never use the previous release as a source of configuration.

Scenario: the tag turned into a number

"image.tag rendered as 1.234e+61." A SHA like 1234e56 parses as scientific notation with --set. Use --set-string, quote the value where the template emits it, and add a values.schema.json that declares tag as a string so the failure is loud and early. The same trap makes 2.10 into 2.1. A strong answer names the whole --set family (--set-string, --set-file, --set-json, --set-literal) and when each is used.

Scenario: Helm and the autoscaler fight

"After moving to Helm 4, new releases fail with a field conflict on .spec.replicas." Helm 4 applies new releases with server-side apply, in which the API server tracks field ownership per manager. The chart sets replicas, the HPA's controller also wants it, and the server reports a conflict rather than letting one overwrite the other silently. Under Helm 3's client-side merge the next upgrade would have reset your scaled replicas without complaint.

The right fix is to stop rendering replicas when autoscaling is enabled, which is exactly what the helm create scaffold does. --force-conflicts is available but hides the real question of who should own the field. A good candidate also explains latching: helm upgrade defaults to --server-side=auto, which reuses the method stored in the previous revision's apply_method, so releases installed by Helm 3 keep using client-side apply until someone passes --server-side=true.

Scenario: pipeline hangs, then says success anyway

"Helm prints deployed and the pods are in ImagePullBackOff." By default, Helm 4 waits only for hooks (hookOnly), so a successful API call is reported as success. The fix is to add --wait (which means the watcher strategy, event-driven with kstatus) and a --timeout, and --rollback-on-failure so a failed wait restores the last revision. Expect the follow-up: "what does watcher need?" It needs list and watch RBAC on the deployed kinds and a non-zero timeout, and custom resources must report readiness through kstatus-style conditions. If watches are unreliable behind some proxy, --wait=legacy polls instead and needs only list.

Trade-off questions

These have no single correct answer. Interviewers grade the reasoning, so state the options, the cost of each, and what would change your choice.

Helm versus Kustomize

Helm templates produce YAML from a parameter set and ship with versioning, dependencies, hooks and release history. Kustomize patches plain YAML without templates, which keeps manifests readable and diffs obvious but has no packaging and no lifecycle. Helm wins when you distribute software to others or need a parameter surface and release tracking. Kustomize wins for your own manifests with a few environment differences. They combine: a Kustomize overlay applied to a Helm chart through a post-renderer plugin (Helm 4 requires the post-renderer to be an installed postrenderer/v1 plugin) or through Argo CD and Flux. Point to the Kustomize guide for the other side.

Helm in CI versus a GitOps controller

A pipeline pushes: it is simple, it gives you Helm's native history and rollback, and credentials to the cluster must exist in CI. A controller pulls: it keeps the cluster equal to Git, corrects drift, and needs no cluster credentials outside the cluster, at the cost of another system to operate. Flux runs the real Helm SDK, so release records exist and helm history works. Argo CD renders with helm template and applies itself, so there are no release Secrets and helm list is empty. Say which you would pick for which team and why. A small team with few clusters may rightly stay with a pipeline; a platform team serving many teams usually wants a controller. See the Argo CD and Flux guides.

One umbrella chart versus a chart per service

An umbrella chart deploys a stack as one release, which is convenient and consistent. It also renders every subchart on every upgrade, fails whole if one subchart fails, rolls back everything together, and accumulates one large history record. Per-service charts deploy and roll back independently. The defensible default is per-service charts, with an umbrella only for components that truly version together.

OCI registry versus a classic chart repository

OCI reuses your existing registry, credentials and retention, supports digests for immutable installs and needs no index file. The costs are no helm search repo over OCI and needing a registry that supports it. A classic repository (index.yaml plus tarballs on GitHub Pages or object storage) is searchable and simple to host but has its own index to maintain and no digest pinning. For new internal charts, OCI is the stronger choice.

Charts for CRDs: crds/, templates, or a separate chart

The crds/ directory installs once and never upgrades or deletes. That is safe, but leaves CRD upgrades to you. Templating CRDs upgrades them with the release but makes helm uninstall delete the CRD and every custom resource of that kind, unless you add helm.sh/resource-policy: keep. A separate platform-owned CRD chart gives the cleanest lifecycle at the price of an ordering rule in your pipeline. The mature answer is the third, with the second handled carefully.

Hook or separate Job for migrations

A pre-upgrade hook blocks the release until the migration completes and stops the upgrade if it fails, which is correct for short, idempotent migrations. It runs outside the release's management and its side effects are not undone by a rollback. A long or risky migration belongs in its own pipeline step with its own rollback plan, and migrations should be compatible across one version so the previous revision still runs.

What a strong answer covers that a weak one misses

The same questions get very different answers depending on whether the candidate has operated Helm under pressure.

Commands to have ready

helm upgrade --install web oci://ghcr.io/acme/charts/web --version 1.4.0 -n web \
  -f values.yaml -f values-prod.yaml --set-string image.tag="$TAG" \
  --rollback-on-failure --wait=watcher --wait-for-jobs --timeout 10m --description "deploy $TAG"
helm template web ./web -f values-prod.yaml --show-only templates/deployment.yaml
helm upgrade web ./web -n web -f values-prod.yaml --dry-run=server --debug --hide-secret
helm history web -n web --show-rollback-revision
helm get values web -n web -a            # all values, including defaults
helm get manifest web -n web | kubectl diff -f -
helm rollback web 4 -n web --wait --description "reason"
helm dependency build ./web              # reproducible, uses Chart.lock
helm package ./web && helm push web-1.4.0.tgz oci://ghcr.io/acme/charts
kubectl get secret -n web -l owner=helm,status=failed

Be ready to explain any flag on these lines. "What does -a do on helm get values?" is a fair question, and so is "why --set-string?".

Questions about Helm 4 specifically

Interviewers in 2026 increasingly check whether candidates learned Helm 4 or are coasting on Helm 3.

"What are the headline changes?" Server-side apply by default for new releases, --wait strategies built on kstatus, plugins redesigned with signing and an optional WebAssembly runtime, install by OCI digest, multi-document values files, renamed flags (--atomic to --rollback-on-failure, --force to --force-replace), and post-renderers as plugins.

"Do existing charts and releases need changes?" No for apiVersion: v2 charts and for release storage, which keeps the same format. Yes for scripts that use renamed or removed flags, registry login with URLs, and path-based post-renderers.

"What is the support position on Helm 3?" v3.22.0 is the final minor release, and security fixes continue until 10 February 2027. After that there are no patches, so the work to move has a deadline.

"Which Kubernetes versions does Helm support?" The version it was built against and the three before, so Helm 4.3 covers 1.34 through 1.37.

Common interview questions

How does helm upgrade --install differ from helm install? It installs if the release does not exist and upgrades if it does, making the command idempotent, which is why it is the CI default. Plain install fails with cannot reuse a name that is still in use if the name exists.

What does helm template do that --dry-run does not? template never contacts the cluster by default; --dry-run=client is similar for install, and --dry-run=server contacts the API server, runs lookup, and validates against CRDs and admission webhooks without persisting.

Why did my lookup return nothing? Offline renders and client dry runs have no cluster, so lookup returns an empty map. Use --dry-run=server or a real install, and write the template to handle the empty case.

How do you make a Deployment restart when only its ConfigMap changed? Put a hash of the rendered ConfigMap in a Pod template annotation, checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}, so the Pod template changes whenever the config does.

Why use include instead of template? include returns a string that can be piped into functions like nindent; template cannot.

What happens to CRDs on upgrade? Nothing. Helm installs crds/ only on first install and never upgrades or deletes them.

What is --history-max for? It caps stored revisions per release, default 10. Unlimited history means many large Secrets in etcd.

Why might helm install fail with Secret ... Too long? The release record exceeds the roughly 1 MiB object limit, typically from embedded files or a huge umbrella chart. Split the chart or stop embedding large files.

How do you verify a chart's integrity? Pin the version or digest, verify provenance with helm verify or --verify when the publisher signs, or verify a cosign signature on the OCI artifact. The senior level covers this in depth.

How would you adopt an existing Deployment into a release? Add the ownership label and annotations (app.kubernetes.io/managed-by: Helm, meta.helm.sh/release-name, meta.helm.sh/release-namespace) or pass --take-ownership. Without them Helm refuses with invalid ownership metadata.

What does helm uninstall do in 4.3 that is new? It verifies ownership before deleting and skips resources that lack matching Helm metadata, warning skipping delete of resource not owned by this release.

Sixty-second self-test

Answer aloud without looking back.

1. Name the ten steps of an upgrade and say from which step on the cluster has been touched.
2. Which values are used on helm upgrade when you pass no flags? One --set? --reuse-values?
3. Why can --reuse-values miss a new chart default?
4. What does --wait with no value mean in Helm 4, and what happens when you omit it?
5. What is latching, and how do you migrate a release to server-side apply?
6. Where does a release record live, and who can read your values?
7. How do you clear a pending-upgrade release, in order?
8. What do hooks not do during a rollback?
9. Why does helm registry login oci://ghcr.io fail in Helm 4?
10. What is the Helm 3 security support end date, and what replaces --post-renderer ./script.sh?

If you stumble on any, reread the matching section of the detailed guide and then try it on a kind cluster. Interviewers can tell the difference between someone who has read about a stuck release and someone who has cleared one.

Part two of three. At this level the commands work, and the problems change character. They stop being "why will this not install" and become "why did that setting vanish", "why is the pipeline green over broken pods", and "why is this release stuck". Start with the error table, then the practices ranked by leverage, then the practice cards and the operational recipes you will paste into runbooks. Everything targets Helm 4.3.

Common errors at this level

Cumulative: the beginner errors still apply, and these are the ones that appear once things mostly work. Error text is what Helm 4 prints.

The practices that pay off most

Ordered by leverage: the first ones remove whole categories of incident.

Why that order

A safety net comes first because it limits the damage of every other mistake. Complete values come second because the most common "mystery" incident is configuration lost between deploys. Promotion and a schema follow because they make the rest of the pipeline trustworthy: you are testing the artifact you ship, and bad input is stopped early. Secrets handling, serialisation and descriptions are smaller in effort, but each of them removes one specific, recurring way of getting hurt.

Practice cards

Do these against a local cluster (kind, k3d or minikube) using a chart from helm create web.

1. Break a deploy on purposeDeploy with an image tag that does not exist and no wait flags. Note Helm reports success. Repeat with --wait --timeout 60s, then with --rollback-on-failure, and compare helm history each time.
2. Lose a setting, then stop losing itInstall with a values file that sets replicaCount: 3. Upgrade with only --set-string image.tag=1.27 and find out what happened to the replicas. Move to passing both values files every time.
3. Expose the trap in --reuse-valuesAdd a new default to values.yaml, bump the chart version, upgrade with --reuse-values, and look at helm get values -a. Then try --reset-then-reuse-values.
4. Open a release recordDecode sh.helm.release.v1.web.v1 with base64 -d | base64 -d | gunzip | jq. Find the values, the manifest and apply_method.
5. Make a stuck release, then clear itStart an upgrade with a bad image and a long timeout, press Ctrl-C, read helm history, and recover with helm rollback.
6. Start a server-side apply fightEnable an HPA and keep replicas in the Deployment template. Upgrade after the HPA has changed the count and read the conflict. Fix the template.
7. Write a hook that cleans upAdd a pre-upgrade migration Job with before-hook-creation,hook-succeeded. Upgrade twice to prove the second run does not collide with the first.
8. Add a schema and trip itWrite values.schema.json that requires image.tag as a string and bounds replicaCount, then break each rule and read the message.
9. Publish to OCI and install by digestRun a local registry:2, push with --plain-http, and install both by version and by @sha256:.
10. Convert a post-rendererTurn a shell script into a postrenderer/v1 plugin and call it with --post-renderer by name.

Reading a failure quickly

Before any recipe, keep this mapping in your head. Each error belongs to a step, and the step tells you whether the cluster has been touched.

Operational recipes

Clear a stuck pending release

helm history web -n web                    # find the pending revision and the last good one
helm rollback web 4 -n web --wait --description "recover from stuck pending-upgrade"
# only if rollback is refused, remove the stuck record (this deletes Helm's memory of one attempt, not any workload)
kubectl delete secret sh.helm.release.v1.web.v7 -n web

Confirm nobody is deploying first. Afterwards check the job timeout is longer than --timeout.

See every failed or pending release in a cluster

helm list -A --failed
helm list -A --pending
kubectl get secret -A -l owner=helm,status=failed

These are the health signals Helm itself offers, so a scheduled job that reports them is a cheap and effective alert.

Find out what a release was really given

helm get values web -n web            # only user-supplied values
helm get values web -n web -a         # merged with chart defaults
helm get manifest web -n web          # what Helm sent
helm get hooks web -n web
helm get manifest web -n web | kubectl diff -f -    # drift between sent and live

Compare the next upgrade with the live release

Install the community helm-diff plugin once, after reviewing it, and then run it on every pull request:

helm plugin install https://github.com/databus23/helm-diff --verify=false
helm diff upgrade web ./web -n web -f values.yaml -f values-prod.yaml

--verify=false is needed because the plugin is installed from a source repository without a signature. That is a decision to make knowingly, not a reflex.

Render as a specific cluster

helm template web ./web -f values-prod.yaml \
  --kube-version 1.34.0 --api-versions monitoring.coreos.com/v1 > rendered.yaml

Use it to test .Capabilities branches and to check that a chart will work on the next Kubernetes version before you upgrade the cluster.

Adopt an object that already exists

helm upgrade --install web ./web -n web --take-ownership -f values.yaml

Or label it by hand:

kubectl label cm app-config -n web app.kubernetes.io/managed-by=Helm
kubectl annotate cm app-config -n web meta.helm.sh/release-name=web meta.helm.sh/release-namespace=web

Keep a resource when the release is removed

metadata:
  annotations:
    "helm.sh/resource-policy": keep

The object becomes orphaned: Helm leaves it alone on uninstall and upgrade, so someone has to own its deletion. Use it for data volumes and for any CRD you template.

Package the same bytes twice

export SOURCE_DATE_EPOCH=$(git log -1 --format=%ct)
helm package ./web -d dist/
sha256sum dist/web-*.tgz

Since Helm 4.3, SOURCE_DATE_EPOCH is honoured, so two builds of the same commit give identical archives. That turns the digest into evidence of what was reviewed.

Upgrade a CRD on its own

kubectl apply --server-side -f charts/web/crds/
kubectl get crd widgets.example.com -o jsonpath='{.spec.versions[*].name}'
helm upgrade --install web ./web -n web -f values.yaml --rollback-on-failure --wait

Export existing custom resources with kubectl get widgets -A -o yaml > widgets-backup.yaml before the first CRD change in a cluster.

Migrate one release to server-side apply

kubectl get secret sh.helm.release.v1.web.v3 -n web -o jsonpath='{.data.release}' \
  | base64 -d | base64 -d | gunzip | jq -r '.apply_method'
helm upgrade web ./web -n web -f values.yaml --server-side=true --dry-run=server
helm upgrade web ./web -n web -f values.yaml --server-side=true --rollback-on-failure

Do the dry run first, read any conflicts, fix them at the source, and do this for one low-risk release before the rest.

Give CI the least privilege that works

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: helm-deployer
  namespace: web
rules:
  - apiGroups: ["", "apps", "batch", "autoscaling", "networking.k8s.io"]
    resources: ["*"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

That is a starting point, not a recommendation: narrow the resources to the kinds your chart actually creates. It includes watch for the watcher strategy and covers Secrets, which are needed for the release records. Anything cluster-scoped belongs to a separate platform pipeline.

Stop leaking secrets into logs

helm upgrade --install web ./web -n web -f values.yaml --dry-run=server --hide-secret

Use --hide-secret on any dry run whose output is logged, and avoid --debug on jobs that print values from a secret store.

Habits worth adopting now

Rename the deprecated flags today. --atomic and --force still work in Helm 4 but warn, and warnings are how removals are announced.

Pin the Helm version, the chart version and the plugin versions in CI. An unpinned tool upgrades itself at the worst moment.

Write --description on every deploy. Six months later it is the only way to answer "what was this revision for?"

Cap history. Leave --history-max at 10 unless you have a reason. Every stored revision is a Secret in etcd.

Test the rollback path. A rollback you have never run is a hope. Practise it in staging on a schedule, including rolling back past a hook.

Keep staging close to production. If the two values files differ in dozens of lines, staging is testing something else.

Read the chart before installing it. helm show values, helm show chart, helm show readme and helm template cost a minute and tell you what you are about to run, including any CRDs, cluster-scoped objects and images. See the Kubernetes guide for understanding what they create.

Decide who owns each field. When Helm, an HPA, a GitOps controller and a person all edit the same object, server-side apply will make the disagreement visible. Settle the owner in the chart.

Final checklist

• Deploys use --rollback-on-failure, an explicit --wait, a --timeout, and --description.
• Values files are versioned and passed in full; no deploy relies on --reuse-values or the previous release.
• Image tags go in with --set-string and the schema declares them as strings.
• A values.schema.json exists and the pipeline runs helm lint --strict.
• Rendered output is diffed in review and scanned for policy problems.
• helm upgrade --dry-run=server --hide-secret runs on pull requests.
• Charts are packaged once, pushed to OCI and promoted by version or digest.
• Secrets are referenced, not passed, and etcd encryption is on.
• CRDs have an explicit lifecycle (a crds/ directory, a separate chart, or a platform step).
• One deploy per release at a time; the CI job timeout exceeds Helm's.
• Helm, chart and plugin versions are pinned; post-renderers are plugins; registry login uses a host.
• Helm 3 is on the retirement plan, ahead of the end of its security fixes on 10 February 2027.

Reviewing a chart pull request

Much of what goes wrong with Helm is visible in review if you know where to look. A chart change has two diffs: the template diff, which is what the author edited, and the rendered diff, which is what the cluster will receive. Review the second one first. If the pipeline does not produce it, ask for golden files or a helm diff upgrade comment, because a one-line template change can rewrite every object in a release.

Then read the change against a short list of questions. Did the chart version move? Helm and your registry treat a version as immutable, so a change published under the same number is either refused or, worse, quietly replaces an artifact that staging has already tested. Did a default change? A new default reaches everyone who passes full values files and nobody who uses --reuse-values, so name the affected deploys in the description. Did a value get renamed? Renaming image.tag to image.version silently breaks every values file that still says tag, unless a schema rejects unknown keys or a required call fails loudly. Did a new object kind appear? A ClusterRole, a webhook or a CRD in an application chart changes who can deploy it, and deserves a platform reviewer.

Look for the traps that hide in small edits. A replicas line that returned next to an HPA will cause a field conflict on the next server-side apply. A hook without a delete policy will collide with its own previous run. A lookup call means the chart now renders differently in different clusters, so the golden files no longer prove much. A tpl on a value that users control means values can execute templates. A helm.sh/resource-policy: keep annotation means somebody must now own deleting that object. An image reference built from .Chart.AppVersion means a chart release and an application release are now coupled, and a hotfix of one needs the other.

Finally, check the operational side of the change. Would a rollback to the previous revision still work against any data or schema this version touched? Does the readiness probe describe the service being ready rather than the process being alive, since --wait=watcher will trust it? Do resource requests and limits exist, because they drive both scheduling and the cloud bill? These questions take five minutes per pull request, and they are the difference between a review that approves the syntax and one that approves the deploy.

This is part three of three. The first two levels taught you to install, template, upgrade and roll back. This one is about what changes when Helm is not your tool but the way forty engineers on six teams put software into production: the release record becomes a database you have to size, the --wait flag becomes a reliability decision, the plugin directory becomes part of your supply chain, and a version bump of the client itself becomes a migration with a hard end-of-support date.

Helm is deceptively small. It has no server, no controller and no database of its own. That makes it easy to adopt and easy to underestimate, because every property you would normally get from a platform (locking, drift detection, audit, tenancy, rollback safety) is either a side effect of where Helm stores its state, or something you have to build around it. A senior owner's job is to know exactly which is which, before the incident that teaches it.

Where this picks up

I am starting with the architecture, because every failure mode in this guide follows from three facts about where Helm's state and authority live.

Architecture and failure modes

Helm is a client. When you run helm upgrade, a short-lived process on your laptop or a CI runner loads a chart, merges values, renders templates, reads the previous release record from the cluster, sends manifests to the Kubernetes API server using the credentials in your kubeconfig, optionally waits for the result, writes a new release record, and exits. Then nothing is running. There is no Tiller (removed in Helm 3), no controller, and no background reconciliation. The same codebase is also a Go library, which is how Flux's helm-controller embeds it, but it behaves identically: it acts only when something calls it.

Three consequences follow, and the rest of this section is about each.

Authority is whatever the caller holds. Helm has no permission model of its own. If the pipeline's service account can create ClusterRoles, a chart can create ClusterRoles, and so can a chart that a teammate pulled from a registry you did not vet. Conversely, if the account cannot list Secrets in a namespace, helm list there shows nothing and helm upgrade cannot find the previous revision, which looks like a "release not found" error and not a permission error. The blast radius of a bad chart is the blast radius of the deploy identity, and that is a design decision you make in RBAC, not in Helm.

There is no lock, only a status field. Two engineers, or two pipeline runs, can start helm upgrade on the same release at the same time. Helm's protection is the release record itself: the first writer marks the new revision pending-upgrade, and a second run that sees a pending revision refuses with Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress. That is a check-then-write on a Secret, not a distributed lock, so it protects you in the common case and not in a tight race. The bigger problem is the converse. If the first run is killed mid-flight (a CI job timeout, a cancelled pipeline, a laptop closing), nothing clears the pending-* status, and the release is wedged until a human intervenes. I come back to this in the incident playbooks, because it is the most common Helm outage in practice.

There is no drift correction. Helm compares the old manifest, the new manifest, and (for client-side apply) the live state at upgrade time, and at no other time. If someone runs kubectl edit deployment on Tuesday and nobody upgrades the release until Friday, the drift sits there silently. helm status will say deployed while the cluster disagrees. If your organisation needs reconciliation, you need a GitOps controller on top, and you need to understand that the two main ones behave differently.

The Argo CD row surprises people. Argo CD uses Helm as a templating engine only, so Helm hooks are translated to Argo sync hooks (partially, not perfectly), helm rollback does not exist for those apps, and helm list -A on a cluster fully managed by Argo CD shows an empty table. Neither model is wrong, but mixing them on the same namespace is a recipe for two systems fighting over the same objects. Pick one owner per namespace and write it down. The neighbouring guides, Argo CD, Flux and Kubernetes, go deeper on the controllers.

The failure modes that follow from this architecture are a short list, and you can name them before they occur:

1. A wedged release (pending-* with no process behind it).
2. A release record that has grown too large to store.
3. A wait that passes when the workload is not healthy, or fails when it is fine.
4. Two field managers fighting over one field.
5. A deploy identity with more power than the change needs.
6. A chart or plugin that was never verified.
7. A client version that falls out of support.

The rest of the guide takes them in roughly that order.

The release record as a database

Every revision of every release is stored as a Kubernetes object, by default a Secret of type: helm.sh/release.v1 named sh.helm.release.v1.<release>.v<revision> and labelled owner=helm, name=<release>, status=<status> and version=<revision>. The payload is the release serialised as JSON, gzipped and base64-encoded. It contains the chart itself (templates and default values), the user-supplied values, the rendered manifest, the hooks and the apply method. You met this at Mid level as "where state lives". Here it matters for a different reason: it is the only persistent state in the whole system, and it lives in etcd, which has opinions about size.

Think about what that implies operationally.

It is bounded by the object size limit. Kubernetes objects, and therefore etcd values, are limited to roughly 1 MiB. A release record holds the chart and the rendered manifest, so a chart with large embedded files, a huge CRD bundle in templates/, or hundreds of rendered resources can push a single revision past the limit. The symptom is an upgrade that renders and applies fine and then fails at the very last step with Secret "sh.helm.release.v1.<release>.vN" is invalid: data: Too long. You are now in a nasty state: the workload changed, the record of the change could not be written.

It is multiplied by history. Each revision keeps a full copy. The default --history-max is 10 (set it with the flag or HELM_MAX_HISTORY; 0 means unlimited). Ten revisions of a 600 KiB record is 6 MiB of etcd per release, and a platform with 400 releases is 2.4 GiB of Secrets that every API server list and every etcd snapshot has to carry. An unlimited history on a release deployed fifty times a day is an etcd incident waiting for a quiet month.

It is readable by anyone who can read Secrets in the namespace. The user-supplied values are in there in clear (after base64 and gzip, which are encodings and not protection). Anyone with get on Secrets in that namespace can reconstruct every value ever passed, including the password someone put in a --set flag in revision 3 and removed in revision 4, for as long as revision 3 is still in history.

# How big are the release records, and which releases are heaviest?
kubectl get secret -A -l owner=helm \
  -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,STATUS:.metadata.labels.status

# Reading one record's values, to understand what an attacker with Secret access sees
kubectl get secret -n web sh.helm.release.v1.my-web.v3 \
  -o jsonpath='{.data.release}' | base64 -d | base64 -d | gunzip | head -c 600

The two base64 -d calls are not a typo: Kubernetes base64-encodes every Secret value, and Helm base64-encodes its own gzip payload inside that. Running that once against a real release is the most effective way to convince a colleague that the release Secret is sensitive data.

Helm offers alternatives through HELM_DRIVER:

The sql driver is the escape hatch for oversize records, but it converts a size problem into an availability and credentials problem, and most teams are better served by shrinking the chart. The rest of the options for a record that is too large are, in order of preference: stop putting large files through .Files and mount them from an object store; move bulky CRDs to a separate chart; split the application into several releases; and only then consider the SQL driver.

Pruning is also worth understanding, because it is automatic and silent. When a new revision is stored and the count exceeds --history-max, Helm deletes the oldest records. That is what you want for etcd. It also means your rollback window is exactly the history you kept: with a limit of 10 and a pipeline that deploys on every merge, "roll back to last Thursday" may be impossible. Decide the window deliberately and record deploy metadata somewhere more durable. --description (available on install, upgrade, uninstall and, since 4.3, rollback, with a 256-character limit) annotates each revision, and writing the commit SHA and pipeline URL into it costs nothing.

helm upgrade --install my-web ./chart -n web -f values-prod.yaml \
  --history-max 10 \
  --description "deploy ${GIT_SHA} pipeline ${CI_PIPELINE_URL}"

helm history my-web -n web --max 5

Finally, observability. Helm emits no metrics. The release Secrets are the closest thing to a fleet-wide status table, because their labels carry the status. Counting releases whose status label is failed or starts with pending- across all namespaces, and alerting when any stays that way for more than one deploy timeout, is the cheapest useful Helm monitor you can build, and kube-state-metrics (which exposes Secret labels only if you configure it to) or a small cron job using helm list -A --failed --pending -o json will do it.

Server-side apply and field ownership

Helm 4 changed how manifests reach the API server. Helm 3 used client-side apply: it computed a three-way merge (old manifest, new manifest, live state) locally and sent a patch. Helm 4 uses server-side apply (SSA, specified in HIP-0023) for new releases: it sends the desired manifest and the API server merges it, tracking which field manager owns each field in metadata.managedFields.

The distinction is invisible until two parties want the same field. Under SSA, the API server records that Helm owns .spec.replicas; if an HPA (or kubectl scale, or Argo CD) also writes it, there is a conflict, and the server tells you exactly who and where:

Error: UPGRADE FAILED: Apply failed with 1 conflict: conflict with "kube-controller-manager" using apps/v1: .spec.replicas

Read that message as "someone else owns this field, and you just tried to set it". The clean fix is the same as it always was, but SSA enforces it: if an HPA manages replicas, the chart must not render replicas at all. The blunt fix is --force-conflicts, which tells the server to take ownership anyway. Use it knowingly and rarely, because it resolves the conflict in Helm's favour and the other controller will take the field back on its next write, leading to a slow flap.

The flag defaults differ by command, and the difference is the source of most migration confusion:

Helm records the method in the release record as apply_method: "ssa" or "csa". The auto behaviour therefore latches: a release created by Helm 3 has csa in its history, so after you move your pipeline to Helm 4 it keeps using client-side apply for every upgrade, silently, until someone passes --server-side=true. A fleet half-migrated to Helm 4 is therefore a mixed fleet, where some releases have SSA semantics and some do not, and you will not see it unless you look.

That mixed state is an argument for an explicit migration rather than an accident. The plan I would run:

1. Upgrade the Helm client in CI first and change nothing else. Existing releases latch to CSA and behave as before, which gives you a safe first step.
2. Pick a low-risk namespace and run helm diff upgrade (the helm-diff plugin) then helm upgrade --server-side=true for one release. Expect conflicts with anything that also writes to the objects: HPAs, VPAs, admission webhooks that mutate defaults, other controllers, and humans with kubectl.
3. Resolve each conflict by removing the duplicate writer (usually deleting replicas from the chart when an HPA exists), not by forcing.
4. Move the remaining releases in batches, tracking which have moved. Record the apply method in your inventory: helm get metadata in Helm 4 prints release metadata you can script against.
5. Only after every release has moved should you consider dropping the --server-side=true flag from pipelines, and even then it is better to leave it explicit.

There is one bug worth knowing about because it bit people in the field: before Helm 4.1, helm upgrade --install did not pass --server-side through to the install half, so a first-time install in a pipeline could ignore your setting. If you are pinned to 4.0.x, upgrade.

# Inspect who owns what on a live object
kubectl get deploy my-web -n web --show-managed-fields -o yaml | grep -A4 'manager:'

Waiting, rollback and what "deployed" means

The word deployed in helm status means that Helm finished its part and stored the record. Whether the pods are running is a separate question, and Helm 4 makes you answer it on purpose, because --wait became a strategy.

A bare --wait means watcher. --rollback-on-failure (the new name for --atomic) implies --wait, defaulting to watcher. --timeout defaults to 5m0s and applies per Kubernetes operation rather than to the whole command, so a release with a long pre-upgrade hook and a slow rollout can run well past five minutes in total.

The design reasoning matters more than the table. The old --wait had a short list of built-in kinds it understood (Deployments, StatefulSets, DaemonSets, Jobs, Pods, Services, PVCs and a few others) and waited on them by polling. A custom resource, such as a Certificate, an InferenceService or a Kafka cluster, was invisible to it, so a release could report success while the operator was still failing to reconcile it. watcher uses kstatus, the convention by which resources report readiness through status conditions, so it can wait on anything that follows the convention. In 4.1 the watching became namespace-scoped, so it no longer needs a cluster-wide LIST; in 4.3 the resync period dropped from one hour to three minutes and status computation became concurrent, which matters for charts with hundreds of objects.

The default of hookOnly deserves a pause because it is a quiet trap for teams moving from scripts that used --atomic. In Helm 3, --atomic implied --wait. In Helm 4 --rollback-on-failure also implies a wait, but now a watcher wait, which is stricter: it inspects custom resources and full reconciliation as well as Deployments. A pipeline that was passing by luck on a Helm 3 wait can start failing on Helm 4 because something that was never checked is now checked. That is the wait being correct, but it is a behaviour change you should announce to the teams whose pipelines it will break.

The rollback side has its own sharp edges:

• --rollback-on-failure on a first install deletes the failed release rather than leaving a failed revision behind, which prevents the has no deployed releases dead end on retry.
• Rollback is a new revision. helm rollback my-web 4 creates revision N+1 whose content equals revision 4. It is applied with the same mechanism as an upgrade, so it can hit the same SSA conflicts, and it re-runs pre-rollback and post-rollback hooks.
• Rollback does not undo everything. It does not reverse a database migration that a hook ran, it does not restore CRDs from the crds/ directory (Helm never touches those after first install), and it does not restore Secrets that an operator regenerated. A rollback is a restore of manifests, not of the world.
• Rollback can fail for the same reason the upgrade did: a full history limit leaving no good revision, a removed API in the stored manifest, or a wait that times out.

A deploy command a platform team can defend looks like this, and every flag in it is a decision:

helm upgrade --install my-web oci://ghcr.io/acme/charts/web \
  --version 1.4.0 \
  -n web --create-namespace \
  -f values-prod.yaml \
  --rollback-on-failure \
  --wait=watcher --wait-for-jobs \
  --timeout 10m \
  --history-max 10 \
  --description "deploy ${GIT_SHA}"

The CI job timeout must be longer than --timeout plus the time for hooks, or the runner will kill Helm mid-operation and leave the release in pending-upgrade, which is the wedge described earlier. A good rule is to set the job timeout to at least twice the Helm timeout and to make the pipeline step non-cancellable while a Helm command is in flight.

Scaling: which component breaks first

Helm does not scale the way a server does, because there is nothing to scale. The limits are in the client, in the API server and etcd that it talks to, and in the humans and pipelines that share them. If you ask "what breaks first?", the honest answer depends on whether you are growing the number of releases, the size of each release, or the number of people deploying, and they fail in a predictable order.

The size of one release breaks first. Long before you have a fleet, a single umbrella chart grows until its record approaches the object size limit, or until rendering takes a minute, or until one failing subchart blocks forty healthy ones. Every subchart renders on every upgrade, and the whole release succeeds or fails as one unit, so the blast radius of a single bad value scales with the size of the chart. This is the strongest argument against the umbrella-everything design, and I come back to it in the chart architecture section.

Client-side throttling breaks next. Helm's Kubernetes client rate-limits itself. The defaults are a burst of 100 (--burst-limit or HELM_BURST_LIMIT, with -1 disabling the limit) and a steady rate set by --qps or HELM_QPS. A chart with many CRDs, or a watcher wait across hundreds of resources, will hit the limiter and slow down with messages about client-side throttling. Raising the limits helps, but the limiter exists to protect the API server, so raise it on a pipeline deliberately and watch the API server's request latency, not only Helm's.

The API server and etcd break after that. The per-release costs are the Secrets discussed above, plus the watches that watcher opens, plus the list calls of helm list -A. A common scaling failure is a dashboard or a cron job that calls helm list -A every minute across a thousand-release cluster: it pages through every release Secret in every namespace. Use label selectors, use -n, and cache the result. The other common one is unbounded history: many revisions per release times many releases.

Concurrency among humans breaks last, and most visibly. The larger the team, the more likely two runs overlap. A list of the controls that scale, in rough order of payoff:

1. One pipeline per release, serialised by the CI system (a concurrency group or a lock keyed on release and namespace), not by Helm's pending check.
2. A non-cancellable deploy step, so a cancelled pipeline cannot orphan a pending-upgrade.
3. Humans do not run helm upgrade in production at all; they merge.
4. A break-glass path for the exception, with its own identity so it shows up in audit logs.

A note on the release count that teams worry about and rarely need to: Helm itself is fine with thousands of releases. What degrades is everything that enumerates them. Design your tooling to ask about one release at a time.

The security model: who can do what

Helm's trust model has one sentence: Helm can do whatever the caller's credentials allow, and a chart can ask it to do anything the Kubernetes API accepts. Everything in this section is a consequence of taking that sentence seriously.

Start with what a chart is. Rendering is pure: templates are Go text/template plus Sprig and Helm's own functions, and they run no arbitrary code on your machine. Two exceptions are worth remembering. lookup and --enable-dns make rendering depend on the environment (the first reads the cluster, the second does DNS lookups, and DNS is off by default for exactly that reason), and plugins and post-renderers are executables that run with your privileges. The dangerous part is not the template, it is what the rendered manifest requests once applied: a ClusterRoleBinding to cluster-admin, a privileged pod, a hostPath mount, a mutating webhook that intercepts every pod. Installing a chart is granting it the ability to ask for those, and the only thing that stops it is RBAC on the caller and admission control in the cluster.

Least-privilege deploy identities

A pipeline should deploy with a namespace-scoped identity that can manage only what its charts contain. The pieces are a service account (or workload identity federated from CI, so there is no long-lived token), a Role in the target namespace, and a RoleBinding. Helm needs more than the workloads: it reads and writes its own release Secrets, and under the watcher strategy it needs list and watch on the kinds it waits for.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: helm-deployer
  namespace: team-a
rules:
  # Helm's own state
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list", "create", "update", "delete"]
  # What the team's charts contain
  - apiGroups: ["", "apps", "batch", "networking.k8s.io", "autoscaling", "policy"]
    resources: ["services", "serviceaccounts", "configmaps", "persistentvolumeclaims",
                "deployments", "statefulsets", "jobs", "cronjobs", "ingresses",
                "horizontalpodautoscalers", "poddisruptionbudgets", "pods"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: helm-deployer
  namespace: team-a
subjects:
  - kind: ServiceAccount
    name: ci-deployer
    namespace: team-a
roleRef:
  kind: Role
  name: helm-deployer
  apiGroup: rbac.authorization.k8s.io

The role above is an illustration, not a template to copy: the right verbs depend on your charts, and SSA needs patch. Start from what the chart renders (helm template | kubectl get -f - ... style analysis, or simply reading the rendered kinds) rather than from a wildcard. Note what is deliberately absent: no ClusterRoles, no CRDs, no namespaces, no webhooks. Those are cluster-scoped, and they are where tenancy is won or lost.

Helm also supports impersonation through --kube-as-user and --kube-as-group. A central pipeline identity can then act as a per-team identity for a given deploy, which gives you one credential to protect and per-team RBAC to enforce. The pipeline identity needs the impersonate verb on the target users or groups, which is itself a powerful permission, so guard it accordingly.

helm upgrade --install my-web ./chart -n team-a \
  --kube-as-user "system:serviceaccount:team-a:ci-deployer" \
  -f values-prod.yaml --rollback-on-failure --wait=watcher

A related mechanism for GitOps: Flux's HelmRelease accepts a serviceAccountName so the controller impersonates that account when it runs Helm for the release, which gives tenants the same RBAC ceiling in a declarative setup. (Verify the field against the Flux version you run; see the Flux guide.)

Admission control as the second wall

RBAC says who may create a kind of object; it says nothing about the contents. A team with legitimate Deployment rights can still deploy a privileged container or a latest tag. Admission controllers (Kyverno, OPA Gatekeeper, or the built-in ValidatingAdmissionPolicy) are where you enforce content: no privileged pods, resource requests and limits required, images only from approved registries, no hostPath. This is deliberately outside Helm. Admission sees the final object no matter how it was produced, so it covers Helm, kubectl, Argo CD and Flux alike, and it cannot be bypassed by editing a chart. A platform team that enforces policy in charts but not in admission has a convention, not a control.

Complement that with shift-left scanning of the rendered output. Scan helm template output, not the chart source, because templates are not YAML and a scanner reading {{ if }} blocks misses what ends up applied.

helm template my-web ./chart -f values-prod.yaml > rendered.yaml
trivy config rendered.yaml            # or checkov, kube-linter

See the Trivy guide for configuration scanning in detail. Render with the same values and the same --kube-version and --api-versions the target cluster has, or the scan checks a different document from the one that will be applied.

Secrets in a chart-driven platform

Helm has a secrets problem because values are how configuration gets in, the release record stores the values, and everyone wants one place to put a database password. The chain of custody you are trying to keep short looks like this: the secret exists in a manager, reaches the cluster without passing through Git, a pipeline log or a release record, and ends up as a mounted file or environment variable in a pod.

Here is how each common pattern fares on the places a secret can leak:

The pattern that keeps secrets out of the release record entirely is the one where the chart never sees the value: it renders a reference (an ExternalSecret, a Vault annotation, a CSI SecretProviderClass) and something in the cluster resolves it. That is also the pattern that makes secret rotation independent of a Helm deploy, which matters the first time you must rotate a leaked credential at 2 a.m. and do not want to redeploy forty charts to do it. See the Vault guide for the injection patterns.

{{- if .Values.externalSecrets.enabled }}
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: {{ include "web.fullname" . }}
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: {{ required "externalSecrets.store is required" .Values.externalSecrets.store }}
  target:
    name: {{ include "web.fullname" . }}-db
  dataFrom:
    - extract:
        key: {{ required "externalSecrets.key is required" .Values.externalSecrets.key }}
{{- end }}

(Check the apiVersion against the External Secrets Operator release installed in your cluster.) The chart renders no secret value, so helm get values, the release Secret and helm template output are all clean.

Some hygiene that applies whatever the pattern:

• helm install --dry-run --hide-secret hides Secret contents in the rendered output, which is what you want in CI logs. A dry run without it prints every Secret manifest.
• helm get values prints whatever was passed, unmasked. Treat access to it as access to the secret.
• If a secret did reach a release record, deleting the release revision is not remediation. Rotate it. Pruned history is not a guarantee that etcd backups, snapshots or a log pipeline never saw it.
• Do not template a Secret with randAlphaNum and expect stability. Every upgrade regenerates it unless you guard with lookup, and lookup returns nothing under helm template and client dry runs (it needs --dry-run=server), so the password changes on the next GitOps render. Use an operator to generate.

Supply chain: charts, registries and provenance

A chart is executable configuration from somewhere else. It arrives from a repository or registry, it may pull subcharts from other places, and the containers it references arrive from yet another. Every hop is a trust decision, and Helm gives you a few tools for making them verifiable.

OCI and immutable references

The modern distribution channel is an OCI registry. A chart is pushed with helm push and referenced as oci://registry.example.com/charts/web. The critical property for a platform is immutability: a tag like 1.4.0 is a name, and registries can be configured, or misconfigured, to let someone re-push it. A digest is content.

helm registry login registry.example.com -u ci --password-stdin < token.txt
helm package ./web                                   # web-1.4.0.tgz
helm push web-1.4.0.tgz oci://registry.example.com/charts

# Resolve the digest once, at promotion time
helm pull oci://registry.example.com/charts/web --version 1.4.0 --untar=false

# Helm 4: install by digest, which cannot be re-pointed
helm install web oci://registry.example.com/charts/web@sha256:<digest> -n web

Two Helm 4 details bite here. helm registry login takes a bare host[:port], and a pipeline that passes oci://registry.example.com or an https:// URL fails. And helm push takes a packaged .tgz with a reference that excludes the chart name, because the repository name comes from the chart and the tag comes from the chart's version. Enforce version bumps with a CI check that refuses to push a chart whose version already exists in the registry, and turn on tag immutability in the registry itself, which every major registry (ECR, ACR, GAR, GHCR, Harbor, Artifactory) offers in some form.

The promotion model that follows from this: CI builds and pushes web:1.4.0 to a staging registry, records the digest, and promotion to production copies that digest, never rebuilds. Deployments reference the digest, or reference a version that a controller has pinned to a digest, and a human-readable tag is for people only. That is the same build-once, promote-the-artifact discipline as container images, described in the Docker guide.

Provenance, signatures and verification

Helm's built-in integrity mechanism is provenance: a .prov file containing a PGP-signed copy of Chart.yaml plus the SHA-256 digest of the archive.

helm package ./web --sign --key 'Platform Team' --keyring ~/.gnupg/secring.gpg
helm verify web-1.4.0.tgz --keyring pubring.gpg
helm install web ./web-1.4.0.tgz --verify --keyring pubring.gpg

On OCI the .prov file is pushed as an extra layer, and helm pull --prov retrieves it. Helm 4.3 also accepts GnuPG keybox (pubring.kbx) keyrings, which matters because modern GnuPG no longer creates the old pubring.gpg format by default. Provenance proves that a chart was signed by a key you trust and has not been modified since; it does not prove the chart is safe, only that it is the one the signer produced.

The operational weakness of PGP is key management: distributing public keys to every verifier, rotating them, and revoking. Many platform teams therefore sign the OCI artifact with cosign (Sigstore), using keyless signing tied to the CI system's identity, and verify at the admission layer or in the deploy job:

cosign sign registry.example.com/charts/web@sha256:<digest>
cosign verify registry.example.com/charts/web@sha256:<digest> \
  --certificate-identity-regexp 'https://github.com/acme/charts/.*' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com

Whichever you pick, the policy decision is the same, and it is worth stating plainly: verification that can be skipped is decoration. A deploy job that verifies and a developer laptop that does not is one bypass away from no control. Make the pipeline the only path to production, and verify there.

Dependencies and the chart itself

Subcharts are the quieter risk. A Chart.yaml dependency pointing at https://some-repo.example.com with a version range of ^2.0.0 will resolve to a different chart next month. Pin exact versions (or tilde ranges you review), commit Chart.lock, use helm dependency build in CI so it uses the lock rather than re-resolving, and vendor or mirror third-party charts into your own registry. Mirroring also protects you against upstream disappearance. The Bitnami chart and image catalogue changes in 2025 left many teams with broken installs overnight, and treating any external chart as a dependency you must be able to rebuild without the upstream being up is the lesson.

Plugins and post-renderers

Plugins extend Helm, and in Helm 4 they are a proper extension system with a proper trust story. This is worth understanding as a platform owner because plugins are the one place where Helm runs code on developer and CI machines that did not come from the chart.

Helm 4 redesigned plugins (HIP-0026). A plugin now has a plugin.yaml with apiVersion: v1, a type and a runtime. There are three types: cli/v1 adds subcommands, getter/v1 adds download protocols such as s3://, and postrenderer/v1 transforms rendered manifests. There are two runtimes: subprocess, which runs a command on the host, and extism/v1, which runs WebAssembly in a sandbox. Legacy Helm 3 plugin.yaml files, with no apiVersion and top-level usage and command, still load, so most existing plugins continue to work during migration.

The security change that matters: helm plugin install --verify defaults to true. Helm 4 expects plugins to be signed and will refuse an unsigned plugin installed from a URL or VCS unless you pass --verify=false. That default turns plugin installation from "run whatever is at this URL" into a decision you must consciously override.

# Verified by default in Helm 4; fails for an unsigned plugin
helm plugin install https://github.com/databus23/helm-diff

# A deliberate override for a source you trust, recorded in your runbook
helm plugin install https://github.com/databus23/helm-diff --verify=false

# Plugins can also come from OCI
helm plugin install oci://ghcr.io/acme/plugins/deploy-policy:1.0.0
helm plugin list

For an organisation, the plugin policy I would write is short. Maintain an approved list (helm-diff, helm-unittest, helm-secrets and your own). Install plugins in CI images at build time, at pinned versions, not at run time. Sign your own plugins with helm plugin package (signing is on by default) and verify them with helm plugin verify. Prefer the Wasm runtime for anything third-party that does not need to execute host binaries, because its runtimeConfig lets you state what it may touch: allowedHosts (none by default), a temp directory, memory, a timeout, and named host functions.

apiVersion: v1
name: "deploy-policy"
version: "1.0.0"
type: cli/v1
runtime: subprocess
config:
  usage: deploy-policy [chart]
  shortHelp: "check a chart against org policy"
  longHelp: |-
    Renders a chart and runs the organisation's policy checks.
  ignoreFlags: false
runtimeConfig:
  platformCommand:
    - os: linux
      command: "sh"
      args: ["-c", "${HELM_PLUGIN_DIR}/check.sh"]

Post-renderers

A post-renderer receives the fully rendered manifests on standard input and writes the modified manifests to standard output. Its classic use is Kustomize: patching a third-party chart without forking it. In Helm 3 you passed --post-renderer ./my-script.sh. In Helm 4 --post-renderer takes the name of an installed postrenderer/v1 plugin, and --post-renderer-args passes arguments (repeat it for several). Every Helm 3 script workflow must be wrapped in a plugin.

apiVersion: v1
name: "kustomize-pr"
version: "1.2.3"
type: postrenderer/v1
runtime: subprocess
runtimeConfig:
  platformCommand:
    - command: "${HELM_PLUGIN_DIR}/kustomize.sh"

helm upgrade --install my-web ./chart -n web \
  --post-renderer kustomize-pr --post-renderer-args=--overlay=prod

Four rules keep post-renderers from becoming a liability:

1. Everyone who touches the release must use the same post-renderer. Post-rendering happens on the client at deploy time, so a teammate who upgrades the same release without it sends the unpatched manifests and quietly reverts your changes.
2. Post-rendered output is invisible to helm template unless you pass the same flags, so CI scans and humans reviewing diffs must render the same way.
3. The SDK has a PostRenderStrategy option controlling how hooks are passed to the renderer: combined (default), separate, or nohooks (the Helm 3 behaviour). It is SDK-only with no CLI flag, so this matters to you only if a platform tool embeds Helm.
4. A post-renderer is a code path with write access to everything that is about to be applied. Review it as you would any deployment code, and keep it in the same repository as the chart it patches.

Multi-tenancy

Helm's tenancy unit is the namespace. Release names are unique per namespace, release records live in the release's namespace, and RBAC is easiest to express per namespace, so the natural design is one or more namespaces per team, a service account per namespace for deploys, and nothing shared except what the platform team owns. That design works well until a chart wants something cluster-scoped.

Cluster-scoped objects are the tenancy boundary that charts cross without announcing it. A CustomResourceDefinition, a ClusterRole or ClusterRoleBinding, a ValidatingWebhookConfiguration, a PriorityClass, an IngressClass: any of them, created by a tenant's chart, affects every other tenant. A mutating webhook with failurePolicy: Fail whose backing service is down can stop pods being created cluster-wide. A CRD upgraded by one team changes the schema another team's resources are validated against.

The rule that keeps tenancy intact is a split of ownership:

Enforce it mechanically. Tenant deploy identities have no cluster-scoped permissions, so a chart that tries to create a ClusterRole fails at install with a forbidden error, and that failure is the control working. Put a chart-lint rule in the golden pipeline that fails any chart containing cluster-scoped kinds unless it is in the platform repository.

Other tenancy details that catch people:

• helm list -A is a privileged query. It requires list on Secrets in every namespace. Give tenants namespaced helm list only.
• Name collisions are per-namespace only, but cluster-scoped objects a chart names with {{ .Release.Name }} can collide across namespaces, because a ClusterRole has no namespace. Two teams installing releases both called web that each render a ClusterRole named web will clash. The ownership check catches it (the error reads exists and cannot be imported into the current release, with a meta.helm.sh/release-namespace mismatch), but a platform that prevents cluster-scoped objects never meets it.
• Quotas and limit ranges live outside Helm. A ResourceQuota in the namespace is what stops a chart that requests 200 replicas. Helm will apply it and the API server will refuse to schedule it, so the release fails under watcher rather than at the apply.
• Impersonation (--kube-as-user, --kube-as-group) lets a central pipeline deploy as a tenant, which is useful for audit: the API server's audit log shows the impersonated identity and the impersonator.
• Adoption across tenants is dangerous. --take-ownership lets a release adopt objects that exist without Helm's ownership metadata. In a shared cluster that is a way for one team's chart to take over an object that belongs to another. Reserve it for migration work done by the platform team.

Since Helm 4.3, uninstall also verifies ownership before deleting: a resource that lacks matching Helm ownership metadata is skipped with the warning skipping delete of resource not owned by this release. That closes one multi-tenant hazard, a release deleting something another release or a human had taken over. Read the warning as information and clean up by hand; do not treat it as noise.

Chart architecture for other teams

If you run the platform, you also own the shape of the charts. The decisions here determine whether forty teams write forty slightly different charts or consume one well-tested one, and they determine how a change in a shared convention reaches the fleet. There are three design axes: how services are packaged, how conventions are shared, and how the chart's interface is protected.

One chart per service, or an umbrella?

An umbrella chart has no templates of its own and lists several subcharts as dependencies, so one helm install deploys a whole system. A per-service chart packages one service with its own release, history and versioning.

The umbrella's costs are the ones from the scaling section: every subchart renders on every upgrade, one failing subchart fails the whole release, the release record holds all of it, and rollback reverts everything, including services that were fine. The common compromise is per-service charts for production, with a thin umbrella (or a Helmfile, or an Argo CD ApplicationSet) that composes them for development and preview environments. Conditions and tags (condition: postgresql.enabled) let an umbrella toggle parts, and conditions override tags.

Library charts and the golden chart

A library chart (type: library) contains only named templates and cannot be installed. It is how you distribute conventions: a helper that renders a Deployment with your security context, probes, labels and resource defaults; a helper for ExternalSecret; a helper for a ServiceMonitor. Service charts depend on it and call its templates.

apiVersion: v2
name: web
version: 1.4.0
appVersion: "2.3.1"
type: application
kubeVersion: ">=1.34.0-0"
dependencies:
  - name: platform-lib
    version: "3.2.1"
    repository: oci://registry.example.com/charts

{{- include "platform-lib.deployment" (dict "root" . "values" .Values) }}

Library charts give you leverage and a new failure mode. A change to the library reaches every chart that depends on it, on the next version bump, which is the point and also the risk. Version the library with SemVer, treat any change to rendered output for the same inputs as a breaking change, and snapshot-test it (see the testing section) so that "we did not change what renders" is a fact you can check.

The alternative to a library is a golden chart: one generic, well-tested application chart that accepts a rich values file, so most teams never write a template at all. It trades flexibility for consistency, and it works well when services are similar (web APIs, workers, cron jobs) and poorly when they are not. Many platforms run both: a golden chart for the eighty percent, and library helpers for teams whose services are the exception.

The values file as an API

Once other teams write values files against your chart, the values are a public interface, and you are subject to all the usual rules of interface design. Keep them small and semantic (ingress.host, not ingress.annotations.nginx.ingress.kubernetes.io/rewrite-target). Deprecate before removing. And protect them with a schema.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "required": ["image", "resources"],
  "properties": {
    "replicaCount": { "type": "integer", "minimum": 1, "maximum": 50 },
    "image": {
      "type": "object",
      "required": ["repository", "tag"],
      "properties": {
        "repository": { "type": "string", "pattern": "^registry\\.example\\.com/" },
        "tag": { "type": "string", "minLength": 1 }
      }
    },
    "resources": {
      "type": "object",
      "required": ["requests", "limits"]
    }
  },
  "additionalProperties": false
}

values.schema.json validates the merged values on install, upgrade, lint and template, and Helm 4 supports JSON Schema 2020-12. A violation fails with values don't meet the specifications of the schema(s) in the following chart(s), which is exactly what you want at pull-request time rather than at 2 a.m. The schema above does three jobs for a platform: it enforces that every release sets resource requests and limits (which is a cost control), it restricts images to your registry, and with additionalProperties: false it turns a typo like replicacount into an error instead of a silently ignored value. --skip-schema-validation exists as an escape hatch (and works on helm lint as of Helm 4); audit its use in pipelines.

CRDs: the lifecycle Helm deliberately does not manage

Helm's rule for the crds/ directory is unusual and worth stating exactly. CRDs placed there are plain, non-templated YAML. Helm installs them before other resources, only on first install, and skips them if they already exist. It never upgrades or deletes them. --skip-crds skips the step, and helm template --include-crds renders them.

The reasoning is caution: deleting a CRD deletes every custom resource of that kind, cluster-wide, so Helm refuses to do it as a side effect of removing a chart. The consequence is that CRD upgrades are yours to manage. The strategies, from least to most robust:

1. crds/ only and apply upgrades with kubectl apply --server-side as a separate pipeline step. Simple and explicit.
2. CRDs in templates/, so Helm upgrades them. This works but uses the release record and makes helm uninstall delete them unless you annotate helm.sh/resource-policy: keep. Data loss on uninstall is the risk.
3. A separate CRD chart, owned by the platform team, released before the application charts. This is the cleanest for shared operators, and the pattern most large projects adopt.

Whichever you choose, write it down, because the failure (an operator new version expecting fields the stale CRD lacks) presents as confusing validation errors weeks after the chart upgrade that should have handled it.

Testing and gating charts in CI

A platform team's charts are code that other teams depend on, so they get tested like code. The useful layers, from cheapest to most expensive:

1. Lint. helm lint --strict --with-subcharts. In Helm 4 lint checks the crds/ directory and no longer ignores *.yml files. Fast, and catches structural mistakes.
2. Render matrix. helm template against every values file the chart supports (defaults, each environment, each feature toggle), with --kube-version and --api-versions matching your clusters. A template that renders only with default values and fails with ingress.enabled=true is the classic bug this finds.
3. Snapshot tests. Commit the rendered output for a set of inputs and diff it on every change. The helm-unittest plugin does this declaratively. For a library chart, this is the test that turns "did I change the output?" into a pull-request diff.
4. Policy and scan on the rendered output. Trivy config scan, checkov or kube-linter, plus conftest or Kyverno CLI for organisation rules.
5. Server-side dry run. helm install --dry-run=server (or helm template --dry-run=server) sends the render to a real API server, which validates against the actual CRDs and admission webhooks without persisting. It also makes lookup work. Helm 4 deprecates helm template --validate in favour of this.
6. Install and test in a kind cluster. chart-testing (ct lint, ct install) installs each changed chart into a throwaway cluster and runs helm test.
7. A diff in the pull request. The helm-diff plugin shows what the upgrade will change against the running release, which is the review artefact humans should read.

# The pull-request pipeline, in order
helm lint ./web --strict --with-subcharts
for f in ci/values-*.yaml; do
  helm template web ./web -f "$f" --kube-version 1.34.0 \
    --api-versions monitoring.coreos.com/v1 > "out/$(basename "$f")"
done
trivy config out/
helm install web ./web -n ci-test --create-namespace --dry-run=server -f ci/values-prod.yaml
helm diff upgrade web ./web -n prod -f ci/values-prod.yaml   # needs the helm-diff plugin and read access

Two subtleties that decide whether these tests are trustworthy. First, render with the same inputs production will use: a scan of a render with default values tells you about the defaults. Second, helm diff compares against the stored manifest of the release (and, with the plugin's options, the live state), so a diff against a drifted cluster can be larger or smaller than the change you think you are making. Read it as "what will this upgrade do to what is there", not "what did this pull request change".

The post-deploy half is helm test, which runs hook resources annotated helm.sh/hook: test. Keep them as cheap smoke tests (a request to the service, a check that a migration completed), and run them after the deploy in the pipeline with --logs so failures carry the pod output. Helm 4 removed --create-pods from helm test, so use standard Pod or Job manifests.

Upgrades and migrations

There are three kinds of upgrade you own, and each needs its own plan: Helm itself, the charts you ship, and the Kubernetes clusters underneath.

Moving from Helm 3 to Helm 4

The dates set the pace. Helm 3's last feature release was 3.22.0 and security fixes end on 2027-02-10, three months later than the originally announced date. After that there are no patches at all, and a CVE in Helm 3 stays open. Helm 4 went GA on 2025-11-12, and its current release is 4.3.0 (2026-09-09), with 4.3.1 scheduled for 2026-10-14 and 4.4.0 for 2027-01-13. Plan to be fully on Helm 4 with margin before February.

The compatibility promise is generous where it matters most. Charts with apiVersion: v2 keep working. Release storage is still sh.helm.release.v1, so Helm 4 manages existing Helm 3 releases in place and, unlike the Helm 2 to 3 change, no migration tool is needed. What changed is the client: flags, output, plugins and the Go SDK. So the migration is a change to your pipelines, scripts and tooling, not to your clusters.

A migration that works in practice, in the order I would run it:

1. Inventory. Grep every pipeline, Makefile, script and wrapper for helm. List the flags in use. Find every --post-renderer, every helm registry login, every --atomic, and every plugin installed in a CI image.
2. Run both clients in parallel in a non-production pipeline. Compare helm template output from 3.22 and 4.3 for each chart and each values file. The diff should be empty or explainable.
3. Rename the flags in the same change that switches the client. --rollback-on-failure and --force-replace do not exist in Helm 3, so they cannot be adopted earlier. Helm 4 still accepts the old names and prints a deprecation warning, so the warnings also find the stragglers you missed.
4. Convert post-renderers to plugins and fix registry logins.
5. Switch non-production first. Existing releases latch to client-side apply, so behaviour should match. Watch for wait behaviour changes (the watcher strictness) and plugin verification failures.
6. Switch production, then migrate releases to server-side apply deliberately, batch by batch, as described earlier.
7. Remove Helm 3 from CI images and laptops' baseline tooling, and set a version-check in the golden pipeline so nothing quietly runs an old client.

Two more upgrade facts matter for the platform. The Kubernetes version skew policy changed: Helm 4 is assumed to work with Kubernetes versions within n-3 of the one it was compiled against (Helm 4.3.x with Kubernetes 1.34 to 1.37), and using Helm against a Kubernetes version newer than the one it was compiled against is not recommended because there is no forward-compatibility guarantee. Pin the client version in CI and tie its update to your cluster upgrade calendar. Second, Helm 4 introduces an experimental chart API v3 behind HELM_EXPERIMENTAL_CHART_V3=1. It is internal and subject to change. Do not build platform standards on it; watch it and keep apiVersion: v2 until it is declared stable.

Upgrading charts and apps

Chart upgrades are where most production incidents actually come from. The disciplines that prevent them:

• Bump the chart version on every change, however small, and keep appVersion for information.
• Never use --reuse-values in automation. It merges the last release's values with new overrides and ignores new chart defaults, so a new default never arrives and a removed key may linger. Pass a complete values file from Git on every upgrade, or use --reset-then-reuse-values when you truly must combine the new defaults, last values and overrides.
• Upgrade CRDs as a separate, earlier step.
• Require a helm-diff output on every production change.
• Write PodDisruptionBudgets, readiness probes and preStop hooks into the chart, so an upgrade does not drop traffic. A rolling update with no readiness probe is an outage with a polite name.
• For hooks that run migrations, make them idempotent and set helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded so a stale Job from the last run does not block this one. Remember that hook resources are not managed as part of the release, so they are not cleaned up by an uninstall unless a delete policy does it.

Upgrading Kubernetes under your releases

A Kubernetes upgrade can remove an API version that your stored releases reference. The release record contains the rendered manifest with the API versions used at install time, so a future helm upgrade or helm rollback may try to reason about objects of a kind the cluster no longer serves, and fails with resource mapping not found ... no matches for kind "PodSecurityPolicy" in version "policy/v1beta1" or similar. The fix is to update the templates, and for records already written, the helm-mapkubeapis plugin rewrites stored manifests to supported API versions. Run it before the cluster upgrade, not after the first failed deploy. Choose APIs in templates with .Capabilities.APIVersions.Has, and keep the kubeVersion constraint in Chart.yaml honest, remembering the -0 suffix (>=1.34.0-0) so managed-cluster pre-release version strings such as v1.34.2-eks-... satisfy it.

Cost

Helm itself is free and Apache-2.0 licensed, and it has no run-time footprint of its own. The costs that land on a platform owner are indirect, and they are worth naming because they are rarely on anyone's dashboard.

The one worth emphasising is the fourth. A chart's values.yaml is where someone decided that a default deployment gets three replicas with 2 CPUs each, and that default is multiplied by every team that does not override it. Changing a default in a shared chart is a spending decision for the whole organisation, so treat such changes as reviewable events. A schema that requires teams to set requests and limits explicitly (and a policy that rejects pods without them) turns cost from an emergent property into a decision. Tag releases with --labels (for owner and cost centre) so helm list -l owner=team-a answers "what does this team run", and propagate the same labels into the workloads, where your cost tooling reads them.

For organisations in the Gulf and Egypt with data-residency requirements, there is one specific consideration. A chart registry is also a place where artifacts live and are replicated. If charts contain environment-specific configuration or internal endpoints, check where your registry stores and replicates them, and prefer a regional registry or one with replication you control. The release records themselves never leave the cluster you deploy to, which is usually the simplest residency story of any tool in the stack.

Governance: making the safe path the default

Governance for a tool like Helm is mostly about deciding what is a convention, what is checked, and what is impossible. The strongest controls are the ones nobody can skip.

A workable governance model has four layers:

1. Impossible. Enforced by the cluster, regardless of tool: tenant identities cannot create cluster-scoped resources, admission refuses privileged pods and unapproved registries, and namespaced RBAC limits who can touch what. People cannot override these by editing a chart.
2. Checked in CI. Enforced by the pipeline, which is the only route to production: lint, render matrix, schema, scan, policy, diff, signature verification, version-bump check. These can be bypassed only by someone who controls the pipeline, which is a short and auditable list.
3. Defaulted. Provided by the golden chart and library: security contexts, probes, PodDisruptionBudgets, labels, resource defaults, ExternalSecret helpers. Most teams never think about these, which is exactly why they work.
4. Documented. Everything else: naming, ownership, deprecation policy, the exceptions process.

Put the policy in the layer that matches its risk. A rule whose violation causes a breach belongs in layer 1. A rule whose violation causes inconsistency belongs in layer 3 or 4. Teams get frustrated, and start working around a platform, when a cosmetic convention is enforced like a security control or a real security control is left as a wiki page.

Some concrete governance artefacts that pay off:

• A chart registry of record, with tag immutability, retention and access control, and a rule that production installs come only from it.
• An exceptions register. Every --skip-schema-validation, --take-ownership, --force-conflicts, --verify=false and --wait=legacy is an exception to a default. You do not need to forbid them; you need to know where they are and why. A grep in CI that requires a linked ticket in a comment is enough.
• Release ownership metadata. Use --labels or chart annotations to record team, on-call channel and tier. Alerts on a wedged release should page the owning team, and they can only do that if the release says who the owner is.
• A deprecation policy for the chart's values interface, with a window (for example two minor versions) during which old keys still work and emit a warning through fail or a NOTES message.
• Audit. Kubernetes audit logs record every Secret write that Helm makes, including impersonated identities, which gives you a complete record of who deployed what and when, without any Helm-side audit feature. Make sure audit logging is on and retained.

Incident playbooks

Most Helm incidents are one of a handful of shapes. What follows are the playbooks, each beginning with how to recognise it. Keep them in the on-call runbook; the commands are the fast path, and the reasoning is what lets you adapt when the situation is slightly different.

A release is stuck in a pending state

Recognise it: Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress, and helm history shows the last revision as pending-install, pending-upgrade or pending-rollback.

Cause: A Helm process died between marking the revision pending and finishing. Nothing will ever clear it.

helm history my-web -n web
kubectl get secret -n web -l owner=helm,name=my-web --show-labels

Fix, in order of preference: first confirm no pipeline or controller is still running a deploy for this release. If the last good revision still exists, helm rollback my-web <last-good-rev> -n web --description "recover from stuck pending". If rollback is refused because the pending state blocks it, as a last resort delete the pending revision's Secret (sh.helm.release.v1.my-web.vN) with kubectl, then retry. Deleting a record is deleting Helm's memory of that revision, so compare helm get manifest and the live state afterwards. Prevent it with a CI timeout longer than --timeout and non-cancellable deploy steps.

Upgrade failed, and the release has no deployed revision

Recognise it: Error: UPGRADE FAILED: "my-web" has no deployed releases.

Cause: Every revision of the release is failed or pending, typically because the first install failed and was left behind.

Fix: if nothing of value is running, helm uninstall my-web -n web and reinstall. If there is a good earlier revision, roll back to it. Prevent it with --rollback-on-failure on installs, which removes a failed first install so the next attempt starts clean.

The release record is too large

Recognise it: The apply succeeds and the upgrade then fails with Secret "sh.helm.release.vN..." is invalid: data: Too long.

Cause: The chart, the rendered manifest or a large embedded file exceeds the object size limit.

Fix: Do not retry blindly; the cluster changed. Compare helm get manifest (the last recorded state) against the live objects to see what is now unrecorded. Reduce the chart: drop .Files bulk, move CRDs to a separate chart, split the release. If the record genuinely cannot shrink, consider the SQL driver with a plan for the credentials it needs. Prevent it with a CI check that measures the compressed size of helm template output against a budget.

A deploy fails on ownership conflict

Recognise it: unable to continue with install: ConfigMap "foo" in namespace "web" exists and cannot be imported into the current release: invalid ownership metadata, with the missing label app.kubernetes.io/managed-by and the meta.helm.sh/release-name and meta.helm.sh/release-namespace annotations listed.

Cause: The object exists and Helm did not create it: created by hand, by another tool, or by another release (the message then says must equal "a": current value is "b").

Fix: If the object should belong to this release, adopt it with --take-ownership or add the label and annotations with kubectl. If it belongs to another release, stop: two releases should not own one object, so remove it from one chart. Prevent it by making the platform the only creator of objects in a namespace, and by using --take-ownership only in reviewed migrations.

A server-side apply conflict

Recognise it: Apply failed with N conflicts: conflict with "<manager>" using apps/v1: .spec.replicas.

Fix: Identify the manager (kubectl get ... --show-managed-fields). If it is an autoscaler or controller, remove the field from the chart. If it is a human, decide whether their change should persist. Only then consider --force-conflicts. Prevent it by giving every field exactly one owner in your chart design.

The wait times out while everything looks healthy

Recognise it: context deadline exceeded or resource not ready, name: ... after the pods are actually up.

Cause: Under watcher, a resource that does not report kstatus conditions never becomes ready; or a hook Job is slow; or the timeout is shorter than a real startup (image pull on a cold node, a model loading into GPU memory).

Fix: Identify which resource Helm is waiting on from the message. Raise --timeout if the startup is legitimately slow. For resources without kstatus support, use --wait=legacy for that chart or a separate readiness check. Prevent it by testing the wait strategy in staging with a cold node.

Clean-up removed something it should not have

Recognise it: An uninstall or a failed-install cleanup deleted objects that teams depend on, or a Helm 4.3 uninstall printed skipping delete of resource not owned by this release.

Fix: Restore from Git (the chart renders the objects) or from the last release record (helm get manifest my-web --revision N | kubectl apply -f -). For things that must survive, annotate helm.sh/resource-policy: keep, which leaves the object in place on uninstall and upgrade as an orphan. Prevent it by keeping persistent state (PVCs, CRDs with data, namespaces) out of application charts or protected with the keep policy.

Where Helm stops

Helm is a package manager with a release history. It is excellent at that. It is not a deployment orchestrator, not a configuration management system, and not a reconciler, and trying to stretch it into those roles is the source of most of the pain people attribute to it.

Templating YAML with a text engine is the fundamental limit. Go templates operate on text, so indentation, quoting and type coercion are constant sources of bugs, and there is no type checking beyond the JSON schema. Charts with deep conditional logic become unreadable. That is a reason to keep charts simple, push logic into the application or an operator, and, for genuinely programmatic configuration, consider a typed tool (cdk8s, Pulumi, Jsonnet or CUE). Helm 4's additions (more template functions, multi-document values files, schema 2020) help at the margin but do not change the nature of the tool.

Kustomize overlays plain YAML without templates, which suits manifests you own and environment differences that are patches. It cannot package and version a chart for outsiders. The two compose: a Helm chart as a base and Kustomize as a post-renderer (or in Flux and Argo CD) for last-mile patches. See the Kustomize guide.

Terraform and the helm_release resource are a fine way to deploy a small number of platform components from the same pipeline that creates the cluster, and a poor way to deploy hundreds of application releases, because every deploy then requires a Terraform plan over a large state. Deploy the platform addons with Terraform and let application teams deploy with the GitOps or CI path. The Terraform guide covers the provider; note that it records its own view of the release alongside Helm's.

Operators are the answer when the thing being deployed needs day-2 logic: backups, failover, schema migrations on version changes. A chart can install an operator, but a chart cannot run a failover. If your templates/ is full of Jobs and hooks that emulate an operator, that is the signal.

GitOps controllers are the answer to reconciliation, and they are complements, not replacements. The question to settle is whether you need real Helm releases (helm list works, helm rollback works, hooks run natively: Flux) or a rendered-manifest model with its own diff and sync engine (Argo CD), because the answer shapes your runbooks and your incident tooling.

Running Helm as a platform

Everything so far adds up to a service you operate. What does the platform offering actually look like? I would build it as five deliverables, because a platform that is only a tool is a wiki, and a wiki does not page anyone.

1. The paved road. A golden chart (and library) that gives a team a working, secure, observable deployment for a typical service from a five-line values file. The measure of success is the number of teams who never wrote a template. Publish it to the registry of record, version it with SemVer, keep a changelog, and announce breaking changes with a deprecation window.

2. The pipeline. A reusable CI job, owned by the platform team, that does lint, render matrix, schema, scan, diff, signature verification, deploy with the safe flags, and test. Teams include it; they do not copy it. This is where the Helm version is pinned, where the serialisation and non-cancellable step live, and where exceptions are checked.

3. The guardrails in the cluster. Namespaces with quotas, network policies and deployer Roles created by the platform; admission policies for content; tag-immutable registries; audit logging. Teams get an isolated place to deploy, and the isolation does not rely on their charts being good.

4. The observability. Release health as a fleet view: counts of failed and pending-* releases by owner, release age, chart-version distribution (how many teams are on an out-of-support chart version), and client-version distribution in CI. Since Helm emits no metrics, you assemble these from release Secrets, helm list -o json, your GitOps controller's metrics and the registry. The second and third are the ones that predict trouble. A fleet where a third of services are three minor versions behind the golden chart is a fleet whose next security fix takes a quarter to land.

5. The support model. A channel, an on-call rotation for the pipeline and the registry, a runbook containing the playbooks above, a documented exceptions process, and a regular review of upgrade lag. Platform work that nobody is on call for decays into a legacy system within a year.

A last point on posture. The platform team's failure mode is becoming a gate rather than a road. Every control above should be measured by how much time it saves teams, not by how many deploys it blocks, and the exceptions process exists so the platform can say "yes, here is how" rather than "no". When the safe route is the fastest route, compliance stops being a topic.

The review checklist

Use this for pull requests that change a chart, a pipeline or a values file, and for design documents that propose a new platform capability. It is ordered the way failures occur, not the way a chart is laid out.

Chart and values

• Is the chart version bumped, and does appVersion match the image?
• Does values.schema.json require what must be set (resources, image) and reject unknown keys?
• Does the chart render under every supported values file, with the target --kube-version and --api-versions?
• Does it contain any cluster-scoped kinds? If so, should it be in the platform repository?
• Are CRDs handled by an explicit strategy (crds/, separate chart, or templated with a keep policy)?
• Are replicas and any other fields owned by an HPA or controller left out of the template?
• Are resource requests and limits set, along with probes, a PodDisruptionBudget and a security context?
• Do ConfigMap and Secret changes roll the pods (checksum/config annotations)?