The Complete Flux Guide

This is part one of three. It covers everything you need to do real work with Flux, not a teaser. By the end you can install Flux on a cluster, point it at a Git repository, and watch it deploy an application, a Helm chart and a change you made with nothing but git push. You will know how to read what Flux is telling you when something is wrong, how to pause it safely, and how to tear it all down without leaving debris. Mid-level and Senior take the same topics further; nothing here is thrown away.

The guide is written against Flux v2.9.5 (released 31 August 2026), which is the current stable release at the time of writing. Flux tutorials age quickly, because several API versions were removed in 2.7, 2.8 and 2.9, and a large share of the YAML you will find on blog posts and forums no longer works. Every manifest in this guide uses the current versions, and a later section explains how to recognise an outdated one.

Each section ends with a Try it task. Do them as you go. They take a few minutes each, and GitOps only makes sense once you have watched a cluster change itself because of a commit you made.

What Flux is, and the problem it solves

Flux is a set of programs that run inside a Kubernetes cluster and keep the cluster matching a description stored somewhere else, usually a Git repository. You write down what should exist (a Deployment, a Service, a Helm release, a namespace), commit it, and Flux notices the commit, applies it, and then keeps checking that the cluster still looks the way the repository says it should. If someone edits the cluster by hand, Flux puts it back. If you delete a file from the repository, Flux deletes the object from the cluster.

This practice has a name, GitOps, and Flux is one of its two best-known implementations (the other is Argo CD, covered in its own guide). Flux is a graduated project of the Cloud Native Computing Foundation, the same foundation that hosts Kubernetes, and it is free and open source.

The diagram is the whole idea. To see why it matters, consider what deploying to Kubernetes looks like without it.

Without Flux you deploy by running kubectl apply -f deployment.yaml from your laptop, or by letting a continuous integration job run the same command. That works on the first day. By the third month the cluster contains objects nobody remembers creating, a hotfix somebody applied directly during an incident that never made it back into the repository, and three different people each believing their copy of a YAML file is the current one. Nobody can answer the simple question "what exactly is running in production, and who changed it?" without detective work.

Flux answers that question structurally. There is one place where the desired state lives, Git, and Git already records who changed what, when, and (if your team writes decent commit messages) why. A rollback becomes git revert. A review becomes a pull request. An audit becomes git log. The cluster stops being a place where people do things and becomes a place where a repository is reflected.

Three consequences of that design explain most of what follows, so notice them now.

Flux pulls, it does not push. A CI job that runs kubectl apply needs credentials to your cluster, stored in your CI system, reachable from wherever the CI runs. Flux inverts this. The controllers live inside the cluster and reach out to Git. The cluster needs only outbound network access to your repository and your container registries. No credential that can change the cluster ever leaves the cluster. For teams whose employers care about where credentials live and which networks can talk to a production environment, which includes many banks, telcos and government-linked employers across the Gulf and Egypt, that is a selling point in its own right.

Flux never stops checking. A deployment script runs once and exits. Flux runs forever. Every few minutes it compares the repository with the cluster again. That is called reconciliation, and it is the word you will see more than any other in Flux documentation and output.

Flux is made of Kubernetes objects. You do not configure Flux through a web form or a config file on a server. You configure it by creating Kubernetes custom resources, which you write as YAML and store in the same Git repository as everything else. Even Flux itself is described in Git, and after installation it manages its own upgrades the same way it manages your applications.

What people use it for:

You need little to follow along: a terminal, kubectl, a Kubernetes cluster you are allowed to break (a local one is perfect) and a GitHub account. We build everything from scratch.

Push versus pull: what came before

It helps to know what Flux replaced, because the old ways still exist in most organisations you will join, and the word "GitOps" is sometimes used loosely for things that are not.

The oldest approach is manual kubectl apply. It is fine for learning and for a one-person prototype. It has no memory and no review.

The next step up is push-based continuous delivery: a pipeline in GitHub Actions, GitLab CI, Jenkins or similar builds your code, and then a final step runs kubectl apply or helm upgrade against the cluster. This is better, because the pipeline definition is in Git and the deploy is repeatable. But it has three weaknesses that Flux was built to remove.

First, the pipeline holds credentials to the cluster. Anyone who can change the pipeline, or who compromises the CI system, can change production. Second, the pipeline runs once. After it finishes, nothing watches the cluster. A colleague who edits a Deployment by hand at 2 a.m. creates a difference that nobody sees until it causes an outage. Third, the pipeline does not know about deletion. If you remove a manifest from the repository, the old object simply stays in the cluster forever, because kubectl apply only creates and updates what you give it.

The original Flux v1 took the pull-based approach, but it was a single monolithic program with its own annotation conventions and limited extensibility. It has been archived, and you should ignore any tutorial that mentions fluxctl or a HelmOperator. Flux v2, which this guide covers and which everyone now just calls Flux, is a rewrite built as a collection of small controllers that cooperate through Kubernetes custom resources. The project calls this collection the GitOps Toolkit, and you will see that name inside resource names such as gotk-components.yaml and in metric names beginning with gotk_. Wherever this guide says "Flux", it means Flux v2.

Flux does not replace your CI system, it changes what CI is for. CI still builds your code, runs tests and pushes a container image. What it stops doing is talking to the cluster. The hand-off happens through Git (or through an OCI registry, which you will meet in the Mid-level guide): CI or a person commits a new image tag, and Flux picks it up.

The mental model: four nouns and one verb

Flux has a lot of custom resource kinds, but a beginner needs only four ideas. Learn these and every command in the rest of the guide will make sense.

A Source says where content lives and how often to check it. The Source you will use most is a GitRepository: a repository URL, a branch, and a polling interval. Other kinds fetch from an OCI registry (OCIRepository), a Helm chart repository (HelmRepository), or an object-storage bucket (Bucket). A Source does not deploy anything. It only fetches.

An Artifact is what a Source produces when it fetches successfully: a compressed archive of the repository contents, stored by Flux and served over HTTP inside the cluster, labelled with a revision. For Git the revision looks like main@sha1:696f056d..., which is the branch and the commit. You never create Artifacts yourself, but you read revisions constantly, because the revision is how you know which commit Flux believes it is running.

A Kustomization (Flux's own kind, in the API group kustomize.toolkit.fluxcd.io) says: take this Source, look in this path, and apply whatever Kubernetes YAML you find there to the cluster. It is the workhorse. Despite the name, it does not require you to know the Kustomize tool, although it runs Kustomize under the hood and understands Kustomize files when they exist.

A HelmRelease says: install this Helm chart, from this Source, with these values. Helm is the most popular way to package Kubernetes applications, and Flux manages Helm releases declaratively instead of you typing helm install. See the Helm guide at /student-guides/helm for Helm itself.

The one verb is reconcile. Every Flux object has a spec.interval. On that interval the controller that owns the object looks at the world, compares it with what the object says, and acts if they differ. A GitRepository with interval: 1m asks Git for new commits every minute. A Kustomization with interval: 10m re-applies its path to the cluster every ten minutes even when nothing in Git changed, which is how manual edits get reverted. The result of each attempt is written into the object's status, where both you and the flux command line tool can read it.

Hold the chain in your head: a Source fetches, producing an Artifact; a Kustomization or HelmRelease takes the Artifact and applies it; and the reconcile loop repeats that forever. Everything else in Flux is a refinement or an add-on to this chain. Flux also has notification objects and image automation objects. We mention them later, but you can be productive without them.

The controllers: who does the work

Flux is not one program. It is several small programs, each a Kubernetes controller running as a Deployment in a namespace called flux-system. A controller is a loop that watches some kinds of objects and makes the world match them. Flux splits its work by kind.

Those four are installed by default. Two further controllers, image-reflector-controller and image-automation-controller, are optional. They watch container registries and can write new image tags back into Git. There is also an optional source-watcher. A beginner does not need any of these, and this guide leaves them alone.

Why split it up? Because each controller can be understood, scaled and secured separately. A failure in fetching shows up on the Source, not on the Kustomization. When something breaks you learn to ask "which stage failed", and the answer is almost always visible on the object that owns that stage. Here is how the pieces cooperate on a normal change.

1. You push a commitA change lands on the branch that your GitRepository tracks.
2. source-controller noticesOn its next interval (or sooner if a webhook arrives) it fetches the repository and stores a new Artifact with the new revision.
3. kustomize-controller is toldIt sees that the Artifact its Kustomization points at has a new revision.
4. It builds and appliesIt renders the YAML at your chosen path and applies it to the cluster with Kubernetes server-side apply.
5. It checks health and prunesIt can wait for your Deployments to become ready, and it deletes objects that used to be in Git but no longer are.
6. It reports backThe result, including the revision now applied, lands in the object's status and in Kubernetes events.

All of the controllers run with restricted security settings, and only kustomize-controller and helm-controller hold broad cluster rights, because applying arbitrary YAML needs them. You do not need to change any of that as a beginner. It is worth knowing now because it explains why Flux is comfortable to run in a locked-down cluster.

Installing the flux CLI and checking your cluster

You interact with Flux through two tools. kubectl you already know. The flux command line tool is new: it installs Flux, prints the state of Flux objects in a readable table, forces a reconcile, and can generate YAML for you. It is a client only. Flux itself runs in the cluster, so you can uninstall the CLI without touching any deployment.

Install it with whichever route suits your machine:

# macOS or Linux with Homebrew
brew install fluxcd/tap/flux

# Linux or macOS with the official install script
curl -s https://fluxcd.io/install.sh | sudo bash

# Windows with Chocolatey
choco install flux

The Homebrew tap and the script are the usual choices. Chocolatey is the only Windows package manager the official documentation lists. If you use something else, such as winget or scoop, treat the package as unofficial and check its version. You can also download a release archive from the fluxcd/flux2 page on GitHub, or run the CLI from the container image ghcr.io/fluxcd/flux-cli, which includes kubectl.

Confirm it works, and turn on shell completion so the long subcommand names become tab-completable:

flux --version
. <(flux completion bash)

flux version 2.9.5

The second command is for bash; zsh, fish and powershell are also supported through flux completion zsh and so on. Put the line in your shell's startup file so it survives a new terminal.

A cluster you are allowed to break

You need a Kubernetes cluster, and for learning it should be disposable. The most convenient option is a local cluster running inside Docker, created with a tool such as kind (Kubernetes in Docker). If you have not met containers yet, read the Docker guide at /student-guides/docker first, since kind runs each cluster node as a container.

kind create cluster --name flux-demo
kubectl cluster-info --context kind-flux-demo
kubectl get nodes

NAME                      STATUS   ROLES           AGE   VERSION
flux-demo-control-plane   Ready    control-plane   40s   v1.35.0

Any cluster works: Minikube, Docker Desktop's built-in Kubernetes, or a managed cluster in the cloud, including the cloud regions in the Gulf that many readers will use at work. What matters is that you have cluster-admin rights, because installing Flux creates custom resource definitions and cluster-wide roles. Flux 2.9 supports Kubernetes 1.34, 1.35 and 1.36, and the project supports only the three newest Kubernetes minor versions, so an old cluster may refuse to cooperate.

The pre-flight check

Before installing anything, ask Flux whether the cluster is ready to receive it:

flux check --pre

► checking prerequisites
✔ Kubernetes 1.35.0 >=1.34.1-0
✔ prerequisites checks passed

Read this output line by line. The first line confirms it could reach the cluster through your current kubectl context, and that the Kubernetes version falls in the supported range. The last line is the summary you care about. If you see a red cross instead, the message tells you whether the problem is an unreachable cluster (check kubectl get nodes works on its own), an unsupported version, or missing rights. The command also tells you when a newer CLI exists.

After Flux is installed, you will run plain flux check with no --pre. It then verifies that each controller is healthy and that the custom resource definitions are present, which makes it the first command to run whenever Flux "feels broken". You can also run flux version to see the CLI version next to the version of the controllers running in the cluster.

Bootstrap: Flux installs itself from Git

There are several ways to put Flux into a cluster. The recommended one is called bootstrap, and it is worth understanding why, because it is the feature that makes Flux different from tools that just apply a YAML bundle.

flux bootstrap does two things in one command. It installs the controllers into the cluster, and it commits the manifests of those controllers into your Git repository, together with a GitRepository and a Kustomization that point back at that same repository. From that moment Flux is managing itself: its own definition lives in Git, and if you later want to upgrade Flux or change one of its settings, you change a file and push. That is the GitOps promise applied to the tool delivering it.

Bootstrap is also idempotent. Running it a second time with the same flags does no harm, and running it with a newer CLI is how you upgrade.

You need a repository host. GitHub is the easiest, and the official commands exist for GitHub, GitLab, Gitea, Bitbucket Server and any generic Git server. We use GitHub.

1. Create a personal access tokenOn GitHub, create a personal access token that has permission to create and write repositories. Bootstrap needs it once, to create the repository and to register a deploy key. The documentation describes the exact scopes.
2. Export itPut the token in an environment variable so it never appears in your shell history or in a file.
3. Run bootstrapGive it your GitHub user, a repository name, a branch and a path inside the repository that represents this cluster.
4. Wait and verifyBootstrap waits for the controllers to become ready. Then you check with flux check.

export GITHUB_TOKEN=<your-token>
export GITHUB_USER=<your-github-username>

flux bootstrap github \
  --owner=$GITHUB_USER \
  --repository=fleet-infra \
  --branch=main \
  --path=clusters/demo \
  --personal=true

Go through the flags, because each one answers a question you will otherwise be asked later.

• --owner is the GitHub user or organisation that will own the repository. With a personal account, add --personal=true, which tells the command the owner is a user, not an organisation. If you want the repository to be public, also pass --private=false; the default is a private repository.
• --repository names the repository. If it does not exist, bootstrap creates it. If it exists, bootstrap uses it.
• --branch is the branch Flux will watch. The default is main.
• --path is the folder inside the repository that belongs to this cluster. One repository can serve many clusters, each with its own folder such as clusters/staging and clusters/production. Flux only reads inside the path you give it for its own self-management.
• --token-auth (not used here) makes Flux authenticate to Git using the token over HTTPS. Without it, bootstrap generates an SSH deploy key and stores its private half in a Secret in the cluster, which is the more common choice because the token then never needs to stay in the cluster.

Typical output, abbreviated:

► connecting to github.com
✔ repository "https://github.com/your-user/fleet-infra" created
► cloning branch "main" from Git repository "https://github.com/your-user/fleet-infra.git"
✔ cloned repository
► generating component manifests
✔ generated component manifests
✔ committed component manifests to "main"
► installing components in "flux-system" namespace
✔ installed components
✔ reconciled components
► determining if source secret "flux-system/flux-system" exists
► generating source secret
✔ public key: ecdsa-sha2-nistp384 AAAA...
✔ configured deploy key "flux-system-main-flux-system-./clusters/demo"
► generating sync manifests
✔ committed sync manifests to "main"
► applying sync manifests
✔ reconciled sync configuration
◎ waiting for GitRepository "flux-system/flux-system" to be reconciled
✔ GitRepository reconciled successfully
◎ waiting for Kustomization "flux-system/flux-system" to be reconciled
✔ Kustomization reconciled successfully
► confirming components are healthy
✔ helm-controller: deployment ready
✔ kustomize-controller: deployment ready
✔ notification-controller: deployment ready
✔ source-controller: deployment ready
✔ all components are healthy

Your exact lines will differ, but the shape is the point. Bootstrap generated manifests, committed them, installed the controllers, created credentials, committed a sync configuration, applied it, and then waited for Flux to reconcile itself. The last line, all components are healthy, is what you need to see.

Two traps cost beginners the most time at this step. The first is a token that lacks repository permissions, which shows up as an authentication or permission error while bootstrap tries to create the repository or the deploy key. Create a fresh token with the right scope and export it again. The second is running bootstrap against the wrong cluster, because kubectl talks to whichever context is current. Always check kubectl config current-context before you bootstrap, and be doubly careful if you have a production context in the same kubeconfig.

If you later need to reuse an existing repository, or to adopt a cluster where Flux was installed another way, the --force flag exists, but read the documentation before using it. For experimenting, a fresh repository and a fresh kind cluster is the safe path.

What bootstrap created, in Git and in the cluster

Open the repository on GitHub. Inside the path you gave, you will find a folder named flux-system containing three files. Understanding them removes most of the mystery about what Flux is.

fleet-infra/
└── clusters/
    └── demo/
        └── flux-system/
            ├── gotk-components.yaml
            ├── gotk-sync.yaml
            └── kustomization.yaml

gotk-components.yaml is a very long file. It contains the definitions of everything Flux needs: the custom resource definitions, the namespace, the roles, the four controller Deployments and their network policies. "gotk" is the GitOps Toolkit. You never edit this file by hand, since bootstrap regenerates it when you upgrade. Reading through it once is a good way to learn that Flux is nothing magical: it is Deployments and CRDs like any other application.

gotk-sync.yaml is short and is the piece that matters most. It holds two objects:

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: flux-system
  namespace: flux-system
spec:
  interval: 1m0s
  ref:
    branch: main
  secretRef:
    name: flux-system
  url: ssh://git@github.com/your-user/fleet-infra
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: flux-system
  namespace: flux-system
spec:
  interval: 10m0s
  path: ./clusters/demo
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system

You now have a complete example of a Source and a Kustomization. The GitRepository named flux-system says: fetch this repository, branch main, every minute, using the credentials in the Secret called flux-system. The Kustomization named flux-system says: take that GitRepository, look inside ./clusters/demo, apply what you find every ten minutes, and prune (delete) anything that used to be there and no longer is.

This one Kustomization is the key to everything. It applies whatever is inside clusters/demo, including the folder flux-system itself, which is how Flux manages its own definition. And it means that any new YAML file you add anywhere under clusters/demo is applied automatically. You will use that in the next section.

kustomization.yaml is a Kustomize file listing the other two, so the folder is buildable.

Now look at the cluster:

kubectl -n flux-system get deployments

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
helm-controller           1/1     1            1           3m
kustomize-controller      1/1     1            1           3m
notification-controller   1/1     1            1           3m
source-controller         1/1     1            1           3m

And ask Flux what it knows:

flux get all

NAME                       REVISION        SUSPENDED  READY  MESSAGE
gitrepository/flux-system  main@sha1:a1b2c3d  False    True   stored artifact for revision 'main@sha1:a1b2c3d'

NAME                       REVISION        SUSPENDED  READY  MESSAGE
kustomization/flux-system  main@sha1:a1b2c3d  False    True   Applied revision: main@sha1:a1b2c3d

This is the format you will read hundreds of times, so learn every column. NAME is the kind and name. REVISION is the exact commit Flux last fetched (for the GitRepository) or last applied (for the Kustomization). SUSPENDED says whether someone paused the object. READY is True when the last reconcile succeeded. MESSAGE is the human sentence explaining the state, and it is where error text appears when something goes wrong.

Notice that the two revisions match. That is what "in sync" looks like: the commit that source-controller fetched is the commit kustomize-controller applied.

There is also one Secret to know about, because people sometimes delete it by mistake:

kubectl -n flux-system get secret flux-system

It holds the credentials that GitRepository flux-system uses to read your repository. If you delete it, Flux can no longer fetch anything and the error will say authentication failed. Re-running the bootstrap command recreates it.

Your first application: a Source and a Kustomization

Time to deploy something. We use podinfo, a tiny web application that the Flux project itself uses in its examples. It is a good teaching target because it is small, it shows its own version number on a web page, and its repository already contains ready-made Kubernetes manifests.

Recall the chain: a Source fetches, and a Kustomization applies. So we need two objects. You could write them by hand, but the CLI can generate them for you. The key flag is --export, which prints YAML to the screen instead of creating the object. That matters for GitOps: you generate the file, put it in your repository, commit it, and let Flux create the object. Creating objects directly in the cluster would leave them outside Git, which is exactly the habit we are trying to avoid.

First, make sure you have a local clone of your repository and are inside it:

git clone https://github.com/$GITHUB_USER/fleet-infra.git
cd fleet-infra

Generate the Source:

flux create source git podinfo \
  --url=https://github.com/stefanprodan/podinfo \
  --branch=master \
  --interval=1m \
  --export > ./clusters/demo/podinfo-source.yaml

Then the Kustomization that applies it:

flux create kustomization podinfo \
  --target-namespace=default \
  --source=GitRepository/podinfo \
  --path="./kustomize" \
  --prune=true \
  --wait=true \
  --interval=30m \
  --retry-interval=2m \
  --health-check-timeout=3m \
  --export > ./clusters/demo/podinfo-kustomization.yaml

Open both files. The first is a GitRepository like the one bootstrap created, pointing at the public podinfo repository on its master branch. The second deserves slow reading:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: podinfo
  namespace: flux-system
spec:
  interval: 30m0s
  path: ./kustomize
  prune: true
  retryInterval: 2m0s
  sourceRef:
    kind: GitRepository
    name: podinfo
  targetNamespace: default
  timeout: 3m0s
  wait: true

Every field is a decision you will make for every app you ever deploy with Flux:

• sourceRef names the Source this Kustomization reads from. It is how the two objects are linked.
• path is the folder inside that Source where the YAML lives, relative to the repository root. It is the most common place for typos.
• interval: 30m says re-apply and re-check every thirty minutes even with no new commits. That period is also the longest a manual edit can survive before being reverted.
• prune: true turns on garbage collection: objects that Flux created from this path and that later disappear from Git are deleted from the cluster.
• targetNamespace: default forces all objects into that namespace, regardless of what the YAML says.
• wait: true tells Flux to wait until every applied object is healthy. timeout: 3m is how long it waits before declaring failure.
• retryInterval: 2m is how soon to try again after a failure, instead of waiting for the full 30 minutes.

Now the GitOps moment. Commit and push both files:

git add -A
git commit -m "Add podinfo application"
git push

You have not run kubectl apply for podinfo, and you will not. The existing flux-system Kustomization already watches clusters/demo, so when its GitRepository next fetches (within a minute) it will find your two new files, apply them, and create the podinfo Source and Kustomization in the cluster. Those then fetch podinfo and deploy it. Two levels of Flux objects, each creating the next.

Watch it happen:

flux get kustomizations --watch

NAME         REVISION             SUSPENDED  READY   MESSAGE
flux-system  main@sha1:d4e5f6a    False      True    Applied revision: main@sha1:d4e5f6a
podinfo      master@sha1:9c8b7a6  False      True    Applied revision: master@sha1:9c8b7a6

Press Ctrl+C to leave the watch. You can see the podinfo revision begins with master@, because it tracks podinfo's own branch, whereas the flux-system Kustomization tracks your main. Finally look at what was created:

kubectl -n default get deployments,services
kubectl -n default rollout status deployment/podinfo

To see the app itself, forward a local port:

kubectl -n default port-forward deployment/podinfo 9898:9898

Visit http://localhost:9898 in a browser. You will see a page showing the podinfo version. Stop the forward with Ctrl+C.

Reading status: conditions, revisions and events

A GitOps tool is only as useful as its ability to tell you what went wrong, and Flux is good at this once you know where to look. There are three places, from quickest to most detailed.

The table from flux get. You have already met it. Every command in the flux get family prints the same columns. The family covers sources and the two workload kinds:

flux get sources git              # only Git sources
flux get sources all -A           # every kind of source in every namespace
flux get kustomizations           # alias: flux get ks
flux get helmreleases -A          # alias: flux get hr
flux get all -A                   # everything Flux owns, all namespaces

The -A flag means "all namespaces". Flux commands default to the flux-system namespace, because that is where Flux keeps its own objects. When you look at objects that live elsewhere, such as a HelmRelease in the default namespace, you either pass -n default or -A. Forgetting this is the classic reason for "flux says there are no helmreleases" when there obviously are.

The most useful filter is --status-selector, which shows only the objects that are not healthy:

flux get all -A --status-selector ready=false

The events. Flux writes a Kubernetes event each time something noteworthy happens, and the CLI reads them for you:

flux events                                  # events in flux-system
flux events -A                               # everywhere
flux events --for Kustomization/podinfo      # one object

LAST SEEN  TYPE    REASON                OBJECT                   MESSAGE
2m         Normal  ReconciliationSucceeded  Kustomization/podinfo  Reconciliation finished in 1.2s, next run in 30m0s with revision master@sha1:9c8b7a6

Events are history: they tell you what happened and when, so they are good for questions like "when was this last applied?".

The controller logs. When the message on the object is not enough, go to the source:

flux logs --all-namespaces --level=error
flux logs --kind=Kustomization --name=podinfo

The conditions on the object. Every Flux object carries a list of conditions in its status. You can view them with kubectl describe, for example kubectl -n flux-system describe kustomization podinfo. The condition called Ready is the headline. When it is False, the message next to it is the reason. You may also see Reconciling, which is true while work is in progress, Stalled, which means Flux has decided that retrying will not help until something changes, and, for workloads with health checks, Healthy.

One more field rewards attention: .status.lastAppliedRevision on a Kustomization, and .status.artifact.revision on a Source. Together they tell you which commit was fetched and which was applied. If the source is on abc123 and the Kustomization on 789def, the Kustomization has not caught up, and the question becomes why.

If you want to see what a Kustomization actually manages, use:

flux tree kustomization flux-system

which prints the objects owned by that Kustomization as a tree, including the Flux objects it creates, such as your podinfo Kustomization. It is a quick way to answer "what does Flux think it is responsible for here?".

Changing things the GitOps way: commit, then reconcile

The most important skill in the whole guide is changing something without touching the cluster directly. Let us change how podinfo is configured, and watch the change flow through.

Podinfo's manifests live in another repository that we do not own, so we cannot edit them. Flux gives a clean way to adjust them anyway: patches in the Kustomization, which Flux applies on top of the fetched YAML. Edit clusters/demo/podinfo-kustomization.yaml and add a patch that raises the replica count:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: podinfo
  namespace: flux-system
spec:
  interval: 30m0s
  path: ./kustomize
  prune: true
  retryInterval: 2m0s
  sourceRef:
    kind: GitRepository
    name: podinfo
  targetNamespace: default
  timeout: 3m0s
  wait: true
  patches:
    - target:
        kind: Deployment
        name: podinfo
      patch: |
        - op: replace
          path: /spec/replicas
          value: 3

The patch uses JSON Patch syntax: it replaces /spec/replicas on the Deployment called podinfo with the value 3. Commit, push and reconcile:

git add -A
git commit -m "Run three replicas of podinfo"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization podinfo
kubectl -n default get deployment podinfo

NAME      READY   UP-TO-DATE   AVAILABLE   AGE
podinfo   3/3     3            3           12m

The path a change takes is always the same: edit a file, commit, push, and wait (or reconcile). Notice the two-step reconcile: the first command updated the podinfo Kustomization object itself from your repository, and the second made that updated Kustomization apply its patch. Normally you would not run them by hand, and the next poll handles it, but it is worth understanding that there are two hops.

Drift: the cluster changes behind Flux's back

Now test the promise that Flux keeps the cluster matching Git. Make a manual change and see what happens:

kubectl -n default scale deployment podinfo --replicas=1
kubectl -n default get deployment podinfo

For a moment the Deployment shows 1 replica. This is drift: the cluster differs from what Git describes. Now ask Flux to reconcile:

flux reconcile kustomization podinfo
kubectl -n default get deployment podinfo

Back to three replicas. Flux re-applied the desired state and overwrote your manual change. If you had not forced it, the same correction would have happened at the next interval, up to 30 minutes later. That is why the interval matters: it is the maximum lifetime of an unofficial edit.

Prune: deleting a file deletes the object

The opposite direction matters too. Because prune: true is set, removing something from Git removes it from the cluster. To see it, delete the podinfo application by removing its two files:

git rm clusters/demo/podinfo-source.yaml clusters/demo/podinfo-kustomization.yaml
git commit -m "Remove podinfo"
git push
flux reconcile kustomization flux-system --with-source
kubectl -n default get deployments

The podinfo Deployment disappears. The top-level flux-system Kustomization noticed the two files were gone, pruned the podinfo Kustomization and Source, and because that Kustomization also had prune: true, all the objects it had created went with it. The deletion cascaded down the chain.

Prune works because every Kustomization keeps an inventory of what it applied, recorded in its status, so it knows exactly what belongs to it and never deletes anything else. You can see it in the status of a Kustomization.

Re-add the two files (restore them with git revert HEAD, then push) before you continue, because the following sections use podinfo again.

Deploying with Helm: HelmRepository and HelmRelease

Not everything you want to run comes as plain YAML. Much of the Kubernetes ecosystem is distributed as Helm charts, which are packaged, parameterised bundles of manifests. Instead of running helm install from your laptop, which would be imperative and leave no record in Git, you describe the release as an object and let Flux's helm-controller manage it. The chain is the same shape as before: a Source that fetches, and an object that applies.

For charts hosted in a classic Helm repository, the Source is a HelmRepository, and the applier is a HelmRelease. Let us install podinfo again, but as a Helm release in its own namespace. Create the files by hand this time, to see the YAML without a generator.

apiVersion: v1
kind: Namespace
metadata:
  name: podinfo

apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: podinfo
  namespace: podinfo
spec:
  interval: 1h
  url: https://stefanprodan.github.io/podinfo

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: podinfo
  namespace: podinfo
spec:
  interval: 10m
  chart:
    spec:
      chart: podinfo
      version: ">=6.0.0"
      sourceRef:
        kind: HelmRepository
        name: podinfo
  values:
    replicaCount: 2

Read the HelmRelease as a sentence. "Every ten minutes, make sure a release called podinfo exists in the namespace podinfo, built from the chart podinfo at any version 6.0.0 or newer taken from the HelmRepository named podinfo, with the value replicaCount set to 2." The values block is exactly what you would put in a values.yaml file for helm install -f, and you can use any value the chart supports. helm show values on a chart lists them.

The version constraint ">=6.0.0" is a semantic-version range. Flux will pick the newest matching chart and will upgrade the release when a newer one is published. For a production system you usually pin an exact version such as 6.7.1 so that upgrades happen only when you change the file. Ranges are convenient for learning and dangerous for production.

To deploy these, tell Flux to apply the folder. A Kustomization does that:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: apps
  namespace: flux-system
spec:
  interval: 10m
  path: ./apps/podinfo-helm
  prune: true
  wait: true
  timeout: 5m
  sourceRef:
    kind: GitRepository
    name: flux-system

This Kustomization reads from flux-system, the GitRepository for your own repository, which is how it reaches the apps folder you just created. Commit and push everything, and then watch:

flux get helmreleases --namespace podinfo --watch

NAME     REVISION  SUSPENDED  READY  MESSAGE
podinfo  6.9.0     False      True   Helm install succeeded for release podinfo/podinfo.v1 with chart podinfo@6.9.0

Read the message. helm-controller fetched the chart, installed release number one (podinfo.v1), and reports the chart version it used. When you later change replicaCount to 3 in release.yaml and push, the release becomes podinfo.v2, and the message becomes "Helm upgrade succeeded".

Some Helm behaviours in Flux 2.9 are worth knowing because older tutorials describe different ones. Flux's helm-controller uses Helm v4. New releases are installed with Kubernetes server-side apply by default, and Flux waits for readiness with the same kstatus health-checking that Kustomizations use. You do not need to configure anything for those defaults, but they explain why an upgrade might behave slightly differently from a Helm 3 tutorial.

There are two commands for Helm releases that beginners reach for constantly:

flux reconcile helmrelease podinfo --namespace podinfo --with-source
flux get helmreleases -A

And when a release is stuck after a failure, flux reconcile helmrelease podinfo -n podinfo --reset clears the failure counter so Flux tries again from scratch.

Order and health: dependsOn, wait and timeouts

Real systems have ordering constraints. An application that needs a database operator cannot be applied before the operator's custom resource definitions exist, and a Deployment that needs a namespace fails if the namespace is not there yet. Inside one Kustomization, Kubernetes server-side apply and Flux handle the common case by applying namespaces and custom resource definitions first. Between Kustomizations, you express ordering explicitly with dependsOn.

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: apps
  namespace: flux-system
spec:
  interval: 10m
  path: ./apps/podinfo-helm
  prune: true
  wait: true
  timeout: 5m
  dependsOn:
    - name: infrastructure
  sourceRef:
    kind: GitRepository
    name: flux-system

Here apps will not be applied until a Kustomization named infrastructure in the same namespace is ready. The usual pattern is a first Kustomization for shared infrastructure (an ingress controller, a certificate manager) and a second for applications that depend on it. dependsOn works on HelmReleases as well. When a dependency is not ready, the dependent object does not fail permanently. It reports a message such as dependency 'flux-system/infrastructure' is not ready and tries again roughly every thirty seconds, so the order sorts itself out as the dependency becomes healthy.

Ordering only helps if "ready" means something. That is what health checking provides. With wait: true, a Kustomization does not consider itself ready once the YAML is applied. It waits until every object it applied reports healthy, using the same readiness logic Kubernetes users know: a Deployment is healthy when its new replicas are available, a Job when it has completed, and so on. If the objects are not healthy within timeout, the Kustomization becomes not ready, with a message like this:

health check failed after 3m0s: timeout waiting for: [Deployment/default/podinfo status: 'InProgress']

Learn to read that line. It says which object, in which namespace, and the last status Flux saw, InProgress, which means the pods were still starting or failing to start. The cause is almost never Flux. It is the workload: a wrong image name, a failing readiness probe, insufficient cluster resources. The next step is kubectl describe deployment and kubectl get pods.

If you would rather not wait on everything, healthChecks lists just the objects you care about:

spec:
  wait: false
  healthChecks:
    - apiVersion: apps/v1
      kind: Deployment
      name: podinfo
      namespace: default

As a beginner, wait: true is the right default for applications: it makes flux get kustomizations tell the truth about whether the app actually works, not merely whether the YAML was accepted by the API server.

Customising manifests: Kustomize overlays and variables

You will soon want the same application in two places with small differences: two replicas in staging and five in production, a different hostname, a different image tag. Flux supports the standard answer to that, Kustomize overlays, and a lighter tool for simple values, post-build variable substitution.

An overlay is a folder containing a kustomization.yaml (the Kustomize file this time, not the Flux one) that points at a shared base and adds changes on top. A typical repository layout looks like this:

apps/
├── base/
│   └── podinfo/
│       ├── deployment.yaml
│       ├── service.yaml
│       └── kustomization.yaml
├── staging/
│   └── kustomization.yaml
└── production/
    └── kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../base/podinfo
patches:
  - target:
      kind: Deployment
      name: podinfo
    patch: |
      - op: replace
        path: /spec/replicas
        value: 5

The staging cluster points its Flux Kustomization at ./apps/staging, the production cluster at ./apps/production, and both share the base. Edit the base once and both change. This is the pattern used in the official flux2-kustomize-helm-example repository, which is worth browsing once you are comfortable with the basics. If there is no kustomization.yaml in a folder you point Flux at, Flux generates one listing every YAML file in it, which is why the simple examples above needed no Kustomize file at all. The moment you need patches, images or overlays, you write one.

Post-build substitution covers the lighter case: replacing placeholders in otherwise identical YAML with values defined in the Flux Kustomization or in a ConfigMap. Write ${variable} where the value goes:

spec:
  rules:
    - host: podinfo.${cluster_domain}

# in the Flux Kustomization
spec:
  postBuild:
    substitute:
      cluster_domain: staging.example.com

There is one behaviour new in Flux 2.9 that will bite you if you follow an older tutorial. Strict substitution is now on by default. If the YAML contains ${something} and no value is supplied, the Kustomization fails instead of quietly leaving the text in place. The error reads:

variable not set (strict mode)

You have three ways out: supply the variable, give it a default with ${something:=default}, or, for a file that legitimately contains a literal dollar-brace (a shell script inside a ConfigMap, for instance), put the annotation kustomize.toolkit.fluxcd.io/substitute: disabled on that object.

Suspend, resume and forcing a reconcile

There will be times you need Flux to stop applying something for a while: you are debugging in production, you are testing a manual change, or you want to freeze a release before a holiday weekend. Flux calls this suspend.

flux suspend kustomization podinfo
flux get kustomizations

NAME     REVISION             SUSPENDED  READY  MESSAGE
podinfo  master@sha1:9c8b7a6  True       True   Applied revision: master@sha1:9c8b7a6

The SUSPENDED column turns to True, and Flux ignores that Kustomization until you resume it. While it is suspended, your manual edits survive, because there is no reconcile to revert them. This is the legitimate way to make an emergency change by hand.

flux resume kustomization podinfo

Resuming triggers an immediate reconcile. If you had changed the cluster by hand and never updated Git, this is the moment your change is reverted. The discipline is: suspend, change, put the same change in Git, resume.

The same verbs exist for other kinds. flux suspend helmrelease podinfo -n podinfo pauses a Helm release. flux suspend source git podinfo stops a Source fetching. Underneath, suspending simply sets spec.suspend: true on the object, which means you can also do it by editing the YAML in Git. In that case the pause is versioned and reviewed, which is better for anything lasting more than an hour.

It helps to keep three related commands apart, because beginners mix them up.

None of the three edits what your application should look like. That edit always happens in Git.

Secrets: never put them in Git as plain text

The moment you deploy a real application, you need a database password or an API token. Flux reads manifests from Git, and Git history is permanent and widely shared, so a plain-text Secret committed to a repository is compromised from that commit onwards, even if you delete it later. Base64, which is how Kubernetes Secrets are stored, is an encoding and not encryption. Anyone who can read the repository can read it.

Flux has a built-in answer, SOPS (Secrets OPerationS), a tool that encrypts the values inside a YAML file while leaving the keys readable, so diffs and reviews still work. The encrypted file is safe to commit. kustomize-controller decrypts it in the cluster at apply time using a key that only the cluster holds. The decryption is enabled on the Kustomization:

spec:
  decryption:
    provider: sops
    secretRef:
      name: sops-age

The setup involves generating a key pair (Flux supports Age keys, among others), storing the private key in a cluster Secret called sops-age, and encrypting files with sops before committing them. The Mid-level guide walks through it. Two alternatives exist: the Sealed Secrets controller, and the External Secrets Operator, which keeps the secret in a cloud vault and only a reference in Git. Which you choose is a team decision.

For now, remember the rule and the symptom. The rule is no plaintext secrets in the repository, ever. The symptom of forgetting to configure decryption is this error when an encrypted file reaches a Kustomization that does not know how to decrypt it:

Secret/default/db-password is SOPS encrypted, configuring decryption is required for this secret to be reconciled

If your employer has data-residency requirements, which is common in Gulf and Egyptian organisations, the key-holding side matters too: decide where the private key lives, and prefer a key held in a vault inside an approved region over a key file copied onto laptops.

Notifications and webhooks: hearing from Flux, and poking it

By default, Flux tells you things only when you ask. flux get and flux events are pull interfaces. In a team you want Flux to tell you when a deploy fails, and sometimes you want Git to tell Flux immediately instead of waiting for the next poll. The notification-controller handles both directions.

Outbound: Provider and Alert. A Provider describes where messages go (Slack, Microsoft Teams, Discord, a GitHub commit status, and many more). An Alert says which events to forward.

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: slack
  namespace: flux-system
spec:
  type: slack
  channel: deployments
  secretRef:
    name: slack-url
---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
  name: on-call
  namespace: flux-system
spec:
  providerRef:
    name: slack
  eventSeverity: error
  eventSources:
    - kind: Kustomization
      name: "*"
    - kind: HelmRelease
      name: "*"

Note the API version: v1beta3. Alert and Provider are still beta in Flux 2.9, which is correct and current, unlike the older v1beta1 and v1beta2 versions that were removed. The slack-url Secret holds the webhook address and must be created without committing it in plain text (see the previous section). With eventSeverity: error, only failures are sent, which keeps the channel quiet enough to be read.

Inbound: Receiver. With the default one-minute poll, a push can wait up to a minute before Flux notices. A Receiver exposes a webhook URL that GitHub (or another system) calls on every push, so Flux fetches at once. It is an optimisation, not a requirement, and it requires that the webhook endpoint be reachable from the internet or from your CI, which has its own security cost. Most beginners skip it and live happily with polling.

Configuration you will touch, and the ones you will not

Flux has a lot of knobs, but a beginner touches a handful. This section collects them so that nothing in the YAML is a mystery.

On a GitRepository, the fields you will use are url, ref (a branch, tag, semver range or exact commit), interval, and secretRef for private repositories. The secret can be created from the CLI with flux create secret git NAME --url=ssh://git@github.com/org/repo, which prints the public key to add as a deploy key on the repository. You can also create a .sourceignore file at the repository root, in .gitignore syntax, to exclude files from the Artifact. That is the fix when a large folder of irrelevant files is slowing things down.

On a Kustomization, the fields are sourceRef, path, interval, retryInterval, timeout, prune, wait, targetNamespace, dependsOn, patches, postBuild and decryption, all of which you have now met. The field suspend pauses the object. Two more are worth reading about later: serviceAccountName for running with restricted permissions, and force: true for recreating objects when an immutable field changes.

On a HelmRelease, you use chart (or chartRef), values, interval, targetNamespace and dependsOn. The remediation settings are the ones to understand early. A HelmRelease can be told to retry a failed install or upgrade and, on upgrade failure, to roll back:

spec:
  install:
    remediation:
      retries: 3
  upgrade:
    remediation:
      retries: 3

Without retries, a failed install stays failed until you intervene. With them, Flux tries again. If all the retries fail, the message contains install retries exhausted or upgrade retries exhausted, and you fix the cause and clear it with flux reconcile helmrelease NAME --reset, or by suspending and resuming.

Intervals deserve a rule of thumb. Sources should poll often enough that a push appears quickly (1m for your own Git repository is typical). Helm repositories, which rarely change, are fine at 1h. Kustomizations that apply your YAML can run every 10m to 30m, since that interval is mostly for drift correction. Shorter intervals are not free: each poll is a request to your Git host, and on a busy organisation with many objects the requests add up and can trip API rate limits.

Everything else, such as controller flags, feature gates, sharding and workload identity, is a platform-engineer concern and is covered in the Senior guide.

The errors you will see, and how to read them

Flux errors fall into a few families. Once you recognise the family, you know where to look. Start every investigation the same way:

flux check
flux get all -A --status-selector ready=false
flux events -A
flux logs --level=error -A

The first confirms the controllers themselves are healthy. The second lists every object that is not ready. The third and fourth show the recent history and the error logs. Usually the answer is now on your screen. Here are the messages beginners meet most, with what they mean.

Two of these teach a broader lesson. The no matches for kind error looks like a Kubernetes problem, but it is a tutorial-age problem: the manifest was written for an older Flux. Flux 2.7 removed the v1beta1 APIs, 2.8 removed source.toolkit.fluxcd.io/v1beta2, kustomize.toolkit.fluxcd.io/v1beta2 and helm.toolkit.fluxcd.io/v2beta2, and 2.9 removed the v1beta2 versions of the image and notification APIs. The current versions are below. Keep the table near you, because you will paste-check other people's YAML against it for years.

The other lesson is in field is immutable. Kubernetes refuses some edits, and Flux faithfully reports the refusal. The fix is a deliberate choice between recreating the object (force) and leaving it alone, and you should make that choice knowingly, since recreating a Job or a StatefulSet-adjacent object can interrupt work.

Removing Flux cleanly

Learning is easier when you know how to start over. Flux has a proper uninstall command, and it is the only supported way to remove it.

flux uninstall

The command asks for confirmation, then removes the controllers, their network policies and roles, the finalizers, the custom resource definitions and the custom resources, and finally the flux-system namespace. Add --keep-namespace if you want the namespace to stay. What it does not remove is the workloads that Flux deployed. Your podinfo Deployments keep running, unmanaged, because pruning them would be a dangerous surprise. If you want those gone too, delete them with kubectl, or delete the Kustomizations first, with pruning on, and wait for them to clean up.

Uninstalling does not touch your Git repository. The fleet-infra repository still has gotk-components.yaml and the rest, and running flux bootstrap again with the same flags will bring everything back and reapply all your applications. That is a good demonstration of the GitOps claim that the repository is the cluster. With a local kind cluster, you can also delete the whole thing with kind delete cluster --name flux-demo and rebuild it in a couple of minutes.

Putting it all together

Let us finish with one small project that uses everything: a fresh repository, two applications deployed in a safe order, one from plain YAML and one from Helm, a change made by commit, and a failure you diagnose. The steps assume a new disposable cluster.

1. PrepareCreate a kind cluster, export GITHUB_TOKEN and GITHUB_USER, and run flux check --pre.
2. BootstrapRun flux bootstrap github with repository platform-demo and path clusters/demo.
3. Lay out the repositoryCreate infrastructure/ and apps/ folders, plus Kustomizations under clusters/demo/ that point at them.
4. Commit and watchPush, then watch flux get kustomizations until everything is ready.
5. Change somethingEdit a value in Git, push, and watch it roll out.
6. Break it, then read itIntroduce a typo and diagnose it from flux get and flux events.

The target layout:

platform-demo/
├── clusters/demo/
│   ├── flux-system/          (created by bootstrap)
│   ├── infrastructure.yaml   (Flux Kustomization -> ./infrastructure)
│   └── apps.yaml             (Flux Kustomization -> ./apps, depends on infrastructure)
├── infrastructure/
│   ├── kustomization.yaml
│   └── namespace.yaml
└── apps/
    ├── kustomization.yaml
    ├── podinfo-source.yaml
    └── podinfo-release.yaml

The two Flux Kustomizations, which are the glue:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: infrastructure
  namespace: flux-system
spec:
  interval: 10m
  retryInterval: 1m
  path: ./infrastructure
  prune: true
  wait: true
  timeout: 3m
  sourceRef:
    kind: GitRepository
    name: flux-system

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: apps
  namespace: flux-system
spec:
  interval: 10m
  retryInterval: 1m
  path: ./apps
  prune: true
  wait: true
  timeout: 5m
  dependsOn:
    - name: infrastructure
  sourceRef:
    kind: GitRepository
    name: flux-system

The infrastructure folder creates a namespace for the application. Its Kustomize file lists the resources:

apiVersion: v1
kind: Namespace
metadata:
  name: demo
  annotations:
    kustomize.toolkit.fluxcd.io/prune: Disabled

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - namespace.yaml

The prune annotation protects the namespace from accidental garbage collection, which is the habit you learned earlier for anything that can hold data. The application folder holds the Helm Source and release, and a Kustomize file that lists both:

apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: podinfo
  namespace: demo
spec:
  interval: 1h
  url: https://stefanprodan.github.io/podinfo

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: podinfo
  namespace: demo
spec:
  interval: 10m
  install:
    remediation:
      retries: 3
  upgrade:
    remediation:
      retries: 3
  chart:
    spec:
      chart: podinfo
      version: "6.x"
      sourceRef:
        kind: HelmRepository
        name: podinfo
  values:
    replicaCount: 2
    ui:
      message: "Hello from Git"

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - podinfo-source.yaml
  - podinfo-release.yaml

Commit and push, then watch both layers come up in order:

git add -A
git commit -m "Add infrastructure and podinfo via Helm"
git push
flux reconcile kustomization flux-system --with-source
flux get kustomizations --watch

You should see infrastructure become ready first, then apps, because of dependsOn. Then confirm the release and look at the page:

flux get helmreleases -n demo
kubectl -n demo port-forward deployment/podinfo 9898:9898

The page should show "Hello from Git", which came from your values. Now change the message in apps/podinfo-release.yaml, push, reconcile and refresh the page. The release upgrades to v2 without you running any Helm command.

To practise diagnosis, introduce a deliberate mistake. Change the chart name to podinfoo, push, and reconcile. Then follow the method from the error section:

flux get all -A --status-selector ready=false
flux events -n demo --for HelmRelease/podinfo

You will see the HelmRelease not ready with a message about the chart not being found, and a HelmChart object failing in flux-system or demo. The error is on the Source side of the chain, because a chart that cannot be located is a fetch failure. Fix the name, push, and it recovers by itself.

The project touches every idea in the guide: bootstrap, Git as the source of truth, Kustomizations as glue, ordering with dependsOn, a Helm release, drift protection through prune and intervals, and the debugging ritual. If you can build it from memory, you have the beginner level.

What you can now do, and what comes next

You can now explain GitOps and why pull beats push. You can install the flux CLI, bootstrap a cluster from a Git repository, and describe the files bootstrap creates. You can connect a Git Source to a Kustomization, deploy a Helm chart with a HelmRepository and a HelmRelease, order work with dependsOn, and use wait and timeout so that "ready" means "working". You can read flux get, flux events and flux logs to tell which stage failed, recognise the most common error messages, and spot an outdated API version in somebody else's YAML. You know how to suspend and resume safely, why manual edits disappear, and why secrets never go into Git in plain text.

What you have not yet done is the work that turns this from a demo into a production setup. The Mid-level guide goes under the hood: how reconciliation really works, OCI artifacts and "Gitless" delivery, SOPS encryption step by step, Helm values from Secrets and ConfigMaps, image automation that commits new tags, notifications and webhooks in depth, multi-environment repository layouts, and debugging beyond the basics. The Senior guide treats Flux as a platform: multi-tenancy and impersonation, scaling and sharding, the security model, upgrades and API migrations (including the flux migrate procedure mentioned above), monitoring, and when another tool is the better choice.

Flux sits in a chain of tools in this catalogue. It deploys onto Kubernetes (/student-guides/kubernetes), very often packages from Helm (/student-guides/helm), and is often used together with Kustomize (/student-guides/kustomize) and provisioned infrastructure from Terraform (/student-guides/terraform). The main alternative that many employers run instead is Argo CD (/student-guides/argo-cd), which is worth reading to understand the trade-offs in an interview. Container basics are in /student-guides/docker.

A practical next step for this week: take one real application you know, write it as plain YAML or a Helm release, and deploy it to a practice cluster entirely through a Flux repository. Do not run kubectl apply once. The first time you resist the temptation and fix a problem through a commit, the pull-based habit starts to feel natural, and that habit is what the next two levels build on.

Sources

• Flux documentation
• Flux installation
• Bootstrap for GitHub
• Upgrading Flux
• Uninstalling Flux
• Flux concepts
• Flux components
• GitRepository API
• OCIRepository API
• HelmRepository API
• Kustomization API
• HelmRelease API
• Notification Alerts
• Notification Providers
• Notification Receivers
• Flux CLI reference
• Troubleshooting cheatsheet
• Manage Kubernetes secrets with SOPS
• Flux security
• Announcing Flux 2.9 GA
• Flux releases on GitHub
• Upgrade procedure for Flux v2.7+
• flux2-kustomize-helm-example

Part one of three. A twenty-five-minute review of the Beginner Detailed track, then the questions interviewers ask. Mid-level reviews this plus its own material; Senior reviews all three. Everything here is written against Flux v2.9.5, so when an interviewer quotes an older tutorial, you can say which parts of it no longer work.

The thirty-second answer

Flux is a GitOps tool that runs inside a Kubernetes cluster and keeps the cluster matching a Git repository. Its controllers pull the desired state from Git, apply it, and keep reconciling on an interval, so manual changes are reverted and files removed from Git are deleted from the cluster. Because the cluster pulls, no CI system needs credentials to it.

Then the line that shows you have used it: "a Source fetches and produces an artifact with a revision, and a Kustomization or HelmRelease applies it, so when something breaks I first ask which stage of that chain failed."

The model

Interviewers want to hear the nouns used precisely. Have these ready.

• Source: a custom resource that says where content lives and how often to check. GitRepository is the common one. Others are OCIRepository, HelmRepository, HelmChart and Bucket. It fetches; it never deploys.
• Artifact: what a Source produces, a tarball with a revision (main@sha1:<commit> for Git) served inside the cluster by source-controller.
• Kustomization (Flux's kind, kustomize.toolkit.fluxcd.io/v1): takes a Source and a path, builds the YAML, and applies it with server-side apply, with optional pruning and health checks.
• HelmRelease (helm.toolkit.fluxcd.io/v2): declares a Helm release, built from a chart supplied by a Source, managed by helm-controller.
• Reconciliation: the loop in which a controller compares desired and actual state on spec.interval and acts on the difference, recording the outcome in status.
• Bootstrap: flux bootstrap installs the controllers and commits their manifests, plus a GitRepository and Kustomization named flux-system, into Git, so Flux manages itself.

The controllers are four Deployments in the flux-system namespace: source-controller, kustomize-controller, helm-controller and notification-controller. An interviewer may ask which one is responsible for something. Fetching is source-controller. Applying plain YAML is kustomize-controller. Helm is helm-controller. Alerts and incoming webhooks are notification-controller.

Pull versus push

This is the most common opener, and the one where a precise answer separates people who have used GitOps from people who have read the slide.

Push means a CI job holds cluster credentials and runs kubectl apply or helm upgrade when a pipeline finishes. Pull means an agent inside the cluster fetches from Git and applies what it finds. Three differences follow from that, and you should be able to give all of them.

The first is credentials. In push, the CI system can change the cluster, so compromising it compromises production. In pull, the cluster only needs outbound access to Git and registries. The second is continuity. A push job runs once, but Flux reconciles forever, so drift is detected and corrected. The third is deletion. kubectl apply never removes objects that you deleted from your manifests. Flux with prune: true does, using an inventory of what it applied.

Expect the follow-up: "Is pull always better?" The honest answer is that pull trades simplicity for safety. It adds an in-cluster component to run and upgrade, and it is slower to show failures, since the feedback goes to a status field rather than to a red pipeline. Teams close that gap with notifications, and by having CI wait for the cluster to reconcile, or by checking flux get output. A strong candidate mentions both sides.

The commands to have ready

flux check --pre                                   # is the cluster ready for Flux?
flux bootstrap github --owner=<user> --repository=<repo> \
  --branch=main --path=clusters/<name> --personal=true
flux check                                         # are the controllers healthy?
flux get all -A                                    # everything Flux owns
flux get sources git -A
flux get kustomizations                            # alias: flux get ks
flux get helmreleases -A                           # alias: flux get hr
flux reconcile source git flux-system
flux reconcile kustomization flux-system --with-source
flux reconcile helmrelease <name> -n <ns> --with-source
flux suspend kustomization <name>  /  flux resume kustomization <name>
flux events -A   /   flux logs --all-namespaces --level=error
flux create source git <name> --url=... --branch=... --export
flux create kustomization <name> --source=GitRepository/<name> --path=./dir --export
flux tree kustomization flux-system
flux diff kustomization <name> --path ./dir
flux uninstall

Be ready to explain one flag in the list that surprises people. --export prints the YAML instead of creating the object in the cluster, which is what makes the CLI a generator for files you commit. An answer that uses flux create without --export to create objects directly works, but it creates state that Git does not know about, and a good candidate says so.

Also be ready for flux get -A. Flux commands default to the flux-system namespace, so flux get helmreleases shows nothing when your releases live elsewhere. The -A flag means all namespaces, and -n <namespace> picks one.

Reading the status table

Interviewers sometimes put a screen of output in front of you and ask what is wrong. Practise reading the columns of flux get.

NAME         REVISION             SUSPENDED  READY  MESSAGE
flux-system  main@sha1:d4e5f6a    False      True   Applied revision: main@sha1:d4e5f6a
podinfo      master@sha1:9c8b7a6  True       True   Applied revision: master@sha1:9c8b7a6

The first line is healthy: it is ready, and it applied the same revision that its source holds. The second looks healthy (READY True) but SUSPENDED is True, so nothing new will be applied. That is the detail a careful candidate catches. Suspension keeps the last result and hides the fact that the object has stopped reconciling.

Other things to read in a status: REVISION is the commit that was fetched or applied. The MESSAGE holds the error text when READY is False. Under the hood, the Ready condition carries the headline, Reconciling is true while work is in progress, Stalled means Flux has decided that retrying will not help until something changes, and .status.lastAppliedRevision says which commit was applied.

The lifecycle of a change

Interviewers like a narrative question: "I merge a pull request that changes a Deployment. What happens?" Answer in order.

1. The merge advances the branch that the GitRepository tracks.
2. source-controller polls on its interval (one minute for the bootstrap source), or is woken by a webhook Receiver, and fetches the repository.
3. It stores a new artifact with a new revision and updates the GitRepository status.
4. kustomize-controller sees that the artifact its Kustomization depends on has changed.
5. It builds the YAML at spec.path, including any Kustomize patches and variable substitution, and applies it with server-side apply.
6. If wait: true, it waits for the applied objects to become healthy, up to timeout.
7. It records the result and revision in the status, emits events, and, if alerts are configured, notifies.

The part that shows depth is the two-level structure. If the changed file is itself a Flux object, say another Kustomization, then the first Kustomization applies it, and only then does the new object start its own reconcile. That is why a fix sometimes takes two hops to arrive.

Drift, prune and the inventory

Drift is any difference between the cluster and Git. With kustomize-controller, drift is corrected every time the Kustomization reconciles, which is spec.interval (or sooner on a new revision), through server-side apply. If an engineer scales a Deployment by hand, the next reconcile sets it back.

Prune is garbage collection. With prune: true, objects that were applied by the Kustomization and no longer appear in its build are deleted. It works because the Kustomization records an inventory of what it applied. It never deletes objects it does not own.

Know the safety valve. Adding the annotation kustomize.toolkit.fluxcd.io/prune: Disabled to an object stops Flux from garbage-collecting it, which is the standard protection for Namespaces and PersistentVolumeClaims. If an interviewer asks "what is the most dangerous default in GitOps?", prune plus a repository path typo is the classic answer, since pointing a Kustomization at an empty path can delete everything it used to own.

For Helm, drift correction is a separate setting. helm-controller corrects drift only when .spec.driftDetection.mode is enabled. The other modes are disabled and warn.

Suspend, resume and emergency changes

If you must change a live object by hand, the correct procedure is to flux suspend the owning Kustomization, make the change, put the same change in Git, and then flux resume. If you skip the Git step, resuming reverts your change. A strong answer explains why: suspended objects are not reconciled, and resume triggers a reconcile.

Also know that suspended objects look healthy in the table, and that a forgotten suspension is a silent pipeline outage.

The three time settings

• interval is how often Flux re-checks and re-applies when everything is fine. On a Kustomization it is also the maximum lifetime of a manual edit.
• retryInterval is the delay before trying again after a failure.
• timeout is the longest a single attempt, including health checks, may take.

A typical beginner configuration is interval: 10m, retryInterval: 2m and timeout: 3m. The interview trap is an interval that is too short: each poll is a request to the Git host, and with many objects the requests add up and can hit API rate limits.

Helm in Flux

A HelmRelease replaces helm install. You declare the chart (chart.spec.chart, a version constraint and a sourceRef), the values, and optionally remediation. Interviewers like to ask three things about it.

How does the chart reach the cluster? A HelmRepository source fetches the repository index; helm-controller then has a HelmChart object created from the HelmRelease's chart template, and renders it.

What is the difference from running Helm yourself? Flux keeps the release declared in Git, reapplies it on an interval, can correct drift in the release's objects when drift detection is enabled, and can roll back or retry when an upgrade fails. helm list still shows the release, so the tools coexist, but hand-running helm upgrade on a Flux-managed release invites a conflict.

What changed in recent versions? Since Flux 2.8, helm-controller uses Helm v4. New releases are installed with server-side apply by default, and health checking uses kstatus. For OCI charts, use an OCIRepository and chartRef, since HelmRepository with type: oci is in maintenance mode. In 2.9 a new field, postRenderStrategy, defaults to combined, which sends hooks through post-renderers. Earlier behaviour was the nohooks setting. A beginner is not expected to recite that, but naming one or two recent changes signals that you keep up.

The classic traps, and why they share one cause

Most Flux traps come from forgetting that the repository, not the cluster, is the source of truth. Go through these and notice the pattern.

• A manual kubectl edit disappears. The next reconcile reapplies Git. Fix: change Git, or suspend first.
• Deleting a file deletes a live object. Because of prune. Fix: think before you delete, and protect stateful objects with the prune annotation.
• A secret committed in plain text. Git history is permanent and Kubernetes Secrets are only base64 encoded. Fix: SOPS, Sealed Secrets or the External Secrets Operator, and rotate the leaked value.
• no matches for kind ... v2beta1. The YAML comes from an old tutorial. Flux 2.7 removed the v1beta1 APIs, 2.8 removed v1beta2 and v2beta2 for sources, Kustomizations and HelmReleases, and 2.9 removed v1beta2 for the image and notification APIs. Fix: use current versions, or run flux migrate.
• variable not set (strict mode). Since Flux 2.9, substitution is strict by default. Fix: supply the variable, give a default with ${var:=default}, or annotate the object with kustomize.toolkit.fluxcd.io/substitute: disabled.
• flux get helmreleases shows nothing. The namespace: add -A.
• A suspended Kustomization shows Ready. Read the SUSPENDED column.
• Health check timeout. The workload is broken, not Flux. Fix the pods.
• The wrong cluster bootstrapped. kubectl uses the current context. Check it first.

The shared cause is a mismatch between what the engineer believes and what Git says. If you can explain each trap as a version of that, you show that you understand the model, not only a list.

Common interview questions

1. What is GitOps?A way of operating infrastructure where the desired state is declarative and stored in Git, agents pull it and apply it automatically, and they continuously reconcile actual state toward it. Git becomes the audit trail and the rollback mechanism.
2. How is Flux different from running kubectl in a pipeline?Flux pulls from inside the cluster, so no pipeline holds cluster credentials. It reconciles continuously, correcting drift, and it prunes deleted objects. A pipeline runs once and only creates or updates.
3. What does flux bootstrap do?Installs the controllers, commits their manifests and a GitRepository and Kustomization called flux-system to your repository, and creates credentials. Flux then manages itself from Git, and rerunning bootstrap is how you upgrade.
4. What are the four default controllers?source-controller, kustomize-controller, helm-controller and notification-controller. Image automation controllers are optional extras.
5. What is the difference between a GitRepository and a Kustomization?The first fetches a repository and publishes an artifact; the second applies part of that artifact to the cluster. Fetching and applying are separate stages, so a failure shows up on the object that owns the stage.
6. Is Flux's Kustomization the same as a kustomization.yaml?No. The file belongs to the Kustomize tool and uses kustomize.config.k8s.io/v1beta1. The Flux kind is kustomize.toolkit.fluxcd.io/v1, a cluster object that tells Flux what to apply. They often work together.
7. How do you make Flux act immediately?flux reconcile source git NAME fetches now; flux reconcile kustomization NAME --with-source fetches and applies. It does not change the desired state, it only brings forward the next check.
8. What is --export for?It prints the generated YAML instead of creating the object, so you can commit it. Creating objects directly leaves them outside Git.
9. How do you order two applications?With dependsOn on the later Kustomization or HelmRelease. The dependent reports that the dependency is not ready and retries about every 30 seconds until it is.
10. What does wait: true do?The Kustomization waits until every applied object is healthy, up to timeout, so Ready means "working" and not just "accepted by the API server".
11. How do you handle secrets?Never plaintext in Git. Use SOPS with a key held in the cluster (kustomize-controller decrypts when spec.decryption is set), or Sealed Secrets, or the External Secrets Operator. Rotate anything that was ever committed.
12. How do you see what went wrong?flux check, then flux get all -A --status-selector ready=false, then flux events -A, then flux logs --level=error -A, then kubectl describe on the object or the workload.
13. What happens if someone edits a Deployment by hand?The next reconcile of the owning Kustomization reverts it. If it is an emergency, suspend first, and put the change in Git before resuming.
14. How would you roll back a bad deploy?git revert the commit and let Flux apply it, or reconcile immediately. For a HelmRelease you can also configure remediation to roll back failed upgrades automatically.
15. How does Flux relate to Argo CD?Both are GitOps controllers. Flux is CLI and custom-resource native, runs as in-cluster agents and has no built-in UI. Argo CD centres on an application UI and a central control plane. Both are CNCF graduated projects, and the choice is usually about team preference and operating model.
16. How do you remove Flux?flux uninstall, which removes the controllers and custom resources but leaves deployed workloads. Deleting Flux with kubectl is not supported.

Questions to ask back

Interviews are two-way, and a beginner who asks good questions stands out. Useful ones: how many clusters does the team run and how are they laid out in the repository; how do changes reach production (a pull request, a promotion, a tag); where do secrets live and who holds the decryption key; how does the team find out that a reconcile failed; and which Flux version is running, and how are upgrades handled. The last one matters, because a team still on Flux 2.6 has a migration ahead of it.

Sixty-second self-test

• Give the thirty-second answer without pausing.
• Name the four nouns and the one verb.
• Name the four default controllers and say what each does.
• Explain push versus pull, with three differences.
• Say what flux bootstrap commits to Git, and why that matters.
• Explain what prune: true does and how to protect a Namespace from it.
• Explain drift, and how to make a manual change safely.
• Give the difference between interval, retryInterval and timeout.
• Say why Flux's Kustomization is not the same as a kustomization.yaml.
• List three error messages and their fixes, including variable not set (strict mode).
• Give the current apiVersion of a Kustomization, a GitRepository and a HelmRelease.
• Say what a health check timeout tells you, and where to look next.

Part one of three. Most beginner trouble with Flux comes from three habits: changing the cluster instead of Git, copying YAML from a tutorial written for an older Flux, and not knowing which stage of the Source-then-apply chain failed. Start with the table, then the habits, the practice cards and the recipes.

Common errors at this level

The pattern is consistent. Most rows are a mismatch between what you believe is in the cluster and what is in Git, or between an object's stage and the place you are looking. The sections below turn the most expensive rows into habits.

The habits that pay off most

1. Check the context firstRun kubectl config current-context before every bootstrap, uninstall or suspend. Flux commands act on whichever cluster kubectl points at, and the wrong one is an expensive mistake.
2. Preview before pushingUse flux diff kustomization NAME --path ./dir and flux build kustomization NAME --path ./dir on a change to an existing app, and read the diff as you would read a pull request.
3. Keep one change per commitWhen a reconcile fails, the commit that broke it should be obvious, and git revert should fix it cleanly.
4. Review Flux changes like codeA merged pull request is a production deploy. Require a review for the branch Flux watches.
5. Read the SUSPENDED columnMake it part of how you glance at the table, so a forgotten suspension is noticed the same day.

Practice cards

Do one card at a time on a disposable cluster. Each takes about ten minutes.

Debugging something that will not reconcile

Follow the same order every time. It is slower to jump around, because Flux errors belong to specific stages, and fixing the wrong stage looks like nothing happened.

1. Is Flux itself healthy?flux check. If a controller is down or a CRD is missing, nothing else matters.
2. What is not ready?flux get all -A --status-selector ready=false. Note the object and its message.
3. Start at the leftmost failing object.If a Source fails, the Kustomization downstream will also fail, with a vaguer message. Fix the Source first.
4. Read events and logs.flux events -A for history, flux logs --level=error -A for detail.
5. Describe it.kubectl -n flux-system describe kustomization NAME shows the conditions in full.
6. If health checks failed, leave Flux.kubectl get pods, kubectl describe pod and kubectl logs for the workload. Flux is only the messenger.
7. Retry after the fix.flux reconcile kustomization NAME --with-source. For a stuck HelmRelease, add --reset.

Two special cases. If a Kustomization keeps saying dependency not ready, the fault is in the dependency; do not touch the dependent. If a HelmRelease says install retries exhausted, Flux gave up after the configured number of tries, so nothing will change until you fix the chart or values and reset it.

Repository layout, from day one

You can start with everything in one folder, but a small amount of structure saves rewrites. A layout that scales from a practice cluster to a real team looks like this:

fleet-infra/
├── clusters/
│   ├── staging/
│   │   ├── flux-system/        (bootstrap output)
│   │   ├── infrastructure.yaml (Flux Kustomization)
│   │   └── apps.yaml           (Flux Kustomization, dependsOn infrastructure)
│   └── production/
├── infrastructure/
│   ├── base/
│   ├── staging/
│   └── production/
└── apps/
    ├── base/
    ├── staging/
    └── production/

The idea: clusters/<name> tells a cluster what to watch; infrastructure/ and apps/ hold what to deploy; each has a base and an overlay per environment. A cluster's bootstrap --path points at its own folder, and the Flux Kustomizations inside that folder point at the overlays. This is the shape used in the official example repository fluxcd/flux2-kustomize-helm-example, and a good one to copy when you are not sure.

Two rules keep it tidy. First, put shared infrastructure before applications, and order them with dependsOn. Second, never let two Kustomizations own the same object. If they both apply a Namespace, one will fight the other, and pruning one can delete what the other needs.

Build time versus reconcile time

A useful mental split explains many confusing failures. Some things are decided when Flux builds your YAML, and others when Flux applies it to the cluster.

At build time, Flux runs Kustomize on your path, merges patches, and substitutes ${variables}. Errors here are about your files: a missing path, a bad patch target, an unset variable in strict mode, an unreadable encrypted file. They appear quickly, on the Kustomization, and they do not touch the cluster at all. flux build and flux diff let you catch them on your laptop.

At apply time, Flux sends the built objects to the Kubernetes API. Errors here are about the cluster: a forbidden action, an immutable field, an admission webhook refusing something, a missing CRD. The message often comes from the API server itself, and Flux relays it.

After applying, there is a third stage, health: did the workload become ready? Errors here are about your application. Keeping the three apart shortens debugging. A variable not set message means build time, so look at your YAML. A field is immutable message means apply time, so look at what you changed. A health-check timeout means your app is failing, so look at pods.

Secrets: a short, safe routine

You will meet secrets early, so have a routine ready.

1. Generate the Secret YAML locally, without committing it: kubectl create secret generic db --from-literal=password=... --dry-run=client -o yaml > secret.yaml.
2. Encrypt it with SOPS, using a key that the cluster also holds.
3. Commit only the encrypted file.
4. Make sure the consuming Kustomization has spec.decryption set with provider: sops.
5. Delete the plaintext file, and check that it was never committed: git log -p -- secret.yaml must show nothing readable.

If you ever commit a secret in plain text, rotate it immediately. Removing the file in the next commit does not help, because the history still has it, and so do any forks and clones. For organisations in the Gulf and Egypt that have data-residency rules, also decide where the decryption key lives: a key held in a vault inside an approved region is easier to defend in an audit than a key file on developers' laptops.

Small things to do from day one

• Put flux check and flux get all -A in a personal cheatsheet. They answer most questions.
• Add shell completion (. <(flux completion bash)). Long subcommands like flux reconcile kustomization become a few keystrokes.
• Use the aliases ks for kustomizations and hr for helmreleases.
• Give every Flux object a clear name that matches what it deploys. apps and infrastructure are fine. test2 is not.
• Keep the README in the repository current: which cluster, which path, how to reconcile, who to ask.
• Write commit messages that say why, since the commit log is your audit trail.
• Keep a disposable cluster for practice, so that mistakes cost nothing.
• Upgrade deliberately. Read the release notes before changing the Flux version, and note that several API versions were removed in 2.7, 2.8 and 2.9.
• Check the docs version. A tutorial from 2022 uses APIs that no longer exist.
• Use labels and namespaces consistently. A search for app.kubernetes.io/name should find everything belonging to an app.

Reading an object's status like a receipt

Every Flux object holds two halves: the spec, which is what you asked for, and the status, which is what Flux did about it. Beginners read only the spec. Reading the status is the faster route to an answer, and it takes one command.

kubectl -n flux-system get kustomization podinfo -o yaml

Scroll to the bottom and look at four things. The conditions list carries the Ready condition whose message is the same text flux get shows, but here you also see a reason and a timestamp, which tell you whether the failure is new or old. The lastAppliedRevision tells you which commit is actually live. The lastAttemptedRevision tells you which commit Flux most recently tried. When these two differ, the most recent attempt failed and the cluster is still on the older revision. That single comparison answers the question "is my latest commit live?" better than any dashboard. Finally, the inventory lists every object the Kustomization owns, which is exactly the list that prune uses.

For a HelmRelease the status shows the Helm release history. You will see the chart version in use, the release revision number, and the remediation counters. If a release is failing, the counters tell you how many retries are left before Flux gives up.

A habit worth building is to glance at the observedGeneration against the object's own generation. If the status lags behind, the controller has not yet processed your latest edit, and any message you read may be about the previous version. This catches a surprising number of "I fixed it but it still shows the old error" moments: the object has not been looked at again yet. Run flux reconcile and read the message again.

Platform notes

Local clusters. kind, Minikube and Docker Desktop all work for practice. On a laptop with limited memory, give the container runtime enough (several gigabytes) or controllers will restart. If the pods crash-loop after a laptop sleeps, check kubectl -n flux-system get pods before assuming Flux is broken.

Corporate networks. The cluster needs outbound HTTPS or SSH to your Git host and to the registries you pull from. A proxy or firewall that blocks them produces fetch errors on the GitRepository, which look like authentication or timeout messages. Test the same URL from a pod inside the cluster before blaming Flux.

Windows. The only official package route is Chocolatey (choco install flux). Line endings can cause odd diffs; configure Git to use LF for YAML.

Public repositories. Practice repositories are fine to keep private. If you make one public, never include tokens or keys, and remember that bootstrap with a personal account defaults to a private repository unless you pass --private=false.

Rate limits. Dozens of objects polling a Git host every minute can hit API rate limits. If you see fetch errors that improve when you lengthen intervals, rate limiting is the cause. Prefer longer intervals for slow-moving sources, and a Receiver for fast feedback.

A checklist before you call it done

• kubectl config current-context shows the cluster you intended.
• flux check ends with "all checks passed".
• Every Flux object in Git uses a current apiVersion.
• Every Kustomization has prune, wait, timeout and a retryInterval chosen on purpose.
• Ordering between infrastructure and apps is expressed with dependsOn.
• No plaintext secrets anywhere in the repository or its history.
• Anything that holds data is protected with the prune-disable annotation.
• Chart versions are pinned, or a range is chosen knowingly.
• flux get all -A shows every object Ready, and none Suspended by accident.
• You changed the app through Git only, and git log explains what and why.
• You can tear the cluster down and rebuild it from the repository alone.

Where to go next

Once the habits above feel natural, the Mid-level guide builds the next layer: how reconciliation works internally, OCI artifacts, SOPS step by step, image automation, notifications in depth and multi-environment layouts. Related guides in this catalogue are Kubernetes (/student-guides/kubernetes), Helm (/student-guides/helm), Kustomize (/student-guides/kustomize) and Argo CD (/student-guides/argo-cd), the tool most often compared with Flux.

This is part two of three. Beginner got a cluster pulling from Git. This level is about the difference between a Flux install that works on the day you demo it and one that still behaves on the day a teammate pushes a bad commit at 17:55 on a Thursday. By the end you will be able to predict what the controllers do before they do it, structure a repository so promotion between environments is a diff rather than a rewrite, keep secrets in Git safely, stop one team's mistake from reaching another team's namespace, and upgrade Flux across the 2.7, 2.8 and 2.9 API removals without an outage. The guide is written against Flux v2.9.5, released on 2026-08-31.

Where this picks up

Each section starts with the problem it solves and ends with a Try it you can run against a practice cluster. A kind or k3d cluster is fine. Nothing here needs a cloud account.

How reconciliation works, precisely

Beginner told you that controllers "compare desired and actual state". To debug Flux you need the exact loop, because nearly every surprise is a consequence of it.

Flux is a set of single-purpose controllers, each watching its own custom resource kind. source-controller turns external content into artifacts. kustomize-controller and helm-controller consume artifacts and change the cluster. notification-controller routes events in and out. The image controllers update Git. None of them call each other's APIs; they communicate through the status of Kubernetes objects and through HTTP downloads of artifacts from source-controller. That is the reason Flux is robust (a dead notification-controller does not stop deployments) and the reason it is sometimes confusing (the cause of a failure is often recorded on a different object from the one you are looking at).

A reconcile of a Kustomization runs these steps in order:

1. Resolve the source. Look up .spec.sourceRef, read the artifact URL and revision from the source's status. If the source is not Ready, stop here with Source artifact not found and retry later.
2. Check dependencies. Every entry in .spec.dependsOn must be Ready (and pass its readyExpr, if any). If not, stop with dependency '<ns>/<name>' is not ready and requeue after --requeue-dependency, 30 seconds by default.
3. Download and unpack the artifact from source-controller's internal HTTP service into a temporary directory. This is why /tmp speed matters for kustomize-controller at scale.
4. Build. Run kustomize build on .spec.path. If the directory has no kustomization.yaml, the controller generates one that includes every manifest it finds. Patches, images, components and name prefixes from the Flux Kustomization spec are applied here.
5. Decrypt SOPS-encrypted files, if .spec.decryption is set. Decryption happens before substitution.
6. Substitute ${var} references from .spec.postBuild.
7. Apply with server-side apply (SSA), first as a dry run for validation, then for real. Objects are applied in a stable order with CRDs and Namespaces first.
8. Prune objects that appear in the previous inventory but not in this one, if prune: true.
9. Health check the applied objects if wait: true or healthChecks is set, up to .spec.timeout.
10. Record status: lastAppliedRevision, the new .status.inventory, conditions, and an event.

Two timing fields control how often all this runs, and people confuse them constantly. .spec.interval is how often the controller re-runs the whole loop even when nothing changed, which is what corrects drift. A new source revision triggers a reconcile immediately, regardless of the interval. .spec.retryInterval is how long to wait before trying again after a failure; if you leave it out, the interval is used. A Kustomization with interval: 1h and no retryInterval therefore waits an hour to retry a failure caused by a dependency that was briefly late. That is one of the most common reasons for "Flux is stuck" that turns out to be "Flux is waiting". Setting retryInterval: 2m alongside a long interval gives you cheap, infrequent drift correction and fast recovery.

The status block is where the loop leaves its footprints, and you should be fluent in reading it:

kubectl -n flux-system get kustomization apps -o yaml | yq '.status'

conditions:
  - type: Ready
    status: "True"
    reason: ReconciliationSucceeded
    message: "Applied revision: main@sha1:3f2a9c1e7b5d..."
lastAppliedRevision: main@sha1:3f2a9c1e7b5d...
lastAttemptedRevision: main@sha1:3f2a9c1e7b5d...
inventory:
  entries:
    - id: default_podinfo_apps_Deployment
      v: v1

lastAttemptedRevision differing from lastAppliedRevision means the controller tried the newer commit and failed; the conditions explain why, and the cluster is still running the old one. That single comparison answers "did my commit land?" faster than any dashboard. Conditions you will meet are Ready, Reconciling (work in progress), Stalled (a failure that retrying will not fix, such as an invalid spec), Healthy (the result of the health checks) and, for HelmReleases since 2.9, Drifted.

The inventory deserves its own mention. It is the list of every object the Kustomization applied, identified by namespace, name, group and kind. Pruning works by set difference between the old and new inventories, not by labels on live objects. Consequences: renaming an object in Git (a new name means a new inventory entry) creates the new object first and then deletes the old one; moving an object between two Kustomizations without care can delete it; and an object Flux did not apply is never pruned even if it carries Flux-looking labels.

Sources in depth: Git, OCI and what an artifact really is

Every reconcile starts from an artifact, so it is worth knowing what one is. When source-controller fetches a GitRepository, it clones, checks out the requested ref, applies the ignore rules, and packs the result into a .tar.gz file. It stores that on its local disk (an emptyDir mounted at /data, which means artifacts are regenerated if the pod restarts) and serves it over HTTP from the source-controller Service. The source's status publishes a URL, a revision and a digest. The revision for Git is <ref>@sha1:<commit>, for example main@sha1:3f2a9c1e..., and for OCI it is <tag>@sha256:<digest>.

Consumers decide whether to do work by comparing revisions. If a Git commit changes only a file outside your path, the GitRepository revision still changes, so every Kustomization pointing at that repository re-runs its build. The controller is smart enough to notice that nothing it renders changed, and SSA makes the apply a no-op, but the build cost is real, and in a monorepo with hundreds of Kustomizations it adds up. Two tools reduce it: .spec.ignore on the source, which uses .gitignore syntax and removes files from the artifact altogether, and splitting sources so that unrelated parts of the repository are not in the same artifact.

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: platform
  namespace: flux-system
spec:
  interval: 5m
  url: ssh://git@github.com/example-org/platform
  ref:
    branch: main
  secretRef:
    name: platform-deploy-key
  ignore: |
    # exclude everything, then include only what clusters consume
    /*
    !/clusters/prod
    !/infrastructure
    !/apps

The pattern above, exclude everything and then re-include, is the cleanest way to keep docs, scripts and CI config out of the artifact. A related trap: if ignore accidentally excludes the directory your Kustomization's path points at, the error is kustomization path not found, which sends people looking at typos in the path instead of at the ignore rules.

The ref field gives you more than branch. You can pin tag, use a semver range (>=1.4.0 <2.0.0) so that a repository of releases is followed automatically, or set a commit for a frozen source. Combining branch with commit checks out the commit from that branch. For authentication, the referenced Secret holds username and password for HTTPS, a bearerToken, or identity plus known_hosts for SSH. Generate SSH ones with flux create secret git, which also prints the public key for you to register as a deploy key. The provider field (generic, azure, github, and aws for CodeCommit since 2.9) selects token-fetching through cloud identity instead of a stored credential.

OCI artifacts: Git is not the only transport

Git is where humans author manifests, but it is not obliged to be where clusters read them. OCIRepository makes a container registry a source. A CI job packages a directory of manifests into an OCI artifact and pushes it; clusters pull that artifact, and the digest you deploy is the digest CI produced.

flux push artifact oci://ghcr.io/example-org/app-manifests:$(git rev-parse --short HEAD) \
  --path=./deploy \
  --source="$(git config --get remote.origin.url)" \
  --revision="$(git branch --show-current)@sha1:$(git rev-parse HEAD)"

flux tag artifact oci://ghcr.io/example-org/app-manifests:$(git rev-parse --short HEAD) --tag=staging

The two flags that record provenance, --source and --revision, matter more than they look. They are stored as OCI annotations and surface in the OCIRepository status, so when an incident happens you can go from a running revision back to a Git commit without guessing. The second command promotes by re-tagging: the bytes do not change, only a pointer moves, which is exactly the promotion model Docker's mid-level guide recommends for images (see the Docker guide).

apiVersion: source.toolkit.fluxcd.io/v1
kind: OCIRepository
metadata:
  name: app-manifests
  namespace: flux-system
spec:
  interval: 5m
  url: oci://ghcr.io/example-org/app-manifests
  ref:
    tag: staging
  verify:
    provider: cosign
    matchOIDCIdentity:
      - issuer: ^https://token\.actions\.githubusercontent\.com$
        subject: ^https://github\.com/example-org/app-repo/.*$

With verify set, source-controller checks the Cosign signature before it produces an artifact, so an unsigned or wrongly-signed push is refused at the door. ref accepts tag, semver, semverFilter or digest; pinning a digest is the immutable option. layerSelector picks which layer of the artifact to use and whether to extract it (the default for manifests) or copy it (what you want for a Helm chart, below).

Do not go gitless for a single cluster and three Kustomizations. The Git-first path is simpler, has a better review story, and is what most of this guide assumes. The OCI path pays off when you run many clusters, want signed artifacts, or cannot give clusters network access to your Git host.

Kustomizations in depth: apply, ownership and prune

The Flux Kustomization is the workhorse, and its behaviour comes from server-side apply. With client-side kubectl apply, the last-applied-configuration annotation records what you sent and the client computes a diff. With SSA, the API server tracks which field manager owns each field. When kustomize-controller applies, it becomes the owner of the fields in your manifest. A field nobody in Git mentions is left to whoever else owns it (an HPA owning replicas, a mutating webhook adding sidecars).

That ownership model explains three behaviours that otherwise look like bugs.

Flux fights you over fields it owns. If Git says replicas: 3 and you run kubectl scale --replicas=5, the next reconcile sets it back to 3. That is drift correction and it is the point. If you really want an HPA to control replicas, remove replicas from the Deployment in Git so Flux never claims the field.

Flux leaves alone fields it never declared. Adding an annotation with kubectl to a Flux-managed object survives, because Flux does not own that annotation. Only the fields in the manifest are enforced.

Changes made with kubectl client-side apply can be overwritten. Under the default policy, Flux overrides conflicts. If you genuinely need to set a field by hand and have it stick, use kubectl apply --field-manager=flux-client-side-apply, which tells the API server the change belongs to a manager Flux treats as its own. Do this only for emergencies, and put the change in Git afterwards.

For objects that should be handled differently, annotate them in Git:

metadata:
  annotations:
    kustomize.toolkit.fluxcd.io/ssa: IfNotPresent

ssa accepts four policies. Override is the default: Flux wins conflicts. Merge preserves fields that are set on the live object but absent from Git. IfNotPresent creates the object once and never touches it again, which is right for Secrets and webhook configurations that another controller populates (cert-manager-managed CA bundles are the classic example). Ignore excludes the object from apply entirely while leaving it in the inventory.

Since 2.9 there is also .spec.ignore on the Kustomization, a list of rules that tell SSA which fields to leave alone for drift purposes. It answers the common question "an operator mutates this field and Flux keeps reverting it" without per-object annotations.

Prune, and protecting what must not be pruned

prune: true is what makes Flux declarative in both directions: delete a file in Git and the object disappears. It is also how a moved directory or a wrong path deletes a namespace, taking its volumes with it. Two guard rails are worth knowing before you need them.

metadata:
  annotations:
    kustomize.toolkit.fluxcd.io/prune: Disabled

Put that annotation on Namespaces and PersistentVolumeClaims, the two kinds you can least afford to lose. The object stays even if it drops out of Git. The second guard rail is .spec.deletionPolicy, which controls what happens to the applied objects when the Kustomization itself is deleted: MirrorPrune (the default, follows prune), Delete, WaitForTermination (delete and wait for finalizers, useful when ordering teardown) and Orphan (leave everything in place). If you ever restructure Kustomizations, setting Orphan first lets you delete and recreate the wrapper without touching the workload.

Immutable fields and force

Kubernetes refuses to change some fields after creation: a Job's pod template, a Service's clusterIP, a Deployment's selector. SSA reports field is immutable and the Kustomization stalls. Two levers exist. .spec.force: true tells the controller to delete and recreate any object whose update fails on an immutable field. The annotation kustomize.toolkit.fluxcd.io/force: Enabled does the same for one object. The cost is real: recreation is a delete, so a Deployment selector change means downtime and a StatefulSet recreation needs thought. Prefer the annotation on the specific Jobs that need it over the Kustomization-wide setting.

Patching without forking

The patches and images fields let you adjust an upstream base without copying it, which is how you keep a third-party manifest pinned and customised at once:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: apps
  namespace: flux-system
spec:
  interval: 30m
  retryInterval: 2m
  timeout: 5m
  sourceRef:
    kind: GitRepository
    name: platform
  path: ./apps/prod
  prune: true
  wait: true
  patches:
    - target:
        kind: Deployment
        labelSelector: "tier=frontend"
      patch: |
        - op: add
          path: /spec/template/spec/topologySpreadConstraints
          value:
            - maxSkew: 1
              topologyKey: topology.kubernetes.io/zone
              whenUnsatisfiable: ScheduleAnyway
              labelSelector:
                matchLabels:
                  tier: frontend
  images:
    - name: ghcr.io/example-org/web
      newTag: 1.14.2

A target can select by kind, name, namespace, labelSelector and annotationSelector, so one patch can apply to every Deployment with a label, which is a neat way to impose an organisation-wide policy such as spread constraints. For how Kustomize itself resolves bases, overlays and patches, see the Kustomize guide.

Variables, environments and repository layout

The problem: staging and production run the same applications with different hostnames, replica counts and registries. Copy-pasting the manifests per environment guarantees drift between them. Flux gives you two complementary tools, Kustomize overlays for structural differences and post-build substitution for scalar values, and a layout convention that uses both.

Post-build substitution

After the build, before the apply, kustomize-controller replaces ${var} references in every manifest with values from .spec.postBuild:

spec:
  postBuild:
    substitute:
      cluster_env: prod
      replicas: "3"
    substituteFrom:
      - kind: ConfigMap
        name: cluster-vars
      - kind: Secret
        name: cluster-secrets
        optional: true

And in the manifest under apps/base:

spec:
  replicas: ${replicas:=2}
  template:
    spec:
      containers:
        - name: web
          image: registry.example.com/${cluster_env}/web:1.14.2

The syntax is the envsubst dialect: ${var}, ${var:=default} for a fallback, ${var:pos:len} for substrings and ${var/sub/repl} for replacement. Variable names must match ^[_[:alpha:]][_[:alpha:][:digit:]]*$, so a dash is not allowed. Values from substitute are literals; values from substituteFrom come from ConfigMaps and Secrets in the Kustomization's own namespace, and when both define a variable, substitute wins. One consequence is an easy pattern for cluster identity: a per-cluster ConfigMap named cluster-vars committed under clusters/<name>/ and read by every shared Kustomization.

Two smaller points. .spec.postBuild.substituteStrategy: Always, new in 2.9, forces substitution even when no variables are defined, which makes the strict check run on Kustomizations that currently have none. And since 2.9.1, every Flux CRD carries the substitute: disabled annotation, so a Flux-managed install of Flux itself no longer has its schemas corrupted by ${...} text in descriptions.

Substitution is string-level, so it cannot inject structure (a list, a nested map), and it cannot be used to choose which resources exist. For that you need overlays.

Overlays and the conventional layout

The layout from the official flux2-kustomize-helm-example is worth adopting because every team that knows Flux will recognise it:

.
├── apps
│   ├── base
│   │   └── web
│   ├── staging
│   └── production
├── infrastructure
│   ├── controllers
│   └── configs
└── clusters
    ├── staging
    │   ├── flux-system
    │   ├── infrastructure.yaml
    │   └── apps.yaml
    └── production
        ├── flux-system
        ├── infrastructure.yaml
        └── apps.yaml

clusters/<name> is the entry point Bootstrap points the flux-system Kustomization at. It contains the Flux-generated flux-system directory plus Kustomizations that point at infrastructure and apps. apps/base holds environment-neutral manifests, and apps/staging and apps/production are Kustomize overlays that reference the base and patch only what differs, typically replica counts, resource requests, ingress hosts and a HelmRelease values patch. infrastructure is split into controllers (operators, ingress controllers, cert-manager, their CRDs) and configs (resources that use those CRDs, such as a ClusterIssuer). The split exists for one reason, which is ordering: a ClusterIssuer cannot be applied until the cert-manager CRDs exist, so the configs Kustomization dependsOn the controllers one.

The effect of this layout is that promotion becomes a matter of diffs. To promote a change from staging to production you copy one line, such as an image tag or chart version, from one overlay to the other, and the review shows exactly that line. Compare that with promoting by branch merge, which moves everything at once.

Promoting with versions instead of copying lines

The alternative to editing overlays is to let each environment track a range rather than a fixed value. A HelmRelease in staging can follow version: ">=1.0.0-0" (pre-releases included) while production follows version: "1.x", or an OCIRepository in production can follow a semver range while staging follows the staging tag. It is powerful but moves the decision out of Git, since the revision the cluster runs is no longer visible in a manifest. Use it where you trust your release process, and record what ran with notifications (covered later).

Branches per environment (a staging branch and a production branch) are the layout to avoid. It makes Git history lie about what is deployed where, forces cherry-picking, and invites merge conflicts in generated files. One branch, directories per environment.

Dependencies, health checks and ordering

Git has no concept of order; a directory is a bag of files. Kubernetes tolerates much of that, since controllers retry until dependencies appear, but two situations need explicit ordering: CRDs before custom resources, and "the database is actually serving before the application migrates against it".

dependsOn is how you say it:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: infrastructure-configs
  namespace: flux-system
spec:
  interval: 1h
  retryInterval: 1m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./infrastructure/configs
  prune: true
  dependsOn:
    - name: infrastructure-controllers
  wait: true
  timeout: 5m

A Kustomization with dependsOn does not start its own reconcile until every dependency is Ready. While waiting it reports dependency 'flux-system/infrastructure-controllers' is not ready and requeues every --requeue-dependency, 30 seconds unless changed. Note that a dependency means ready at this moment, not ready once. If the dependency later flips to not-ready during a reconcile, the dependent will wait again on its next run. Also note the trap of circular dependencies: they do not get detected at apply time, they just make both sides wait forever.

What "ready" means: wait, healthChecks and kstatus

Without wait: true, a Kustomization is Ready as soon as the apply succeeds, which only means the API server accepted your objects. A Deployment that will never start is still "Ready". With wait: true, the controller waits for every applied object to become healthy, using kstatus, a library that understands the status conventions of built-in kinds (Deployments finishing their rollout, Jobs completing, PVCs binding) and of any custom resource that follows the standard Ready/Reconciling/Stalled condition conventions.

wait: true is convenient, but it checks everything, including slow objects you do not care about. The narrower alternative lists the objects that define readiness:

spec:
  wait: false
  healthChecks:
    - apiVersion: apps/v1
      kind: Deployment
      name: web
      namespace: production
  timeout: 3m

If the objects do not become healthy within timeout, the reconcile fails with health check failed after 3m0s: timeout waiting for: [Deployment/production/web status: 'InProgress']. The part after status: tells you what to do: InProgress means the workload is starting too slowly or is crash-looping, Failed means a controller reported a terminal failure, and NotFound means the object was never created, which usually means a namespace typo in the health check itself.

Custom health logic with CEL

kstatus does not know what "healthy" means for a custom resource that does not follow the condition conventions, for example a Crossplane claim or a database operator's own CRD. healthCheckExprs lets you teach it with CEL expressions:

spec:
  wait: true
  healthCheckExprs:
    - apiVersion: databases.example.com/v1
      kind: PostgresCluster
      current: "status.phase == 'Running' && status.readyReplicas == spec.replicas"
      inProgress: "status.phase == 'Creating'"
      failed: "status.phase == 'Failed'"

current is the healthy state, inProgress means keep waiting and failed means stop waiting immediately. Since 2.9 you may omit kind for some uses, but write it anyway unless you know you need that. HelmReleases got the same field in 2.8, so a chart that installs a custom resource can declare its own health logic.

Shorter feedback loops

By default, when a health check is waiting on a broken revision and a new fixed commit arrives, the controller first finishes waiting on the broken one, up to the full timeout. That is a slow way to ship the fix. The CancelHealthCheckOnNewRevision feature gate, available on both kustomize-controller and helm-controller since 2.8, cancels the in-flight wait as soon as a newer revision appears. It is off by default, and it is one of the highest-value flags to turn on for clusters where people iterate on live environments.

Dependencies can also carry a readyExpr, a CEL expression evaluated over the dependency's object, for when "Ready" is not the right gate. A typical example is gating on a specific status field a migration job publishes, rather than on the job's container state. If the expression evaluates to false, the dependent reports dependency '...' is not ready according to readyExpr eval.

HelmRelease in depth

Beginner showed a HelmRelease that installs a chart. The mid-level questions are about what happens when an install or upgrade goes wrong, how Flux decides a release has drifted, and which defaults changed when helm-controller moved to Helm v4. Flux 2.9.5 bundles Helm v4.2.4, and the defaults below reflect that.

Chart source: chart versus chartRef

A HelmRelease can describe its chart in two ways. .spec.chart.spec is a template: helm-controller creates a HelmChart object from it (named <namespace>-<release>), source-controller fetches and packages the chart, and the release consumes the result. This works with HelmRepository and GitRepository sources. .spec.chartRef is a direct reference to an existing OCIRepository or HelmChart, skipping the template step.

For charts in OCI registries, prefer OCIRepository with chartRef. HelmRepository with type: oci has been in maintenance mode since source-controller 1.8:

apiVersion: source.toolkit.fluxcd.io/v1
kind: OCIRepository
metadata:
  name: podinfo
  namespace: default
spec:
  interval: 10m
  url: oci://ghcr.io/stefanprodan/charts/podinfo
  layerSelector:
    mediaType: "application/vnd.cncf.helm.chart.content.v1.tar+gzip"
    operation: copy
  ref:
    semver: ">= 6.0.0"
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: podinfo
  namespace: default
spec:
  interval: 10m
  chartRef:
    kind: OCIRepository
    name: podinfo
  values:
    replicaCount: 2

Two implications. First, the chart's digest is tracked, so a republished chart under the same version is detected, where a plain HelmRepository index would not show it. Second, the chart is a first-class source object, which means you get flux get sources oci and Cosign verification on it for free.

When you do use .spec.chart.spec, note reconcileStrategy. The default, ChartVersion, re-packages the chart only when its version changes. For a chart living in a Git repository whose version: you never bump, that means edits to templates are never noticed; set reconcileStrategy: Revision so that any new source revision produces a new chart.

Values from everywhere

values carries inline overrides, and valuesFrom merges ConfigMaps and Secrets, in order, before inline values (which therefore win):

spec:
  valuesFrom:
    - kind: ConfigMap
      name: web-common-values
    - kind: Secret
      name: web-secret-values
      valuesKey: values.yaml
    - kind: Secret
      name: db
      valuesKey: password
      targetPath: database.password
      optional: true

A targetPath sets a single value at a path, which is how a one-line Secret becomes one key in the chart's values. Since 2.9, literal: true on an entry is the equivalent of Helm's --set-literal, meaning the value is taken as-is and commas and dots in it are not parsed, which fixes the long-standing pain of passing a certificate or a password with special characters. To see the merged result that the controller will feed to Helm, use the debug command:

flux debug hr podinfo --show-values

Remediation: what Flux does when Helm fails

A Helm release can fail at install or upgrade: a hook job errors, pods never become ready, the API server rejects a rendered object. Remediation is how the controller reacts. Configure it explicitly, because the defaults are conservative:

spec:
  install:
    remediation:
      retries: 3
  upgrade:
    remediation:
      retries: 3
      remediateLastFailure: true
      strategy: rollback
  rollback:
    cleanupOnFail: true
  test:
    enable: false

With retries: 3, a failed upgrade is retried up to three times, each failing attempt followed by a rollback (strategy: rollback) or an uninstall (strategy: uninstall). remediateLastFailure: true makes the last failed attempt be remediated too, so the release ends up in a known-good state instead of stranded in failed. When retries are exhausted the release reports upgrade retries exhausted and the controller stops trying until something changes.

The two named strategies for actions differ in what they do between retries. RemediateOnFailure is the classic behaviour: remediate, then retry with the same chart and values. RetryOnFailure, added in 2.7, simply retries the action after retryInterval with no rollback or uninstall in between. It suits installs where a hook needs a few tries to succeed, or upgrades you would rather not roll back because the rollback itself is risky. Choose it per action:

spec:
  install:
    strategy:
      name: RetryOnFailure
      retryInterval: 5m

When a release is stuck after exhausting retries, the recovery toolkit is, in order of increasing force:

flux reconcile hr podinfo -n default --reset    # clear the failure counter and try again
flux reconcile hr podinfo -n default --force    # force a one-off upgrade
flux suspend hr podinfo -n default && flux resume hr podinfo -n default

Drift detection

Plain Helm has no runtime drift correction: helm upgrade compares the last manifest Helm stored against the new one, so a manual kubectl edit to a deployed object is invisible to it. helm-controller can do better with .spec.driftDetection:

spec:
  driftDetection:
    mode: enabled
    ignore:
      - paths: ["/spec/replicas"]
        target:
          kind: Deployment

The modes are disabled (the default), warn (report drift in events and the Drifted condition without fixing it) and enabled (correct it). Start with warn on an existing release to see how noisy it is, because webhooks, HPAs and operators often change fields legitimately; the ignore list with JSON Pointer paths and a target silences known, legitimate differences. The old feature gates DetectDrift and CorrectDrift are deprecated; do not use them in new configuration.

Server-side apply and waiting

When helm-controller moved to Helm v4 in Flux 2.8, two defaults changed, and they matter on upgrade day. New releases are installed with server-side apply (.spec.install.serverSideApply defaults to true). For upgrades and rollbacks, serverSideApply takes enabled, disabled or auto, defaulting to auto, which reuses whatever method the last release used. So existing releases stay on client-side apply until you opt them in with enabled. Switching is worthwhile once you are comfortable, because SSA gives Helm-managed objects the same ownership model as everything else Flux applies.

Health assessment also changed. The default wait strategy is now poller, which uses kstatus, the same library kustomize-controller uses. Set .spec.waitStrategy.name: legacy to go back to Helm's built-in waiting. If you want all the Helm 3 behaviours back at once while you stage a migration, there is a UseHelm3Defaults feature gate for helm-controller.

Flux 2.9 adds a second behaviour to know about. The new .spec.postRenderStrategy field says whether hooks are passed through post-renderers. It defaults to combined, meaning hooks are sent to post-renderers along with the regular manifests. Before 2.9, hooks were excluded, which is the nohooks strategy. If you have a Kustomize post-renderer with patches that assumed only regular manifests, for example a blanket patch on every Job, it now also patches hook Jobs. Preserve the old behaviour with postRenderStrategy: nohooks.

Release bookkeeping

Helm stores release history as Secrets in the release's storage namespace, and keeps five revisions by default (maxHistory: 5). The Secret holds the rendered manifest and the values, and it has a size limit. A chart that bundles large CRDs or thousands of lines of generated YAML can exceed it and fail with Request entity too large: limit is 3145728. Shrink the chart with a .helmignore, or move the big CRDs out of the chart and manage them as plain manifests. CRD handling has its own knobs: install.crds and upgrade.crds accept Skip, Create and CreateReplace. The default for upgrade is Skip, meaning Helm does not upgrade CRDs in a chart's crds/ directory, which is how many people end up with old CRDs and new controllers. Set upgrade.crds: CreateReplace for charts where you want CRD updates, or manage CRDs separately in the infrastructure/controllers Kustomization.

The Helm mechanics underneath this, such as values precedence, hooks and templating, are covered in the Helm guide.

Secrets in Git: SOPS and the alternatives

The problem: GitOps says everything lives in Git, but a database password in a public or widely-readable repository is a breach. You have two families of answer. Either encrypt the secret and commit the ciphertext, or keep the secret elsewhere and commit only a reference. Flux supports the first natively and composes with the second.

SOPS built into kustomize-controller

SOPS encrypts the values of a YAML file and leaves the keys in plain text, so diffs and reviews stay useful: a reviewer sees that data.password changed without seeing what it changed to. kustomize-controller decrypts during the build step, after fetching and before substitution. The decryption backends are Age (including the post-quantum hybrid keys added in 2.9), OpenPGP, AWS KMS, Azure Key Vault, GCP KMS and OpenBao or Vault transit. Age is the simplest and is where you should start.

age-keygen -o age.agekey
# prints: Public key: age1q2l8...

kubectl create secret generic sops-age \
  --namespace=flux-system \
  --from-file=age.agekey=age.agekey

cat > .sops.yaml <<'YAML'
creation_rules:
  - path_regex: .*\.enc\.yaml$
    encrypted_regex: ^(data|stringData)$
    age: age1q2l8...
YAML

kubectl create secret generic db-credentials \
  --from-literal=password='s3cr3t' --dry-run=client -o yaml > apps/prod/db-credentials.enc.yaml
sops --encrypt --in-place apps/prod/db-credentials.enc.yaml

The private key goes into the cluster once, by hand, as the Secret sops-age; it is the one secret that must never enter Git. You can also generate secrets for Flux itself the same way, for example flux create secret git ... --export | sops -e. Then the Kustomization declares that decryption is expected:

spec:
  decryption:
    provider: sops
    secretRef:
      name: sops-age

The key names in the decryption Secret are significant: anything ending .agekey is an Age identity, sops.asc an OpenPGP key, sops.vault-token a Vault token, and sops.aws-kms, sops.azure-kv and sops.gcp-kms carry cloud credentials. With cloud KMS you usually prefer workload identity over stored credentials: annotate the kustomize-controller ServiceAccount and the controller authenticates as the cloud identity, so no long-lived key sits in the cluster. Since 2.9 Vault and OpenBao decryption can also use Kubernetes workload identity through a JWT auth method, replacing a static token. On a single-tenant cluster, the controller flag --sops-age-secret sets one global Age key, so individual Kustomizations need not name one.

Two mistakes recur. If a Kustomization applies a SOPS-encrypted file but has no decryption block, the error is <kind>/<ns>/<name> is SOPS encrypted, configuring decryption is required for this secret to be reconciled; the fix is to add the block, not to re-encrypt. If decryption is configured but fails, failed to decrypt and format or cannot get sops age decryption Secret points at the wrong or missing key Secret, or, for KMS, a permissions problem on the controller's identity.

When to choose something else

SOPS has one structural limit: the secret value is in Git, encrypted. If your policy says secrets must never be in the repository in any form, or you rotate secrets frequently and do not want a commit per rotation, use an external secret manager and commit a pointer. The External Secrets Operator reads from AWS Secrets Manager, Azure Key Vault, GCP Secret Manager or HashiCorp Vault and creates Kubernetes Secrets, with only an ExternalSecret manifest in Git. Sealed Secrets encrypts to a controller-held key, so the ciphertext is safe in Git but tied to one cluster. All three work well beside Flux; ordering is the only thing to get right, because the operator's CRDs must exist before the manifests that use them (the controllers and configs split again). See the Vault guide for the secret-manager side. In the Gulf and Egypt, where data-residency rules often dictate which region a KMS key can live in, check that the region of the key matches your policy before you standardise; SOPS with a regional cloud KMS key is an easy way to keep key material in-country.

Multi-tenancy: limiting what a team can deploy

By default, kustomize-controller and helm-controller are bound to cluster-admin, and a Kustomization applies objects with that authority. On a cluster shared by several teams, that means anyone who can commit a Kustomization to a path Flux watches can create a ClusterRoleBinding and own the cluster. Multi-tenancy in Flux is the set of controls that turn this into "a team can deploy only into its own namespaces".

The mechanism is impersonation. Setting .spec.serviceAccountName on a Kustomization or HelmRelease makes the controller apply with that ServiceAccount's permissions instead of its own. A tenant's ServiceAccount, bound by a RoleBinding to their namespace, can create Deployments there and nothing else. The Flux CLI generates the scaffolding:

flux create tenant team-a --with-namespace=team-a --export > tenants/team-a/rbac.yaml

This produces the Namespace, a ServiceAccount, and a RoleBinding. Its --cluster-role flag sets which ClusterRole the binding grants inside the tenant namespace (it defaults to cluster-admin, which is cluster-admin within that namespace only, but you may prefer a narrower role), and since 2.8 --skip-namespace omits creating the Namespace when something else owns it. The tenant's Kustomization then uses the account:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: team-a-apps
  namespace: team-a
spec:
  interval: 10m
  serviceAccountName: team-a
  sourceRef:
    kind: GitRepository
    name: team-a-repo
  path: ./deploy
  prune: true

Impersonation alone is not a lock, because a tenant can simply omit serviceAccountName. The lock is at the controller. Patch the controllers (in the flux-system kustomization file in Git) with these flags:

• --no-cross-namespace-refs=true on the controllers, so a tenant's object cannot reference a source, Secret or ConfigMap in another namespace.
• --no-remote-bases=true on kustomize-controller, so a tenant cannot pull a Kustomize base from an arbitrary URL.
• --default-service-account=default on kustomize-controller and helm-controller, which makes objects that set no serviceAccountName run as default in their own namespace rather than with the controller's cluster-admin. The related --default-decryption-service-account and --default-kubeconfig-service-account flags do the same for those two features.

The flux-system Kustomization, which bootstraps everything, keeps cluster-admin because platform owners write it. Everything below it is scoped. To close the last gap, enforce with an admission policy (Kyverno or Gatekeeper) that every Kustomization and HelmRelease outside flux-system sets a serviceAccountName, since a policy engine is the only thing that can reject a non-compliant manifest at admission time. The official flux2-multi-tenancy repository shows the whole arrangement.

Also know what this does not cover. Impersonation limits what objects can be applied; it does not limit CPU, and it does not stop a tenant from writing a Kustomization that reconciles every 10 seconds. Pair it with resource quotas, and a floor on interval enforced by policy. For the Kubernetes side of RBAC and quotas, see the Kubernetes guide.

Notifications, webhooks and what to alert on

Polling has a latency cost (a five-minute interval means up to five minutes before a push is noticed) and an API cost (every poll is a Git or registry call, and GitHub rate limits are real). Receivers remove the polling latency: an inbound webhook that tells Flux to reconcile now. Providers and Alerts send events outward so people learn about failures without running flux get.

Sending events: Provider and Alert

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: slack
  namespace: flux-system
spec:
  type: slack
  channel: deployments
  secretRef:
    name: slack-webhook-url
---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
  name: failures
  namespace: flux-system
spec:
  providerRef:
    name: slack
  eventSeverity: error
  eventSources:
    - kind: Kustomization
      name: "*"
    - kind: HelmRelease
      name: "*"
      namespace: "*"

Note the apiVersion: Alert and Provider are still v1beta3. That is correct and current, even though most Flux APIs are v1; do not "fix" it. An Alert filters events by eventSeverity (info or error), by source kind, name and namespace, and optionally by inclusion and exclusion regular expressions on the message. The Provider's type covers chat (Slack, Teams, Discord, Google Chat), incident tools (PagerDuty, Opsgenie), Alertmanager and Grafana annotations, and Git hosts for commit status (GitHub, GitLab, Gitea, Bitbucket, Azure DevOps).

Commit status is the most underused. A Provider of type github plus an Alert on the Kustomization writes the reconcile result onto the commit that was deployed, so the GitHub commit list shows a green or red mark saying "this commit reached this cluster". Since 2.8, pull-request comment providers also exist. Pair this with eventSeverity: info for the Kustomizations that matter, because the commit-status Provider needs the success event, not only failures.

A sensible starting policy: error severity from everything to a chat channel; info from production Kustomizations to a separate "deploys" channel as an audit trail; and page on Stalled, not on any single failure, because transient errors self-heal.

Receiving events: Receiver

apiVersion: notification.toolkit.fluxcd.io/v1
kind: Receiver
metadata:
  name: github-push
  namespace: flux-system
spec:
  type: github
  events:
    - "ping"
    - "push"
  secretRef:
    name: webhook-token
  resources:
    - kind: GitRepository
      name: platform

The Secret holds a token key, and notification-controller derives a URL of the form /hook/<sha256> (hashing the token, name and namespace) and publishes it in the Receiver's status. You register that URL, exposed through an Ingress or Gateway route, as the webhook in your Git host. When a push arrives, Flux annotates the listed resources with a reconcile request, which is reconcile.fluxcd.io/requestedAt, the same thing flux reconcile does. Keep a long interval as a safety net, since webhooks can be lost, and use the Receiver for speed. The 2.9 additions are generic-oidc Receivers that need no shared secret and a resourceFilter CEL expression to decide which resources an event applies to. Receivers for container registries (dockerhub, quay, harbor, gcr, acr, nexus) make image updates near-instant. Note that GCR Receivers now require email and audience keys in their Secret since 2.9, a security fix. Test a Receiver from a terminal with flux trigger receiver.

The only inbound traffic Flux ever needs is for Receivers. Bootstrap's NetworkPolicies already allow that port, and since 2.9.4 the allow-webhooks policy is restricted to the receiver port.

Image automation: Flux commits image updates to Git

The problem: CI builds an image, and now a human has to edit a tag in Git. Image automation closes that loop while keeping Git as the source of truth: the cluster detects a new image, and Flux commits the change to Git, which Flux then deploys like any other commit. Nothing is ever changed in the cluster behind Git's back.

It is an optional set of controllers, installed at bootstrap with --components-extra=image-reflector-controller,image-automation-controller. The APIs are image.toolkit.fluxcd.io/v1, GA since Flux 2.7. The v1beta2 versions were removed in 2.9, which is a common source of errors in older tutorials.

Three objects cooperate. An ImageRepository scans a registry repository for tags. An ImagePolicy picks the newest acceptable tag. An ImageUpdateAutomation rewrites manifests in Git where markers say to.

apiVersion: image.toolkit.fluxcd.io/v1
kind: ImageRepository
metadata:
  name: web
  namespace: flux-system
spec:
  image: ghcr.io/example-org/web
  interval: 5m
---
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImagePolicy
metadata:
  name: web
  namespace: flux-system
spec:
  imageRepositoryRef:
    name: web
  filterTags:
    pattern: '^main-[a-f0-9]+-(?P<ts>[0-9]+)$'
    extract: '$ts'
  policy:
    numerical:
      order: asc
---
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImageUpdateAutomation
metadata:
  name: flux-system
  namespace: flux-system
spec:
  interval: 30m
  sourceRef:
    kind: GitRepository
    name: flux-system
  git:
    checkout:
      ref:
        branch: main
    commit:
      author:
        name: fluxcdbot
        email: fluxcdbot@example.org
      messageTemplate: "chore: update images {{range .Changed.Changes}}{{.OldValue}} to {{.NewValue}}; {{end}}"
    push:
      branch: main
  update:
    path: ./apps/staging
    strategy: Setters

The policy block is where design matters. semver with a range (>=1.0.0 <2.0.0) follows release tags. alphabetical and numerical sort tag strings or numbers. For CI-built tags that are not semver, such as main-3f2a9c1-1718000000, the trick above is to filterTags with a regular expression that extracts a timestamp and sort numerically on it. Sorting on a commit SHA is meaningless because SHAs are not ordered.

The markers go in the manifest you want rewritten:

containers:
  - name: web
    image: ghcr.io/example-org/web:main-3f2a9c1-1718000000 # {"$imagepolicy": "flux-system:web"}

The marker names the policy as namespace:name. You can also mark only the tag with :tag or only the name with :name, for manifests that split them. The automation controller edits the file, commits and pushes, and the next Git reconcile deploys.

Operational points that catch people out. The automation needs write access to Git, which is a deploy key or token with push rights, not the read-only credential of your normal source. Bootstrap's --read-write-key flag exists for this. If branch protection blocks direct pushes to main, push to a separate branch through push.branch and let a pull request promote it, which also gives you a review gate on automated bumps. In 2.9 the commit can be signed with an SSH key, and refspecs can no longer force-update or delete. And restrict the automation to apps/staging: pointing an automation at production is possible but rarely a good idea, because it removes the human from the path you most want a human on. Promote to production with a reviewed pull request or a re-tag instead.

The image-reflector-controller stores what it scans in a small embedded database; on small nodes it can be memory-starved and fail with unable to open the Badger database. Give it a bit more memory before assuming anything deeper is wrong.

Previewing and testing changes before they merge

The cheapest bug is the one CI finds. Flux ships three commands for that, and a reviewer who has seen flux diff output will never go back to reading raw YAML.

flux build ks apps --path ./apps/prod > /tmp/rendered.yaml
flux diff ks apps --path ./apps/prod
flux envsubst --strict < apps/base/web/deployment.yaml

flux build ks renders what kustomize-controller would apply, including patches, decryption (if you have the key) and substitution from the live cluster's ConfigMaps. flux diff ks compares that rendered output with the live cluster using a server-side dry-run, and prints a diff, so you see exactly which fields a pull request would change. Useful flags include --recursive to follow nested Kustomizations, --ignore-not-found when a dependency does not exist yet, --in-memory-build to skip writing to disk, and --local-sources GitRepository/flux-system/platform=./ so the diff uses your working copy instead of what the cluster last fetched. In a CI job with read access to the target cluster's API, this produces a pull-request comment showing the real effect of a change.

For validation without a cluster, flux envsubst --strict reproduces the 2.9 strict-substitution failure locally, and the schema plugin (installed with flux plugin install schema@<version>) validates manifests against JSON schemas and CEL rules. Plugins, new in 2.9, are installed, listed and updated with flux plugin, and can be pinned by version or by digest. Combine client-side checks with kustomize-controller's own dry-run, which catches what static validation cannot: admission webhooks, immutable fields, quota.

A pull-request pipeline that pays for itself has four steps: render every Kustomization (flux build), validate the output against schemas, run flux diff against the target cluster for the changed environment only, and post the diff as a comment. A merge pipeline then does nothing Flux-specific at all. The cluster pulls the commit.

Debugging beyond the basics

Beginner's triage was "look at flux get all". At this level the failures are subtler. The habit that pays off most is to work from the symptom to the layer: Is the source ready? Did the build succeed? Did the apply succeed? Did the health check pass? Each layer has a distinct error family, and the first not-ready object in that chain is nearly always the real problem.

flux check
flux get all -A --status-selector ready=false
flux events -A
flux logs --level=error -A
kubectl get events -n flux-system --field-selector type=Warning

Run these in that order. The status selector filters to the broken objects across all namespaces, events add the recent history, and logs give controller-level detail.

Tracing backwards from a resource. flux trace deployment/podinfo -n default walks from a live object to the Kustomization or HelmRelease that manages it, and on to the source and revision. flux tree ks flux-system goes the other way, printing everything a Kustomization applied in a hierarchy. Between them you can answer "who owns this?" and "what does this own?" without reading any YAML.

Seeing the inputs. flux debug ks my-app --show-vars prints the substitution variables the controller resolved, which is decisive for "why did it render that hostname". flux debug hr podinfo --show-values does the same for a HelmRelease's merged values. flux export source git --all dumps source definitions so you can diff them against Git.

Forcing the question. When a change seems not to arrive, distinguish the layers: flux reconcile source git <name> tells you whether the source has the commit, then flux reconcile ks <name> whether the build and apply succeed. flux get sources all -A shows each source's revision so you can compare it with git rev-parse HEAD.

A few specific failure patterns:

• kustomization path not found. The path is relative to the artifact root, and the source's ignore or sparseCheckout may have removed it. Compare against flux pull artifact or the unpacked artifact.
• webhook does not support dry run. SSA validates with a dry run, and an admission webhook that does not declare sideEffects: None or NoneOnDryRun rejects it. The fix is in the webhook's configuration, not in Flux.
• Constant "configured" noise on every reconcile. Empty fields (null, {}, []) in a manifest make the API server return a slightly different object each time. Remove the empty fields from the source.
• A Helm install that hangs at InProgress. Since helm-controller 1.5, health uses kstatus, and a chart whose CRD does not report standard conditions may wait the full timeout. Use healthCheckExprs or set the wait strategy as described earlier.
• values files merge error: no values file found at path. A valuesFiles path is wrong; fix it or set ignoreMissingValuesFiles.
• Git errors such as failed to checkout and determine revision. This is almost always auth, the branch name, or egress; check .spec.ref, the Secret, and whether the cluster can reach the Git host.

When all that fails, raise the log level on a controller temporarily (--log-level=debug), ideally by patching the Deployment through Git so the change is tracked, and read the reconcile in full. Remove it afterwards; debug logs are large.

Observability: metrics, logs and status

Status and events tell you about one object at a time. To know about a fleet, you need metrics. Every Flux controller exposes Prometheus metrics on :8080/metrics. The ones that matter are gotk_reconcile_duration_seconds_bucket{kind,name,namespace}, a histogram of how long reconciles take, controller_runtime_reconcile_total{controller,result}, a count of reconciles by outcome (watch the error result), and the workqueue_* family, whose depth and latency tell you whether a controller is keeping up.

There is a subtlety that catches people: Flux itself does not export "this Kustomization is not Ready" as a metric. The gotk_resource_info series, with labels such as ready and suspended, comes from kube-state-metrics configured with Custom Resource State for Flux's CRDs. The official flux2-monitoring-example repository provides the whole stack, kube-prometheus-stack plus PodMonitors and Grafana dashboards, and it is itself deployed with Flux, so it is a good worked example of everything in this guide. For the tools involved see the Prometheus guide and the Grafana guide.

Alerts worth having, in order of value: any Flux resource with ready="False" for longer than a threshold (longer than the interval plus the retry interval, to avoid paging on transient retries); any resource suspended for more than a day, because someone suspended a Kustomization to debug and forgot; reconcile duration p99 rising steadily, an early signal of a controller running out of CPU or a repository growing too large; and work queue depth growing without draining. Logs are structured JSON when --log-encoding=json is set, which makes them searchable by kind and name in whatever log backend you use.

Upgrading Flux and the API removals

Flux upgrades are routine when you are current and painful when you are two releases behind, because removed API versions make old manifests and stored objects unusable. Learn the removal schedule once:

Current versions: sources, Kustomization and Receiver are v1, HelmRelease is v2, image APIs are v1, and Alert and Provider are v1beta3. A manifest with a removed version fails with the standard Kubernetes no matches for kind "HelmRelease" in version "helm.toolkit.fluxcd.io/v2beta1".

The official upgrade procedure from 2.6 or earlier to 2.9 has four steps, and the order matters:

1. Fix Git first. Run flux migrate -v 2.6 -f . in your repository to rewrite manifests to current API versions, commit, push, and run flux reconcile ks flux-system --with-source. Use --dry-run first, and --extensions=.yml,.yaml,.tpl if your manifests use other suffixes.
2. Migrate stored objects. Run flux migrate against each cluster (it is idempotent, and accepts --context and --kubeconfig). This rewrites objects already stored in etcd so they no longer reference removed versions. It can also run as a Job under the kustomize-controller ServiceAccount.
3. Upgrade the controllers. Re-run flux bootstrap with the original flags, or merge the pull request opened by the Flux GitHub Action. Verify with flux check.
4. If you use image automation, run flux migrate -v 2.8 -f . in Git and flux migrate on the clusters again.

If you run Flux through the Flux Operator, it must be v0.53.0 or later before you move to 2.9. It migrates stored objects itself, but you still migrate manifests in Git. For routine updates after you are current, flux install --export > clusters/prod/flux-system/gotk-components.yaml with your usual --components-extra, committed and reconciled, is the Git-native way; the Flux GitHub Action can open that pull request for you on a schedule, which turns upgrades into a reviewable diff. Always upgrade CRDs with the controllers: 2.9.4 changed the ArtifactGenerator and ImageUpdateAutomation schemas.

Before you upgrade to 2.9 specifically, review the behaviour changes from earlier in this guide, because they do not raise API errors, they change outcomes: strict post-build substitution, the postRenderStrategy default of combined, helm-controller's kstatus waiting and SSA defaults for new releases, and 2.9.5's rejection of kubeconfig Secrets that reference local files (credentials must be embedded inline). Upgrade staging first, let it soak for a few days of real reconciliation, and then promote. Also keep the Kubernetes version within what 2.9 supports, which is 1.34 to 1.36, since the project supports only the latest three minors.

Putting it all together

Here is a compact platform setup that uses most of what this guide covered, described as a sequence you could follow on a practice cluster.

1. Bootstrap the cluster with the four default controllers plus the image controllers, pointing at clusters/staging, with a read-write key so image automation can push.
2. Lay out the repository as clusters/, infrastructure/controllers, infrastructure/configs, apps/base and apps/staging.
3. Create two infrastructure Kustomizations: controllers (with wait: true) and configs (dependsOn: controllers, with retryInterval: 1m).
4. Create an apps Kustomization with prune: true, wait: true or explicit healthChecks, timeout: 5m, retryInterval: 2m, and postBuild.substituteFrom pointing at a cluster-vars ConfigMap.
5. Deliver a chart through an OCIRepository and chartRef with remediation retries: 3, driftDetection.mode: warn, and Secrets supplied by valuesFrom from a SOPS-encrypted Secret.
6. Add notifications: error alerts to chat, info events for production to a deploy channel, and a GitHub commit-status Provider.
7. Add a Receiver so pushes reconcile immediately, keeping a long interval as a fallback.
8. Turn on image automation for the staging path only, with an ImagePolicy tuned to your tag format.
9. Turn on CancelHealthCheckOnNewRevision for both controllers.
10. Wire CI: render, validate and flux diff on pull requests, pin the Flux CLI version with the GitHub Action, and open an upgrade pull request on a schedule.
11. Scope tenants: flux create tenant per team, controller flags for no cross-namespace references and a default ServiceAccount, and an admission policy.
12. Monitor: kube-state-metrics Custom Resource State, PodMonitors, and alerts on not-ready and long-suspended resources.

A staging commit then flows like this. CI builds an image and pushes it; image automation commits the tag to apps/staging; the Receiver triggers a reconcile; kustomize-controller builds, substitutes, applies and waits; a Slack message or a commit status reports the result. Promotion to production is a pull request that copies one tag line from the staging overlay to the production overlay, reviewed as a diff, merged, reconciled, and alerted. Nothing in that path needs a CI job to hold cluster credentials.

# the morning-after check for a platform you own
flux get all -A --status-selector ready=false     # anything broken?
flux get ks -A --show-source                      # what is each Kustomization following?
flux stats -A                                     # how many objects, how many suspended?
kubectl get kustomization -A -o json \
  | jq -r '.items[] | select(.spec.suspend==true) | .metadata.name'   # forgotten suspensions

Where you are now

You can now describe exactly what a reconcile does and use that to predict behaviour: what "Ready" does and does not promise, why a long interval needs a retry interval, and how the inventory drives pruning. You can structure a repository so that environment differences are a diff, use variables without being bitten by strict mode, order deployments with dependencies and real health checks, and tune a HelmRelease's remediation, drift detection and apply mode. You can keep secrets in Git with SOPS, scope teams with impersonation and controller flags, automate image updates through Git, preview changes in CI, trace problems across layers, watch a fleet with metrics, and upgrade across the API removals without surprises.

What is left for the senior level is the platform view: sharding and vertical scaling of the controllers, the trust model and supply-chain verification, running Flux for many teams and clusters, the Flux Operator, and deciding when another tool is the better answer. Related guides in this catalogue are Kubernetes, Helm, Kustomize, Argo CD for the other major GitOps tool, and Terraform, which commonly provisions the cluster Flux then manages.

Sources

• Flux documentation
• Flux concepts
• Flux components
• GitRepository
• OCIRepository
• HelmRepository
• Kustomization
• HelmRelease
• Receivers
• Alerts
• Providers
• Image automation components
• Automate image updates to Git
• Manage Kubernetes secrets with SOPS
• Secrets management
• Security
• Multi-tenancy configuration
• Helm drift detection
• CEL health checks
• Troubleshooting cheatsheet
• Flux CLI reference
• Upgrading Flux
• Upgrade Procedure for Flux v2.7+
• Flux 2.9 release announcement
• Flux 2.8 release announcement
• Flux 2.7 release announcement
• Flux monitoring metrics
• Flux GitHub Action
• flux2 releases on GitHub
• flux2-kustomize-helm-example
• flux2-multi-tenancy
• flux2-monitoring-example

Part two of three. A cumulative review of Beginner and Mid-level material, organised around the questions a hiring panel asks someone who already runs Flux at work. Interviewers at this level do not ask "what is GitOps". They describe a broken situation and watch how you narrow it down, then ask you to defend a design choice against an alternative. Senior reviews all three levels. The guide is written against Flux v2.9.5.

Where this picks up

The thirty-second answer, updated

Say this when asked what Flux is, and notice that it goes a level deeper than the beginner version.

Flux is a set of Kubernetes controllers that continuously reconcile a cluster to a declared state stored in Git, an OCI registry or a bucket. Source controllers fetch content into versioned artifacts, and the Kustomize and Helm controllers apply those artifacts with server-side apply, prune what was removed, and check health. Because the agents run inside the cluster and pull, CI never holds cluster credentials.

Two phrases carry the weight: "server-side apply" shows you know why Flux enforces only fields it declared, and "CI never holds cluster credentials" shows you know why pull beats push for security, not just for convenience.

The reconcile loop, from memory

Expect to be asked to walk through what happens between a git push and a running pod. A strong answer is an ordered list, with the failure you would see at each step:

1. The GitRepository fetches on its interval, or immediately if a Receiver fired, and publishes a new artifact with a revision such as main@sha1:abc123. If this fails, flux get sources git shows it and nothing downstream moves.
2. The Kustomization notices the new revision. If it has dependsOn, the dependencies must be ready, otherwise you see dependency '...' is not ready.
3. It downloads the artifact, runs kustomize build on .spec.path, decrypts SOPS files, and substitutes ${var} references.
4. It applies with server-side apply, dry run first. Webhook and immutable-field errors appear here.
5. It prunes objects in the old inventory that are not in the new build.
6. It waits for health if wait: true or healthChecks is set, failing with a timeout message that names the object and its status.
7. It writes the result to .status and emits an event, which an Alert may forward.

A weak answer says "Flux syncs Git to the cluster". The difference is that the strong answer tells the interviewer which step to look at when something goes wrong, and so is obviously the answer of someone who has debugged it.

A follow-up you should expect: "What is the difference between interval and retryInterval?" The interval is how often the whole loop re-runs to correct drift, and a new source revision triggers it early. The retry interval is how long to wait after a failure. If you omit it, the interval is used, so a one-hour interval means a transient failure is retried an hour later.

The commands to have ready

flux get all -A --status-selector ready=false       # what is broken, across the cluster
flux get ks -A --show-source                        # what is each thing following
flux events -A                                      # recent history
flux logs --level=error -A                          # controller errors
flux trace deployment/web -n prod                   # from a live object back to Git
flux tree ks flux-system                            # from a Kustomization down to its objects
flux build ks apps --path ./apps/prod               # what would be applied
flux diff ks apps --path ./apps/prod                # how that differs from the cluster
flux debug ks apps --show-vars                      # resolved substitution variables
flux debug hr web --show-values                     # merged Helm values
flux reconcile ks apps --with-source                # go now, source included
flux reconcile hr web --reset                       # clear a stuck release's failure count
flux suspend ks apps && flux resume ks apps         # pause and continue
flux migrate -v 2.6 -f .                            # rewrite manifests for removed APIs

If you can say what each one is for without looking, you pass the "have you actually operated it" filter. The two that separate candidates are flux trace and flux diff.

Scenario questions

"I pushed a fix and nothing changed in the cluster. What do you check?"

Weak answer: "Run flux reconcile." Strong answer: work down the layers.

First, did the source see the commit? Compare flux get sources git revision to git rev-parse HEAD. If they differ, the problem is the fetch: credentials, branch name, or a long interval with no Receiver. Then check the Kustomization: flux get ks, and compare lastAttemptedRevision with lastAppliedRevision in its status. If the attempted revision is newer than the applied one, Flux tried and failed, and the message says why. If the attempted revision is still the old one, the Kustomization has not picked up the new source revision: it may be suspended, waiting on a dependency, or following a different source than you assumed (--show-source tells you). If both are current and Ready, the commit did land, and the question becomes whether it changed anything you expected: a wrong path, an ignore rule that removed the file from the artifact, or a field Flux does not own. Finally, run flux diff ks to see the difference between what Git renders and what the cluster holds.

What a strong answer includes: the two revisions comparison, suspension as a candidate, and ending with a diff rather than a guess.

"A Kustomization is stuck with health check failed after 5m0s ... InProgress. What do you do?"

The message means the apply worked and a workload did not become healthy in timeout. Do not touch Flux first. The object named in the message is a Deployment or similar, so investigate it as Kubernetes: kubectl describe, the pod events, the logs. Usual causes are an image that cannot be pulled, a failing readiness probe, insufficient resources, or a missing Secret. Once the workload is fixed, either wait for the next reconcile or run one. Then address the pain: a five-minute wait for each bad commit is slow, so mention the CancelHealthCheckOnNewRevision feature gate, which cancels the wait when a newer revision arrives. Also mention whether the timeout is sensible for the workload's real start time, and whether wait: true is checking too much when only healthChecks on the critical Deployment would do.

"After the 2.9 upgrade, a Kustomization fails with variable not set (strict mode)."

Kustomize-controller 1.9 turned on StrictPostBuildSubstitutions by default. Something in the rendered output contains ${something} with no value and no default. Find it with flux envsubst --strict < file.yaml locally, or flux debug ks ... --show-vars to see what variables exist. The cause is often a legitimate ${VAR} in a shell script inside a ConfigMap. Options in order of preference: add a default with ${var:=default} if the variable should exist; annotate the object with kustomize.toolkit.fluxcd.io/substitute: disabled if the text is not meant to be substituted at all; and only as a last resort turn the feature gate off on the controller. A strong answer explains that this is a safety improvement (a missing variable used to silently become an empty string and produce a broken manifest) and does not recommend simply disabling it.

"A HelmRelease keeps failing with upgrade retries exhausted."

The controller attempted the upgrade the configured number of times, remediated between attempts, and gave up. Look at the failure message on the release and its events (flux events --for HelmRelease/name), because the underlying cause is a Helm error: a failing hook, a timeout waiting for pods, a rejected object. Fix the cause in Git. The release will not retry by itself, because it is stalled until something changes, so recover with flux reconcile hr name --reset, which clears the failure counter. If it is still stuck, --force forces a one-off upgrade, and suspend then resume is the blunt alternative. Then discuss configuration: whether remediateLastFailure is set so failure ends in a rolled-back state, and whether RetryOnFailure would suit an install that needs several tries.

What a strong answer adds: the health of the release after failure (Helm's failed status versus rolled back), and mention of the Helm v4 changes, since a release upgraded to Flux 2.8 or later now uses kstatus for waiting and may fail on a custom resource that does not report conditions.

"Flux keeps reverting a change an operator makes to a Deployment."

Flux owns the fields in the manifest and corrects drift on every interval. If the operator (an HPA, a service mesh injector, a VPA) legitimately owns a field, there are three options. Remove the field from the manifest in Git, so Flux never claims it; the classic case is replicas when an HPA manages scale. Use .spec.ignore on the Kustomization, new in 2.9, to tell server-side apply to leave certain fields alone. Or annotate that object with kustomize.toolkit.fluxcd.io/ssa: Merge or IfNotPresent if Flux should stop managing it. The weak answer is "suspend the Kustomization", which turns off all enforcement to fix one field.

"We need to give team A access to their namespace only."

Impersonation plus controller hardening. Create a tenant with flux create tenant team-a --with-namespace=team-a --export, giving a namespace, ServiceAccount and RoleBinding. Their Kustomizations and HelmReleases set serviceAccountName: team-a, so the apply is limited to that account's permissions. The second half is what candidates forget: make the controllers enforce it, with --default-service-account=default so objects without an account do not run as cluster-admin, --no-cross-namespace-refs=true so a tenant cannot reference another namespace's sources or Secrets, and --no-remote-bases=true so they cannot pull arbitrary Kustomize bases. Then an admission policy (Kyverno or Gatekeeper) to require the field. A strong answer also says what this does not cover: resource consumption and reconcile frequency still need quotas and policy.

"How do you handle secrets?"

Start with principle: never plaintext in Git. Flux decrypts SOPS files natively in kustomize-controller, with Age, OpenPGP or cloud KMS and Vault backends. The private key is the one secret created by hand, as a Secret in flux-system, referenced from .spec.decryption.secretRef. Encrypt only data and stringData with encrypted_regex so manifests stay valid and diffs readable. For cloud KMS, prefer workload identity over stored credentials. Then say when you would choose something else: External Secrets Operator when secrets live in a cloud secret manager and rotate frequently, and Sealed Secrets when you want ciphertext bound to a cluster. The best answers include the failure modes: is SOPS encrypted, configuring decryption is required (you forgot the block) and failed to decrypt (wrong key or missing KMS permission).

"The pipeline takes five minutes to notice a push. How do you improve that?"

Do not shorten the interval, because that multiplies Git API calls across every source and risks rate limits. Add a Receiver so the Git host calls Flux on push, and keep a long interval as a safety net. For registries, use registry Receivers for image pushes. If the delay is after the fetch, look at reconcile duration metrics, the dependency requeue (--requeue-dependency, 30 seconds by default) and whether the health check is waiting out a bad revision.

"How would you promote a release from staging to production?"

Describe the model you chose and its failure mode. With a single Git branch and overlays, promotion is a pull request that copies the changed tag or chart version from the staging overlay to the production overlay, reviewed as a diff. With OCI artifacts, CI pushes an artifact per commit, and promotion is re-tagging (flux tag artifact), which moves a pointer without rebuilding; production's OCIRepository follows that tag and can verify a Cosign signature. Avoid branch-per-environment, which makes Git history misleading. Mention image automation as a staging-only feature, and that production should keep a human in the loop.

"We are on Flux 2.6. How do you get to 2.9?"

The 2.7, 2.8 and 2.9 releases each removed API versions, so the order matters. First rewrite manifests in Git with flux migrate -v 2.6 -f ., commit, and reconcile. Then run flux migrate against every cluster to rewrite objects stored in etcd. Then upgrade the controllers, by re-running bootstrap with the same flags or merging a Flux GitHub Action pull request, and verify with flux check. If you use image automation, run the migration again with -v 2.8. Upgrade the CRDs together with the controllers. Do staging first. Then list the behaviour changes you would check before promoting: strict substitution, postRenderStrategy now defaulting to combined, helm-controller on Helm v4 with kstatus waiting and server-side apply for new releases, and rejection of kubeconfig Secrets that reference local files in 2.9.5. A strong answer also mentions that the Kubernetes version must be in the supported window, 1.34 to 1.36.

Trade-off questions

Flux versus Argo CD

Do not pick a winner; pick by constraint. Flux is composed of controllers and CRDs, has no built-in UI in the core project, runs as agents inside each cluster that pull, and leans on Kubernetes RBAC and impersonation for tenancy. Argo CD is centred on an application model with a UI, and a central control plane that can push to many clusters. Flux fits platform teams that want everything declarative and composable, and environments where clusters must not be reachable from a central system. Argo CD fits teams that value a UI and an application-centric view. The strong answer notes that both are CNCF projects, and that the UI gap for Flux is filled by third-party tools and the Flux Operator's web UI. See the Argo CD guide.

Git as the only source, or OCI artifacts?

Git is simpler, has a native review workflow, and is enough for most teams. OCI makes sense when you have many clusters, want signed, immutable artifacts promoted by tag, or cannot expose Git to clusters. The costs of OCI are an extra pipeline step and a place where the artifact can differ from what a human reviewed, which you mitigate with --source and --revision provenance on push and Cosign verification on pull.

wait: true or explicit healthChecks?

wait: true is simple and checks everything, which means a slow unrelated object can fail the whole Kustomization. healthChecks names the objects that define readiness, which is faster and more precise but can miss objects you forgot to list. For a large Kustomization, name the critical ones.

SOPS or External Secrets Operator?

SOPS keeps everything in Git and works with no extra component, at the cost of committing ciphertext and a commit per rotation. ESO keeps values out of Git entirely and suits frequently rotated secrets, at the cost of another controller and a dependency on the secret manager's availability. Mature platforms often use both: SOPS for bootstrap secrets such as registry credentials, ESO for application secrets.

Image automation or CI committing the tag?

Image automation keeps the loop inside the cluster's ecosystem and needs no CI access to the repository, but it requires a write credential held by the cluster and commits that nobody authored. CI that commits the tag into Git (a pipeline step opening a pull request) keeps authorship and review in your normal flow. A sensible split is automation for staging, pull requests for production.

One Kustomization or many?

Fewer Kustomizations are easier to reason about, but a failure in one object can block the rest and each reconcile rebuilds everything. Many Kustomizations isolate failures and let different teams own different paths, at the cost of more objects, more ordering, and more reconcile overhead. Split along ownership and lifecycle boundaries (controllers, configs, each team's apps), not per manifest.

What a strong answer covers that a weak one misses

Common interview questions

What is the difference between a source and a Kustomization? A source produces an artifact and knows nothing about Kubernetes objects. A Kustomization consumes an artifact and applies it. Keeping them separate lets several Kustomizations share one fetch.

How does prune work? Through the inventory. Each reconcile records the objects applied; on the next, anything in the old inventory and absent from the new build is deleted. Objects Flux did not apply are never pruned. Protect Namespaces and PVCs with kustomize.toolkit.fluxcd.io/prune: Disabled.

What does bootstrap do that flux install does not? Bootstrap commits the controller manifests and a GitRepository plus Kustomization called flux-system to Git, so Flux manages itself, and it is also the upgrade path. flux install applies directly to the cluster with no Git, suited to tests, and does not upgrade itself.

Why is HelmRepository with type: oci discouraged? It is in maintenance mode since source-controller 1.8. Use OCIRepository with HelmRelease.spec.chartRef, which tracks the chart digest and supports signature verification.

How does Flux correct drift in a Helm release? Via .spec.driftDetection.mode: disabled, warn or enabled. Plain Helm does not do this. The old DetectDrift and CorrectDrift gates are deprecated.

What is force: true and why is it risky? It recreates an object when an update fails because of an immutable field. Recreation is a delete, so it can cause downtime. Prefer the per-object annotation for the specific Jobs that need it.

How do you see what Flux would change before merging? flux build to render and flux diff to compare against the cluster with a dry run, in CI, posted to the pull request.

How do you monitor Flux? Controller metrics on :8080/metrics, such as gotk_reconcile_duration_seconds_bucket, plus the gotk_resource_info series, which comes from kube-state-metrics Custom Resource State rather than from Flux. Alert on not-ready resources and, separately, on long-suspended ones.

How many controllers are there and which have cluster-admin? The defaults are source, kustomize, helm and notification controllers, with optional image-reflector, image-automation and source-watcher. Only kustomize-controller and helm-controller are bound to cluster-admin.

What is the API version of an Alert? notification.toolkit.fluxcd.io/v1beta3, and the same for a Provider. A Receiver is v1. This is correct for 2.9.

Questions to ask the interviewer

Good candidates end with questions that show operational maturity. Ask how they structure repositories and whether environments are directories or branches, how they promote between environments, how secrets are managed and who holds the decryption key, what their upgrade cadence for Flux is and whether they have hit an API removal, and how they know a deploy failed without watching a terminal. The answers tell you whether the team runs Flux as a disciplined platform or as a script that happens to work.

Sixty-second self-test

Answer aloud, then check.

1. Name the seven steps of a Kustomization reconcile in order.
2. Which two revision fields in .status tell you whether a commit landed?
3. What does retryInterval default to, and why does that matter?
4. Why does Flux revert a kubectl scale but not a kubectl annotate?
5. Name the three ways to stop Flux fighting an HPA.
6. What does wait: true check, and what does healthChecks check instead?
7. What changed about ${var} handling in 2.9, and how do you opt one object out?
8. Which HelmRelease field chooses whether hooks go to post-renderers, and what is its new default?
9. Which two controllers are cluster-admin, and which flags stop a tenant using that?
10. Name the four steps to upgrade from 2.6 to 2.9 in order.

Answers in brief: (1) resolve source, check dependencies, build, decrypt and substitute, apply, prune, health check and status; (2) lastAttemptedRevision and lastAppliedRevision; (3) the interval, so a long interval delays retries; (4) SSA ownership, Flux owns replicas but not your annotation; (5) omit the field from Git, .spec.ignore, or an ssa annotation; (6) everything applied versus only the listed objects; (7) strict mode is on and errors on unset variables, opt out with kustomize.toolkit.fluxcd.io/substitute: disabled; (8) postRenderStrategy, now combined; (9) kustomize-controller and helm-controller, contained by --default-service-account, --no-cross-namespace-refs and --no-remote-bases; (10) migrate manifests in Git, migrate stored objects, upgrade controllers, then migrate again for image APIs if you use them.

Part two of three. At this level your cluster already syncs from Git, so the problems change character. They are no longer "why is nothing happening" but "why did that commit take ten minutes to arrive", "why does Flux keep undoing what the operator does", and "why did the upgrade break a cluster that was fine yesterday". Start with the error table, then the practices ranked by leverage, then the cards and recipes you can copy. Everything is checked against Flux v2.9.5.

Common errors at this level

Cumulative: the beginner errors still apply, and these are the ones that appear once things work.

Read the table as a map of categories. The first three rows are upgrade debt, the next few are timing, several are ownership conflicts between Flux and something else that writes to the cluster, and the rest are Helm and secrets. When you meet a new error, asking which category it belongs to usually shortens the search.

The practices that pay off most

In order of leverage

1. Learn the two revision fields. lastAttemptedRevision against lastAppliedRevision answers "did my commit land" in seconds, and most "Flux is broken" reports end there. It costs nothing and saves the most time.
2. Put retryInterval next to every interval. Without it, the slowest recovery in your system is the interval you picked for drift correction.
3. Preview before merging. flux build and flux diff in CI move bugs from production to the pull request. Pin the CLI version with the Flux GitHub Action so the preview matches the controllers.
4. Make readiness real. wait: true with a sensible timeout, plus CancelHealthCheckOnNewRevision where people iterate quickly.
5. Keep environment differences in overlays and variables. One base, overlays for structure, postBuild.substituteFrom for scalars, never a branch per environment.
6. Decide who owns each field. Every conflict between Flux and an HPA, an operator or a webhook ends by choosing an owner. Remove the field from Git, ignore it, or set an ssa policy.
7. Lock down tenancy early. Adding --default-service-account to a cluster with twenty Kustomizations is a migration. Adding it on day one is a one-line patch.
8. Upgrade monthly, not yearly. Every Flux release you skip is API removals and default changes you will face together later.
9. Alert on the silent states. Suspended objects and stalled Kustomizations do not page anyone unless you ask.

Practice cards

Each card is a five-minute exercise on a practice cluster. Do them in order.

Operational recipes

Finding what is broken, fast

flux check
flux get all -A --status-selector ready=false
flux events -A
flux logs --level=error -A
kubectl get events -n flux-system --field-selector type=Warning

Run these in order. flux check verifies the controllers and CRDs, the selector shows only the failing objects, events show the recent history, and logs show controller detail. If the first command reports a problem, fix that before reading anything else, because a crashlooping controller explains every downstream symptom.

Forcing a change through, properly

flux reconcile source git flux-system
flux reconcile ks apps --with-source
flux reconcile hr web -n prod --with-source

Reconcile the source first, then the consumer, so you learn which layer is at fault. --with-source does both in order. To recover a stuck Helm release, use flux reconcile hr web -n prod --reset to clear the failure count, --force for a one-off upgrade, or suspend and resume.

Suspending safely

flux suspend ks apps
# ... investigate, apply a hotfix by hand ...
flux resume ks apps

Suspension stops reconciliation for that object only. It is the right tool for a short investigation, and the wrong tool for living with a conflict. Put a calendar reminder on it, or better, an alert for resources suspended longer than a day, because a suspended Kustomization keeps reporting its last status and nobody notices.

Tracing ownership in both directions

flux trace deployment/web -n prod     # live object -> Kustomization or HelmRelease -> source -> revision
flux tree ks apps                     # Kustomization -> everything it applied
flux get ks -A --show-source          # which source does each one follow

Use trace when you are holding a live object and want to know where it came from, and tree when you are holding a Kustomization and want to know what it does.

Seeing the resolved inputs

flux debug ks apps --show-vars
flux debug hr web --show-values
flux envsubst --strict < apps/base/web/deployment.yaml

Most "why did it render that" problems are a variable or a value that is not what you believed. Print them instead of reasoning about them.

Generating YAML instead of typing it

flux create source git platform \
  --url=ssh://git@github.com/example-org/platform --branch=main --interval=5m --export > sources/platform.yaml

flux create kustomization apps \
  --source=GitRepository/platform --path=./apps/prod \
  --prune=true --wait=true --interval=30m --retry-interval=2m --health-check-timeout=5m \
  --export > clusters/prod/apps.yaml

flux create secret git platform-deploy-key --url=ssh://git@github.com/example-org/platform
flux create tenant team-a --with-namespace=team-a --export > tenants/team-a/rbac.yaml

The --export flag prints the manifest instead of creating it. Commit the output so Git remains the source of truth, which is the difference between using the CLI well and using it as a second way to change the cluster.

Making Flux's own settings reviewable

Controller flags and feature gates live in a patch in clusters/<name>/flux-system/kustomization.yaml, committed to Git, rather than edited on the Deployment:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - gotk-components.yaml
  - gotk-sync.yaml
patches:
  - target:
      kind: Deployment
      name: "(kustomize-controller|helm-controller)"
    patch: |
      - op: add
        path: /spec/template/spec/containers/0/args/-
        value: --default-service-account=default
      - op: add
        path: /spec/template/spec/containers/0/args/-
        value: --no-cross-namespace-refs=true
      - op: add
        path: /spec/template/spec/containers/0/args/-
        value: --feature-gates=CancelHealthCheckOnNewRevision=true

The patch survives upgrades, because upgrading regenerates gotk-components.yaml and leaves your patches alone. The file at the top is the Kustomize tool's Kustomization (kustomize.config.k8s.io), not the Flux CRD of the same name; mixing the two up is a classic beginner confusion that still bites at this level. Note also that --no-remote-bases applies only to kustomize-controller, so it would need a separate patch.

A safe 2.9 upgrade

flux migrate -v 2.6 -f . --dry-run                # what would change in Git?
flux migrate -v 2.6 -f .                          # rewrite manifests
git commit -am "flux: migrate manifests" && git push
flux reconcile ks flux-system --with-source
flux migrate --context staging                    # migrate stored objects
flux install --export --components-extra=image-reflector-controller,image-automation-controller \
  > clusters/staging/flux-system/gotk-components.yaml
git commit -am "flux: upgrade to 2.9.5" && git push
flux check

Do staging first and let it run for a few days. Read the release notes for behaviour changes that do not raise errors: strict substitution, postRenderStrategy, Helm v4 defaults, and rejection of kubeconfig Secrets with local file references. If you use the Flux Operator, it must be v0.53.0 or later first.

Restoring after a bad prune

If objects were pruned by mistake, do not recreate them by hand. Fix path or restore the files in Git, reconcile, and let Flux re-apply. For PVC data, that is why the prune annotation matters: a deleted PVC is gone, and with it the volume unless its reclaim policy retained it. Check the StorageClass's reclaim policy on anything stateful before you rely on prune.

Removing Flux without losing workloads

flux uninstall --keep-namespace

This removes the controllers, CRDs and custom resources, but not the workloads Flux deployed. Deleting Flux with kubectl is not supported and leaves finalizers and CRDs behind. If you only want Flux to stop managing an application, set the Kustomization's deletionPolicy: Orphan first, delete the Kustomization, and the workload stays.

Metrics and alerts worth adding

# Prometheus alert sketch: requires kube-state-metrics Custom Resource State for Flux
- alert: FluxResourceNotReady
  expr: gotk_resource_info{ready="False", suspended="false"} == 1
  for: 15m
- alert: FluxResourceSuspendedTooLong
  expr: gotk_resource_info{suspended="true"} == 1
  for: 24h

The for on the first alert should exceed your interval plus retry interval, or it will page on transient failures that heal themselves. The series comes from kube-state-metrics configured for Flux's CRDs, which the flux2-monitoring-example repository sets up; Flux itself does not export it.

Habits worth adopting now

Read before you run. Before flux reconcile, read the status. Half the time the answer is already in lastAttemptedRevision or the condition message, and a forced reconcile only rewrites the same error.

Change Git, not the cluster. A kubectl edit is either reverted or creates drift you will rediscover during an incident. If you must act on the cluster in an emergency, suspend the owner, make the change, commit the same change to Git, and resume.

Keep the tutorial skeptic's eye. Much of what search engines return about Flux is from before 2.7. If a snippet uses v2beta1, v2beta2, v1beta2, a HelmRepository with type: oci for charts, or the DetectDrift gate, treat it as out of date and check the current API reference.

Make every object traceable. Use labels and naming that say which team and environment own an object, so flux trace and flux tree lead to a person.

Write the error down. Flux messages are precise and stable: dependency ... is not ready, health check failed, retries exhausted, strict mode. A running list in your team's runbook of "message, meaning, fix" is worth more than any dashboard, and the tables in this guide are a good starting point.

Separate platform from product. Platform owners write flux-system, the controllers and policy. Product teams write their own paths under scoped ServiceAccounts. Keeping that boundary in the repository layout makes code owners files and review rules straightforward.

Practise the recovery, not only the deploy. Once a quarter, break a staging cluster on purpose: delete a controller Deployment, corrupt a SOPS key, push a bad commit. Discovering that your runbook is wrong in a drill costs an hour; discovering it in an incident costs a night.

Checklist before you call a Flux setup production-ready

• Every Kustomization has interval, retryInterval and timeout set deliberately.
• prune: true is on, and Namespaces and PVCs carry kustomize.toolkit.fluxcd.io/prune: Disabled.
• Readiness is real: wait: true or healthChecks, with CEL expressions for custom resources.
• Environments are overlays plus variables on one branch, with no hand-edited copies.
• Secrets are SOPS-encrypted (only data) or fetched from a secret manager, and the decryption key is stored outside Git and backed up.
• HelmReleases set remediation explicitly, use chartRef for OCI charts, and use drift detection at least in warn.
• Tenants are scoped with serviceAccountName, and the controllers carry --default-service-account and --no-cross-namespace-refs.
• A pull-request job runs flux build and flux diff, with the CLI version pinned.
• Alerts cover errors, stalled and long-suspended resources, and a Receiver covers push latency.
• The upgrade path is rehearsed on staging, and no manifest uses a removed API version.
• Controller flags and feature gates are patched in Git, not on the live Deployments.
• A teammate other than you has done a recovery drill from the runbook.

For the pieces around Flux, see Kubernetes, Helm, Kustomize, Argo CD, and Terraform for provisioning the cluster itself.

This is part three of three. It takes every topic from Beginner and Mid one level further and adds the work you own when Flux is not just your tool but the delivery path for every team on the cluster: its failure modes, its blast radius, its upgrade calendar and its rules.

The stakes change at this level. When one engineer breaks a Kustomization, one application is late. When the platform owner breaks Flux, nothing new ships anywhere, and worse, the thing that would normally repair a bad change is the thing that is broken. Flux also holds more power than almost any other workload in the cluster, because two of its controllers are bound to cluster-admin. You are the person who decides what that power may be used for, who may borrow it, how it is verified and how you get it back when it misbehaves. This guide is written against Flux v2.9.5 (released 2026-08-31).

Where this picks up

I am starting with the architecture, because every later decision depends on knowing where state lives and what happens when a piece of it is gone.

Architecture and failure modes

Flux is a set of independent controllers, each a single-replica Deployment in flux-system that uses leader election. They communicate through the Kubernetes API and, for artifacts, through one HTTP service. That shape is the source of both its robustness and its particular failure modes, so it is worth holding precisely.

source-controller is the only component that talks to Git, OCI registries, Helm repositories and buckets. It fetches content, packages it as an artifact (a tarball with a revision and a digest), and stores it on an emptyDir volume at /data. It serves those artifacts over HTTP through the source-controller Service. kustomize-controller and helm-controller never clone anything: they read the source's status for an artifact URL, download the tarball from source-controller, and work from that. notification-controller receives events from the other controllers and forwards them to Providers, and it is also the only component that listens for inbound webhooks.

Read the diagram as a list of dependencies. If the Git host is unreachable, source-controller keeps serving the last artifact it has, so the cluster keeps its current state and only new changes are delayed. If source-controller restarts, the emptyDir is empty until it re-fetches every source, and during that window dependents report that the artifact is not found. If kustomize-controller is down, nothing is applied or corrected, but nothing already running stops running. Flux failing is almost always a delivery outage, not a runtime outage, and that distinction is the most important sentence in your incident communication: the applications are fine, the pipeline is not.

What each failure actually costs you

The useful exercise is to walk the components and ask what the blast radius is, what the symptom looks like and how long recovery takes. The answers differ sharply.

• Git host down. Sources go Ready=False with failed to checkout and determine revision, but the last good artifact is retained and still served. Running workloads are unaffected, and drift correction keeps running against the stored artifact. You lose the ability to deploy. If your rollback plan is "revert the commit", you have also lost the rollback, which is a reason to keep a documented break-glass path (covered later).
• Registry down. The same, for OCI sources and charts. The difference is that a registry outage also stops nodes pulling images, so a Flux problem and an application problem arrive together, and you must separate them quickly.
• source-controller restart. Artifacts are rebuilt from scratch, which means a burst of clones and pulls against your Git host and registries. With hundreds of sources this is a self-inflicted rate-limit event. Jitter and longer intervals exist for exactly this reason.
• kustomize-controller crash-looping. Commonly an out-of-memory kill on a large build. The symptom is that every Kustomization stops moving at once, with no status change, because the thing that would write a status is dead. You find it with kubectl get pods -n flux-system, not with flux get.
• helm-controller out-of-memory. Helm renders charts in memory, and a large chart or many concurrent releases can exceed the limit. Flux ships an OOMWatch feature gate that shuts the controller down gracefully before the kernel kills it, so in-flight releases are not abandoned in a half-upgraded state.
• notification-controller down. Alerts and Receivers stop. Reconciliation itself is unaffected. This is the quietest failure, because the thing that would tell you about it is the thing that is broken, so monitor it from outside.
• A CRD removed. Deleting a Flux CRD deletes every object of that kind, and with prune semantics downstream this can cascade. This is the single worst manual error, and the reason uninstall has its own command.

State you can and cannot lose

It helps to classify where state lives, because recovery is a question of which classes survive. The desired state lives in Git or an OCI registry and is durable by the standards of those systems. The Flux objects (sources, Kustomizations, HelmReleases) live in etcd but are also in Git, because bootstrap committed them, so they are reproducible. The inventory of what each Kustomization owns lives in that Kustomization's .status.inventory and is what makes pruning safe; lose the object and its inventory is lost, which is why deleting a Kustomization with prune: true deletes the workloads it owns. The artifacts live on an emptyDir and are disposable. The Helm release history lives in Secrets in the release namespace, governed by maxHistory (default 5), and is what rollback depends on. Only the secrets that decrypt your encrypted manifests, meaning the Age private key or the KMS permissions, live outside Git by design, and losing them is the one unrecoverable case.

How a change travels, and where time goes

A commit does not reach the cluster instantly, and as the owner you will be asked, repeatedly, why a merge took twelve minutes to appear. You need a latency budget in your head.

The path has four stages. First, detection: source-controller polls at the source's .spec.interval, or is told by a Receiver that something changed. With a one-minute interval, the average wait is thirty seconds and the worst case is a full minute. Second, artifact production: a clone or fetch, a verification if configured, and packaging. Third, the dependent's wake-up: kustomize-controller is notified through the source's status change and queues the Kustomization. Fourth, apply and health: build, substitute, dry-run, server-side apply, then waiting for health, which can dominate everything else if a Deployment takes minutes to become ready.

Three settings shape that budget. .spec.interval on a Kustomization is the drift-correction period, not the detection period, which surprises many people: a change in the source triggers a reconcile immediately, while the interval governs how often Flux re-applies with no change. retryInterval governs how soon a failed reconcile tries again, and is distinct from interval so that you can have a long steady-state interval and a short retry. timeout is the ceiling on the apply and health check, and is the reason a slow rollout reports health check failed after 3m.

Dependencies add latency by design. When a Kustomization lists dependsOn, it waits until the named ones are ready, then requeues after --requeue-dependency, which defaults to thirty seconds. A chain of four dependent layers (CRDs, controllers, configuration, applications) therefore spends at least a few requeue cycles waiting even when everything is healthy. Platform teams shorten this, as the scaling section describes, but the better lever is to reduce the depth of the chain: two layers that are each well health-checked beat five that each wait on the last.

A fast feedback path for the people who merge is worth real engineering. Receivers cut detection from a polling interval to seconds. The flux reconcile ks X --with-source command gives an engineer an immediate answer. And a Provider of type github posts commit statuses, so the pull request itself shows whether the change was applied. The goal is that nobody has to ask you whether their change landed.

Scaling: which component breaks first

The documentation gives no hard object limit, and you should be wary of any number quoted without a workload. What it does give is the set of levers, and the order in which constraints appear is predictable.

The first wall is the API server, not Flux

Every reconcile involves reads and writes against the Kubernetes API. The controllers rate-limit themselves with --kube-api-qps and --kube-api-burst, and a large fleet of Kustomizations on short intervals turns into a steady stream of server-side-apply dry runs and applies. The symptom is rising reconcile durations with low CPU on the controller, because it is waiting on the API. The answer is usually fewer, longer-interval reconciles rather than a bigger controller.

Then controller concurrency and memory

The --concurrent flag sets how many objects a controller reconciles in parallel. The defaults are 4 for kustomize-controller and helm-controller and 2 for source-controller. The official vertical-scaling guidance raises concurrency to around 10, or 20 when builds happen in memory, lowers --requeue-dependency to five seconds, and lifts limits to as much as 2 CPUs and 2 GiB. It also advises backing /tmp for kustomize-controller with a memory-backed emptyDir, because disk contention caps the benefit of concurrency. You apply all of this by patching the controllers in clusters/<cluster>/flux-system/kustomization.yaml so that the patch survives upgrades.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - gotk-components.yaml
  - gotk-sync.yaml
patches:
  - patch: |
      - op: add
        path: /spec/template/spec/containers/0/args/-
        value: --concurrent=10
      - op: add
        path: /spec/template/spec/containers/0/args/-
        value: --requeue-dependency=5s
      - op: add
        path: /spec/template/spec/containers/0/args/-
        value: --kube-api-qps=100
      - op: add
        path: /spec/template/spec/containers/0/args/-
        value: --kube-api-burst=200
      - op: replace
        path: /spec/template/spec/containers/0/resources/limits
        value:
          cpu: 2000m
          memory: 2Gi
    target:
      kind: Deployment
      name: "(kustomize-controller|helm-controller)"

Treat the numbers as a starting point to measure against, not as a recommendation: the QPS values here are illustrative, and you should confirm the flags and defaults with kubectl get deploy kustomize-controller -n flux-system -o yaml and the vertical-scaling page before copying them.

Source-controller and the Helm cache

Helm repositories are the quiet scaling cost. Each HelmRepository index can be large, and each HelmChart resolves against it. source-controller offers a Helm index cache, controlled by --helm-cache-max-size (default 0, meaning off), --helm-cache-ttl (default 15 minutes) and --helm-cache-purge-interval (default 1 minute). The documented scaling values are a maximum size of 10, a TTL of 60 minutes and a purge interval of 5 minutes. Turning the cache on is often the cheapest win on a fleet that uses many public chart repositories. The longer-term fix, and the direction of the project, is to move charts to OCI and use OCIRepository with chartRef, which avoids index downloads altogether.

Sharding is horizontal scale, with rules

When one controller instance is not enough, Flux shards by label. You label objects with sharding.fluxcd.io/key: shard1, run an additional controller Deployment whose --watch-label-selector is sharding.fluxcd.io/key=shard1, and run the main controllers with the negation, !sharding.fluxcd.io/key, so they ignore sharded objects. The rules that trip people are these. A source and every object that consumes it must be on the same shard, because a controller only sees objects that match its selector. For a HelmRelease you must also label the generated HelmChart through .spec.chart.metadata.labels. And notification-controller and source-watcher do not support sharding, so they remain single instances whatever else you do.

Sharding is the right tool when one team or one environment generates most of the load and you want to isolate its noisy reconciles from everyone else. It is the wrong tool for a problem that is actually API-server saturation, which adding controllers makes worse.

Spreading load in time

Two levers spread work rather than add capacity. --interval-jitter-percentage randomises intervals so that a thousand objects created together do not all wake together, which matters after a restart. And webhooks replace short polling: a Receiver on the Git host makes detection event-driven, so you can lengthen every source interval to tens of minutes without hurting latency.

The trust model

Security for a GitOps controller starts with one uncomfortable fact: whoever can get a change merged into the path Flux watches can run anything cluster-admin can run. Flux did not create that power; it concentrated it and made it auditable. Your job is to decide where the boundary sits and to make it hard to cross by accident.

What the controllers can do

Only kustomize-controller and helm-controller are bound to cluster-admin. The other controllers get the narrower crd-controller role, which covers Flux's own custom resources. The pods themselves are hardened by default: they follow the restricted Pod Security Standard, drop all capabilities, use a read-only root filesystem, run as a non-root user (UID and GID 65534, with an fsGroup of 1337) and use the RuntimeDefault seccomp profile. Bootstrap also installs NetworkPolicies that deny ingress from other namespaces, with allow-egress, allow-scraping and allow-webhooks carving out what is needed. Since Flux 2.9.4 the allow-webhooks policy is restricted to the receiver port. That is a good baseline, and your review should confirm you have not weakened it.

The direction of trust matters. Flux is pull-based: the cluster reaches out to Git and registries, and nothing outside holds cluster credentials. Compare a push-based pipeline in which a CI system stores a kubeconfig. There, compromising CI compromises every cluster it can reach. With Flux, the attacker must compromise the source of truth, which is where your review gates, branch protection and signed commits already live. That is a real improvement, but only if those gates exist.

The realistic threats

Five threats are worth naming, because each has a different control.

1. A malicious or mistaken merge into a watched path. Controlled by branch protection, required reviews, code owners per path and, for high-stakes paths, signed commits verified by GitRepository.spec.verify.
2. A compromised tenant who can write to their own repository and tries to reach beyond their namespace. Controlled by impersonation, which the multi-tenancy section covers.
3. A poisoned dependency: a tampered image, chart or artifact. Controlled by pinning and verification, in the supply-chain section.
4. Stolen decryption material: an Age key or cloud credential. Controlled by scoping and workload identity, in the secrets section.
5. An abused inbound webhook: a Receiver that anyone can trigger. Controlled by using secret tokens or OIDC and restricting the network path. Receivers are the only part of Flux that listens, so they deserve their own threat review.

Receivers as an attack surface

A Receiver (notification.toolkit.fluxcd.io/v1) exposes a URL of the form /hook/<sha256>, where the digest is derived from the token, the Receiver name and its namespace (for generic-oidc it is derived from the name and namespace alone, since there is no token). The types include generic, generic-hmac, generic-oidc, github, gitlab, bitbucket, harbor, dockerhub, quay, nexus, acr, gcr and cdevents. A Receiver can only ask Flux to reconcile the resources it lists, so it cannot inject content, but it can be used to hammer your Git host or to force reconciles.

Flux 2.9 tightened this. Receivers of type gcr now require email and audience keys in the referenced Secret, closing CVE-2026-40109. The new generic-oidc type lets the caller prove its identity with a token from an OIDC issuer, so you can drop the shared secret entirely, and a resourceFilter (a CEL expression over the request, the resource and the token claims) lets you drop events you do not care about. Use HMAC or OIDC types wherever the sender supports them, and keep the webhook route private where you can.

apiVersion: notification.toolkit.fluxcd.io/v1
kind: Receiver
metadata:
  name: github-push
  namespace: flux-system
spec:
  type: github
  events:
    - ping
    - push
  secretRef:
    name: receiver-token
  resources:
    - apiVersion: source.toolkit.fluxcd.io/v1
      kind: GitRepository
      name: fleet

flux trigger receiver github-push --token="$(cat token.txt)" --url=https://flux-hooks.example.com
kubectl -n flux-system get receiver github-push -o jsonpath='{.status.webhookPath}'

Secrets in a GitOps world

The principle is simple and strict: never commit a plaintext secret. The difficulty is that GitOps wants everything declarative, including secrets, so you need a mechanism that keeps the encrypted form in Git and the plaintext only in the cluster.

SOPS, built in

kustomize-controller decrypts SOPS-encrypted manifests natively. It supports Age (including the post-quantum hybrid keys added in 2.9), OpenPGP, AWS KMS, Azure Key Vault, GCP KMS and OpenBao or Vault transit. You enable it with .spec.decryption on the Kustomization, and the key material lives in a Secret whose keys have conventional names: *.agekey, sops.asc, sops.vault-token, sops.aws-kms, sops.azure-kv and sops.gcp-kms.

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: apps
  namespace: flux-system
spec:
  interval: 30m
  path: ./apps/prod
  prune: true
  sourceRef:
    kind: GitRepository
    name: fleet
  decryption:
    provider: sops
    secretRef:
      name: sops-age

The everyday workflow generates a Secret manifest and encrypts it before it ever touches Git.

kubectl create secret generic db-credentials \
  --namespace=payments \
  --from-literal=password="$DB_PASSWORD" \
  --dry-run=client -o yaml > secret.yaml
sops --encrypt --age "$AGE_PUBLIC_KEY" \
  --encrypted-regex '^(data|stringData)$' \
  --in-place secret.yaml
git add secret.yaml

The failure modes are instructive. If you apply an encrypted Secret without .spec.decryption, reconciliation fails with is SOPS encrypted, configuring decryption is required for this secret to be reconciled. If the key Secret is missing or wrong you will see cannot get sops age decryption Secret or failed to decrypt and format. Those messages point to the key Secret or the workload identity, not to the manifest.

Whose key is it?

This is the governance decision. A global Age key, set with the kustomize-controller flag --sops-age-secret (added in 2.7), is convenient for a single-tenant cluster and dangerous for a shared one, because every Kustomization, including a tenant's, can then decrypt anything encrypted to that key. For a multi-tenant cluster, give each tenant its own key Secret in its own namespace, reference it from that tenant's Kustomization, and let impersonation and RBAC stop one tenant from reading another's key. The related flag --default-decryption-service-account makes decryption use a tenant's identity when workload identity is involved.

With cloud KMS, prefer workload identity over static credentials. Since 2.9, decryption through OpenBao or Vault can use Kubernetes workload identity through a JWT auth method instead of a long-lived token, which removes a secret that would otherwise need rotating.

The alternatives, and when they win

SOPS is not the only answer, and the senior question is which fits the organisation. External Secrets Operator keeps secrets in a managed store (a cloud secret manager or Vault) and syncs them into the cluster, so nothing encrypted sits in Git; it fits organisations that already run a central secret store and need rotation handled outside Git. Sealed Secrets encrypts to a cluster's public key, which is simple but ties each ciphertext to one cluster. The Secrets Store CSI driver mounts secrets straight into pods without creating a Kubernetes Secret. The decision turns on rotation (SOPS requires a commit to rotate, ESO does not), on audit (a central store logs reads, Git logs writes) and on data residency: for employers in the Gulf or Egypt with rules about where secrets may live, a store that keeps keys in a regional HSM or vault is often a requirement that decides the matter.

Supply chain

GitOps moves the question of "what is running" to "what did Flux fetch", so the supply chain you must defend is the chain from author to artifact to cluster.

Trusting Flux itself

The Flux project signs its release images with Cosign keyless, and publishes SPDX SBOMs and SLSA provenance. You can verify a controller image before it ever runs.

cosign verify ghcr.io/fluxcd/source-controller:v1.9.5 \
  --certificate-identity-regexp='^https://github\.com/fluxcd/.*$' \
  --certificate-oidc-issuer=https://token.actions.githubusercontent.com

A platform team does this in its upgrade pipeline: pull the new controller images, verify signatures, mirror them into a registry you control, and point --registry (or the bootstrap --registry flag) at that mirror so the cluster never reaches the public registry at runtime. The mirror also protects you from registry outages and pull limits.

Verifying what Flux deploys

Flux can verify the content it is about to apply. On an OCIRepository, .spec.verify with provider: cosign or notation checks a signature and can require a specific OIDC identity through matchOIDCIdentity. On a GitRepository, .spec.verify checks OpenPGP commit or tag signatures, and since 2.9 SSH signatures too. When verification fails, the source does not become ready, so nothing downstream applies the unverified content.

apiVersion: source.toolkit.fluxcd.io/v1
kind: OCIRepository
metadata:
  name: platform-manifests
  namespace: flux-system
spec:
  interval: 10m
  url: oci://ghcr.io/example-org/platform-manifests
  ref:
    semver: ">=1.0.0"
  verify:
    provider: cosign
    matchOIDCIdentity:
      - issuer: ^https://token\.actions\.githubusercontent\.com$
        subject: ^https://github\.com/example-org/platform-repo/.*$

For air-gapped environments, 2.9 added support for a custom Sigstore trusted root, so keyless verification works without reaching the public transparency infrastructure. That matters for regulated environments in the region where egress to the public internet is restricted.

The push side: signing in CI

The publishing half of the chain is flux push artifact, which packages a directory as an OCI artifact tagged with the commit, carrying the source URL and revision as metadata. You sign the result with Cosign in the same pipeline, so that what Flux verifies is exactly what CI produced.

flux push artifact "oci://ghcr.io/example-org/platform-manifests:$(git rev-parse --short HEAD)" \
  --path=./deploy \
  --source="$(git config --get remote.origin.url)" \
  --revision="$(git branch --show-current)@sha1:$(git rev-parse HEAD)"
flux tag artifact "oci://ghcr.io/example-org/platform-manifests:$(git rev-parse --short HEAD)" --tag=latest
cosign sign --yes "ghcr.io/example-org/platform-manifests:$(git rev-parse --short HEAD)"

This pattern, sometimes called Gitless GitOps, changes the trust boundary: the cluster trusts a signed artifact rather than a Git host, which shrinks the set of systems that must be reachable and trusted at runtime. The cost is that Git is no longer what the cluster reads, so you need a clear story for how an engineer finds the change that produced an artifact; the --source and --revision metadata are that story, and you should treat them as mandatory.

Images and charts

Flux does not scan images. Pair it with an admission policy (Kyverno or Gatekeeper) that verifies image signatures and forbids unpinned tags, and with a scanner in CI (see the Trivy guide). Pin charts to exact versions in production HelmReleases rather than ranges, and use image automation only with policies that cannot move backwards or sideways onto an unintended tag stream.

Multi-tenancy without illusions

Flux supports multi-tenancy, but it is a pattern you assemble, not a switch. The honest framing: Flux tenants share a control plane and, usually, a cluster, so the isolation is logical, enforced by Kubernetes RBAC and admission, not physical.

The lockdown, step by step

The mechanism is impersonation. A Kustomization or HelmRelease with .spec.serviceAccountName is applied using that ServiceAccount's permissions rather than the controller's own. If the tenant's ServiceAccount can only touch its own namespace, the tenant's manifests can only affect that namespace, whatever they contain. To make this mandatory rather than optional, you patch the controllers.

• --no-cross-namespace-refs=true on all controllers stops a tenant referencing a source, secret or other object in another namespace.
• --no-remote-bases=true on kustomize-controller stops builds fetching remote Kustomize bases, which would be an unverified fetch.
• --default-service-account=default makes every Kustomization and HelmRelease without a serviceAccountName impersonate the default ServiceAccount in its own namespace, which has no permissions, so forgetting the field fails safe.
• --default-decryption-service-account and --default-kubeconfig-service-account do the same for SOPS decryption and remote-cluster access.

The flux-system Kustomization, which you own, keeps cluster-admin and is the one place platform changes are made.

flux create tenant team-payments \
  --with-namespace=payments \
  --cluster-role=payments-deployer \
  --export > tenants/team-payments/rbac.yaml

The command generates the Namespace, the ServiceAccount and the RoleBinding. Note two details that changed recently: since 2.8 you can pass --skip-namespace to avoid creating the namespace, and --cluster-role sets which role is bound, defaulting to cluster-admin within the tenant namespace. The default is far too generous for production. Create a narrow ClusterRole, bind it only inside tenant namespaces, and pass its name.

Admission policy closes the gap

The flags make a missing service account safe, but they cannot stop a tenant explicitly naming a powerful one. An admission policy in Kyverno or Gatekeeper can. Typical rules: a tenant's Kustomization must set serviceAccountName to an account in its own namespace; sourceRef must point to a source in the same namespace; tenants cannot create cluster-scoped resources; and targetNamespace must be inside the tenant's allowed set. The flux2-multi-tenancy repository is the reference implementation of this combination.

The things tenancy does not isolate

Be explicit with leadership about what stays shared. The controllers' queues are shared, so one tenant's thousand objects can slow another's reconcile; sharding with labels is the remedy. The API server is shared, so a tenant that creates expensive objects affects everyone. Cluster-scoped resources such as CRDs, ClusterRoles and webhooks are platform-owned, so tenants who need a CRD must ask. And notification-controller is a single instance, so a noisy tenant's alerts share it with everyone. If a tenant's requirements exceed this (hostile code, a regulator requiring separation), the answer is a separate cluster, not more policy.

Repository topology and promotion

How you lay out repositories is the decision that is hardest to change later, because it encodes who can change what. There is no universally correct layout, only trade-offs you should be able to state.

Three shapes

A monorepo puts clusters, infrastructure and applications in one repository, following the flux2-kustomize-helm-example: a clusters/<env> directory holding each cluster's entry point, an infrastructure/ directory, and apps/base plus apps/<env> overlays. It gives one review surface, atomic cross-cutting changes and a simple mental model. It struggles at scale because everyone shares one history and one set of permissions, and path-level ownership must be enforced with code owners.

Repository per team gives each team its own repository, which a platform repository references through a GitRepository and a Kustomization in the team's namespace. Ownership is clean and blast radius is small. The price is that cross-cutting changes span repositories, and you need tooling to keep the many small repositories consistent.

Repository per environment separates production from everything else, so that different people and different protections apply. It pairs naturally with promotion by pull request between repositories, but it multiplies the places a change must be made.

OCI artifacts pushed from CI is the fleet-scale shape: CI builds the rendered or templated manifests, pushes them as a signed artifact, and clusters follow a tag or semver range. Promotion becomes retagging, not merging. It is the right answer when many clusters share the same content, or when you want the cluster to depend on a registry rather than a Git host.

Promotion

Promotion is moving a known-good change from one environment to the next. Flux gives you three honest mechanisms. Overlays: each environment is a Kustomize overlay, and promotion is a pull request that changes the overlay. Version ranges: an environment follows a semver ref on an OCI or Helm source, so dev takes >=1.0.0-0 and production takes an exact version that you bump deliberately. Image automation: an ImageUpdateAutomation commits new tags, but only to the lower environments, and promotion to production remains a human-reviewed change.

The discipline that matters is that promotion moves a version, not a diff. If staging and production are different configurations that happen to share a name, "it worked in staging" tells you little. Keep the environment-specific surface small and explicit, using post-build substitution from a per-cluster ConfigMap (cluster-vars) for the few values that really differ.

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: apps
  namespace: flux-system
spec:
  interval: 30m
  retryInterval: 2m
  timeout: 10m
  wait: true
  prune: true
  path: ./apps/base
  sourceRef:
    kind: GitRepository
    name: fleet
  dependsOn:
    - name: infrastructure
  postBuild:
    substitute:
      cluster_env: prod
    substituteFrom:
      - kind: ConfigMap
        name: cluster-vars

Dependency layering

The layers you will almost always want are: CRDs and controllers (cert-manager, ingress, the policy engine), then configuration that uses those CRDs (issuers, policies), then applications. Each layer is a Kustomization that dependsOn the previous one and uses wait: true or explicit healthChecks, so that "ready" means workloads are available, not merely applied. Since 2.7 a dependsOn entry can carry a CEL readyExpr, which lets you express readiness more precisely than the default ready condition.

For custom resources that kstatus does not understand, write health expressions with healthCheckExprs, using CEL to define when an object is current, inProgress or failed. Keep them small and tested; a wrong expression either blocks forever or waves broken things through.

Upgrades and migrations

Upgrades are where a platform owner earns their keep, and the last year gave a perfect case study: three releases in a row removed old API versions, and each one broke any cluster that had not migrated.

The API removals you must be able to recite

Flux 2.7 (September 2025) removed the v1beta1 versions of the source, Kustomization, image and notification APIs and the v2beta1 HelmRelease. Flux 2.8 (February 2026) removed v1beta2 of the source and Kustomization APIs and v2beta2 of HelmRelease. Flux 2.9 (June 2026) removed v1beta2 of the image and notification APIs. After 2.9 the only served versions are v1 for sources, Kustomization, Receiver and the image APIs, v2 for HelmRelease, and v1beta3 for Alert and Provider, which is still beta and is correct. Any YAML using a removed version now fails with the standard no matches for kind error.

The official procedure

For a cluster on 2.6 or earlier moving to 2.9, the project's "Upgrade Procedure for Flux v2.7+" has four steps, and the order matters because the two kinds of migration fix two different places.

1. Migrate the manifests in Git, with flux migrate -v 2.6 -f ., commit and push, then run flux reconcile ks flux-system --with-source. This rewrites API versions in your repository.
2. Migrate the stored objects in etcd, by running flux migrate against each cluster. It is idempotent, accepts --context and --kubeconfig, and can also run as a Job under the kustomize-controller ServiceAccount.
3. Upgrade the controllers, by rerunning flux bootstrap with your original flags or by merging the pull request from the Flux GitHub Action. Verify with flux check.
4. If you use image automation, run flux migrate -v 2.8 -f . in Git and then flux migrate on the clusters again.

# in the fleet repository
flux migrate -v 2.6 -f . --dry-run
flux migrate -v 2.6 -f .
git commit -am "flux: migrate manifests to current API versions" && git push
flux reconcile ks flux-system --with-source

# per cluster
flux migrate --context prod-1
flux check

If you use the Flux Operator, it must be at v0.53.0 or later before you move to 2.9; it migrates the in-cluster storage itself, but you still migrate Git. The useful flags on flux migrate are -f, --version, --dry-run and --extensions.

Behaviour changes that are not API removals

Some changes break a working setup without changing any API version, and these are the ones that cause incidents because nothing rejects the manifests; behaviour simply differs.

• Strict post-build substitution (kustomize-controller 1.9, Flux 2.9). The StrictPostBuildSubstitutions gate is now on by default. A ${var} with no value and no default fails the reconcile with variable not set (strict mode). Fix by supplying the variable, writing ${var:=default}, annotating the object with kustomize.toolkit.fluxcd.io/substitute: disabled, or turning the gate off as a temporary measure. Audit with flux envsubst --strict in CI before upgrading.
• Post-renderer strategy (helm-controller 1.6, Flux 2.9). The new .spec.postRenderStrategy defaults to combined, so hooks are now sent to post-renderers, where before they were not. Releases that patch hooks through postRenderers may render differently. Keep the old behaviour with postRenderStrategy: nohooks or the UseHelm3Defaults gate while you review.
• Helm 4 defaults (helm-controller 1.5, Flux 2.8). New releases install with server-side apply and health checks use kstatus (waitStrategy.name: poller). Existing releases stay on client-side apply under upgrade.serverSideApply: auto until you change it. The UseHelm3Defaults gate restores the older defaults in one move.
• Kubeconfig Secrets (Flux 2.9.5). A .spec.kubeConfig Secret whose kubeconfig refers to local files (certificate-authority, tokenFile, client-certificate, client-key) is now rejected. Credentials must be embedded inline.
• CRD schema changes (Flux 2.9.4). The ArtifactGenerator and ImageUpdateAutomation schemas changed, so the CRDs must be upgraded along with the controllers. And ImageUpdateAutomation refspecs can no longer force-update or delete.

The 2.9.1 regression with openapi.path URLs (failed to read OpenAPI schema) is a reminder to stay on the latest patch: it was fixed in 2.9.2.

An upgrade process for a fleet

Treat a Flux upgrade as a release with stages, not a one-line change.

1. Read the release notes and the controller changelogs, looking for removals, changed defaults and feature-gate changes. Write the delta down.
2. Run the migration and a dry build in CI. flux migrate --dry-run, flux build ks and flux envsubst --strict against each cluster's path catch most breakage before any cluster is touched.
3. Upgrade a canary cluster and let it bake through a few reconcile cycles of every tenant. Watch flux get all -A --status-selector ready=false, controller restarts and reconcile-duration metrics.
4. Roll the fleet in rings, development, then staging, then production clusters, with a pause between rings. Automation does the mechanical part: the Flux GitHub Action opens a pull request that regenerates gotk-components.yaml, or a Flux Operator FluxInstance with distribution.version: "2.9.x" tracks patch releases automatically.
5. Keep a rollback. Upgrading is reverting the same commit, but only if the CRDs did not migrate storage versions irreversibly. Read the notes before assuming that a downgrade is safe.

Note the supported Kubernetes window. Flux 2.9 supports Kubernetes 1.34, 1.35 and 1.36 (at their documented minimum patch levels), and the project supports only the latest three minors. Your Kubernetes upgrade calendar therefore constrains your Flux calendar, and the reverse, so plan them together. The Installation page still shows the 2.8 table; use the 2.9 release notes.

Observability and service levels

A platform you cannot measure is a platform you will defend by anecdote. Flux exposes what you need, with one important catch about where.

Two sources of metrics

Each controller exposes Prometheus metrics on port 8080 at /metrics. These include gotk_reconcile_duration_seconds_bucket (labelled by kind, name and namespace), controller_runtime_reconcile_total (labelled by controller and result), gotk_cache_events_total, gotk_token_cache_events_total and the standard workqueue_* family. They describe the controllers' own behaviour: how long reconciles take, how many fail, how deep the queues are.

The per-object status metrics, such as whether a given Kustomization is ready or suspended, do not come from Flux. They come from kube-state-metrics with a Custom Resource State configuration, exposing series like gotk_resource_info with labels such as customresource_kind, name, ready and suspended. This catches people out: they scrape the controllers, find no readiness metric and conclude it does not exist. The reference implementation is the flux2-monitoring-example repository, which ships kube-prometheus-stack, the PodMonitors and Grafana dashboards. See the Prometheus and Grafana guides for the query and dashboard side.

What to alert on

Alert on symptoms of delivery health, not on every transient failure. A few rules carry most of the value.

• A Kustomization or HelmRelease that has been not-ready for longer than a threshold, using the status metric. Transient failures resolve themselves; ten minutes of Ready=False does not.
• Reconcile duration at a high percentile rising, from the histogram. This is your early warning for the scaling walls.
• Controller restarts and out-of-memory kills, from the pod metrics, since a crash-looping controller cannot report its own failure.
• Suspended resources older than a threshold. Suspension is the tool an engineer reaches for during an incident and forgets afterwards. A suspended Kustomization is silently not enforcing anything.
• Source failures, because every downstream object depends on them.

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: flux-delivery
  namespace: monitoring
spec:
  groups:
    - name: flux
      rules:
        - alert: FluxReconciliationFailing
          expr: max by (customresource_kind, namespace, name) (gotk_resource_info{ready="False", suspended="false"}) == 1
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "{{ $labels.customresource_kind }} {{ $labels.namespace }}/{{ $labels.name }} not ready for 10m"
        - alert: FluxResourceSuspended
          expr: max by (customresource_kind, namespace, name) (gotk_resource_info{suspended="true"}) == 1
          for: 24h
          labels:
            severity: info
          annotations:
            summary: "{{ $labels.customresource_kind }} {{ $labels.namespace }}/{{ $labels.name }} suspended for a day"

The label values in this sketch depend on your kube-state-metrics configuration, so check them against what your installation emits before you rely on the rule.

Two layers of notification

Metrics tell you the platform owner something is wrong. Alerts and Providers tell the owning team. An Alert (notification.toolkit.fluxcd.io/v1beta3) filters events by severity and source and sends them through a Provider to Slack, Teams, PagerDuty or similar, and a Provider of type github sets commit statuses so the pull request shows the outcome. Give each tenant an Alert scoped to their own objects, so that a team learns about its own failures without you relaying them. Since 2.7 an OpenTelemetry tracing Provider exists for tracing deployments, and since 2.8 Providers can post pull-request comments.

Service levels for delivery

Once the data exists, promise something. A reasonable service level for a platform is expressed as a time from merge to applied at a given percentile, and a percentage of reconciles that succeed over a window. State them, measure them, and publish them. The act of writing "ninety-five percent of merges to a tenant path are applied within five minutes" forces the conversation about intervals, webhooks and health-check timeouts that otherwise never happens.

Cost, honestly

Flux is free, open source under Apache-2.0 and a CNCF graduated project. The cost is everything around it, and it is worth naming so that a finance conversation does not surprise you.

Compute. Controllers are small Deployments, but they scale with the number of objects and with concurrency. A large fleet raises the limits (up to a couple of CPUs and gigabytes per controller in the vertical-scaling guidance), and if you shard you multiply those. Run the numbers per cluster, because a fleet of small clusters each carrying the full control plane of controllers spends proportionally more than one large cluster.

API calls. Short intervals against a hosted Git provider count against its rate limits, and pulls from a hosted registry can incur charges or hit limits, including pulls from managed registries on a cloud you pay for by request. The cheapest saving is almost always to lengthen intervals and add Receivers.

Egress and storage. Large artifacts pulled repeatedly across regions cost egress. For organisations in the Gulf or Egypt, where cross-region transfer to a distant region carries both cost and latency, keeping a registry mirror in the same region as the clusters is usually worth it on both counts.

People. The dominant cost is the platform team's time: upgrades, tenant support, incident response. This is the number that decides whether a managed or supported offering is worth buying. Commercial support exists, for example ControlPlane Enterprise for Flux, and the decision is about engineer-hours, not licence fees.

A useful framing for leadership is cost per cluster and cost per deployment. If adding a cluster requires a platform engineer for a day, your cost structure is wrong, and the remedy is automation of bootstrap (through Terraform's fluxcd/flux provider with its flux_bootstrap_git resource, or a Flux Operator FluxInstance), not cheaper controllers.

Governance

Governance is the set of rules that keep a shared platform from drifting into an unmanageable state. The aim is small, enforceable rules, not a document nobody reads.

Ownership of paths. Every directory Flux watches has an owner, written in a code-owners file and enforced by branch protection. Changes to clusters/ and to cluster-scoped infrastructure need platform approval; changes inside a tenant's path need only that tenant.

Change classes. Distinguish routine changes (an image tag in a tenant's overlay) from platform changes (controller flags, RBAC, policies). Routine changes should merge and apply without the platform team's involvement. Platform changes get review, a canary and an announcement.

Who may suspend. flux suspend is powerful and silent. Decide who may use it, require that a suspension names an incident or a ticket in an annotation, and alert on age, as in the observability section.

Break-glass. Sooner or later someone must change the cluster without Git, because Git or Flux itself is the problem. Define the procedure: who is authorised, how the action is recorded, and how the change is back-ported to Git. Flux's own annotation model helps. kustomize.toolkit.fluxcd.io/reconcile: disabled on an object stops Flux managing it while an engineer repairs it by hand, and removing the annotation restores management. What you want to avoid is a break-glass that becomes routine, which is a sign that the normal path is too slow or too fragile.

Policy as code. Kyverno or Gatekeeper rules are versioned in Git and applied by Flux in the infrastructure layer, so the policy that constrains tenants is itself delivered by the pipeline it constrains. That is correct and worth testing: the policy layer must be applied before tenant layers, which is a dependsOn.

Audit. Git history records who asked for a change, Kubernetes audit logs record what was applied and under which identity, and Flux events record the reconcile outcome. Together these answer "who changed this and when". Retain them for as long as your regulator requires, and confirm that the Flux events are actually being kept, since Kubernetes Events expire quickly by default.

Incident playbooks

Incidents with Flux fall into a few recurring shapes. Having a playbook for each means the on-call engineer spends the first minutes doing, not thinking. Every playbook starts from the same triage.

flux check
flux get all -A --status-selector ready=false
flux events -A
flux logs --level=error -A
kubectl get events -n flux-system --field-selector type=Warning
kubectl get pods -n flux-system

The last command matters as much as the others: if a controller is crash-looping, the flux get output is stale, because nothing is updating the statuses.

Playbook one: nothing is deploying

Symptoms: merges do not appear, and several teams report it. First decide whether the source or the consumer is stuck. Run flux get sources git -A. If the source's revision is old and its message reads failed to checkout and determine revision, the problem is upstream: credentials, the branch, network egress or the Git host. Check the Secret, the URL, the .spec.ref and whether the provider is having an outage. If the source is current but Kustomizations are stale, look at the controller pods and logs: out-of-memory kills and leader-election problems are the usual causes. If only one Kustomization is stuck, read its message; a dependency '...' is not ready points to the dependency, not to the stuck object.

Playbook two: a bad change is live and pruning

Symptoms: a merge removed or broke resources, and Flux is faithfully applying the damage. The first move is to stop the bleeding without losing state. Suspend the affected Kustomization with flux suspend ks <name>, which halts further applies and prunes, then decide the remedy. Reverting the commit and letting Flux reconcile is the clean path and is usually right, because it restores from history. If pruning already deleted a stateful object, recovery depends on whether the volume was protected by kustomize.toolkit.fluxcd.io/prune: Disabled or by the storage class's reclaim policy; this is why those protections are set before the incident, not during. After recovery, write down why the change passed review.

Playbook three: a HelmRelease is stuck

Symptoms: install retries exhausted or upgrade retries exhausted, or a release stuck in a failed state. Read the message to see the Helm error, then fix the cause, whether values, chart version or a cluster-side problem such as a webhook. To clear the failure count, use flux reconcile hr <name> --reset; to force a single install or upgrade, flux reconcile hr <name> --force; and flux suspend hr <name> followed by flux resume hr <name> also clears it. Consider strategy: RetryOnFailure on installs and upgrades where retrying is safer than remediating. If the error is Request entity too large: limit is 3145728, the release Secret has outgrown the roughly 3 MiB limit; trim the chart with .helmignore or .sourceignore.

Playbook four: every reconcile fails after an upgrade

Symptoms: right after upgrading Flux, several Kustomizations fail. Match the message to the changes. variable not set (strict mode) is the 2.9 strict substitution default. no matches for kind ... v2beta1 or similar is an API removal, so you skipped a migration. Hooks behaving differently in Helm releases that use post-renderers points at postRenderStrategy. An ArtifactGenerator or ImageUpdateAutomation failing points at the 2.9.4 CRD schema change, so check the CRDs were upgraded. Roll the controllers back to the previous version by reverting the gotk-components.yaml commit, then fix forward in a canary.

Playbook five: drift that keeps coming back

Symptoms: a field is changed on the cluster and reverted by Flux every interval, or Flux reports a change on every run. There are two causes. Either someone or something (an autoscaler, an operator, a mutating webhook) legitimately owns that field, or the manifest contains empty values (null, {}, []) that the API server normalises away, causing perpetual "configured". For the first, tell Flux to leave the field alone: Kustomization .spec.ignore (new in 2.9) holds server-side-apply ignore rules, and HelmRelease .spec.driftDetection.ignore does the same for Helm. For the second, remove the empty fields from the manifest.

Playbook six: a controller is out of memory

Symptoms: a controller restarts repeatedly and the pod shows OOMKilled. For helm-controller, enable the OOMWatch gate, so that it shuts down gracefully before the kernel intervenes, with --oom-watch-memory-threshold (default 95) and --oom-watch-interval. For kustomize-controller, find the largest builds; a Kustomization whose path includes a giant rendered directory is the usual culprit. Split it into several Kustomizations, raise the limit, and consider the memory-backed /tmp. For image-reflector-controller on small nodes, the error unable to open the Badger database means it needs more memory or swap.

Playbook seven: lost decryption key

Symptoms: Secrets fail with decryption errors across many Kustomizations. If the key Secret was deleted or overwritten, restore it from your secure backup of the Age private key and reconcile. If the key is genuinely lost, every secret encrypted only to it is unrecoverable from Git, and you must re-create the values from their origin (the database, the provider) and re-encrypt to a new key. This is why the key has a backup owned by more than one person, and why rotating away from a single shared key is worth the effort.

After every incident

Close the loop. Every incident should leave behind at least one of: a new alert, a new policy, a changed default, a runbook edit or a documented decision not to act. Incidents that teach nothing recur.

Where Flux stops

A platform owner must know where the tool ends, both to avoid forcing it into the wrong job and to argue honestly when someone proposes replacing it.

Flux is not a build or test system. It applies what it is given. Building images, running tests and producing the manifests belong in CI, and the boundary is the commit or the pushed artifact. Attempts to make Flux do CI work make both worse.

Flux is not a progressive-delivery engine by itself. It applies the desired state and waits for health, but it does not shift traffic gradually or roll back on a metric. For canary, blue-green and A/B releases, the project family's answer is Flagger, which works alongside Flux, or another rollout controller. Flux health checks tell you a rollout finished, not whether it was good.

Flux has no built-in UI. This is a deliberate design choice, not an omission, and it is a real adoption cost for organisations used to a dashboard. Options exist: the Flux Operator ships a Flux Web UI with workload dashboards and pod logs, and third-party tools such as Headlamp, Capacitor and Weave GitOps (community-maintained) read Flux objects. If your stakeholders need a central console across many clusters, weigh that seriously.

Flux is not a secrets manager, a policy engine or an inventory of your infrastructure. It integrates with those but does not replace them.

Flux is per-cluster by design. Each cluster runs its own agents, which is excellent for failure isolation and for egress-only networking, and less convenient when you want one place to see and push to many clusters. Argo CD's model is the opposite: a central control plane that pushes to many clusters and presents them in one UI. Neither is wrong. See the Argo CD guide for the comparison from the other side. The honest summary is that Argo CD suits organisations that want an application-centric console and a central view, and Flux suits those that want Kubernetes-native composition, per-cluster autonomy and no inbound access to clusters.

Infrastructure below Kubernetes is someone else's job. Flux manages objects in a cluster. Creating the cluster, the network and the cloud accounts is for Terraform or OpenTofu, which is also how you bootstrap Flux into a new cluster. The clean split is that Terraform builds the platform and hands over to Flux, which takes over everything inside the cluster.

When to choose something else

Choose Argo CD when the organisation needs a central UI and multi-cluster management from one place. Choose plain CI-driven kubectl apply or helm upgrade only for a single small cluster where the ceremony of GitOps exceeds its benefit, and accept that you then own drift. Choose a managed GitOps offering from your cloud provider when you want someone else to run the controllers and your compliance regime allows it. And choose a mature platform product on top of Flux (the Flux Operator being the project-adjacent example) when what you really need is fleet management rather than the raw toolkit.

Running Flux as a platform for other teams

Everything so far is mechanism. Running Flux as a product is about people: what you promise, what you provide, and how you keep the promise as the number of teams grows.

Decide the contract

A platform has a contract, even when nobody wrote it down. Write it. What does a team get (a namespace, a service account with defined permissions, a place in Git, an alert route), what must they do (use the template, pin versions, label for cost), what you guarantee (the delivery service levels) and what they may not do (cluster-scoped changes, unreviewed suspensions). A written contract turns arguments about exceptions into conversations about a document.

Provide the paved road

Most support load comes from teams making reasonable-looking mistakes. Reduce it with templates and generation. A tenant repository template with a working Kustomization, a HelmRelease and an Alert. A command or pipeline that runs flux create tenant and commits the result. A CI check that runs flux build ks, flux envsubst --strict and a schema validation on every pull request, so tenants see their errors in the pull request rather than in a cluster event. Since 2.9 the schema plugin (flux plugin install schema) validates manifests against JSON schema and CEL, which is a good fit for that CI check; pin it by version or digest like any other tool.

flux plugin search
flux plugin install schema@<version>
flux plugin list

Replace <version> with a release you have checked; the plugin catalogue also offers mirror (for mirroring charts, artifacts and images) and operator (the Flux Operator CLI).

A CI check for tenant pull requests might look like this.

set -euo pipefail
flux build ks tenant-app --path ./apps/base --dry-run >/dev/null
flux envsubst --strict < rendered.yaml > /dev/null
flux diff ks tenant-app --path ./apps/base --in-memory-build --ignore-not-found

The diff output is the most valuable part: it shows the reviewer exactly what will change on the cluster, which is a far better review artefact than a YAML diff.

Give tenants visibility

A tenant who cannot see what Flux is doing will ask you. Give them read access to their own Flux objects and events, a route for their alerts, and a dashboard filtered to their namespace. Give them the commands: flux get ks -n <ns>, flux events -n <ns>, flux trace deployment/<name> -n <ns> (which walks from a workload back to the Kustomization and source that own it) and flux tree ks <name>. Teams that can answer "why did this deploy fail" themselves generate a fraction of the tickets.

Manage the fleet, not the cluster

Past a handful of clusters, treat clusters as cattle. Bootstrap should be one automated step (Terraform with flux_bootstrap_git, or a FluxInstance), each cluster's differences should be a small overlay plus a cluster-vars ConfigMap, and upgrades should roll through rings. The Flux Operator's FluxInstance is worth evaluating for exactly this: it manages the install declaratively, tracks versions with distribution.version ("2.x" for minor upgrades, "2.9.x" for patch-only, an exact version to freeze) and makes the controller configuration part of a manifest rather than a patch on generated files.

apiVersion: fluxcd.controlplane.io/v1
kind: FluxInstance
metadata:
  name: flux
  namespace: flux-system
spec:
  distribution:
    version: "2.9.x"
    registry: ghcr.io/fluxcd
  components:
    - source-controller
    - kustomize-controller
    - helm-controller
    - notification-controller
  cluster:
    type: kubernetes
    multitenant: true
  sync:
    kind: GitRepository
    url: https://github.com/example-org/fleet
    ref: refs/heads/main
    path: clusters/prod-1

The Operator is a separate project maintained by ControlPlane, not part of the Flux core, and you should treat adopting it as a decision with its own upgrade cadence, as noted earlier for the 0.53.0 requirement. Confirm the exact FluxInstance fields against its documentation before use.

Measure the platform as a product

Track a few numbers monthly: the merge-to-applied latency, the percentage of reconciles that succeed, the number of tenant tickets and their causes, the age of your oldest suspension, the time to onboard a new team and the time to bootstrap a new cluster. The ticket categories tell you what to automate next. A platform team whose tickets are dominated by "why is my release stuck" needs better visibility and alerts. One dominated by "how do I set this up" needs a better template.

Keep the bus factor above one

The last habit is the unglamorous one. At least two people must be able to upgrade Flux, rotate the decryption key, run the break-glass procedure and restore the cluster from Git alone. Test that claim by having the second person do each task while you watch. A platform that only its author can repair is a liability, however good it is.

The review checklist

Use this for design documents and for pull requests that touch platform paths. Each item is a question with a reason behind it.

Architecture and scaling

• Does every Kustomization have an interval that matches its purpose, with sources using webhooks rather than very short polling?
• Are --concurrent, limits and the /tmp volume set from measurement, and are the patches in the flux-system kustomization so they survive upgrades?
• Is the Helm index cache enabled if many HelmRepository sources are in use, and are charts moving to OCIRepository with chartRef?
• If sharding is used, are sources and their consumers on the same shard, and are HelmRelease chart labels set?

Delivery correctness

• Does every layer with consumers use wait: true or explicit health checks, and is each ordering dependency written as dependsOn?
• Are namespaces and volume claims protected from pruning, and is deletionPolicy considered for anything whose deletion must not cascade?
• Do substitutions use defaults or pass flux envsubst --strict, given that strict mode is now the default?
• Do HelmReleases pin chart versions in production, and have postRenderStrategy and server-side apply settings been reviewed after the 2.8 and 2.9 default changes?

Security

• Does every tenant Kustomization and HelmRelease impersonate a namespace-scoped service account, enforced by --default-service-account and by admission policy?
• Are --no-cross-namespace-refs and --no-remote-bases set, and is the tenant ClusterRole narrower than the cluster-admin default?
• Are decryption keys per tenant, backed up, rotatable and out of Git?
• Are Receivers using HMAC or OIDC, with the network path restricted?
• Are artifacts and controller images verified, pinned and mirrored?

Operations

• Is there an alert for not-ready, for long suspensions and for controller restarts, with a route to the owning team?
• Is the upgrade path written, rehearsed on a canary and rolled through rings, with the Kubernetes support window considered?
• Is break-glass documented, authorised and followed by a back-port to Git?
• Can a second person perform each operational task?

The complete picture

Put the pieces together as one system and the shape becomes clear. Git or a signed OCI artifact holds the desired state, protected by review, code owners and, where it matters, signatures. source-controller pulls and verifies it, and produces artifacts. kustomize-controller and helm-controller apply it, with impersonation confining each tenant, decryption handing secrets to the right identity, health checks defining "done" and pruning keeping the cluster equal to the source. notification-controller reports outcomes to the people who own them and accepts inbound triggers through hardened Receivers. Metrics from the controllers and from kube-state-metrics feed alerts and service levels. Upgrades roll through rings, governed by the project's procedure, and a small set of rules and a written contract keep the whole thing manageable as teams are added.

The failure model is equally simple to state. Flux failing is a delivery outage, not a runtime outage, so its recovery has a different urgency from the applications it serves, but it removes your fastest way to fix things, which is why break-glass matters. The first scaling wall is usually the API server and the Helm index rather than CPU. The most dangerous features are pruning and the power of the two cluster-admin controllers. The most common self-inflicted wounds are skipped migrations, shared decryption keys, forgotten suspensions and lockdown flags that nobody verified.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - gotk-components.yaml
  - gotk-sync.yaml
patches:
  - patch: |
      - op: add
        path: /spec/template/spec/containers/0/args/-
        value: --no-cross-namespace-refs=true
      - op: add
        path: /spec/template/spec/containers/0/args/-
        value: --default-service-account=default
    target:
      kind: Deployment
      name: "(kustomize-controller|helm-controller)"
  - patch: |
      - op: add
        path: /spec/template/spec/containers/0/args/-
        value: --no-remote-bases=true
    target:
      kind: Deployment
      name: kustomize-controller

This is the smallest useful lockdown: cross-namespace references blocked, a default service account, and no remote bases. Everything else in this guide is a reason the file exists and a set of checks that it stays that way. To go further, the natural neighbours are Kubernetes for the RBAC and admission machinery that tenancy rests on, Helm and Kustomize for what the controllers run, Terraform for what bootstraps them, Vault for the secret stores that complement SOPS, and Argo CD for the main alternative.

Where the series leaves you

At the start you typed flux bootstrap and watched a cluster take itself over. At the middle level you learned to shape what it does: sources, Kustomizations, HelmReleases, substitution, health checks and debugging. At this level the question changed from "how do I make it work" to "how do I make it safe, measurable and boring for everyone who depends on it".

That shift is most of what seniority means with this tool. You can now predict where Flux will break before it does, argue the trade-offs between repository shapes and between Flux and its alternatives, run an upgrade as a staged release, constrain tenants in a way you can demonstrate, and respond to incidents from a playbook rather than from panic. You know where it stops, and you can say so without sounding like you are attacking it.

Keep learning the way the project itself moves. Read the release notes of each new minor, particularly the sections on removed APIs and changed defaults, because those are the changes that hurt. Watch the controller changelogs. Keep your migration muscle exercised, since the pattern of deprecating and removing beta APIs will recur. And keep your runbook honest: the platform that serves you best is the one whose next incident you have already rehearsed.

Sources

• Flux documentation
• Installation
• Upgrading Flux
• Uninstalling Flux
• Bootstrap for GitHub
• Multi-tenancy configuration
• Workload identity
• Vertical scaling
• Sharding
• Helm OOM detection
• Helm drift detection
• Optional components
• Concepts
• Components
• GitRepositories
• OCIRepositories
• HelmRepositories
• Kustomizations
• HelmReleases
• Receivers
• Alerts
• Providers
• Image automation components
• CLI reference
• Troubleshooting cheatsheet
• CEL health checks
• Security
• Secrets management
• Manage Kubernetes secrets with SOPS
• Automate image updates to Git
• Metrics
• Flux GitHub Action
• Flux releases
• Announcing Flux 2.9 GA
• Announcing Flux 2.8 GA
• Announcing Flux 2.7 GA
• Upgrade Procedure for Flux v2.7+
• flux2 releases
• kustomize-controller changelog
• helm-controller changelog
• source-controller changelog
• flux2-kustomize-helm-example
• flux2-multi-tenancy
• flux2-monitoring-example
• Flux Operator documentation

A senior interview about Flux is rarely about syntax. The interviewer already assumes you can write a Kustomization. What they are testing is whether you can be trusted to own a delivery platform: whether you reason from failure modes, whether you can draw a boundary and defend it, and whether you can explain a trade-off to someone who will not read the YAML. This file is organised around those three kinds of question: design prompts, incident prompts and the conversations with leadership. The facts below are for Flux v2.9.5.

How to answer at this level

Three habits separate a senior answer from a mid-level one, and they apply to every question below.

State the constraint before the design. "It depends" is only a good answer if you then say what it depends on. For Flux the usual dependencies are the number of clusters, the number of teams, how hostile the tenants are (an internal team is different from a customer running code), the regulatory regime and whether the organisation wants a central console. Name two or three of these out loud, pick an assumption and proceed.

Name the failure mode of your own design. Every design has one. If you propose sharding, say that notification-controller and source-watcher cannot be sharded. If you propose a global SOPS key, say it is a shared identity. Interviewers notice when candidates volunteer the weakness, and it is the fastest way to signal experience.

Separate delivery from runtime. When Flux breaks, the applications keep running. Saying so early, and then explaining what you cannot do while it is broken, shows you understand what the tool is.

Design prompt: a platform for forty teams

The prompt. "Your company runs eight Kubernetes clusters across two regions and has forty product teams. Design the GitOps platform with Flux."

A strong answer starts with questions. How hostile are the tenants? Are there regulatory boundaries between teams? Does anyone require a central dashboard? How many clusters will there be in two years? Assume internal teams who are trusted not to be malicious but who make mistakes, eight clusters growing to thirty, and a requirement for an audit trail.

Then the shape. A platform repository owned by the platform team holds cluster entry points under clusters/<name>/, the infrastructure layer (controllers, policy engine, issuers) and the tenant definitions. Each team has its own repository, referenced from the platform repository by a GitRepository and a Kustomization that lives in the team's namespace. Each cluster is bootstrapped by automation (the Terraform flux_bootstrap_git resource or a Flux Operator FluxInstance), never by hand.

Tenancy. Each team gets a namespace, a ServiceAccount and a RoleBinding created with flux create tenant, using a narrow ClusterRole instead of the cluster-admin default. The controllers are patched with --no-cross-namespace-refs=true, --no-remote-bases=true and --default-service-account=default, so forgetting to impersonate fails safe, and a Kyverno or Gatekeeper policy stops a tenant naming a more powerful account or creating cluster-scoped objects.

Ordering. Layers are separate Kustomizations with dependsOn and wait: true: CRDs and controllers, then policies and issuers, then tenants. The policy layer must be ready before any tenant layer, so the constraints exist before the tenants they constrain.

Secrets. SOPS with a key per tenant, in the tenant's namespace, rather than one global Age key, because a global key lets every Kustomization decrypt everything. If the organisation already runs a central secret store, External Secrets Operator is the alternative, and I would ask about rotation and regional storage requirements before deciding.

Promotion. Environments are overlays or version ranges on an OCI source; production follows an exact version that a person bumps. Image automation is allowed to commit only to lower environments.

Observability. Controller metrics plus kube-state-metrics Custom Resource State for per-object status, alerts on long not-ready, long suspensions and controller restarts, and a per-tenant Alert routing failures to the owning team.

What I would say about the weak points. The controllers' queues are shared, so a noisy tenant can slow others; I would measure reconcile duration and be ready to shard by label. Flux has no built-in UI, so I would ask whether the organisation needs one and, if so, evaluate the Flux Operator's web UI or a third-party tool. And tenancy is logical, not physical: if a tenant's code is hostile, they get their own cluster.

What separates strong from weak

A weak answer draws one repository and one Kustomization and stops. A strong answer has layers, names the lockdown flags, pairs them with admission policy, explains the secret-key decision and ends on the honest limits.

Design prompt: Flux or Argo CD

The prompt. "We are choosing between Flux and Argo CD for twenty clusters. Recommend one."

The mistake is to pick a side. The senior move is to identify the deciding axis. Architecture: Flux runs agents in each cluster that pull from Git or OCI, so the clusters need only egress access and there is no central component holding credentials for all of them. Argo CD typically runs a central control plane that pushes to clusters, which gives one place to see and manage everything but concentrates access and depends on that control plane. Interface: Flux is CLI and CRD native with no built-in UI, composing with Kubernetes RBAC; Argo CD is UI-centric and application-oriented. Composition: Flux's Kustomization and HelmRelease objects are ordinary Kubernetes resources, so you can template, patch and generate them with the same tools as everything else.

Then the recommendation with a condition: if the organisation needs a central console that application teams use daily, or wants one control plane to manage all clusters, Argo CD is the natural fit; if it values per-cluster autonomy, egress-only networking and Kubernetes-native composition, and is comfortable with a CLI-and-metrics operating model, Flux is. Mention that Flux's UI gap is closable (the Flux Operator ships a web UI, and community tools exist) and that neither choice is irreversible because the manifests are in Git. See the Argo CD guide for the other side of the argument.

Design prompt: upgrading a fleet across the 2.7 to 2.9 removals

The prompt. "You inherit twelve clusters on Flux 2.6 with five hundred manifests in Git. Get them to 2.9 safely."

The answer is the official procedure, plus the fleet discipline around it. Start by reading what was removed: the v1beta1 APIs in 2.7, v1beta2 of sources and Kustomization and v2beta2 of HelmRelease in 2.8, and the v1beta2 image and notification APIs in 2.9. Then:

1. Migrate the manifests in Git with flux migrate -v 2.6 -f ., using --dry-run first, commit, and reconcile the flux-system Kustomization.
2. Migrate the objects stored in etcd with flux migrate against each cluster; it is idempotent.
3. Upgrade the controllers by rerunning bootstrap with the original flags or merging the GitHub Action's pull request, and verify with flux check.
4. If image automation is in use, run flux migrate -v 2.8 -f . and migrate the clusters again.

The fleet discipline: a canary cluster first, then rings, pausing between them; flux build, flux envsubst --strict and flux migrate --dry-run in CI for every cluster's path; and a written list of behaviour changes that do not fail validation. The ones to name are strict post-build substitution on by default in 2.9 (variable not set (strict mode)), the new postRenderStrategy defaulting to combined, the Helm 4 defaults from 2.8 (server-side apply for new installs, kstatus health checks), and the kubeconfig Secret rule in 2.9.5. If the Flux Operator is in use, it has to be at v0.53.0 or later first.

Close on the Kubernetes window: 2.9 supports 1.34, 1.35 and 1.36, and only the latest three minors are ever supported, so the two upgrade calendars are coupled.

Follow-up: "What if a cluster breaks after the upgrade?"

Revert the commit that regenerated gotk-components.yaml so the controllers return to the previous version, but say plainly that a downgrade is only safe if the migration did not change stored versions in a way the older controllers cannot read, so you would check the release notes first. Then reproduce the failure on the canary. Do not claim rollback is always trivial.

Design prompt: secrets for a regulated employer

The prompt. "A bank in the Gulf requires that encryption keys never leave a regional HSM and that every secret read is logged. Design the secrets story for Flux."

The constraint is in the prompt, and it decides the answer. SOPS with a cloud KMS key in the required region satisfies the residency part: the key never leaves the KMS and kustomize-controller calls it to decrypt, authenticating with workload identity rather than a static credential. But SOPS writes are logged, not reads of the decrypted secret in the cluster, so the read-audit requirement pushes towards a central store: External Secrets Operator syncing from a vault with audit logging, or Vault with transit and, since 2.9, Kubernetes workload identity through a JWT auth method rather than a static token.

A good answer compares the two on rotation and audit. SOPS rotates through a commit and re-encryption; ESO rotates in the store without a commit. SOPS leaves ciphertext in Git, which some policies forbid; ESO leaves only references. And name what remains true either way: anyone who can create a pod in a namespace can read that namespace's Secrets, so Kubernetes RBAC and, where required, encryption at rest in etcd matter as much as the delivery mechanism. See the Vault guide.

Design prompt: making a slow fleet fast

The prompt. "Reconciliation is slow on a cluster with three thousand Flux objects. Where do you look, and what do you change?"

Measure before changing. Look at gotk_reconcile_duration_seconds_bucket and the workqueue_* metrics on each controller, along with CPU and memory. The pattern tells you where the wall is.

• High queue depth, low CPU. The controller is waiting, most likely on the API server. Reduce the work: lengthen Kustomization intervals (the interval is only the drift-correction period), add Receivers so source intervals can also grow, and check the --kube-api-qps and --kube-api-burst limits.
• High CPU or memory pressure. Raise --concurrent (the docs use values around 10, and 20 with in-memory builds), raise limits, and back /tmp with a memory-medium emptyDir for kustomize-controller. Enable OOMWatch on helm-controller so it exits cleanly before an OOM kill.
• Slow sources and Helm. Turn on the source-controller Helm cache (--helm-cache-max-size, --helm-cache-ttl, --helm-cache-purge-interval), and move charts to OCI with OCIRepository and chartRef, since HelmRepository of type oci is in maintenance mode.
• One noisy tenant. Shard by label: sharding.fluxcd.io/key, a dedicated controller set with --watch-label-selector, the main set with the negated selector. Sources and their consumers must share a shard, and notification-controller and source-watcher cannot be sharded.

Finish with the warning that there is no documented object limit, so the answer is always measured, not quoted.

Incident prompt: nothing has deployed for an hour

The prompt. "Several teams report that merges have not reached the cluster for an hour. Walk me through your first fifteen minutes."

First, say what is and is not at risk: running applications are unaffected, because Flux failing is a delivery outage. Then triage in order.

1. kubectl get pods -n flux-system. If a controller is crash-looping or not running, flux get output may be stale, since nothing is writing statuses.
2. flux check, then flux get all -A --status-selector ready=false to see the shape of the failure: one object, one team or everything.
3. Decide source or consumer. flux get sources git -A: if the revision is old and the message is failed to checkout and determine revision, it is upstream (credentials, branch, network, the Git host). If the source is current but Kustomizations are stale, the consumer or its controller is at fault.
4. For one stuck Kustomization, read the message. dependency '...' is not ready sends you to the dependency. health check failed after ... sends you to the workload. variable not set (strict mode) sends you to a substitution.
5. flux events -A and flux logs --level=error -A for the specifics.

Communicate early: say that delivery is delayed, that applications are healthy, and when you will update. If the Git host is down, mention the break-glass procedure and that you are not using it yet.

What a weak answer misses

A weak answer runs flux reconcile repeatedly, which changes nothing if the cause is upstream. It also skips the pod check, and it does not communicate.

Incident prompt: a bad merge is deleting production resources

The prompt. "A commit removed a directory and Flux is pruning. What do you do?"

Stop further damage first. flux suspend ks <name> halts applies and prunes for that Kustomization. Then find out what was already deleted: the Kustomization's inventory and the previous revision tell you. Revert the commit and resume, which restores from history. For stateful objects the recovery depends on protections set beforehand: the annotation kustomize.toolkit.fluxcd.io/prune: Disabled on namespaces and volume claims, and the storage class's reclaim policy. Say that these are design-time controls, and that the post-incident actions are to add them, to tighten review on that path, and possibly to set deletionPolicy on Kustomizations whose removal must not cascade.

The follow-up is often "why did it pass review?" An honest answer names the gap: a diff of deleted files is easy to skim past. The fix is a CI step that runs flux diff ks with --in-memory-build and posts the list of objects that would be deleted on the pull request.

Incident prompt: reconciles fail after a patch upgrade

The prompt. "You upgraded to 2.9.x and now some Kustomizations fail with variable not set (strict mode). Why, and what now?"

Kustomize-controller 1.9, shipped in Flux 2.9, turned on the StrictPostBuildSubstitutions gate by default. Any ${var} with no supplied value and no default now fails the reconcile; previously the variable quietly became an empty string. Strictly, the manifests were relying on undefined behaviour. The fix, in order of preference, is to supply the variable in substitute or substituteFrom, to give a default with ${var:=default}, or to annotate an object that legitimately contains ${...} text with kustomize.toolkit.fluxcd.io/substitute: disabled. Turning the gate off with --feature-gates=StrictPostBuildSubstitutions=false is an acceptable temporary measure while you audit. Prevent recurrence by running flux envsubst --strict in CI.

A nuance worth adding: the new .spec.postBuild.substituteStrategy: Always makes substitution happen even when no variables are defined, which is relevant if some manifests have ${...} text that is meant to be literal.

Incident prompt: a leaked Git credential

The prompt. "A deploy key for the fleet repository has leaked. What do you do?"

Distinguish a read key from a write key. A read-only deploy key lets an attacker read your manifests, which may contain encrypted secrets and certainly reveals your topology, but it cannot change what is deployed. A write key lets them push and, through Flux, deploy. Treat the write case as a potential cluster compromise.

Revoke the key first, then create a new one with flux create secret git and update the Secret the GitRepository references (Flux will report authentication required until you do). Review the repository history for commits made with the key, and review what Flux applied in the window, using Git history and Kubernetes audit logs. If commit signature verification (GitRepository.spec.verify) is configured, unsigned commits from the leaked key would have been refused, which is the argument for configuring it. Finally, reduce the blast radius for next time: per-repository keys, read-only keys wherever writing is not needed (image automation is the legitimate writer), and workload identity or GitHub App authentication instead of long-lived keys where possible.

Incident prompt: a tenant escaped their namespace

The prompt. "A tenant's HelmRelease created a ClusterRoleBinding. How did this happen and how do you stop it?"

The likely cause is that the HelmRelease or its Kustomization had no serviceAccountName, or named a powerful one, and so was applied with the controller's own cluster-admin rights. Check whether --default-service-account was set on the controllers. If it was not, every object without an explicit account ran as cluster-admin. If it was set, check whether the tenant named another account explicitly, which the flag does not prevent.

The fix has three layers. The flags (--default-service-account, --no-cross-namespace-refs, --no-remote-bases) make the safe path the default. An admission policy forbids tenant objects from naming accounts outside their namespace and from creating cluster-scoped kinds. And the tenant's ClusterRole, bound through flux create tenant --cluster-role, is narrow enough that even a mistake cannot escalate. Then remove the offending binding, review what it was used for, and run the detection rule that would have caught it.

Arguing trade-offs to leadership

The last category is not technical at all. You will be asked to justify spend, defend a delay or decline a request. The pattern that works is to restate the goal in their terms, present two or three options with costs and risks, recommend one and say what would change your mind.

"Why are we spending engineer time on an upgrade that adds nothing?"

Reframe: the upgrade removes risk. Flux removes old API versions on a schedule (three releases in a row, 2.7, 2.8 and 2.9), so an unmaintained cluster is one upgrade from a forced, larger migration, and it also falls out of the supported Kubernetes window, since only the latest three minors are supported. Small regular upgrades are cheaper than one big one, and the evidence is the migration procedure itself. Offer a number: what a patch upgrade costs with automation, against what a four-version catch-up costs.

"Can we skip the review gate for a faster release?"

Explain what the gate buys. Flux applies whatever is merged into a watched path with the permissions of its controllers, so review is the security boundary. Propose faster alternatives that keep it: Receivers to cut detection latency, flux diff in pull requests so review is quicker, and automated promotion to lower environments. Offer to measure merge-to-applied time and make it a published service level, so speed is a number rather than a complaint.

"Why can't we just give every team cluster-admin?"

Translate into impact: cluster-admin on a shared cluster means one team's mistake can delete another's namespace, and the audit trail cannot distinguish them. Show the cost of the alternative honestly. Namespaced tenants need templates and a short onboarding, and there is an occasional request for a cluster-scoped resource that goes through the platform team. Offer the paved road as the answer to the friction.

"Should we buy a commercial offering?"

Flux itself is free (Apache-2.0, a CNCF graduated project). What you are buying is support, fleet management and people-time. Frame it as a build-versus-buy on engineer-hours: how many platform engineers does the current approach require, how many clusters are coming, and what is the cost of an outage on the day nobody who understands it is available. Vendors such as ControlPlane offer enterprise support for Flux. Recommend a decision criterion, not a product: "if onboarding a cluster takes more than a day of engineer time, or fewer than two people can run an upgrade, we should buy".

"Should we adopt the Flux Operator?"

It is a separate project, maintained by ControlPlane, that manages the Flux install through a FluxInstance object, tracks versions declaratively and ships a web UI. The benefit is fleet-friendly management and a console; the costs are another component with its own upgrade cadence and a dependency on a project that is not Flux core. Recommend a pilot on a non-production cluster and a decision after you have measured onboarding time and upgrade effort.

Short-answer questions you should be able to handle

These are quick questions that test depth. A sentence or two each is enough.

What is the difference between interval on a source and on a Kustomization? On a source it is how often Flux checks for new content. On a Kustomization it is how often Flux re-applies and corrects drift when nothing changed; a source change triggers a reconcile immediately. Different jobs, so different values.

Why is retryInterval separate? So a failed reconcile can retry soon while a healthy one runs rarely. One interval cannot serve both.

Which controllers hold cluster-admin, and why does that matter? kustomize-controller and helm-controller. They must be able to apply any resource, so anything that reaches them without impersonation has that power; the lockdown flags and policy exist to prevent that.

What happens to applications if Flux is uninstalled? With flux uninstall, the controllers, CRDs and custom resources are removed but the workloads Flux deployed remain. Deleting Flux with kubectl is not supported and risks cascading deletes through finalizers and pruning.

Why is HelmRepository of type oci discouraged? It is in maintenance mode since source-controller 1.8. Use an OCIRepository with a HelmRelease chartRef.

How do you stop Flux reverting a field that an autoscaler owns? Use the Kustomization .spec.ignore rules (new in 2.9) or HelmRelease .spec.driftDetection.ignore.

How does Flux know what to delete when prune is on? It records every applied object in .status.inventory and deletes inventory entries that no longer appear in the build.

What are the served API versions for Alert and Provider? notification.toolkit.fluxcd.io/v1beta3, which is still beta and current. Receivers are v1.

Where do per-object readiness metrics come from? Not from Flux. From kube-state-metrics with a Custom Resource State configuration, as in the monitoring example.

Can you shard notification-controller? No. Neither can source-watcher.

What changed about Helm installs in 2.8? helm-controller moved to Helm 4: new releases install with server-side apply and health checks use kstatus by default. Existing releases stay on client-side apply under upgrade.serverSideApply: auto. The UseHelm3Defaults gate restores the old behaviour.

How do you verify an artifact before Flux applies it? OCIRepository.spec.verify with cosign or notation, and GitRepository.spec.verify for signed commits or tags.

What a strong answer covers that a weak one misses

Across every question above, the same differences recur.

• Failure modes. The strong answer states what happens when a component fails. The weak one describes only the happy path.
• Blast radius. The strong answer says who is affected and how far. The weak one says "it breaks".
• Order of operations. The strong answer triages in a sequence and explains why that sequence. The weak one tries commands at random.
• Prevention. The strong answer ends with the control that would have prevented the incident. The weak one ends at the fix.
• Honesty about limits. The strong answer says where Flux stops and what it would add. The weak one defends the tool.
• Facts that are current. The strong answer uses v1 and v2 APIs, knows the 2.9 defaults and does not cite removed versions. Quoting v2beta1 in an interview dates you immediately.

Practice prompts to rehearse out loud

Try these with a colleague, for five minutes each, without notes.

1. Design the GitOps platform for a company with ten clusters and thirty teams, starting from the constraints.
2. A HelmRelease has been failing for a day with upgrade retries exhausted. Walk through diagnosis and recovery.
3. Explain to a product manager why a change merged five minutes ago is not yet live, and what you could do to shorten it.
4. Argue for and against a monorepo for the platform, then recommend one.
5. The security team asks you to prove that tenants cannot escalate. What evidence do you produce?
6. Describe the upgrade from 2.6 to 2.9 to an auditor.

The final self-test

Answer these in under ten minutes, aloud, and mark any you hesitate on.

1. Which two controllers are bound to cluster-admin, and what stops a tenant reaching that power?
2. What are the four steps of the official upgrade procedure for moving from 2.6 to 2.9, and what does each one fix?
3. Which three Flux 2.9 changes alter behaviour without changing an API version?
4. Name the first three things you check when nothing is deploying, in order.
5. Why does deleting a Kustomization with prune: true delete workloads, and how do you prevent it?
6. Name two scaling levers that add capacity and two that spread load in time.
7. What are the sharding rules, and which components cannot be sharded?
8. What is the difference between a global SOPS key and a per-tenant one, and when is each acceptable?
9. Why do per-object readiness metrics not appear on the controllers' own /metrics endpoints?
10. Where does Flux stop, and what would you add for canary releases and for a central UI?

If you can answer eight of these without hesitation and explain the reasoning behind each, you are ready for the senior conversation. The Kubernetes, Argo CD and Terraform guides cover the neighbouring topics an interviewer is likely to branch into.

This file collects what only shows up once Flux carries real load for many teams: production traps that do not appear in a demo, habits that keep a shared platform healthy, and review checklists you can paste into a pull-request template. It is written against Flux v2.9.5, and it assumes you have read the senior detailed guide, so explanations here are short and the emphasis is on recognising the problem quickly.

Production traps that appear only at scale

Each trap below has a symptom you will see, the cause and the fix. They are ordered roughly by how often they hurt.

Every Kustomization on a one-minute interval

Symptom. The API server is busy, reconcile durations climb, and Git-provider rate limits trigger. Cause. The interval on a Kustomization is the drift-correction period, not the change-detection period, so a short value re-applies unchanged state constantly. Fix. Give sources a short interval or a Receiver, and give Kustomizations a long interval such as thirty minutes. A change in the source still triggers an immediate reconcile.

The restart thundering herd

Symptom. After source-controller restarts, dozens of sources fail for several minutes with checkout errors or rate-limit responses. Cause. Artifacts live on an emptyDir at /data, so a restart empties them and every source fetches at once. Fix. Use --interval-jitter-percentage, lengthen intervals, use Receivers, mirror heavy dependencies to a registry you control, and avoid unnecessary restarts by setting sensible memory limits.

Sources and consumers on different shards

Symptom. After enabling sharding, a Kustomization reports that its source cannot be found, even though the source is healthy. Cause. A controller only sees objects that match its --watch-label-selector, so a source and everything that uses it must carry the same shard label. For HelmReleases, the generated HelmChart needs the label too, through .spec.chart.metadata.labels. Fix. Label the source, the consumer and the chart template together, and remember that notification-controller and source-watcher cannot be sharded.

Dependency chains that are too deep

Symptom. A full rebuild takes many minutes even when everything is healthy. Cause. Each dependsOn hop waits for readiness and then requeues after --requeue-dependency, which defaults to thirty seconds. Fix. Reduce the number of layers, lower --requeue-dependency to a few seconds on busy clusters, and write precise health checks so a layer becomes ready as soon as it really is.

Health checks that hide a failure

Symptom. A Kustomization reports ready while an application is broken. Cause. Without wait: true or healthChecks, "ready" only means the apply succeeded. Custom resources that kstatus does not understand are treated as healthy when they exist. Fix. Enable wait: true on layers with consumers, list explicit healthChecks, and write healthCheckExprs in CEL for custom kinds.

The first incident with pruning

Symptom. A refactor moved files, and the resources they defined were deleted and recreated, or deleted outright. Cause. prune: true deletes anything in the inventory that is no longer in the build, and a moved or renamed object looks like a removal. Fix. Annotate namespaces and volume claims with kustomize.toolkit.fluxcd.io/prune: Disabled, use deletionPolicy thoughtfully, and preview changes with flux diff ks in pull requests.

A global SOPS key on a shared cluster

Symptom. Nobody notices, until a security review finds that every tenant's Kustomization can decrypt every other tenant's secrets. Cause. The --sops-age-secret flag configures one Age key for all Kustomizations. Fix. Use it only on single-tenant clusters. On shared clusters, one key Secret per tenant, referenced by that tenant's .spec.decryption.secretRef.

Forgotten suspensions

Symptom. A fix someone applied by hand during an incident is reverted weeks later, or a change never applies. Cause. Someone ran flux suspend and never resumed, so the object is not being enforced. Fix. Alert on any suspended object older than a day, and require a ticket reference in an annotation whenever a suspension is created.

Lockdown flags that were never verified

Symptom. A tenant creates a cluster-scoped resource despite the documented lockdown. Cause. The flags were set on one controller but not the other, or were overwritten by an upgrade that regenerated gotk-components.yaml without the patch. Fix. Keep patches in the flux-system kustomization, check them with kubectl get deploy -n flux-system -o yaml, and add a test that attempts a forbidden action and expects a failure.

Upgrades that skip the migration

Symptom. After upgrading, reconciles fail with no matches for kind "HelmRelease" in version "helm.toolkit.fluxcd.io/v2beta1" or the same for another removed version. Cause. The 2.7, 2.8 and 2.9 releases removed old API versions, and the YAML or the stored objects still use them. Fix. flux migrate -v 2.6 -f . in Git (and -v 2.8 for image APIs), then flux migrate against the cluster, then upgrade the controllers.

Behaviour changes that no validator rejects

Symptom. After the 2.9 upgrade, a Kustomization fails with variable not set (strict mode), or a Helm release with post-renderers produces different output. Cause. Strict post-build substitution is now on by default, and postRenderStrategy now defaults to combined, so hooks reach post-renderers. Fix. Supply variables or defaults (${var:=default}), set postRenderStrategy: nohooks to keep the old behaviour while you review, and catch both in CI before the upgrade.

Controller out-of-memory kills in the middle of a release

Symptom. A HelmRelease is left in a strange state after helm-controller restarts. Cause. The controller was killed mid-action when rendering exceeded its memory limit. Fix. Enable the OOMWatch gate so helm-controller shuts down gracefully beforehand, raise limits, and find the oversized chart.

Oversized Helm release Secrets

Symptom. Request entity too large: limit is 3145728. Cause. Helm stores release state in a Secret, and a chart with large embedded files exceeds roughly 3 MiB. Fix. Trim the chart with .helmignore or .sourceignore, and move large static content out of the chart.

Perpetual drift from empty fields

Symptom. Flux reports a change on every reconcile and events fill with "configured". Cause. Manifests contain empty fields (null, {}, []) that the API server normalises away, so the desired and actual state never quite match. Fix. Remove empty fields from the manifests. For fields legitimately owned by another actor, use .spec.ignore on a Kustomization or .spec.driftDetection.ignore on a HelmRelease.

Admission webhooks that break dry runs

Symptom. webhook does not support dry run on apply. Cause. Flux's server-side-apply dry run calls webhooks, and a webhook that does not declare sideEffects: None or NoneOnDryRun rejects it. Fix. Correct the webhook configuration, which is almost always a third-party chart default.

Kubeconfig Secrets that point at files

Symptom. From 2.9.5, a Kustomization that manages a remote cluster fails when it reads its kubeconfig Secret. Cause. Kubeconfigs that reference local files (certificate-authority, tokenFile, client-certificate, client-key) are now rejected. Fix. Embed credentials inline, or use workload identity through .spec.kubeConfig.configMapRef.

Platform-ownership habits

Platform owners are defined less by what they know than by what they do regularly.

Read release notes the week they ship. Flux's recent history shows that breaking changes arrive in minor releases, and some arrive in patches (2.9.4 changed CRD schemas; 2.9.5 tightened kubeconfig handling). Reading the notes takes twenty minutes and saves the incident.

Rehearse what you will need during a bad day. Bootstrap a cluster from nothing. Restore a decryption key from backup. Run the break-glass procedure. The first time you do any of these should never be during an incident.

Keep the lockdown in code and test it. The patches live in Git, the admission policy lives in Git, and a test proves they work. A control nobody tests is an assumption.

Publish service levels and report them. A platform team that tells its users "ninety-five percent of merges apply within five minutes, last month was ninety-seven" gets trusted. One that says nothing gets blamed for every slow deploy.

Make the paved road easier than the dirt road. Templates, a tenant generator and a CI check that previews the diff cost a week to build and save months of tickets.

Own the exit. Know what it would take to leave Flux: the workloads keep running, and the declarative state is already in Git. Being able to say so makes your recommendations more credible.

Keep two people able to do everything. Upgrades, key rotation, break-glass and restore. Test the claim by having the second person do it.

Close every incident with a change. A new alert, a new policy, a changed default, a runbook edit. Incidents that teach nothing recur.

Watch your own controllers as closely as you watch tenants'. Alert on controller restarts, reconcile duration and queue depth. A crash-looping controller cannot report its own failure.

Practice cards

Short drills you can run on a scratch cluster. Each takes under half an hour and teaches something the documentation does not make vivid.

Card one: break and recover the source. Point a GitRepository at a wrong branch. Observe the message, what flux get sources git shows, and what happens to dependent Kustomizations. Fix it and time the recovery.

Card two: the prune drill. Create a Kustomization with prune: true that owns a namespace and a deployment. Delete the file from Git and watch the deletion. Repeat with kustomize.toolkit.fluxcd.io/prune: Disabled on the namespace and compare.

Card three: the strict-mode rehearsal. Write a manifest with ${cluster_env} and no supplied value. See variable not set (strict mode). Fix it three ways: supply the value, use a default, and annotate with substitute: disabled. Then confirm flux envsubst --strict catches the first case in CI.

Card four: the tenant escape attempt. Set up a tenant with impersonation, then try to apply a ClusterRoleBinding, to reference another namespace's source and to name another service account. Note which attempts fail with the flags alone, then write the admission policy for the ones that do not.

Card five: the key rotation. Encrypt a Secret to one Age key, add a second recipient, re-encrypt with sops updatekeys, switch the cluster's key Secret, then retire the first key. Practise until it is boring.

Card six: the shard experiment. Label one tenant's source and Kustomization with sharding.fluxcd.io/key: shard1, run a second kustomize-controller with the matching selector, and patch the main one with the negated selector. Verify which controller reconciles which object, then deliberately mislabel the source and see the failure.

Card seven: the upgrade dry run. Take a repository with an old API version in it. Run flux migrate -v 2.6 -f . --dry-run, read the output, run it for real and look at the diff. Then run flux migrate against a cluster and confirm it is idempotent by running it twice.

Card eight: the alert that tells you something. Break a Kustomization and time how long your alert takes to fire. Read the notification as if you were the on-call engineer who has never seen this cluster. Does it name the object, the namespace and the reason? If not, fix the annotation.

Card nine: the receiver test. Create a Receiver, trigger it with flux trigger receiver <name> --token=... --url=..., and confirm the source reconciles. Then use a wrong token and see what you observe. Consider how you would limit who can reach that URL.

Card ten: the rebuild from Git. In a scratch cluster, run flux uninstall, confirm the workloads survive, and rebuild by bootstrapping again. Record how long it took, what was missing and which secrets you needed from outside Git.

Review checklist for pull requests that touch Flux

Paste this into a pull-request template for the platform repository. Reviewers tick what applies.

Every Kustomization or HelmRelease

• Uses current API versions: v1 sources and Kustomization, v2 HelmRelease, v1beta3 Alert and Provider, v1 Receiver and image APIs.
• Has an interval that matches its purpose, and a retryInterval where failures are plausible.
• Has wait: true or explicit healthChecks if anything depends on it, and a timeout long enough for the slowest rollout.
• Names its dependsOn explicitly rather than relying on ordering.
• For tenants, sets serviceAccountName to an account in its own namespace.

Pruning and deletion

• A deleted or moved file has been checked against what it will remove, using the output of flux diff ks.
• Namespaces and volume claims carry kustomize.toolkit.fluxcd.io/prune: Disabled.
• Removing a Kustomization has considered deletionPolicy.

Variables and values

• Every ${var} is supplied, defaulted or intentionally escaped, and flux envsubst --strict passes.
• Production HelmReleases pin chart versions, not ranges.
• postRenderStrategy, upgrade.serverSideApply and waitStrategy are deliberate, not inherited by accident after the 2.8 and 2.9 changes.
• Charts come from OCI with chartRef where possible, not HelmRepository of type oci.

Secrets

• No plaintext secrets, including in comments and examples.
• Encrypted files use the correct tenant's key and .spec.decryption is set.
• Rotation has been considered.

Platform changes (controller flags, RBAC, policies, Receivers)

• The patch lives in clusters/<cluster>/flux-system/kustomization.yaml, not in a hand-edited gotk-components.yaml.
• Lockdown flags (--no-cross-namespace-refs, --no-remote-bases, --default-service-account) are intact on every controller that needs them.
• A canary cluster has applied the change first.
• Receivers use HMAC, OIDC or a secret token, and the network path is restricted.

Review checklist for design documents

Use this when someone proposes a new platform feature, a new cluster type, a new tenancy model or an integration.

Problem and alternatives

• What problem does it solve, for whom, and how will we know it worked?
• What alternatives were considered, including "do nothing" and "use a different tool", and why were they rejected?
• Does it keep Flux within its role (applying desired state), or does it make Flux do CI or rollout analysis?

Failure modes

• What happens when each involved component is down: the Git host, the registry, source-controller, kustomize-controller, notification-controller?
• What is the blast radius of a mistake, and who is affected?
• How is it rolled back, and has the rollback been tried?

Scale

• How many objects will this add, and what is the effect on reconcile duration and API load?
• Does it need sharding, and do the sharding rules hold?
• Does it add a polling source that should be a Receiver?

Security

• What identity does it run as, and is that identity the narrowest that works?
• What new trust does it introduce, and how is it verified (signatures, pinning, policy)?
• Where do its secrets come from, and who can read them?

Operations

• What alerts does it add, and who receives them?
• Who owns it, and can a second person operate it?
• How is it upgraded, and what does the Kubernetes and Flux support window mean for it?
• What does it cost in compute, API calls and engineer time, and how will we measure that?

Exit

• If we remove it in a year, what is the procedure and what remains?

Operational recipes

Short, copyable procedures for things you will do repeatedly.

Find everything that is unhealthy

flux get all -A --status-selector ready=false
flux get ks -A --status-selector ready=false --show-source
flux events -A
flux logs --level=error -A

Trace a workload back to its owner

flux trace deployment/podinfo -n default
flux tree ks flux-system

flux trace walks from a running object to the Kustomization or HelmRelease that owns it and on to the source and revision, which answers "where did this come from" in one command.

See what a change will do before merging

flux diff ks apps --path ./apps/prod --in-memory-build --ignore-not-found
flux build ks apps --path ./apps/prod
flux debug ks apps --show-vars

Unstick a HelmRelease

flux reconcile hr podinfo -n default --reset
flux reconcile hr podinfo -n default --force
flux suspend hr podinfo -n default && flux resume hr podinfo -n default

Use --reset to clear the failure count, --force for a one-off install or upgrade, and the suspend-resume pair as a last resort. Read the error message first, since none of these fixes a bad chart.

Pause and resume safely

kubectl annotate ks apps -n flux-system platform.example.com/suspended-because="INC-1234: investigating bad rollout"
flux suspend ks apps
# ... investigate ...
flux resume ks apps
kubectl annotate ks apps -n flux-system platform.example.com/suspended-because-

The annotation key is your own convention. The point is that a suspension always says why, so your alert on old suspensions has something to show.

Roll out a controller setting to the fleet

# edit clusters/<cluster>/flux-system/kustomization.yaml in Git, then
git commit -am "flux: raise kustomize-controller concurrency" && git push
flux reconcile ks flux-system --with-source
kubectl -n flux-system get deploy kustomize-controller -o jsonpath='{.spec.template.spec.containers[0].args}'

Always confirm the setting reached the running deployment; a patch that does not match its target fails silently.

Upgrade the controllers

flux check --pre
flux install --export --components-extra=image-reflector-controller,image-automation-controller \
  > clusters/prod-1/flux-system/gotk-components.yaml
git commit -am "flux: upgrade to v2.9.5" && git push
flux reconcile ks flux-system --with-source
flux check

Keep --components-extra identical to your bootstrap flags, and run the migration steps first if you are coming from 2.6 or earlier. Rerunning flux bootstrap with the original flags is the alternative.

Verify a Flux image before mirroring it

cosign verify ghcr.io/fluxcd/kustomize-controller:v1.9.5 \
  --certificate-identity-regexp='^https://github\.com/fluxcd/.*$' \
  --certificate-oidc-issuer=https://token.actions.githubusercontent.com

Take Flux out of the loop for one object

Annotate the object with kustomize.toolkit.fluxcd.io/reconcile: disabled to stop Flux managing it, repair it by hand, then remove the annotation. Record the action and back-port the fix to Git, or Flux will revert it once management resumes.

A closing checklist for the platform owner

Before you call the platform production-ready, you should be able to say yes to each line.

• Every controller has limits and concurrency chosen from measurement, and the patches live in Git.
• Sources detect changes through Receivers, and Kustomization intervals are long.
• Each layer has health checks, and dependsOn is explicit.
• Tenants impersonate narrow service accounts, the lockdown flags are on, and admission policy backs them up.
• Decryption keys are per tenant, backed up and rotatable.
• Artifacts and controller images are verified, pinned and mirrored.
• Alerts exist for not-ready, long suspensions, controller restarts and source failures, each with an owner.
• The upgrade procedure is written, rehearsed and staged, and the migrations for 2.7, 2.8 and 2.9 are complete.
• Break-glass is documented, authorised and followed by a back-port.
• At least two people can run every operational task.

Flux rewards this kind of boring diligence. The Kubernetes, Helm and Kustomize guides cover what the controllers run, Prometheus covers the metrics side, and Argo CD is the main alternative if the trade-offs here start to point elsewhere.