The Complete Argo CD Guide

This is part one of three. It covers everything you need to do real work with Argo CD, not a teaser. By the end you can install Argo CD into a cluster, connect it to a Git repository, deploy an application from that repository, watch it drift and heal, roll a bad change back, and explain to a colleague why the cluster looks the way it does. Mid-level and Senior take the same topics further; nothing here is thrown away.

The guide is written against Argo CD v3.5.3, the current stable release at the time of writing (3.5 went generally available on 4 August 2026, and the next minor, 3.6, is scheduled for 3 November 2026). Most tutorials on the web still describe the 2.x series, and a few of their commands no longer work. Where that matters, the text says so and shows the current form.

Each section ends with a Try it task. Do them as you go. They take a few minutes each, and the ideas only stick once you have watched your own application go out of sync and come back.

You will need a Kubernetes cluster you are allowed to break (a local one such as kind, minikube or Docker Desktop's built-in Kubernetes is perfect), kubectl configured to talk to it, and a GitHub account for the later sections. If Kubernetes itself is new to you, read the Kubernetes guide first; Argo CD is a program that manages Kubernetes objects, so it assumes you know what a Deployment and a Service are. If you write your manifests with Helm or Kustomize, the Helm guide and Kustomize guide cover the tools Argo CD will run on your behalf.

What Argo CD is, and the problem it solves

Argo CD is a continuous delivery controller for Kubernetes that uses Git as the source of truth. You describe what should run in the cluster as files in a Git repository. Argo CD watches that repository, compares what it says with what is actually running, tells you where the two differ, and, if you allow it, changes the cluster until they match.

To see why this is useful, look at what teams did before. The classic approach to deploying to Kubernetes is a script in your continuous integration system. The pipeline builds an image, then runs kubectl apply or helm upgrade against the cluster. That works, and it is how many teams still operate, but it has three structural problems.

The first is credentials. For the pipeline to change the cluster, the CI system must hold a credential that can change the cluster. CI systems are large, shared, internet-facing programs that run other people's code in pull requests. Handing them the keys to production is a real risk, and it is one that many security reviews flag.

The second is drift. After the pipeline finishes, nobody is watching. Somebody runs kubectl edit deployment at two in the morning to fix an incident and forgets to put the change back in the repository. A controller modifies an object. A teammate scales something by hand. The repository now says one thing and the cluster does another, and the next pipeline run silently overwrites the hotfix, or does not touch it at all. You only discover the difference when something breaks.

The third is visibility. "What is running in production right now, and is it what we merged?" should be a thirty-second question. With push-based scripts, the honest answer is often "read the pipeline logs and hope nobody changed anything since".

Argo CD attacks all three by reversing the direction. It runs inside the cluster and pulls from Git. The pipeline's only job becomes building an image and committing a new version number to the repository. It never talks to the cluster at all, so it needs no cluster credentials. Argo CD keeps looking at both sides, so drift is detected rather than discovered. And the whole state of the system is one screen: every application, whether it matches Git, and whether it is healthy.

This way of working has a name, GitOps. The idea is simple enough to state in one sentence: the desired state of your system lives in Git, and software continuously makes reality match it. Git then gives you, for free, everything it already gives you for code: history, review, blame, revert, and a permission model.

Argo CD is one implementation of the idea. Its closest neighbour is Flux, which takes the same approach with a different design: Flux is a set of small controllers with no built-in web interface, while Argo CD is a single product with a strong UI and a command-line client. Both are open source projects of the Cloud Native Computing Foundation. This guide only covers Argo CD.

A note on what Argo CD is not. It does not build your container images; that remains the job of your CI system and of tools like Docker. It does not create your cluster; that is the job of something like Terraform. It does not replace Helm or Kustomize; it runs them. It is the last step in the chain: given a repository that already describes the desired state, get it into the cluster and keep it there.

The GitOps loop: desired state, live state, and the difference

Two words carry most of Argo CD's vocabulary, and the official documentation uses them precisely. The target state (also called the desired state) is what the files in Git describe. The live state is what is actually running in the cluster right now: which Deployments exist, how many replicas they have, which image they use.

Argo CD's whole job is a loop with four steps, repeated forever:

1. Fetch the files from the Git repository at the revision you asked for.
2. Render them into plain Kubernetes manifests. If the files are already plain YAML, this is nothing. If they are a Helm chart or a Kustomize overlay, Argo CD runs helm template or kustomize build to produce YAML.
3. Compare those manifests with the live objects in the cluster. This comparison is called a diff.
4. Report the result, and, only if configured to, apply the manifests to remove the difference.

The first three steps are always on. Step four is where you choose how much to trust the tool. You can let Argo CD only report and press a button yourself to apply (manual sync, the default), or let it apply automatically every time Git changes (automated sync). Starting with manual is wise, because it lets you see what Argo CD would do before it does it.

The word for step four is sync. "To sync an application" means to make the live state match the target state, in practice by applying the rendered manifests to the cluster. You will type and click that word hundreds of times.

There is an important consequence of the pull model that beginners often miss. Argo CD does not run when you push; it runs on its own schedule. By default it checks Git roughly every three minutes (the exact number is 120 seconds plus up to 60 seconds of random jitter, so the checks from many applications do not all land at once). You can make that near-instant with a webhook, which the later section on refreshing covers. Until then, if you push a commit and nothing happens for two minutes, nothing is broken.

The nouns: Application, Project, status, health

Argo CD has four core nouns. Learn them now, because every screen, command and error message uses them.

An Application is the central object. It is a record that says: take these files from this repository, and deploy the result to this namespace in this cluster. The file location is called the source, and the place it is deployed is called the destination. The Application does not contain your manifests; it only points at them. Technically an Application is a Kubernetes custom resource (kind Application, API group argoproj.io), which means you can create one the same way you create any other Kubernetes object, with YAML and kubectl. That detail turns out to matter a lot, because it means you can store the Applications themselves in Git too.

The source of an application has a type, called the tool. Argo CD detects it from the files: a directory of plain YAML files, a Kustomize directory (it has a kustomization.yaml), a Helm chart (it has a Chart.yaml), or Jsonnet. You rarely pick this yourself.

An AppProject, usually just called a project, is a group of Applications with shared rules. A project answers questions such as "which repositories may these applications deploy from?" and "which clusters and namespaces may they deploy to?" Every Application belongs to exactly one project. A brand-new installation has one project, named default, which allows everything. In a company, projects are how you stop the payments team from deploying into the security team's namespace. As a beginner you will use default and learn to recognise the errors that come from a stricter one.

The sync status says whether the live state matches Git. It has three values: Synced (they match), OutOfSync (they differ) and Unknown (Argo CD could not tell, usually because it could not read the repository or render the manifests).

The health status says whether the application is actually working. It has six values: Healthy, Progressing (still starting up), Degraded (something has failed), Suspended (paused on purpose), Missing (the resources do not exist) and Unknown. Argo CD decides this by looking at the resources: a Deployment is healthy when its replicas are available, a Pod is degraded when it is crash-looping, and so on.

Keep those two statuses separate in your head. They answer different questions, and mixing them up is the most common beginner misreading of the dashboard.

The third row is the result of the last sync operation rather than a property of the application as a whole. You see it in the sync history and when a sync fails.

Two more words appear constantly. A refresh means "compare the latest Git contents with the live state again". A hard refresh does the same but also throws away Argo CD's cached copy of the rendered manifests, which matters when a cached error is stuck. You will use it in the troubleshooting section.

What is inside Argo CD: the components you will meet

You can use Argo CD for months without thinking about its internals, but three of its pieces appear in every error message and log line, so it is worth knowing what they do. When you install Argo CD, you get a set of pods in a namespace (conventionally called argocd).

The API server (argocd-server) is what you talk to. It serves the web interface and the API that the argocd command-line tool uses. It handles logging in and decides what each user may do. It is stateless: nothing precious lives in it.

The repo server (argocd-repo-server) is the component that touches your Git repository. It clones the repository, keeps a cached copy, and produces the rendered manifests by running Helm, Kustomize or plain file reading. Notably it has no permissions in your cluster. It only reads Git and writes YAML. That separation is deliberate: the component that processes untrusted repository content cannot modify your cluster.

The application controller (argocd-application-controller) is the brain. It watches your cluster, asks the repo server for the target state, compares it with the live state, updates each application's sync and health status, and performs syncs. It runs as a StatefulSet.

Everything else is supporting cast. The ApplicationSet controller stamps out many Applications from one template (a mid-level topic). The notifications controller sends messages to Slack, email and so on. Dex is a bundled single sign-on broker. Redis is a cache, and it is disposable: Argo CD keeps no important state there. All the real state (your Applications, Projects, repository credentials, configuration) is stored as ordinary Kubernetes objects and ConfigMaps. That is why you can back up Argo CD by exporting its objects, and why deleting a pod never loses your setup.

When something goes wrong, this map tells you where to look. A repository that cannot be read or a chart that will not render is a repo-server problem: look at its logs. An application that will not sync or shows the wrong health is a controller matter. A login or permission failure is the API server.

kubectl logs -n argocd deploy/argocd-repo-server
kubectl logs -n argocd statefulset/argocd-application-controller
kubectl logs -n argocd deploy/argocd-server

Installing Argo CD into a cluster

Argo CD installs as a set of Kubernetes manifests into its own namespace. Start by checking that kubectl points where you think it does. Installing into the wrong cluster is the classic first mistake, and it is cheap to avoid.

kubectl config current-context
kubectl get nodes

If you do not have a cluster, create a local one. With kind (Kubernetes in Docker) it is one command, and you can throw it away afterwards:

kind create cluster --name argo-demo

Now create the namespace and install. Note the flags carefully, because this is the most commonly outdated command in tutorials:

kubectl create namespace argocd
kubectl apply -n argocd --server-side --force-conflicts \
  -f https://raw.githubusercontent.com/argoproj/argo-cd/v3.5.3/manifests/install.yaml

The --server-side --force-conflicts part is mandatory since Argo CD 3.3. Argo CD's ApplicationSet definition became larger than the amount of data that client-side kubectl apply is allowed to store in an annotation (262,144 bytes). A plain kubectl apply -f install.yaml, which is what older tutorials show, now fails with this message:

The CustomResourceDefinition "applicationsets.argoproj.io" is invalid: metadata.annotations: Too long: may not be more than 262144 bytes

Server-side apply moves the bookkeeping from an annotation to the API server, which has no such limit. If you meet that error, you have copied an old command; add the two flags and run it again.

Also notice the version in the URL. The official documentation uses stable in its examples, which always points to the newest release. For anything beyond a throwaway experiment, pin a version tag as shown, so that re-running the command next month does not quietly upgrade you.

The install.yaml manifest is the standard, non-high-availability installation, good for learning and evaluation. The project also publishes a high-availability variant (ha/install.yaml, which needs at least three nodes) for production, a namespace-scoped variant for restricted setups, and a headless core-install.yaml. A community-maintained Helm chart (argo/argo-cd from https://argoproj.github.io/argo-helm) is another popular route; note that it is maintained by the community, not by the Argo project itself. All of those are mid-level and senior topics.

Wait for the pods to come up:

kubectl get pods -n argocd

NAME                                                READY   STATUS    RESTARTS   AGE
argocd-application-controller-0                     1/1     Running   0          90s
argocd-applicationset-controller-6c8f7c4b9d-x2k7p   1/1     Running   0          90s
argocd-dex-server-7d9f5b8c6f-q4m8n                  1/1     Running   0          90s
argocd-notifications-controller-5f7b9d6c4-h7t2v     1/1     Running   0          90s
argocd-redis-66b7c5d8f9-w9r4z                       1/1     Running   0          90s
argocd-repo-server-5d8c7f9b6-m3n5p                  1/1     Running   0          90s
argocd-server-6b9d8c7f5-k8j2l                       1/1     Running   0          90s

Your pod name suffixes will differ. What matters is that every pod is Running and READY shows all containers ready. The first minute often shows ContainerCreating or Init, which is normal while images download. If something stays in ImagePullBackOff, your cluster cannot reach the registry; if it stays Pending, the cluster has no room.

You can also confirm that the custom resource definitions, the new object types Argo CD adds to Kubernetes, exist:

kubectl get crd | grep argoproj.io

applications.argoproj.io      2026-10-01T09:14:02Z
applicationsets.argoproj.io   2026-10-01T09:14:02Z
appprojects.argoproj.io       2026-10-01T09:14:02Z

Installing the CLI and checking the setup

The web interface is enough to get started, but the argocd command-line tool is how you script, automate and debug. It is a separate download from the server, and its version should match the server's closely.

On macOS, and on Linux or WSL if you use Homebrew, the simplest route is:

brew install argocd

On Linux without Homebrew, download the pinned binary directly and install it:

VERSION=v3.5.3
curl -sSL -o argocd-linux-amd64 https://github.com/argoproj/argo-cd/releases/download/$VERSION/argocd-linux-amd64
sudo install -m 555 argocd-linux-amd64 /usr/local/bin/argocd
rm argocd-linux-amd64

On Apple Silicon without Homebrew the binary is named argocd-darwin-arm64, and on Intel Macs argocd-darwin-amd64. On Windows, the official documentation describes downloading argocd-windows-amd64.exe from the releases page and adding its folder to your Path; package-manager packages for Windows exist, but they are maintained by the community rather than the Argo project, so check their version before trusting them. If you use WSL, simply follow the Linux steps.

Confirm the client works:

argocd version --client

The output names the client version, such as argocd: v3.5.3+.... The server half of the answer only appears once you are logged in, which is the next step.

Now reach the server. The installation creates a Service called argocd-server, but by default it is only reachable from inside the cluster. The quickest way to get at it from your laptop is a port-forward, which tunnels a local port to the service:

kubectl port-forward svc/argocd-server -n argocd 8080:443

Leave that command running in its own terminal. The interface is now at https://localhost:8080. Your browser will warn about the certificate. That is expected: a default installation generates a self-signed certificate that no browser trusts. For a local experiment, click through the warning. For a real deployment you would configure a proper certificate or terminate TLS at an ingress; that is an operations topic, and the warning is not a bug.

Other ways to expose the server exist (changing the Service to a LoadBalancer, or configuring an Ingress), and they matter for shared installations. For learning, the port-forward is ideal because nothing is exposed beyond your own machine.

Logging in for the first time

A fresh installation has one user, admin, with a randomly generated password. The password is stored in a Kubernetes Secret named argocd-initial-admin-secret. The CLI can read it for you:

argocd admin initial-password -n argocd

Copy the password it prints. (Older tutorials tell you the password is the name of the argocd-server pod. That was true before version 1.9 and has not been true for years.)

Log in with the CLI, with the port-forward still running. Because of the self-signed certificate, the CLI will refuse to connect at first, printing a message like x509: certificate signed by unknown authority. For local learning only, tell it to skip verification:

argocd login localhost:8080 --username admin --insecure

It prompts for the password. On success you see 'admin:login' logged in successfully and a line naming the context. The --insecure flag disables certificate checking, so never use it against a server you do not control; in production you fix the certificate instead.

Now argocd version shows both halves:

argocd version

argocd: v3.5.3+abc1234
  BuildDate: 2026-09-14T10:02:11Z
  GitCommit: abc1234
  ...
argocd-server: v3.5.3+abc1234
  ...

If the client and server minor versions differ, the tool usually still works, but mismatches are a source of odd behaviour, so keep them aligned.

The first thing to do after logging in is change the generated password, because the initial one is only meant to get you in:

argocd account update-password

It asks for the current password and the new one twice. After that, the initial secret is no longer needed, and the official guidance is to delete it:

kubectl -n argocd delete secret argocd-initial-admin-secret

Log into the web interface at the same address with username admin and your password. You will see an empty applications page with a + New App button. That emptiness is about to change.

Your first application: the guestbook

The Argo project maintains a public repository of example applications, argocd-example-apps. Its guestbook directory contains a tiny web application as two plain Kubernetes manifests, which makes it the ideal first deployment. You will deploy it into the default namespace of the same cluster Argo CD runs in.

One detail first. Argo CD calls the cluster it runs in the in-cluster destination, and its API address from inside the cluster is always https://kubernetes.default.svc. You can see it is registered already:

argocd cluster list

SERVER                          NAME        VERSION  STATUS      MESSAGE  PROJECT
https://kubernetes.default.svc  in-cluster  1.35     Successful

The version shown will be whatever your cluster runs. Now create the Application. The CLI command spells out the four things an Application needs: where the files are, which folder, and where to put the result.

argocd app create guestbook \
  --repo https://github.com/argoproj/argocd-example-apps.git \
  --path guestbook \
  --dest-server https://kubernetes.default.svc \
  --dest-namespace default

You should see application 'guestbook' created. Notice what has not happened: nothing is deployed yet. You have only told Argo CD about the application. Ask it what it sees:

argocd app get guestbook

Name:               argocd/guestbook
Project:            default
Server:             https://kubernetes.default.svc
Namespace:          default
URL:                https://localhost:8080/applications/guestbook
Source:
- Repo:             https://github.com/argoproj/argocd-example-apps.git
  Target:
  Path:             guestbook
SyncWindow:         Sync Allowed
Sync Policy:        Manual
Sync Status:        OutOfSync from  (53e28ff)
Health Status:      Missing

GROUP  KIND        NAMESPACE  NAME          STATUS     HEALTH   HOOK  MESSAGE
       Service     default    guestbook-ui  OutOfSync  Missing
apps   Deployment  default    guestbook-ui  OutOfSync  Missing

Read that output line by line, because this is the screen you will read most. Sync Policy: Manual means Argo CD will not change anything until told to. Sync Status: OutOfSync means Git says there should be a Service and a Deployment and the cluster has neither. Health Status: Missing means the resources do not exist. The table at the bottom lists each resource Argo CD found in the repository, and each is OutOfSync and Missing for the same reason. This is exactly what "desired but not live" looks like. The commit hash in brackets is the revision Argo CD compared against.

Before syncing, ask for the diff, to see what would change:

argocd app diff guestbook

Because nothing exists yet, the output shows both objects as additions. The command's exit code carries information too: 0 means no difference, 1 means a difference was found, and 2 means an error. Scripts rely on that.

Now sync:

argocd app sync guestbook

The output streams each resource as Argo CD applies it, ending with a table whose rows say Synced and Healthy. Check the result in two places:

argocd app get guestbook
kubectl get deploy,svc,pods -n default

The status is now Synced and Healthy, and Kubernetes shows a guestbook-ui Deployment, Service and Pod that you never applied yourself. Open the web interface and click the guestbook tile: you get the resource tree, a graph from the Application down to the Deployment, ReplicaSet and Pod, with green hearts for healthy nodes. That picture is the reason many teams adopt Argo CD.

To see the application itself in a browser, forward its service:

kubectl port-forward svc/guestbook-ui -n default 8081:80

Then visit http://localhost:8081.

Reading the UI: sync status, health and the resource tree

The command line and the web interface show the same facts, and you should be comfortable reading both. The applications page shows one tile per application with two badges: the sync status and the health status. A green check means Synced, a yellow circle with an arrow means OutOfSync, a green heart means Healthy, a blue spinning circle means Progressing, and a broken red heart means Degraded.

Click into an application and you get the resource tree. On the left is the Application; to its right are the Kubernetes objects it manages, and to their right the objects those created. A Deployment owns a ReplicaSet, which owns Pods. Every node has its own small sync and health icons, so when an application is unhealthy you follow the red icons down the tree to the single pod or object that is the cause. That is usually faster than reading a dozen kubectl describe outputs.

Three buttons at the top of the application page do the everyday work. Refresh re-compares Git and the cluster (a normal refresh; the dropdown offers a hard refresh). Sync opens a panel where you choose what to sync and with which options, then applies. History and Rollback lists previous syncs, which you will use shortly.

Click any resource node and a panel slides out with tabs: Summary, Events, Logs (for pods) and Diff. The Diff tab deserves special attention. It shows the live object beside the desired one, highlighting what differs. When an application says OutOfSync and you do not know why, the diff tab answers the question, and it is also how you learn which fields the cluster adds to your objects (defaults, annotations) that can cause confusing permanent differences.

There is also a Details panel for the application itself, showing its source, destination and sync policy, and a Parameters tab for Helm or Kustomize settings. You can edit the Application from there, and Argo CD will store the change, but keep in mind the declarative approach in the next section: edits made in the UI live only in the cluster, not in Git.

One last reading skill: the sync status colours can disagree with what you expect for a moment after a change. Git is polled, the live cluster is watched, and there is a small cache in between. If a status looks stale, press Refresh before concluding anything.

An Application as a YAML file

You created the guestbook with a command, which is fine for exploring. But the command only stored the Application inside the cluster. If you deleted the cluster, the record of what you had deployed would be gone. The GitOps answer is to write the Application itself as a file and keep it in Git too.

Remember that an Application is a Kubernetes custom resource. Here is the guestbook as a manifest:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: guestbook
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: https://github.com/argoproj/argocd-example-apps.git
    targetRevision: HEAD
    path: guestbook
  destination:
    server: https://kubernetes.default.svc
    namespace: guestbook
  syncPolicy:
    syncOptions:
      - CreateNamespace=true

Each part matters, so go through it slowly.

The metadata.namespace is argocd, the namespace where Argo CD is installed. This catches people out. The Application object itself lives in Argo CD's namespace, while the things it deploys go to spec.destination.namespace. They are different namespaces with different purposes. (Later versions allow Applications in other namespaces, but only when an administrator enables it.)

The finalizer, resources-finalizer.argocd.argoproj.io, controls what happens on deletion. With it, deleting the Application also deletes every resource it deployed (a cascading delete). Without it, deleting the Application object leaves the deployed resources running in the cluster, orphaned. Neither is wrong, but you should choose deliberately; the deletion section returns to this.

spec.project names the project. default is the permissive one that exists from the start.

spec.source says where the files are. repoURL is the Git repository. targetRevision is which version to track: a branch name, a tag, a commit hash, or HEAD (the default branch's latest commit). path is the folder inside the repository. If you deploy a Helm chart from a chart repository instead of a folder in Git, you write chart: <name> in place of path, and targetRevision becomes the chart version.

spec.destination says where the result goes. server is the cluster's API address and namespace is the target namespace. You can use name: in-cluster instead of server, but never both.

spec.syncPolicy.syncOptions holds per-application tweaks. CreateNamespace=true tells Argo CD to create the destination namespace if it does not exist. Without it, syncing into a missing namespace fails with a "namespace not found" error. That is a very common first failure, and this option is the fix.

Apply the file with kubectl, exactly as you would any other Kubernetes object:

kubectl apply -f guestbook-app.yaml

If the CLI-created guestbook still exists, this updates it (the names match). Kubernetes stores it, Argo CD notices it within seconds, and the Application appears in the UI. Now the Application definition is a file you can review, version and recreate.

The CLI can show you the YAML for an existing Application too, which is a handy way to learn the format:

argocd app get guestbook -o yaml

kubectl get application guestbook -n argocd -o yaml

The second command proves the point made earlier: an Application is just a Kubernetes object, and kubectl can read it.

Deploying your own repository

The guestbook is someone else's repository. The real value appears when Argo CD watches yours. The workflow from here is the one you will use at work: put manifests in a Git repository, point an Application at them, then change the deployed state only by committing.

Create a new public GitHub repository (public keeps the credentials question out of the way for now; the private case is coming) with a folder app/ containing two files.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello
  labels:
    app: hello
spec:
  replicas: 2
  selector:
    matchLabels:
      app: hello
  template:
    metadata:
      labels:
        app: hello
    spec:
      containers:
        - name: hello
          image: nginx:1.27
          ports:
            - containerPort: 80

apiVersion: v1
kind: Service
metadata:
  name: hello
spec:
  selector:
    app: hello
  ports:
    - port: 80
      targetPort: 80

Commit and push them. Then create the Application, replacing the URL with your repository's:

argocd app create hello \
  --repo https://github.com/YOUR-USER/YOUR-REPO.git \
  --path app \
  --dest-server https://kubernetes.default.svc \
  --dest-namespace hello \
  --sync-option CreateNamespace=true
argocd app sync hello

Argo CD clones your repository (through the repo server), sees plain YAML files in app/, and treats the directory as its source. You now have two pods running from files in your repository.

Now make a change the GitOps way. Edit replicas: 2 to replicas: 3 in deployment.yaml, commit and push. Do not touch the cluster. Watch what Argo CD does:

argocd app get hello --refresh

The --refresh flag asks Argo CD to re-check Git right now instead of waiting for the polling interval. The status becomes OutOfSync, because Git says three replicas and the cluster has two. Look at the difference, then apply it:

argocd app diff hello
argocd app sync hello

The diff shows replicas: 2 becoming replicas: 3. After the sync, kubectl get pods -n hello lists three pods. Nothing touched the cluster except Argo CD, and the record of why there are three is a commit with an author and a message. That is the GitOps loop in its smallest form.

Argo CD is not limited to plain YAML folders. If the path contains a kustomization.yaml, it runs Kustomize; if it contains a Chart.yaml, it runs Helm; if you point at a Helm chart repository with --helm-chart, it pulls that chart. The bundled tools in v3.5 are Helm 4 and Kustomize 5. You still write the application definition the same way; Argo CD figures out the tool.

argocd app create nginx-chart \
  --repo https://charts.bitnami.com/bitnami \
  --helm-chart nginx \
  --revision 18.2.0 \
  --dest-name in-cluster \
  --dest-namespace web \
  --sync-option CreateNamespace=true

Treat the repository address and chart version above as placeholders for a chart you actually use; chart repositories change over time and publishers sometimes retire versions, so check a current version before relying on one. The important shape is that --helm-chart replaces --path, --revision is the chart version, and --dest-name in-cluster uses the cluster's registered name in place of its address. See the Helm guide for writing charts.

Automated sync: prune and self-heal

So far every change needed a manual sync. That is the safest starting point, but it is also the thing GitOps tries to remove: a human pressing a button is one more step that can be skipped or forgotten. Automated sync tells Argo CD to apply differences by itself.

argocd app set hello --sync-policy automated

Or in the YAML:

spec:
  syncPolicy:
    automated: {}

With that set, pushing a commit leads, within the polling interval, to Argo CD syncing the change without anyone pressing anything. Two further switches refine the behaviour, and both are off by default even after automation is on. The defaults are cautious because both can be destructive.

Prune controls what happens to things that disappear from Git. Suppose you delete service.yaml from the repository. By default, even with automation, Argo CD will not delete the live Service; it will mark the application OutOfSync and leave the object running. The reasoning is that deleting things is riskier than creating them, so you must opt in. With prune on, removing a file from Git removes the resource from the cluster:

argocd app set hello --auto-prune

Self-heal controls what happens when the live state changes without a Git change. Without it, if someone runs kubectl scale deployment hello --replicas=10, the application becomes OutOfSync and stays that way, waiting for a human. With self-heal, Argo CD notices the drift and reverts it:

argocd app set hello --self-heal

The two together are the usual configuration for a fully GitOps application:

spec:
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Watch self-heal work. With it enabled, scale the deployment by hand and look immediately afterwards:

kubectl scale deployment hello -n hello --replicas=10
kubectl get pods -n hello -w

For a moment there are ten pods, then Argo CD notices and scales back to what Git says, terminating the extras. The manual change is gone without a trace except in events. This is the feature that surprises people during incidents: an engineer hand-edits a production object to fix an outage and watches the fix vanish. The lesson is not to turn self-heal off; it is that the emergency change has to go into Git (or be made by pausing automation first), which is exactly the discipline GitOps exists to enforce.

There is one more rule worth memorising now. Rollback is not allowed while automated sync is on, because the automation would immediately undo the rollback by re-syncing to Git. To roll back an automated application you revert the commit in Git, which is the GitOps way anyway, or temporarily disable automation. The rollback section shows both.

Automated sync can also retry failures. A sync that fails on a transient error (an API timeout, say) can be retried with backoff:

spec:
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m

limit is the number of attempts (-1 means retry forever, which is rarely wise), and the backoff block makes each wait longer than the previous one, so a struggling cluster is not hammered.

The everyday commands, grouped by what you are trying to do

You now know enough to use most of the argocd app commands. Grouped by intent, rather than alphabetically, this is the set to keep at hand.

See what exists. List applications and inspect one:

argocd app list
argocd app list -l team=payments
argocd app get hello
argocd app get hello -o tree
argocd app get hello --refresh
argocd app get hello --hard-refresh

-o tree prints the resource tree in the terminal, -o wide adds columns, and -o yaml or -o json gives machine-readable output. -l key=value filters by label. --refresh asks Argo CD to re-compare now; --hard-refresh also clears its cached manifests.

See what would change. Before applying, always be able to answer "what will this do?":

argocd app diff hello
argocd app manifests hello
argocd app sync hello --dry-run

app diff compares live and target. app manifests prints the rendered YAML Argo CD derived from Git, which is how you confirm what a Helm chart actually produced. --dry-run previews a sync without applying it.

Apply changes. Sync everything, or just part of an application:

argocd app sync hello
argocd app sync hello --prune
argocd app sync hello --resource apps:Deployment:hello
argocd app sync hello --revision v1.2.0
argocd app sync hello --async

--resource GROUP:KIND:NAME syncs a single object (an empty group, for core kinds, is written :Service:hello). --revision syncs a specific branch, tag or commit instead of the tracked one, which is useful for testing a branch. --async returns immediately instead of waiting for the sync to finish.

Wait for a result. Scripts need a blocking command:

argocd app wait hello --health --sync --timeout 300

It returns once the application is healthy and synced, or fails at the timeout. CI jobs that test a deployment use it.

Read history and logs:

argocd app history hello
argocd app logs hello --kind Deployment --name hello
argocd app logs hello --kind Deployment --name hello -f

-f follows the log stream, like tail -f.

Change settings on an existing application:

argocd app set hello --sync-policy automated
argocd app set hello --sync-policy none
argocd app set hello --revision main
argocd app set hello -p image.tag=v2

The last form overrides a Helm parameter directly. Careful: settings changed with app set are stored in the Application object in the cluster, so if the Application is itself defined in a file in Git, the file and the cluster now disagree. Prefer editing the file.

Remove an application:

argocd app delete hello
argocd app delete hello --cascade=false

The default deletion cascades and removes the deployed resources; --cascade=false deletes only the Application record and leaves the resources running.

A few global flags apply to every command. --server names the server if you are not using your saved login. --grpc-web helps when a proxy in front of Argo CD does not pass gRPC traffic. --insecure skips certificate checks (never in production). -o picks the output format. Tab completion makes all of this easier, and the CLI generates it for your shell:

argocd completion zsh > ~/.argocd-completion.zsh

Replace zsh with bash, fish or powershell, and source the file from your shell's startup file.

Sync history and rolling back

Every successful sync is recorded. The history shows which Git revision was deployed, when, and by whom:

argocd app history hello

ID  DATE                           REVISION
0   2026-10-01 09:41:12 +0300 +03  a1b2c3d
1   2026-10-01 10:05:47 +0300 +03  e4f5a6b
2   2026-10-01 10:30:03 +0300 +03  9c8d7e6

Each row is a numbered deployment. To go back to an earlier one, give its ID:

argocd app rollback hello 1

This re-deploys the manifests that revision produced. There is an important catch, which is the rule from the automated sync section: the command is refused if automated sync is enabled. The rollback would change the cluster away from Git, and automation would immediately change it back. The error reads along the lines of "rollback cannot be initiated when auto-sync is enabled".

It gets more subtle after that. Even when a rollback works on a manual application, it only moves the cluster; Git still contains the bad commit. The application will show OutOfSync because it no longer matches the tracked branch. A rollback is therefore a temporary measure that buys time during an incident, and it is not a fix. The real fix, and the fix for automated applications, is to revert the bad commit in Git:

git revert HEAD
git push

Argo CD then syncs the reverted state like any other change. Because the change is in Git, the history stays honest: the repository records that the change was made and undone, and who did each.

This is a good moment to appreciate what GitOps gives you over kubectl. With hand-applied changes, "what was running yesterday?" has no reliable answer. Here the answer is a revision hash in the history, and a git show of that hash.

Connecting a private repository

Everything so far used public repositories. Real repositories are private, and Argo CD needs credentials to read them. A missing or wrong credential produces an error that looks alarming to beginners:

rpc error: code = Unknown desc = ... authentication required

or in the UI, a repository not found warning on the application. The fix is to register the repository with Argo CD along with a credential.

The simplest credential for HTTPS is a personal access token. On GitHub, create a fine-grained token with read-only access to the contents of just that repository, then register it:

argocd repo add https://github.com/YOUR-ORG/private-repo.git \
  --username your-github-user \
  --password ghp_yourTokenHere

For SSH you use a private key instead:

argocd repo add git@github.com:YOUR-ORG/private-repo.git \
  --ssh-private-key-path ~/.ssh/id_ed25519

Check that it connected:

argocd repo list

The STATUS column should say Successful. If it says Failed, the message beside it is the cause, usually a wrong token, insufficient token scope or a typo in the URL.

What argocd repo add actually does is worth understanding: it stores the credential as a Kubernetes Secret in the argocd namespace, labelled argocd.argoproj.io/secret-type: repository. So you can also do it declaratively, and it is how a GitOps setup keeps repository definitions in code (taking care never to commit a real token; secrets management is a mid-level topic).

apiVersion: v1
kind: Secret
metadata:
  name: private-repo
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repository
stringData:
  type: git
  url: https://github.com/YOUR-ORG/private-repo.git
  username: your-github-user
  password: REPLACE_WITH_TOKEN

Never commit that file with a real token in it. Keep the secret out of Git, or use one of the secret-management approaches covered later in this series. A token pasted into a repository is a leaked token.

Two more points. First, SSH connections are checked against known host keys. Since Argo CD 3.5, a repository without configured credentials also validates the host key against the argocd-ssh-known-hosts-cm ConfigMap, so a knownhosts: key mismatch error means the server's key is not registered; add it with argocd cert add-ssh --batch. Second, for Helm or OCI registries, add --type helm or --type oci to argocd repo add, plus a --name.

Projects: the rules an application plays by

Every Application belongs to an AppProject. Until now everything used default, which allows any repository, any cluster and any namespace. That is convenient for learning and dangerous for sharing, so in a team setting each group gets its own project with restrictions. As a beginner you will mostly meet projects through their error messages, so this section teaches you to read them.

A project can restrict four main things: which source repositories its applications may use (sourceRepos), which destinations they may deploy to (destinations, pairs of cluster and namespace), which cluster-wide kinds of object they may create (cluster-scoped resources such as Namespaces are denied unless whitelisted), and which namespaced kinds are blocked.

Here is a small project:

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: team-a
  namespace: argocd
spec:
  description: Applications owned by team A
  sourceRepos:
    - https://github.com/YOUR-ORG/team-a-*
  destinations:
    - server: https://kubernetes.default.svc
      namespace: team-a-*
  clusterResourceWhitelist: []

Apply it with kubectl apply -f team-a-project.yaml, then create an Application with --project team-a. If the application strays outside the rules, Argo CD refuses with a message that names the exact rule:

application repo https://github.com/someone/else.git is not permitted in project 'team-a'

application destination server 'https://kubernetes.default.svc' and namespace 'prod' do not match any of the allowed destinations in project 'team-a'

resource :Namespace is not permitted in project team-a

The first means the repository is not in sourceRepos. The second means the destination is not in destinations. The third means the repository contains a cluster-scoped object (here a Namespace) that the project does not allow. Each has a matching fix, done either by editing the project file or with argocd proj add-source, argocd proj add-destination or argocd proj allow-cluster-resource. The key reading skill is that these messages are not bugs; they are the project doing its job, and the answer is to decide whether the rule or the application is wrong.

A related surprise: the CreateNamespace=true option creates a Namespace, but a project that blocks cluster-scoped kinds can still allow it because the option is handled by Argo CD rather than by a manifest in the repository. If instead you commit a Namespace object into Git inside a restricted project, you will meet the third error above.

Projects also carry roles, sync windows and more, which the Mid-level guide covers. For now remember that default is unrestricted, that you cannot delete it, and that a platform team usually locks it down.

Deleting applications safely

Deleting is where the finalizer from earlier earns its place. There are two separate things you might want to delete: the Application (Argo CD's record) and the resources it deployed (your Deployments and Services). Which happens depends on a setting.

With the finalizer resources-finalizer.argocd.argoproj.io on the Application, deletion is cascading: Argo CD deletes the deployed resources first, and then removes the Application. This is what people usually expect, and what argocd app delete does by default on an application created through the CLI.

Without the finalizer, deleting the Application object (for example with kubectl delete application hello -n argocd) removes only the record. Your resources keep running, and nothing manages them any more. That can be exactly what you want when you are handing resources over, and it can be a surprise that leaves forgotten workloads running and costing money.

argocd app delete hello
argocd app delete hello --cascade=false

Two rules keep you safe. First, put the finalizer in every Application you write as a file, unless you have a reason not to. Second, never delete a production application without first reading what is in its resource tree: cascading delete removes whatever the tree shows, which can include persistent volume claims.

Deleting an application from a parent "app of apps" (an advanced pattern where one Application creates others) cascades too, which can take out many applications at once. That is one of the reasons the mid-level guide spends time on prune protection.

Also remember that with automated sync and prune on, deleting a file from Git is itself a deletion. The destructive power is the same; it just arrives through a commit.

How Argo CD notices changes: polling, refresh and webhooks

Newcomers often ask why Argo CD did not react the instant they pushed. The answer is the polling behaviour introduced earlier, and knowing the options makes the tool feel predictable.

By default, Argo CD re-checks each application's Git repository every 120 seconds plus up to 60 seconds of jitter, so "about three minutes" at the outside. The interval is the timeout.reconciliation setting in the argocd-cm ConfigMap, with timeout.reconciliation.jitter controlling the random spread. Setting the interval to 0 disables polling and is not recommended.

You have three ways to bring that delay down.

Manual refresh. Clicking Refresh in the UI, or running argocd app get NAME --refresh, makes Argo CD check Git immediately for that application. Use --hard-refresh when you suspect a stale cache: it also discards the repo server's saved rendering.

Webhooks. Your Git host can notify Argo CD when a push happens, and Argo CD refreshes the affected applications straight away. The receiver is at /api/webhook on your Argo CD address, and it supports GitHub, GitLab, Bitbucket, Bitbucket Server, Gogs and Azure DevOps (and, since 3.5, GitHub Container Registry). You add the webhook in your Git host pointing at https://YOUR-ARGOCD/api/webhook and, if you set a shared secret, store the same secret in Argo CD's argocd-secret. This is an administrator task that needs a reachable address, so a laptop cluster behind a port-forward generally cannot receive webhooks.

Annotation. You can trigger a refresh of an Application by annotating it, which is how automation does it:

kubectl annotate application hello -n argocd argocd.argoproj.io/refresh=normal --overwrite

Use hard instead of normal for a hard refresh.

It helps to keep straight that refresh (compare) and sync (apply) are different things. A refresh only updates the status. If automated sync is off, a refresh will show OutOfSync and change nothing in the cluster. If automated sync is on, the detected difference is then applied by the controller without you pressing anything.

# argocd-cm excerpt (administrator setting)
data:
  timeout.reconciliation: 180s
  timeout.reconciliation.jitter: 60s

Changing these values is an operations matter, and lowering the interval raises load on both Argo CD and your Git host, so prefer webhooks over aggressive polling.

Configuration you will meet as a beginner

Argo CD is configured through Kubernetes ConfigMaps and Secrets in its own namespace, each with a fixed name. You do not need to edit them yet, but you should know they exist and what lives where, because documentation and error messages refer to them constantly.

Two practical habits. The first: after editing argocd-cmd-params-cm, restart the affected component, because those values are read at start-up. For example, after changing the server's settings:

kubectl rollout restart deployment argocd-server -n argocd

The second: a few settings are used so often that beginners hit them early. url in argocd-cm is required for single sign-on and for links in notifications. Setting server.insecure: "true" in argocd-cmd-params-cm makes the API server serve plain HTTP, which is what you want when TLS is terminated by an ingress in front of it (and never when it is exposed directly). admin.enabled: "false" in argocd-cm switches off the built-in admin account once real accounts exist.

You can check a configuration without applying it, using the administrator command:

argocd admin settings validate -n argocd

It reads the settings from the cluster and reports problems. It needs direct access to Kubernetes through your kubeconfig, which a cluster administrator has.

Finally, you can read what is deployed from any ConfigMap with kubectl get cm -n argocd argocd-cm -o yaml. Get in the habit of looking at real configuration rather than trusting a tutorial's description; since 3.0 several defaults changed (for instance, Argo CD now tracks the resources it owns with an annotation named argocd.argoproj.io/tracking-id instead of a label, and logs require an explicit permission), and older articles describe the 2.x behaviour.

Common errors, and how to read them

Most Argo CD failures fall into a handful of families. The skill is to read the message, decide which component produced it, and look in the right place. Here are the ones beginners meet, with what each really means.

The CustomResourceDefinition "applicationsets.argoproj.io" is invalid: metadata.annotations: Too long. You installed or upgraded with a plain kubectl apply. Re-run with --server-side --force-conflicts.

x509: certificate signed by unknown authority (from the CLI). The server uses its default self-signed certificate. For a local experiment add --insecure; for anything real, install a proper certificate.

rpc error: code = Unavailable desc = transport is closing. A proxy or ingress between you and the server does not carry gRPC. Add --grpc-web to the CLI command.

repository not found, or authentication errors on an application. The credential is missing, wrong, or scoped too narrowly, or the URL has a typo. Register the repository with argocd repo add and check argocd repo list for Successful.

app path does not exist. The path in the Application is wrong for that revision. Check the folder name and the targetRevision; a branch that does not contain the folder produces this message, and so does a capital-letter mismatch.

unable to resolve '<revision>' to a commit SHA. The branch or tag in targetRevision does not exist or is not readable. Fix the name, or the credentials.

Manifest generation error (cached). The repo server tried to render your manifests, failed, and cached the failure so it does not retry constantly. Read the repo server logs for the real cause, fix the source, and then run argocd app get NAME --hard-refresh to clear the cached error.

application repo ... is not permitted in project, application destination ... do not match any of the allowed destinations, and resource ... is not permitted in project. The project's rules are refusing the application, as described in the projects section.

there are no clusters with this name. A destination.name does not match any registered cluster. Run argocd cluster list and copy the name exactly, or use server.

existing application spec is different, use upsert flag to force update. You ran argocd app create for a name that exists with different settings. Use --upsert if you mean to overwrite, or use argocd app set.

Sync fails with namespace ... not found. The destination namespace does not exist. Add CreateNamespace=true to the sync options.

The application is permanently OutOfSync right after a successful sync. Something in the cluster keeps changing a field that your manifest also sets: a mutating webhook, a controller that adds defaults, or another tool. Open the Diff tab on the resource to see which field differs. The fix, usually an ignoreDifferences rule, is covered at the Mid-level; as a beginner your job is to find the field.

The application is stuck in Progressing. Argo CD is waiting for a health check that never finishes. A Service of type LoadBalancer or an Ingress with no address yet is a common cause on local clusters that cannot allocate one. Read the resource's message in the tree.

You cannot see the Logs tab. Since Argo CD 3.0, viewing logs needs its own explicit permission (logs, get). That is an RBAC matter, and it confuses people who used 2.x.

Sync operation blocked by sync window. An administrator has defined a time window that denies syncs right now. Wait, or ask whether manual syncs are permitted during the window.

A forgotten admin password. The initial secret may already be deleted. The administrator reset procedure generates a bcrypt hash with argocd account bcrypt --password NEW and stores it in argocd-secret; do that only on a cluster you own, and follow the official documentation for the exact keys.

A general-purpose debugging order, which works for almost everything, goes like this. First argocd app get NAME for the statuses and the message. Second, argocd app diff NAME for the difference. Third, the repo server logs if the problem smells like rendering, Git or credentials, and the controller logs if it smells like comparing or applying. Fourth, kubectl describe on the failing resource, because Kubernetes events often explain a stuck pod better than Argo CD can. Finally, try a hard refresh before concluding that a problem is real, since a stale cache explains a surprising number of ghosts.

argocd app get hello
argocd app diff hello
kubectl logs -n argocd deploy/argocd-repo-server --tail=50
kubectl logs -n argocd statefulset/argocd-application-controller --tail=50
kubectl describe deploy hello -n hello
argocd app get hello --hard-refresh

Putting it all together

Here is one small end-to-end project that uses everything above. You will run a web application whose configuration lives in Git, with Argo CD deploying and healing it, and you will change it only by committing.

Step 1: the repository. Create a public repository argo-demo with this layout:

argo-demo/
  apps/
    hello/
      deployment.yaml
      service.yaml
  argocd/
    hello-app.yaml

The apps/hello files are the Deployment and Service from earlier (two replicas of nginx:1.27). The argocd/hello-app.yaml file is the Application definition itself, stored next to what it deploys:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: hello
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: https://github.com/YOUR-USER/argo-demo.git
    targetRevision: main
    path: apps/hello
  destination:
    server: https://kubernetes.default.svc
    namespace: hello
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Step 2: bootstrap once. The Application is the only thing you apply by hand, and only once. Everything afterwards flows from Git:

kubectl apply -f argocd/hello-app.yaml
argocd app wait hello --health --sync --timeout 180
kubectl get pods -n hello

Within a minute the application is Synced and Healthy with two pods. You have not run a sync command; automation did it.

Step 3: change through Git. Edit deployment.yaml to set replicas: 3 and the image to nginx:1.27.1. Commit with a meaningful message, push, and watch:

git commit -am "Scale hello to 3 and bump nginx" && git push
argocd app get hello --refresh
argocd app history hello

The application rolls forward, and the history gains an entry pointing at your commit.

Step 4: prove drift is corrected. Hand-edit the live object, then watch it revert:

kubectl scale deployment hello -n hello --replicas=1
sleep 15
kubectl get deploy hello -n hello

Self-heal restores three replicas.

Step 5: break it, then recover through Git. Push an image tag that does not exist. The application goes Degraded (pods in ImagePullBackOff), while the status stays Synced: Git is wrong, and the cluster faithfully matches it. Fix it with:

git revert HEAD --no-edit && git push

Argo CD syncs the revert and health returns.

Step 6: prune. Delete service.yaml from Git, push, and confirm that kubectl get svc -n hello no longer lists it, because prune is on. Restore it with another commit.

Step 7: clean up.

argocd app delete hello
kind delete cluster --name argo-demo

The finalizer removes the deployed resources before the Application goes away.

You have now done the full cycle: bootstrap, deploy, change, heal, break, recover, prune and delete. Notice that after step 2, the only write path into the cluster was a Git commit. That constraint is the point.

What you can now do, and what comes next

If you worked through the Try it tasks, you can now do the following, and can explain it to someone else:

• Explain why pull-based delivery removes the need for cluster credentials in CI, and what GitOps means.
• Name the core nouns (Application, Project, sync status, health, target and live state) and the three components (server, repo server, controller).
• Install Argo CD with the current server-side command, install the CLI, log in, and change the initial password.
• Create an Application from the CLI and from YAML, and read its status, diff and resource tree.
• Deploy your own repository, and change a cluster only by committing.
• Choose between manual and automated sync, and explain prune and self-heal.
• Roll back an incident correctly, and know why a revert in Git is the real fix.
• Connect a private repository, read project errors, and delete applications without losing data by accident.
• Diagnose the common errors by reading the message and choosing the right component to inspect.

The Mid-level guide picks up with how Argo CD decides what is different (diffing and resource tracking), projects and role-based access in depth, single sign-on, ApplicationSets that generate many applications from a template, the app-of-apps pattern, sync waves and hooks for ordering, sync windows, notifications, and the integration with CI pipelines. The Senior guide covers running Argo CD as a platform: high availability, scaling, multi-cluster management, security and upgrade strategy.

For neighbouring topics in this catalogue, Kubernetes is the foundation everything here sits on, Helm and Kustomize are the tools you will most often point Argo CD at, Terraform creates the clusters Argo CD runs in, and Flux is the alternative GitOps controller worth comparing once you are comfortable here.

One regional note for readers deploying in the Gulf and Egypt. Data-residency rules often dictate which cloud region your clusters run in. Because Argo CD pulls from Git and needs no inbound credentials in CI, the control plane can sit in the same in-country region as your workloads, and only Git traffic crosses the boundary. Decide where your repositories are hosted with the same care as where your clusters are, since the repository contents (including any configuration values) are data too.

Sources

• Argo CD documentation (stable)
• Getting started
• CLI installation
• Core concepts
• Understand the basics
• Architecture
• Installation manifests
• Declarative setup
• Automated sync policy
• Projects
• Private repositories
• Webhooks
• Upgrade guides
• Argo CD releases on GitHub
• Example applications repository

Part one of three. A twenty-five-minute review of the Beginner Detailed track, then the questions interviewers ask about Argo CD. Mid-level reviews this plus its own material; Senior reviews all three. Everything here is written for Argo CD v3.5, and where older answers from the 2.x era are now wrong, the text says so.

The thirty-second answer

Argo CD is a GitOps continuous-delivery controller that runs inside a Kubernetes cluster. You describe the desired state in a Git repository; Argo CD pulls it, renders it with Helm, Kustomize or plain YAML, compares it with the live cluster, reports any difference, and optionally syncs the cluster to match. Because it pulls from inside the cluster, your CI system needs no cluster credentials.

Then the line that shows you have used it: "Synced means the cluster matches Git, and Healthy means it is actually working, and they are independent, so an app can be Synced and Degraded at the same time."

The model

Two sentences to be able to say without hesitation. First, "the Application does not contain the manifests; it points at them." Second, "Argo CD stores all its state as Kubernetes objects, so Redis is only a disposable cache."

Pull versus push

The most common opening question is how this differs from a normal pipeline deployment.

A strong answer adds the trade-off honestly: pull-based delivery needs a separate step to update the Git repository (usually the pipeline commits the new image tag), and feedback is slightly slower because of polling.

The components

Be ready to name what runs after installation. The API server (argocd-server) serves the web UI and the API the CLI uses, and handles authentication and permissions. The repo server (argocd-repo-server) clones Git and renders manifests, and notably has no Kubernetes privileges. The application controller (argocd-application-controller, a StatefulSet) compares live and target state and performs syncs. Supporting pieces are the ApplicationSet controller, the notifications controller, Dex for single sign-on, and Redis as a cache.

The follow-up that separates people who have read about it from people who have debugged it: "Where would you look if an app shows a manifest error?" The answer is the repo server logs, because rendering happens there. "If it will not sync?" The controller logs.

The commands to have ready

# install (server-side apply is required since 3.3)
kubectl create namespace argocd
kubectl apply -n argocd --server-side --force-conflicts \
  -f https://raw.githubusercontent.com/argoproj/argo-cd/v3.5.3/manifests/install.yaml

# first login
argocd admin initial-password -n argocd
argocd login localhost:8080 --username admin --insecure
argocd account update-password

# create and inspect
argocd app create guestbook \
  --repo https://github.com/argoproj/argocd-example-apps.git \
  --path guestbook --dest-server https://kubernetes.default.svc --dest-namespace default
argocd app list
argocd app get guestbook --refresh
argocd app diff guestbook

# change
argocd app sync guestbook --prune --dry-run
argocd app sync guestbook
argocd app set guestbook --sync-policy automated --auto-prune --self-heal
argocd app wait guestbook --health --sync --timeout 300

# undo and remove
argocd app history guestbook
argocd app rollback guestbook 1
argocd app delete guestbook

If you can explain what each line does and why --server-side is there, you have covered most practical questions.

Installation, and the flag every old tutorial misses

Expect a question about installing. The current command uses kubectl apply --server-side --force-conflicts. The reason is that since Argo CD 3.3 the ApplicationSet definition exceeds the 262,144-byte limit that client-side apply imposes on the last-applied annotation, and a plain apply fails with metadata.annotations: Too long: may not be more than 262144 bytes. Server-side apply stores that bookkeeping elsewhere. Mention pinning a version tag instead of stable, and you show production instinct.

Know the manifest variants at headline level: install.yaml (standard), ha/install.yaml (high availability, at least three nodes), a namespace-scoped install that deploys only to external clusters, and core-install.yaml (headless, no UI or API server). A community-maintained Helm chart also exists; say "community-maintained" if you mention it.

For access, the first-login path is: the initial admin password lives in the argocd-initial-admin-secret Secret (read it with argocd admin initial-password), you change it with argocd account update-password, and you then delete the secret. A default install uses a self-signed certificate, hence the CLI's --insecure for local testing only.

An Application, in YAML

You may be asked to write one on a whiteboard. Practise this until it is automatic:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: hello
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: https://github.com/YOUR-USER/argo-demo.git
    targetRevision: main
    path: apps/hello
  destination:
    server: https://kubernetes.default.svc
    namespace: hello
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Points worth saying aloud while writing. The Application lives in the argocd namespace, while the workload goes to destination.namespace. destination takes server or name, never both. A Helm chart repository uses chart: in place of path:. The finalizer makes deletion cascade to the deployed resources.

Manual versus automated sync

The default sync policy is manual: Argo CD reports OutOfSync and waits. Automated sync applies differences by itself. Two sub-options are off by default and this is a classic question:

• Prune is off by default, so removing a file from Git does not remove the live object unless you enable it.
• Self-heal is off by default, so a manual change in the cluster shows as OutOfSync but is not reverted unless you enable it.

Explain why they are off: deletion and overwriting human changes are riskier than creation. Then explain the combination used in real GitOps setups: automated with prune: true and selfHeal: true.

A trap follows naturally. You cannot roll back while automated sync is on, because automation would undo it. The correct answer to "how do you roll back an automated app?" is "revert the commit in Git", with the honest addition that the quick argocd app rollback only moves the cluster, leaves Git unchanged, and so only buys time during an incident.

Status versus health

This is asked in nearly every interview because it tests understanding rather than memory. Sync status answers "does the cluster match Git?" with three values. Health answers "is it working?" with six: Healthy, Progressing, Degraded, Suspended, Missing, Unknown. Give both examples without being asked: Synced and Degraded (Git describes a broken thing, such as an image tag that does not exist), and Healthy and OutOfSync (everything runs, but someone edited it by hand).

The operation phase is a third, separate notion: whether the last sync attempt Succeeded, Failed or Errored.

Polling, refresh and webhooks

Argo CD polls Git about every three minutes by default (120 seconds plus up to 60 seconds of jitter, set by timeout.reconciliation and timeout.reconciliation.jitter in argocd-cm). So a push does not apply instantly. Three ways to shorten that: a manual refresh, a webhook to /api/webhook from the Git host, or annotating the Application with argocd.argoproj.io/refresh. Distinguish refresh from sync: a refresh only re-compares, it does not apply anything unless automated sync is enabled.

Projects and the errors they produce

A project restricts source repositories, destinations and resource kinds. The default project is unrestricted and cannot be deleted. Be able to map each message to a fix:

The traps, and why they share one cause

Almost every beginner failure comes from forgetting that Argo CD reconciles the cluster toward Git, and that Git is the only input it trusts.

Note also what changed in the 3.0 major release, because interviewers who know the tool like to test it. Resource tracking switched from the label app.kubernetes.io/instance to an annotation, argocd.argoproj.io/tracking-id. Viewing logs needs the explicit logs, get permission. Fine-grained permissions no longer inherit from the application to its child resources. The password for the initial admin is in argocd-initial-admin-secret, not the pod name.

Debugging order

When an application misbehaves, give an ordered answer, since interviewers listen for method. First argocd app get NAME for the statuses and the message. Second argocd app diff NAME to see the difference. Third, the repo server logs for rendering, Git and credential problems, or the controller logs for comparing and applying problems. Fourth, kubectl describe on the failing resource for events. Fifth, a hard refresh to rule out a stale cache, since a render failure is cached and shows as Manifest generation error (cached).

Common interview questions

What problem does Argo CD solve? It removes the gap between what the repository says and what the cluster runs. Pipelines that push changes need cluster credentials and have no ongoing view; Argo CD pulls from inside the cluster, continuously compares, and gives one place to see whether everything matches Git and works.

What is GitOps? A way of operating where the desired state of a system is stored in Git and software continuously reconciles reality to it. Git supplies history, review, and revert.

Does Argo CD build images? No. CI builds and pushes images. Argo CD only deploys what the repository describes; a pipeline usually commits the new image tag to the repository.

What is an Application? A Kubernetes custom resource pointing at a source in Git and a destination in a cluster. It references manifests rather than containing them.

Difference between OutOfSync and Degraded? OutOfSync is about difference from Git; Degraded is about health. They are independent.

What does sync do? It renders the manifests from the source and applies them so the live state matches the target state.

What are prune and self-heal? Prune deletes live resources removed from Git; self-heal reverts changes made outside Git. Both are off by default with automated sync.

How do you roll back? With manual sync, argocd app rollback NAME ID using the sync history; with automated sync, revert the commit in Git, because rollback is blocked while automation is on. In both cases, Git is the real fix.

Why is the repo server unprivileged? It processes repository content, which could be hostile, so it only reads Git and writes YAML and has no Kubernetes permissions. The controller is the only component that changes the cluster.

Why did installing fail with "Too long"? Client-side apply stores the last applied configuration in an annotation with a size limit, and the ApplicationSet CRD exceeds it since 3.3. Use --server-side --force-conflicts.

How does Argo CD know which resources it owns? By default in 3.x it adds an annotation, argocd.argoproj.io/tracking-id, to each resource it manages. Older versions used a label.

How does Argo CD find out about a push? It polls about every three minutes, or a webhook to /api/webhook triggers an immediate refresh.

What happens if I delete an Application? With the resources-finalizer.argocd.argoproj.io finalizer it cascades and deletes the deployed resources; without it, only the Application record is removed.

How do you connect a private repository? argocd repo add URL with a token or SSH key. It is stored as a Secret labelled argocd.argoproj.io/secret-type: repository. Never commit the real credential.

Can Argo CD deploy Helm charts? Yes. It detects a Chart.yaml or takes --helm-chart from a chart repository and renders with the bundled Helm (Helm 4 in 3.5). It renders the chart rather than running helm install, so the release is not a Helm release you manage with the helm command.

What is a project for? To restrict which repositories, clusters, namespaces and resource kinds a group of applications can use. The default project allows everything.

What is stored in Redis? Only cache. Losing it loses nothing permanent; real state is in Kubernetes objects.

Why might an app stay OutOfSync right after a sync? Something keeps modifying a field your manifest sets, such as a mutating webhook or controller default. The diff tab shows which field, and an ignoreDifferences rule handles it.

What would you check when pods run but the app says Progressing? The health check is waiting, often on an Ingress or LoadBalancer without an address. Read the resource message in the tree.

Name one thing that changed in 3.5. It bundles Helm 4; spec.source.helm.version: v3 is ignored; plain-HTTP OCI registries need an explicit opt-in flag; and the ApplicationSet web UI appeared in Alpha. Say this only if you are sure, and otherwise describe the install change, which is certain.

Questions to ask back

Interviews are two-way, and a good question shows experience. "Is sync automated here, and with prune and self-heal?" reveals how much they trust the tool. "How do image tags get into Git: a pipeline commit or an image updater?" shows you understand the last-mile problem. "How are Applications themselves stored, and who can change a project?" probes governance. "Do you use one Argo CD per cluster or a central one?" opens the architecture discussion.

Sixty-second self-test

Answer aloud, without looking back.

1. Give the two-sentence definition of Argo CD.
2. Why does a pull model need no cluster credentials in CI?
3. Name the three main components and what each does.
4. What is the difference between sync status and health? Give one example of Synced and Degraded.
5. Why is prune off by default?
6. Why does self-heal revert your kubectl scale?
7. Why can't you roll back an automated application, and what do you do instead?
8. Which flags does the install command need, and why?
9. A push happened two minutes ago and nothing deployed. Is that a bug?
10. What does the finalizer resources-finalizer.argocd.argoproj.io do?

If any answer takes more than a few seconds, reread the matching section of the Beginner Detailed track. The next guide, Mid-level, builds on these same ideas with diffing, projects, ApplicationSets and sync ordering.

Part one of three. Three mistakes produce most beginner trouble with Argo CD: forgetting that Git is the only input the tool trusts, confusing "synced" with "working", and following a tutorial written for version 2.x. Start with the table, then the habits and practice cards.

Common errors at this level

The first row deserves a moment, because almost everyone meets it. The old install command simply stopped working in 3.3, and a large share of blog posts still show it. If a command from an article fails, check the article's date and the Argo CD version it used before you blame your cluster.

Mistake: treating "Synced" as "it works"

Symptom. The dashboard shows a green tick for sync and a broken heart for health, or the team says "Argo CD says it is fine" while users see errors.

Cause. Synced answers only one question: does the cluster match Git? A Deployment pointing at an image that does not exist matches Git perfectly and still fails. The two statuses are independent, and some people never learn to look at the second.

Fix. Read both badges, every time. When health is Degraded, click down the resource tree to the red node, open its Events tab, and read the message. Nine times out of ten it is a bad image tag, a missing Secret or ConfigMap, or an unmet resource request, and it is a Kubernetes problem rather than an Argo CD problem. Use kubectl describe on the failing object for the full events.

Mistake: fixing things with kubectl

Symptom. You scale a Deployment or edit a ConfigMap by hand to fix something, and the change reverts, or worse, the application sits OutOfSync for weeks and nobody remembers why.

Cause. Argo CD reconciles toward Git. With self-heal on, it reverts the hand edit. With self-heal off, it leaves the edit in place and reports a difference, which everyone then learns to ignore.

Fix. Make the change in Git. For a real emergency where you must act first, pause automation on that application (argocd app set NAME --sync-policy none), make the cluster change, and then immediately commit the same change so Git catches up and you can re-enable automation. A difference you leave unresolved is a debt: the next sync will overwrite it, often at a bad moment.

Mistake: enabling prune before understanding it

Symptom. After a refactor, an object, sometimes one holding data, is gone from the cluster.

Cause. With prune on, anything missing from Git is deleted. Renaming a file is harmless, but renaming the metadata.name inside it is a delete of the old object and a create of the new one. If the old object was a PersistentVolumeClaim, the storage behind it may go too, depending on its reclaim policy.

Fix. Run argocd app sync NAME --prune --dry-run and read what it would delete before turning prune on. Keep stateful objects out of prune's reach when you can: annotate a resource with argocd.argoproj.io/sync-options: Prune=false so Argo CD never deletes it automatically. Look at the whole resource tree first, so you know what the application owns.

Mistake: editing the application settings in the UI

Symptom. You change the revision or a Helm parameter through the UI or argocd app set, and later someone applies the Application file from Git and the change disappears, or the reverse.

Cause. The Application is itself a Kubernetes object. The UI and app set change the copy in the cluster; the file in Git is a separate copy. Two sources of truth is exactly the situation GitOps exists to avoid.

Fix. Treat the Application YAML as code. Edit the file, commit it, and apply it (or have a parent Application apply it for you, which the next level covers). Use the UI and app set for experiments and throwaway applications only.

Mistake: deleting without knowing the finalizer

Symptom. Either pods survive after you deleted their Application, or a deletion removes far more than you intended.

Cause. Cascading deletion depends on the finalizer resources-finalizer.argocd.argoproj.io. Present, it deletes everything the application deployed. Absent, kubectl delete application removes only the record and leaves the workloads running with nothing managing them.

Fix. Decide on purpose. Put the finalizer in every Application file you write. Before deleting anything important, run argocd app get NAME -o tree and read what the delete will take. To keep the workloads and remove only the record, use argocd app delete NAME --cascade=false.

Mistake: following a 2.x tutorial

Symptom. A command fails, a setting does nothing, a screen differs, or an old password trick does not work.

Cause. Argo CD 3.0 changed several defaults, and later releases removed more. The ones you are most likely to meet: install needs server-side apply; resource tracking uses an annotation rather than the app.kubernetes.io/instance label; viewing logs needs the explicit logs, get permission; the initial admin password is in the argocd-initial-admin-secret Secret and not the pod name; and Helm 4 replaced Helm 3 in the image in 3.5, making the helm.version: v3 field meaningless.

Fix. Always note the version. Check argocd version and read the page from the official documentation's stable site, which tracks the current release. When you install, pin a tag such as v3.5.3, so the version you read about is the version you run.

The habits that pay off most

Practice cards

Work these in order on a throwaway local cluster. Each takes about ten minutes.

1. Break a path. Set an application's path to a folder that does not exist. Find the error in the UI and in argocd app get. Fix it and hard-refresh.
2. Break an image. Push an image tag that does not exist. Confirm the app is Synced and Degraded. Find the cause in kubectl describe pod. Revert the commit.
3. Watch self-heal. With self-heal on, scale a Deployment by hand and time how long it takes to revert. Turn self-heal off and repeat.
4. Watch prune. Delete a file from Git with prune off and see the object survive. Turn prune on and see it go.
5. Try the finalizer. Delete an application with and without --cascade=false and count the surviving pods each time.
6. Hit a project wall. Create a project that allows one repository, point an application at another, and read the refusal.

Debugging an application that will not sync

Follow the same order every time. It is slower on the first few tries and faster than guessing forever after.

1. Read the status. argocd app get NAME prints the sync status, health status and the message. Often the message names the cause outright.
2. Read the diff. argocd app diff NAME shows the difference Argo CD wants to fix. An empty diff with OutOfSync usually means a stale cache: hard-refresh.
3. Look at the right logs. Rendering, Git and credential failures show up in the repo server; comparing and applying failures show up in the controller.
4. Ask Kubernetes. kubectl describe on the failing resource, and kubectl get events -n NAMESPACE --sort-by=.lastTimestamp, often explain a stuck object better than Argo CD can.
5. Hard-refresh. argocd app get NAME --hard-refresh clears the cached rendering. Render failures are cached on purpose, so a fixed source can keep showing the old error until you do this.

argocd app get hello
argocd app diff hello
kubectl logs -n argocd deploy/argocd-repo-server --tail=100
kubectl logs -n argocd statefulset/argocd-application-controller --tail=100
kubectl get events -n hello --sort-by=.lastTimestamp
argocd app get hello --hard-refresh

Reading a diff

The diff is the most useful thing Argo CD shows you, and it rewards a little practice. Lines prefixed with < come from the live cluster and lines with > come from Git, so a change reads as "live has this, Git wants that". A typical output looks like this:

===== apps/Deployment hello/hello ======
157c157
<   replicas: 2
---
>   replicas: 3

That says the live Deployment has two replicas and Git wants three. The number before the c is a line in the rendered document, and it only matters for orientation.

Sometimes the diff shows fields you never wrote: defaults Kubernetes added, or annotations a controller attached. Argo CD already ignores many of these, and you will meet the rest as permanent differences. When a field differs on every refresh, find out who keeps writing it (a mutating webhook, an autoscaler changing replicas, an operator adding an annotation) before deciding what to do. An autoscaler is the classic case: if a HorizontalPodAutoscaler manages the replica count, do not also set replicas in Git, or the two will fight.

Habits for repositories

Good Argo CD usage is mostly good repository hygiene.

Keep the application definition separate from the code it deploys, or at least in a different folder, so a change to "what is deployed" is obvious in review. Many teams use one repository for application source and a second for deployment manifests, so a pipeline that commits an image tag does not trigger a build in the source repository.

Give every commit that changes the cluster a message that explains why. The repository is now your change log for production, and "update" is a useless entry at two in the morning.

Use branch protection and pull requests on the repository Argo CD tracks. The repository is the control panel for your cluster, so anyone who can push to the tracked branch can change production. Reviewing a manifest change is reviewing a deployment.

Prefer a tag or a branch name for targetRevision in real applications, and understand what each implies. A branch means "follow the latest commit", which is what you want for a continuously deployed environment. A tag or commit hash means "stay on exactly this version", which suits a release you have promoted on purpose. HEAD means the repository's default branch.

Keep one concern per Application at the beginning. A single application containing a database, a message queue and three services is hard to reason about and sync. Smaller applications give clear health signals and let you sync or roll back one thing at a time.

Mistake: expecting Argo CD to do CI's job

Symptom. You push application code and the new version never appears in the cluster.

Cause. Argo CD deploys what is in the repository. It does not build images, and it does not notice that a new image exists in a registry. If the Deployment still names myapp:1.4.0, nothing has changed as far as Argo CD is concerned.

Fix. Close the loop: the pipeline builds and pushes the image, then commits the new tag into the deployment repository (by editing the manifest, the Kustomize images section or the Helm values file). That commit is what Argo CD reacts to. Teams that want to automate the tag update often add a helper for it; that design choice belongs to the next level. Until then, committing the tag by hand is perfectly good practice.

Platform gotchas

Local clusters and images. Some example applications are built only for AMD64. On an Apple Silicon laptop the pod may fail with an exec format error. Use an image built for your architecture.

Local clusters and load balancers. A Service of type LoadBalancer never gets an address on many local clusters, so the application stays Progressing. Use a ClusterIP Service and kubectl port-forward, or install a local load balancer.

Corporate networks. A proxy that does not pass HTTP/2 or gRPC breaks the CLI with transport is closing; use --grpc-web. A network that inspects TLS traffic can break the CLI's certificate checks in the same way as it breaks other tools, because the certificate the CLI sees is not the one the server sent. Fix the trust on your machine rather than disabling verification everywhere.

Webhooks and laptops. A cluster behind a port-forward has no public address, so Git hosts cannot reach its webhook endpoint. Rely on manual refresh for local work.

Windows. Only argocd-windows-amd64.exe is published. If you prefer a Linux toolchain, WSL follows the Linux instructions.

Keeping your practice environment clean

Practice clusters accumulate clutter. Delete applications with their finalizers so resources are removed, then delete the cluster when you are done:

argocd app list -o name
argocd app delete hello
kind delete cluster --name argo-demo

If you installed Argo CD into a shared cluster by mistake, remove the namespace with kubectl delete namespace argocd. The custom resource definitions stay behind, since they are cluster-wide; delete them only if you are certain nothing else uses them. Also remove any tokens you created for private-repository tests, both in Argo CD and in your Git host.

Small things to do from day one

• Change the admin password straight after first login, then delete argocd-initial-admin-secret.
• Run argocd version and note the client and server versions. Keep them close.
• Enable shell completion so you can tab-complete application names.
• Store your practice Applications in a Git repository, not just in the cluster.
• Name applications and namespaces predictably, for example team-service-environment, so lists stay readable.
• Label applications (team=payments) so argocd app list -l team=payments works.
• Read the upgrade notes before changing versions. Patch releases carry no breaking changes, while minor releases can carry small ones.
• Bookmark the official documentation and check the page's version before trusting an example.

argocd completion zsh > ~/.argocd-completion.zsh
echo 'source ~/.argocd-completion.zsh' >> ~/.zshrc

A starter setup to copy

A minimal layout that scales from a first project to a real team is one deployment repository with an Applications folder and a folder per service:

deploy-repo/
  argocd/
    hello-app.yaml        # Application definitions
  apps/
    hello/
      deployment.yaml
      service.yaml

And the Application that points at it:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: hello
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: https://github.com/YOUR-USER/deploy-repo.git
    targetRevision: main
    path: apps/hello
  destination:
    server: https://kubernetes.default.svc
    namespace: hello
  syncPolicy:
    syncOptions:
      - CreateNamespace=true

It starts with manual sync on purpose. Once you have watched several syncs and read several diffs, add the automation block:

  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

A short checklist

Before you call an application "done", run down this list.

1. The Application is a YAML file in Git, with the finalizer, and it is reviewed like code.
2. The version of Argo CD is pinned, and you know it.
3. The application shows Synced and Healthy, and you have looked at both badges.
4. You know whether automated sync, prune and self-heal are on, and why.
5. You have read a --dry-run or diff before any prune.
6. No token or password is committed anywhere.
7. You know how to roll back: revert in Git, with argocd app rollback as the stopgap for manual applications.
8. You can find the logs for the repo server and the controller without looking them up.

If all eight hold, you are using Argo CD the way it is meant to be used. The next step is the Mid-level guide, which covers how Argo CD decides what is different, how to handle permanent differences, how to organise many applications, and how to order deployments.

This is part two of three. It picks up where Beginner ended and takes every topic from there further, then adds the machinery you have not met yet. Everything here is checked against Argo CD v3.5.3 (released 14 September 2026; the 3.5 line went GA on 4 August 2026). Most tutorials on the web still describe 2.x behaviour, so where a default changed in 3.x this guide says so and shows the current form.

The stakes are practical. At Beginner level, Argo CD is a green tick in a UI. At work it is the thing standing between a merged pull request and a production cluster, which means its failure modes are your failure modes: an application that is forever OutOfSync for no visible reason, a sync that deletes a namespace you did not mean to delete, a team that can see another team's logs, a controller that falls over at 400 applications. By the end of this guide you should be able to predict what Argo CD will do before it does it, and to explain why.

Where this picks up

Each section starts with the problem it solves and ends with a Try it you can run against a scratch cluster in a few minutes. If you do not have one, kind or minikube with the install command from the Beginner guide is enough. Two neighbouring guides pair well with this one: Kubernetes for the objects Argo CD applies, and Helm and Kustomize for the renderers it runs.

How reconciliation really works

The problem this section solves is that "Argo CD compares Git with the cluster" hides a pipeline with several stages, each with its own cache, timeout and failure mode. When something is slow or stale, you need to know which stage to look at.

Start with who does what. argocd-server is the API and UI; it is stateless and handles authentication, RBAC and the webhook receiver. argocd-repo-server clones repositories and renders manifests by running helm template, kustomize build, Jsonnet or a plugin, and it deliberately has no Kubernetes privileges. argocd-application-controller, a StatefulSet, watches the clusters, compares live state to rendered state, and executes syncs and hooks. Redis sits in the middle as a cache. The important fact about Redis is that it is disposable: all durable state lives in Kubernetes objects (Applications, AppProjects, Secrets and ConfigMaps in etcd), so a lost Redis costs you warm caches, not data.

Now follow one change through the system.

1. A refresh is triggered, either by polling or by a webhook. By default Git is polled every 120 seconds plus up to 60 seconds of jitter (timeout.reconciliation and timeout.reconciliation.jitter in argocd-cm), so "about three minutes" in the worst case. A webhook to /api/webhook makes it near-instant.
2. The controller asks the repo-server for the application's manifests at the resolved commit. The repo-server resolves the revision to a commit SHA, fetches it, renders the manifests with the right tool, and caches the result keyed by commit and source parameters. The default cache lifetime is 24 hours (reposerver.repo.cache.expiration).
3. The controller compares those target manifests to the live state, which it does not fetch on demand: it keeps a watch-based cluster cache of every resource kind in every cluster it manages. That cache is why the controller's memory grows with the number of cached resources, not the number of applications.
4. The result is written to the Application's status as sync status (Synced, OutOfSync, Unknown) and health. If automated sync is on and the app is out of sync, an operation is queued.
5. The operation runs: hooks, waves, apply, prune, health assessment, history entry.

Two distinctions matter constantly in day-to-day work. The first is refresh versus hard refresh. A normal refresh re-compares using the manifest cache; a hard refresh also invalidates the repo-server's cached manifests for that application. If you changed something the repo-server cannot see in Git, such as a Helm chart published to a repository under a floating version, a plugin's output, or a cached error, a normal refresh will happily reuse the stale render.

argocd app get guestbook --refresh          # re-compare, use the manifest cache
argocd app get guestbook --hard-refresh     # invalidate the cache, re-render, re-compare

The second is sync status versus operation status. Synced means live equals target at this moment. The operation phase (Running, Succeeded, Failed, Error, Terminating) says whether the last sync attempt worked. An application can be Synced with a failed last operation (someone fixed it by hand) and OutOfSync with a succeeded one (something drifted afterwards). Keep the two axes separate in your head and most "but it says succeeded" confusion disappears.

There are two queues worth knowing by name. The status processors (default 20) do comparisons; the operation processors (default 10) run syncs. They are set with controller.status.processors and controller.operation.processors in argocd-cmd-params-cm, and the docs' guidance for roughly a thousand applications is 50 and 25. Raising them without raising the repo-server's capacity just moves the bottleneck, which is a theme we return to in the performance section.

Diffing, drift and phantom OutOfSync

The problem: the most common complaint about Argo CD is an application that is permanently OutOfSync although nothing seems to differ. To fix that you need to know what is being compared and by which algorithm.

Argo CD normalises both sides and compares them. The target is the rendered manifest. The live object is what the API server returns, which includes everything the cluster added: defaulted fields, status, metadata.managedFields, injected sidecars, and fields written by controllers such as an HPA changing spec.replicas. A raw comparison would never match, so Argo CD applies rules to decide what counts.

Three diff strategies exist. The default is the legacy three-way diff using the last-applied-configuration annotation, which the docs call the main strategy used by default. Structured-merge diff has been discontinued. The third is Server-Side Diff, Stable since 3.1, which asks the API server to run a server-side apply dry run and compares that result. Because the real API server does the merging, it accounts for defaulting and for admission webhooks, which is exactly the class of phantom drift that the client-side algorithm cannot predict.

Enable it for everything, or per application:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cmd-params-cm
  namespace: argocd
data:
  controller.diff.server.side: "true"

metadata:
  annotations:
    argocd.argoproj.io/compare-options: ServerSideDiff=true

Server-side diff costs an extra API call per comparison, and it lets mutating admission webhooks participate in the diff. That is usually what you want, but it means a webhook that injects something non-deterministic becomes visible as drift.

The controlled ways to ignore a difference, from narrowest to broadest:

1. Per application, in spec.ignoreDifferences. Targets a group, kind and optionally a name, using JSON pointers, JQ path expressions or managedFieldsManagers.
2. Per resource type, system-wide, with resource.customizations.ignoreDifferences.<group>_<kind> in argocd-cm (or .all for everything).
3. On the object itself, with the annotation argocd.argoproj.io/compare-options: IgnoreExtraneous, which stops an out-of-sync resource from turning the whole application OutOfSync (it only hides the effect on the parent, it does not fix the diff).

spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas          # an HPA owns this field
    - group: admissionregistration.k8s.io
      kind: MutatingWebhookConfiguration
      jqPathExpressions:
        - '.webhooks[]?.clientConfig.caBundle'   # injected by cert-manager
  syncPolicy:
    syncOptions:
      - RespectIgnoreDifferences=true

That last option is the one people miss. ignoreDifferences only affects the comparison. At sync time Argo CD still applies the full target manifest, so your ignored replicas field is reset to whatever Git says on the next sync, and the HPA and Argo CD fight. RespectIgnoreDifferences=true makes the sync leave ignored fields alone. The pair is what you actually want.

Since 3.0 the defaults changed in ways that remove whole categories of noise. The .status field is now ignored for all resources, not just CRDs (restore the old behaviour with ignoreResourceStatusField: crd under resource.compareoptions). ignoreDifferences now also suppresses resource update events so ignored changes do not trigger a reconcile at all, and the defaults for resource.exclusions ship with sensible entries such as Endpoints, EndpointSlice, Lease and the *Review kinds. If an upgrade from 2.x made you re-examine a diff, this table of 3.0 changes is where the answer usually lives.

Resource tracking and ownership

The problem: Argo CD has to decide, for every object in a cluster, whether an application owns it. It needs that to know what to prune, what to show in the tree, and what counts as an orphan.

Until 2.x it used a label, app.kubernetes.io/instance. That collided with everything else that uses the same label (Helm charts, other tools), causing applications to claim each other's resources. Since 3.0 the default is an annotation:

metadata:
  annotations:
    argocd.argoproj.io/tracking-id: guestbook:apps/Deployment:guestbook/guestbook-ui

The value encodes the application name, the group and kind, the namespace and the object name. Two applications cannot silently claim the same object because the annotation names exactly one owner. The method is configured with application.resourceTrackingMethod in argocd-cm: annotation (default), annotation+label, or label, whose value is truncated to 63 characters.

Two practical consequences. First, if you run several Argo CD instances against one cluster, set a distinct installationID on each so they do not treat each other's objects as their own. Second, switching from label to annotation tracking has a documented edge case: if the first sync after the switch deletes a resource, that resource can be orphaned. After such an upgrade, sync every application once; applications that use ApplyOutOfSyncOnly=true need an explicit full sync.

Orphaned resources are objects in a destination namespace that no application claims. Monitoring is switched on per project with orphanedResources, and it produces a warning condition rather than deleting anything.

spec:
  orphanedResources:
    warn: true
    ignore:
      - kind: ConfigMap
        name: kube-root-ca.crt

Use it as a hygiene signal in namespaces that a team is supposed to manage entirely through Argo CD. argocd app resources NAME --orphaned lists them.

Sync in depth: policy, options and failure handling

The problem: a sync is not "apply the manifests". It is a sequence with decisions in it, and most production incidents involving Argo CD are a wrong decision in that sequence: a prune that removed too much, a Replace that recreated a stateful object, a retry loop that hammered the API server.

Automated sync is configured under syncPolicy.automated. Its three controls are independent:

• prune: true lets a sync delete live resources that are no longer in Git. It is off by default even with automation on, because deletion is destructive.
• selfHeal: true re-syncs when live state drifts without a Git change, for example after a manual kubectl edit. Without it, automation only reacts to new commits.
• allowEmpty: true permits a sync whose target renders to zero resources. The default refuses, which protects you from a bad path wiping an entire application.

Self-heal has a pacing mechanism so a fight between Argo CD and another controller does not spin. In 3.x it uses exponential backoff controlled by controller.self.heal.backoff.timeout.seconds (2), controller.self.heal.backoff.factor (3) and controller.self.heal.backoff.cap.seconds (300). One doc conflict to know about: the auto-sync page still says the self-heal timeout is five seconds by default, while the parameter reference shows a timeout of zero with backoff on. Teach the backoff settings, and treat the effective retry pacing as exponential.

Automated sync has rules you should know before relying on it. It will not retry a failed sync for the same commit and parameters indefinitely on its own: after a failed automated attempt, it waits for a change. Add a retry block if you want persistence, and note that argocd app rollback is refused while automated sync is enabled, because automation would immediately undo it.

spec:
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    retry:
      limit: 5                # -1 means unlimited
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m
    syncOptions:
      - CreateNamespace=true
      - PruneLast=true
      - ApplyOutOfSyncOnly=true

Sync options can be set on the application (syncPolicy.syncOptions) or on a single resource with the annotation argocd.argoproj.io/sync-options. The ones you will actually use:

There is also a server-side apply nuance specific to Argo CD itself. Since 3.3 the ApplicationSet CRD is bigger than the 262,144-byte annotation limit that client-side apply needs, so installing or upgrading with a plain kubectl apply -f install.yaml fails with metadata.annotations: Too long: may not be more than 262144 bytes. The current form is:

kubectl apply -n argocd --server-side --force-conflicts \
  -f https://raw.githubusercontent.com/argoproj/argo-cd/v3.5.3/manifests/install.yaml

If Argo CD manages its own installation through an Application, that application needs ServerSideApply=true in its sync options. helm upgrade users are unaffected.

Finally, deletion. Without the resources-finalizer.argocd.argoproj.io finalizer on an Application, deleting the Application leaves its resources behind. With it, deletion cascades. argocd app delete NAME --cascade=false does the opposite on demand. Decide this deliberately for each application, because the wrong default deletes a production namespace when someone tidies up a UI.

Hooks, waves and ordering

The problem: some changes must happen in a particular order. A database migration runs before the new application version; a CRD is created before its custom resources; a smoke test runs after everything is up. Argo CD gives you two separate ordering tools that people routinely mix up.

Phases and hooks are set with the annotation argocd.argoproj.io/hook. The phases are PreSync, Sync, PostSync, SyncFail, Skip, and, since 2.10, PreDelete and PostDelete for cleanup around application deletion. A hook is an ordinary manifest, usually a Job, that Argo CD creates at that phase. Its lifecycle is governed by argocd.argoproj.io/hook-delete-policy: HookSucceeded, HookFailed or BeforeHookCreation.

apiVersion: batch/v1
kind: Job
metadata:
  name: db-migrate
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
spec:
  backoffLimit: 0
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: migrate
          image: registry.example.com/shop/migrate:1.14.2
          args: ["./migrate", "up"]

BeforeHookCreation is the delete policy you almost always want. The Job name is fixed, so on the next sync the previous Job has to go before a new one can be created, and this policy removes it automatically. Without any policy, the second sync fails because the object already exists.

Sync waves order resources within a phase. Set argocd.argoproj.io/sync-wave: "<integer>"; the default is 0 and negative numbers are allowed. Argo CD applies wave by wave, lowest first, and a wave must be healthy before the next one starts. There is a two-second pause between waves (controller.sync.wave.delay.seconds, or ARGOCD_SYNC_WAVE_DELAY).

The two tools combine in a fixed order: phases run PreSync, then Sync, then PostSync, and waves order things inside each phase. The mental model is a grid, not a line.

Two subtleties bite in practice. First, a wave only waits for health, and health is only as good as the health check for that kind. A custom resource with no health logic reports healthy immediately, so the next wave starts too early; the next section shows how to fix that. Second, a PreSync hook runs before anything in the Sync phase is applied, including a new ConfigMap or Secret your hook may need. If the hook depends on those, put the prerequisite in the PreSync phase with an earlier wave, or restructure.

Helm charts add a wrinkle: Helm hooks are translated to Argo CD hooks, and helm.sh/hook annotations are mapped, but Argo CD is not running Helm's own lifecycle. The FAQ is explicit that the application is rendered with helm template, so helm ls shows nothing for an Argo CD-managed chart. That surprises people who reach for it while debugging; use argocd app manifests NAME instead.

Health assessment and resource actions

The problem: "healthy" is not something the Kubernetes API reports uniformly. A Deployment has a rollout status; a Certificate has conditions; a custom resource from an operator you installed last week has nothing Argo CD understands. Health drives sync waves, the green tick in the UI, the result of argocd app wait --health, and your alerts, so a wrong answer cascades.

Argo CD's health values are Healthy, Progressing, Degraded, Suspended, Missing and Unknown. Built-in checks cover the core kinds, and Lua scripts ship for many popular CRDs. The application's health is an aggregate of its resources, with one caveat that changed in 3.4: an application is Missing only when all of its resources are missing. A partially missing application now shows the aggregate health of what exists, so automation that keyed off Missing should check for OutOfSync instead.

A few built-in behaviours surprise people. An Ingress with no load balancer address yet stays Progressing. Since 3.2 a CronJob whose most recent Job failed can turn the application Degraded; annotate a CronJob with argocd.argoproj.io/ignore-healthcheck: "true" to opt it out. And in 3.5.3, Crossplane managed resources report Progressing while they are provisioning, which is correct but can look like a stuck sync.

For a kind with no check, or a wrong one, write your own in Lua in argocd-cm. The key is resource.customizations.health.<group>_<kind>. The script receives the object as obj and returns a table with status and message.

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  resource.customizations.health.database.example.com_PostgresCluster: |
    hs = {}
    if obj.status ~= nil and obj.status.conditions ~= nil then
      for _, c in ipairs(obj.status.conditions) do
        if c.type == "Ready" and c.status == "True" then
          hs.status = "Healthy"
          hs.message = "cluster ready"
          return hs
        end
        if c.type == "Ready" and c.status == "False" then
          hs.status = "Degraded"
          hs.message = c.message
          return hs
        end
      end
    end
    hs.status = "Progressing"
    hs.message = "waiting for Ready condition"
    return hs

You do not have to test this against a live cluster. The CLI evaluates a script against a saved object:

argocd admin settings resource-overrides health ./postgres.yaml --argocd-cm-path ./argocd-cm.yaml

Resource actions are the same mechanism pointed at operations rather than health. An action is a Lua script registered under resource.customizations.actions.<group>_<kind> that returns a modified object. Built-in actions exist for things like restarting a Deployment, and since 3.5 an Application itself has a toggle-auto-sync action. From the CLI:

argocd app actions list guestbook --kind Deployment
argocd app actions run guestbook restart --kind Deployment --resource-name guestbook-ui

Through the API, use POST /api/v1/applications/{name}/resource/actions/v2. The original /resource/actions endpoint is deprecated (since 3.1) in favour of v2, which takes a JSON body and supports resourceActionParameters. RBAC for actions is fine-grained: action/<group>/<kind>/<name> in the object position of a policy, covered below.

Projects as the tenancy boundary

The problem: the default project allows every repository, every destination and every cluster-scoped kind. On a shared instance that is a platform where any team can deploy anything anywhere, including into kube-system. The AppProject is where you draw the lines.

A project restricts six things: which repositories applications may pull from (sourceRepos), which clusters and namespaces they may deploy to (destinations), which cluster-scoped and namespace-scoped kinds they may create (clusterResourceWhitelist/Blacklist, namespaceResourceWhitelist/Blacklist), which namespaces may host Application objects (sourceNamespaces), when syncs may run (sync windows), and who can do what (project roles). It also carries destinationServiceAccounts for impersonation and sourceIntegrity for signed commits, both covered below.

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: team-a
  namespace: argocd
spec:
  description: Team A services
  sourceRepos:
    - https://github.com/acme/team-a-*
  destinations:
    - server: https://kubernetes.default.svc
      namespace: team-a-*
  clusterResourceWhitelist:
    - group: ''
      kind: Namespace
  namespaceResourceBlacklist:
    - group: ''
      kind: ResourceQuota
    - group: ''
      kind: LimitRange
  orphanedResources:
    warn: true

Read this the way Argo CD does. Applications in team-a may only pull from repositories matching the glob, may only target namespaces beginning team-a- on the local cluster, may create Namespaces as their only cluster-scoped kind (cluster-scoped kinds are denied unless whitelisted), and may not touch quotas, which belong to the platform team. Violations produce specific messages you will learn to recognise:

• application repo <url> is not permitted in project '<proj>'
• application destination server '<s>' and namespace '<ns>' do not match any of the allowed destinations in project '<p>'
• resource <group>:<kind> is not permitted in project <p>

The CLI mirrors the spec for quick changes, though in practice you should keep projects in Git:

argocd proj create team-a -d https://kubernetes.default.svc,team-a-* -s 'https://github.com/acme/team-a-*'
argocd proj allow-cluster-resource team-a '' Namespace
argocd proj deny-namespace-resource team-a '' ResourceQuota

Lock down the default project. It can be edited but not deleted. A common hardening is to empty its sourceRepos and destinations and blacklist all namespace resources, so that forgetting to set spec.project yields a refusal rather than an unrestricted deployment.

Project roles and JWT tokens give automation a narrow identity. A role grants policies on one project; a token minted for the role carries exactly those permissions and can be revoked without touching anyone else.

argocd proj role create team-a ci
argocd proj role add-policy team-a ci --action sync --permission allow --object '*'
argocd proj role create-token team-a ci -e 24h       # default: no expiry. Always set one.

There is another layer worth a mention, Applications in any namespace. By default Application objects must live in Argo CD's own namespace. Set application.namespaces: team-a, team-b in argocd-cmd-params-cm (plus the extra ClusterRole from the examples/k8s-rbac/argocd-server-applications manifests) and list the allowed namespaces under the project's spec.sourceNamespaces, and teams can own their Application objects in their own namespaces with ordinary Kubernetes RBAC. The error when one side is missing is application '<app>' in namespace '<ns>' is not permitted to use project '<p>'; the fix is to add the namespace to sourceNamespaces and to application.namespaces.

RBAC, SSO and impersonation

The problem: the question "who can do what" has three separate answers in Argo CD: how users prove who they are, what Argo CD lets them do, and what identity Argo CD uses when it acts on a cluster. Treating them as one leads to either an overpowered controller or an unusable permission model.

Authentication. Argo CD uses JWTs only. The local admin account is for bootstrapping: after SSO works, set admin.enabled: "false" in argocd-cm and delete argocd-initial-admin-secret. SSO uses either the bundled Dex (federating GitHub, LDAP, SAML, Microsoft and others) or direct OIDC with oidc.config for providers like Okta, Entra ID, Keycloak and Google. Whichever you use, url must be set in argocd-cm, or SSO login fails, because Argo CD builds its callback URL from it. Since 3.1 the server handles OIDC with PKCE, so your identity provider must allow the redirect URI https://<argocd>/auth/callback. For automation, create a local account with only API-key capability rather than reusing a user:

data:
  url: https://argocd.example.com
  accounts.ci-bot: apiKey
  accounts.ci-bot.enabled: "true"

argocd account generate-token --account ci-bot -e 30d

Authorization is Casbin-based and lives in argocd-rbac-cm. A policy line has the form p, <subject>, <resource>, <action>, <object>, <effect>, and group membership is g, <group>, <role>. The resources are applications, applicationsets, clusters, projects, repositories, accounts, certificates, gpgkeys, logs, exec and extensions. For application-scoped resources the object is <project>/<app>, or <project>/<app-namespace>/<app> when apps-in-any-namespace is on.

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-rbac-cm
  namespace: argocd
data:
  policy.default: ''
  scopes: '[groups, email]'
  policy.csv: |
    p, role:team-a-dev, applications, get,    team-a/*, allow
    p, role:team-a-dev, applications, sync,   team-a/*, allow
    p, role:team-a-dev, logs,         get,    team-a/*, allow
    p, role:team-a-dev, applications, action/apps/Deployment/restart, team-a/*, allow
    p, role:team-a-lead, applications, update, team-a/*, allow
    p, role:team-a-lead, applications, delete, team-a/*, allow
    g, team-a-developers, role:team-a-dev
    g, team-a-leads,      role:team-a-dev
    g, team-a-leads,      role:team-a-lead

Four things in that file are 3.0-era behaviour changes that catch people migrating from older tutorials. First, logs, get must be granted explicitly; applications, get no longer implies the Logs tab, and the old enforcement flag has been removed. Second, update and delete on an application no longer cascade to its managed resources: the fine-grained permission applies only to the Application object, so granting a team the right to delete pods needs delete/*/Pod/*/* or similar. Third, use all four slashes in fine-grained objects, because glob matching does not treat / as a separator. Fourth, policy.default cannot be revoked by a deny rule, so if you set it to role:readonly, every authenticated user keeps read access whatever else you write. Use '' for no default access and grant through roles.

Policies are code, and code should be tested before it is applied:

argocd admin settings rbac validate --policy-file policy.csv
argocd admin settings rbac can role:team-a-dev sync applications 'team-a/payments' --policy-file policy.csv
argocd admin settings rbac can role:team-a-dev sync applications 'team-b/payments' --policy-file policy.csv

If a Dex or OIDC user sees a 4 KB cookie error, invalid cookie, longer than max length 4093, the JWT carries too many groups. Reduce the groups the provider emits (scope them to Argo CD) rather than looking for a server setting. On Microsoft Entra ID, group overflow can be resolved through Microsoft Graph since 3.5.

Impersonation addresses the third question. By default the application controller applies manifests with its own very powerful service account, so anything a project permits runs as that account. With sync impersonation (Beta), enabled through application.sync.impersonation.enabled: "true" in argocd-cm, each destination gets a service account chosen from the project:

spec:
  destinationServiceAccounts:
    - server: https://kubernetes.default.svc
      namespace: team-a-*
      defaultServiceAccount: team-a-deployer

argocd proj add-destination-service-account team-a https://kubernetes.default.svc team-a-payments team-a-deployer

Now a sync into team-a-payments runs as team-a-deployer, so Kubernetes RBAC on that namespace, not just the Argo CD project, limits what the sync can do. In 3.5 this extends beyond syncs to server operations: logs, events, delete, patch and resource actions also run as the impersonated identity, and enforcement is on by default (application.sync.impersonation.enforced, settable to false to fall back to the controller account). The consequence is that the service account needs more than apply rights. It needs get, patch, delete and list on the relevant objects and events, and get on pods/log, or the UI will break for users the project otherwise allows.

Repositories, credentials and multiple sources

The problem: applications need to read private Git, Helm and OCI sources, and at scale you cannot create a credential per repository or scatter tokens across Secrets nobody tracks.

Repositories and credentials are Secrets in Argo CD's namespace, distinguished by the label argocd.argoproj.io/secret-type. The values are repository (a single repo), repo-creds (a credential template matched by URL prefix), repository-write (used by the source hydrator to push) and cluster. The older approach of listing repositories in argocd-cm was removed in 3.0, so if an old guide tells you to edit repositories: there, it is obsolete.

A credential template is the efficient pattern. One Secret covers every repository under an organisation:

apiVersion: v1
kind: Secret
metadata:
  name: acme-org-creds
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repo-creds
stringData:
  type: git
  url: https://github.com/acme
  githubAppID: "123456"
  githubAppInstallationID: "7890123"
  githubAppPrivateKey: |
    -----BEGIN RSA PRIVATE KEY-----
    ...
    -----END RSA PRIVATE KEY-----

In practice you would not commit that file with a private key in it; create it from your secret manager, or use the CLI form argocd repocreds add https://github.com/acme --github-app-id 1 --github-app-private-key-path key.pem. A GitHub App is preferable to a personal access token because it is not tied to a person and has rotatable scoped access. Azure DevOps Service Principal authentication is new in 3.5 (--azure-service-principal-client-id and --azure-service-principal-client-secret on argocd repo add).

SSH has a 3.5 change. Repositories without explicitly configured credentials now validate host keys against argocd-ssh-known-hosts-cm, not the container's own ~/.ssh/known_hosts. The symptom after an upgrade is knownhosts: key mismatch on a repository that worked yesterday. Add the host key:

ssh-keyscan git.example.com | argocd cert add-ssh --batch
argocd cert add-tls git.example.com --from ca.pem         # for a private CA over HTTPS

OCI and Helm. An application can pull Helm charts from an HTTP chart repository, or from an OCI registry with repoURL: oci://registry.example.com/charts. Register an OCI repo with argocd repo add oci://registry.example.com/charts --type oci --name myoci --username u --password p. Since 3.5 bundles Helm 4, plain-HTTP registries need an explicit opt-in (--insecure-oci-force-http, or insecureOCIForceHttp: "true" in the repo Secret), and setting it together with --insecure-skip-server-verification produces http: server gave HTTP response to HTTPS client with no workaround, so do not combine them. The field spec.source.helm.version: v3 is now ignored, since Helm 4 is always used.

Multiple sources solve a very common need: a chart from one place, values from another. Declare spec.sources[], give the values repo a ref, and refer to it with $ref/:

spec:
  project: team-a
  sources:
    - repoURL: https://charts.example.com
      chart: payments
      targetRevision: 2.4.1
      helm:
        valueFiles:
          - $values/payments/prod/values.yaml
    - repoURL: https://github.com/acme/deploy-config.git
      targetRevision: main
      ref: values
  destination:
    name: prod-eu
    namespace: team-a-payments

This separates the artifact (an immutable chart version) from the configuration (a Git file you review). It also means a change to either source produces a new target state. Since 3.5, valueFiles accept wildcard globs, and you can build multi-source applications from the UI. CLI helpers: argocd app add-source and argocd app remove-source.

Helm parameter overrides are a tempting shortcut: argocd app set NAME -p image.tag=v2 writes the override into the Application. That is fine for an experiment, but the override now lives in a cluster object, not in Git, which defeats the reason you adopted GitOps. Keep overrides in values files in Git, and use CLI parameters only for throwaway work. Kustomize works the same way: the bundled version is 5.8.1, a different one can be registered with kustomize.path.<ver> in argocd-cm and selected with spec.source.kustomize.version, and since 3.2 that can also be set in a .argocd-source.yaml file next to the manifests.

ApplicationSets and app-of-apps

The problem: one Application per service per cluster per environment is hundreds of near-identical YAML files, and a change to the pattern means editing all of them. You want to declare the pattern once and let Argo CD stamp out the instances.

There are two tools, and they solve different problems. App of apps is a parent Application whose source directory contains child Application manifests. It is how you bootstrap a cluster: one kubectl apply of the root application and everything else follows from Git. It is simple, visible and has no new concepts, but it is static: you add a child by adding a file.

An ApplicationSet is a template plus generators. The controller evaluates the generators, produces a list of parameter sets, and renders one Application per set. There are nine generators: List, Cluster, Git, Matrix, Merge, SCM Provider, Pull Request, Cluster Decision Resource and Plugin. The ones you will use first:

• Cluster produces one set per registered cluster, with name, server and labels, optionally filtered by a label selector. This is how "deploy the monitoring stack to every production cluster" works, including clusters added next month.
• Git directories produce one set per directory in a repository, so adding a folder adds an application. Git files read parameters from JSON or YAML files.
• Matrix combines two generators into a cross product (every cluster times every directory).
• Pull Request creates a preview application per open pull request, and removes it when the PR closes.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: platform-addons
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - matrix:
        generators:
          - clusters:
              selector:
                matchLabels:
                  env: prod
          - git:
              repoURL: https://github.com/acme/platform-addons.git
              revision: main
              directories:
                - path: addons/*
  template:
    metadata:
      name: '{{.name}}-{{.path.basename}}'
    spec:
      project: platform
      source:
        repoURL: https://github.com/acme/platform-addons.git
        targetRevision: main
        path: '{{.path.path}}'
      destination:
        server: '{{.server}}'
        namespace: '{{.path.basename}}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true

Use Go templates (goTemplate: true) with missingkey=error. The older fasttemplate syntax silently renders an empty string for a typo in a parameter name, which produces an Application with a blank destination and a confusing error. With missingkey=error a misspelt parameter stops the whole render with a clear message.

Know what an ApplicationSet controls and what it does not. The generated Applications are owned by the ApplicationSet, and their spec is overwritten from the template. Changing syncPolicy.automated directly on a generated Application has no lasting effect; change the ApplicationSet instead. The ApplicationSet-level policy (spec.syncPolicy.applicationsSyncPolicy: create-only, create-update, create-delete or sync) decides whether the controller may update or delete Applications once created, and is your safety net against a bad template deleting applications: create-only or create-update mean the controller never deletes them.

Progressive syncs (Beta, off by default, enabled through applicationsetcontroller.enable.progressive.syncs) add spec.strategy so a change rolls out in steps, for example dev, then staging, then a percentage of production clusters, with each step gated on the health of the previous one. It is the ApplicationSet answer to "do not update all fifty clusters at once".

The 3.5 release adds an ApplicationSet web UI (Alpha) and an AppSet node in the resource tree, and the controller reports status.resources capped at 5,000 entries by default (applicationsetcontroller.status.max.resources.count). For controller limits, applicationsetcontroller.concurrent.reconciliations.max bounds parallel work.

Finally, which to use. Use app-of-apps for a small, fixed, hand-curated set of things with distinct configuration, especially the bootstrap layer. Use an ApplicationSet whenever the thing you are declaring is "the same pattern, many targets". A common, robust design is both: a root app-of-apps that installs the ApplicationSets.

Secrets, signed commits and the rendered-manifests pattern

The problem: GitOps says everything lives in Git, yet you cannot commit a database password. The official position on this is firm, and it shapes which tool you choose.

Populate secrets on the destination cluster, not during manifest generation. The recommended tools are Sealed Secrets, the External Secrets Operator, the Secrets Store CSI Driver and the Vault Secrets Operator. Each keeps the secret out of Git (or encrypted in it) and materialises a Kubernetes Secret in the target cluster, and Argo CD manages the small, non-sensitive reference: a SealedSecret or an ExternalSecret object.

The docs "strongly caution against" the alternative of injecting secrets while rendering, for example with a Vault plugin run as a config management plugin. The reasons are concrete. The rendered manifests, secrets included, are cached in plaintext in Redis and served over the repo-server's gRPC interface. Rotation is coupled to syncs, since nothing changes until the next render. And it is incompatible with the rendered-manifests pattern described below. The pragmatic rule: if the secret needs to be in the rendered output, it will sit in Redis, so avoid it.

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: payments-db
  namespace: team-a-payments
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: corp-vault
  target:
    name: payments-db
  data:
    - secretKey: password
      remoteRef:
        key: secret/payments/db
        property: password

Because these objects are reconciled by another controller, Argo CD needs to see them as healthy and not fight them. Check that your operator's kinds have health checks, and use resource.sensitive.mask.annotations in argocd-cm to mask annotations that might carry sensitive values in the UI and diffs. For Argo CD's own secrets (repository credentials, cluster credentials, the argocd-secret with its admin hash and signing key), protect them with etcd encryption and tight RBAC on the argocd namespace. On the region side, teams with data-residency obligations in the Gulf or Egypt usually point External Secrets at a secret manager hosted in a regional cloud or on-prem Vault, which is a choice made in the SecretStore, not in Argo CD.

Signed commits: Source Integrity. Before 3.5, enforcing commit signatures used AppProject.spec.signatureKeys and argocd proj add-signature-key. Those are deprecated; GnuPG verification has been replaced by Source Integrity, configured under AppProject.spec.sourceIntegrity. Legacy configurations keep working but log a warning, and the REST field verifyResult is replaced by sourceIntegrityResult.

spec:
  sourceIntegrity:
    git:
      policies:
        - repos:
            - url: https://github.com/acme/*
          gpg:
            mode: head
            keys:
              - 4AEE18F83AFDEB23

The mode is none, head (the tip commit must be signed by a trusted key) or strict (every new commit since the last sync must be). When the check fails the application cannot sync and reports a ResourceComparison error, and local syncs are blocked while it is enforced. Repository patterns accept globs and a ! prefix to exclude.

The rendered-manifests pattern and the Source Hydrator. With Helm or Kustomize, what Argo CD applies is generated at sync time, so the output a reviewer approved in Git is not the output that reached the cluster; you approved inputs, not results. The Source Hydrator (introduced in 2.14 as Alpha, Beta since 3.5.0) implements the rendered manifests pattern: Argo CD renders your "dry" source, commits the result to a separate branch (or, since 3.5, a separate repository via syncSource.repoURL), and syncs from that. Reviewers can now diff the exact YAML a change will apply.

spec:
  sourceHydrator:
    drySource:
      repoURL: https://github.com/acme/payments.git
      targetRevision: main
      path: environments/prod
    syncSource:
      targetBranch: environments/prod-hydrated
      path: prod

Turning it on takes three things: hydrator.enabled: "true" in argocd-cmd-params-cm, the commit server (present in the *-install-with-hydrator.yaml manifests), and a repository-write Secret with push credentials. Changes across 3.x that affect operations: from 3.3 the hydrator tracks state in git notes and commits only when manifests change, and it no longer cleans the application path before writing, so stale files from a deleted resource must be removed manually; from 3.2 syncSource.path must be a non-root path ("" and "." are rejected). Because the hydrated branch is machine-written, protect it from human pushes.

Sync windows and change control

The problem: sometimes "when" matters as much as "what". You do not want automated syncs at 17:00 on a Thursday, or during a regional trading peak, or while a platform team is changing the cluster.

A sync window is a schedule during which syncs are allowed (allow) or blocked (deny). It has a cron schedule, a duration, and selectors for applications, namespaces and clusters. Windows live on the project:

argocd proj windows add team-a --kind deny --schedule "0 14 * * 4" --duration 18h \
  --applications '*' --manual-sync

The rules compress into a handful of facts. deny always beats allow. If a project has any allow window, syncs are permitted only inside one. The selectors are OR-ed unless "Use AND operator" is set, which is a frequent source of a window matching more than intended. manualSync: true lets a human sync during a window that blocks automation, which is the usual pattern for an emergency fix. syncOverrun, new in 3.5, lets a sync that started inside the window finish after it ends. The message you see when blocked is cannot sync: blocked by sync window (or similar), and the UI shows which window applies.

Pair windows with Prune=confirm for destructive changes and with Git-side controls (branch protection, required reviews, CODEOWNERS on the environment directories). Argo CD enforces when and where; Git enforces who approved it.

CI/CD integration and promotion

The problem: Argo CD handles delivery from Git to cluster, but something has to get the new image into Git, and something has to move a change from staging to production. Getting this boundary wrong is how teams end up with either a CI job holding cluster credentials (defeating the pull model) or a release process nobody can reason about.

The clean division is: CI builds and tests, then commits; Argo CD deploys. CI never talks to the cluster. After building an image, CI updates a file in the deploy repository, such as the image tag in a Kustomize kustomization.yaml or a Helm values file, and pushes a commit. Argo CD sees the commit (by polling or webhook) and syncs. The Docker guide's advice on immutable tags and digests matters here, because you should commit a sha-… tag or digest, never latest.

name: release
on:
  push:
    branches: [main]
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          repository: acme/deploy-config
          token: ${{ secrets.DEPLOY_REPO_TOKEN }}
      - name: Bump image tag
        run: |
          cd payments/overlays/staging
          kustomize edit set image registry.example.com/shop/payments=registry.example.com/shop/payments:sha-${GITHUB_SHA::7}
      - name: Commit and push
        run: |
          git config user.name "release-bot"
          git config user.email "release-bot@acme.example"
          git commit -am "payments: staging -> sha-${GITHUB_SHA::7}"
          git push

Notice what the workflow does not have: no kubectl, no kubeconfig. To wait for the result, it may call the Argo CD API with a narrowly scoped token (the project role or ci-bot account from earlier), for example argocd app wait payments-staging --health --sync --timeout 300, using ARGOCD_AUTH_TOKEN. Keep that token to get and sync on one project.

Webhooks cut latency and Git load. Configure the provider to post to https://<argocd>/api/webhook with a shared secret stored in argocd-secret. Argo CD supports GitHub, GitLab, Bitbucket, Bitbucket Server, Gogs and Azure DevOps, with GHCR added in 3.5. Once webhooks are reliable, raise timeout.reconciliation so polling becomes a backstop rather than the main mechanism, and use webhook.refresh.jitter (3.5) to smooth a burst of refreshes after a large merge.

For monorepos, argocd.argoproj.io/manifest-generate-paths tells Argo CD which directories affect an application, so a commit elsewhere in the repository does not trigger a re-render:

metadata:
  annotations:
    argocd.argoproj.io/manifest-generate-paths: .;../shared

. means the application's own path, semicolons separate entries, and it works without webhooks since 2.11. It does not help with shallow clones (depth > 0).

Promotion across environments can be done three ways. The simplest is directory-per-environment in one repository (overlays/staging, overlays/prod), promoted by a pull request that copies the tag. A second is branch-per-environment, which tends to cause merge pain and is the pattern the hydrator's separate hydrated branches address in a cleaner way. A third is to use a tool built for it: the argoproj-labs GitOps Promoter integrates with the hydrator, and Argo CD Image Updater writes new tags back for you. Whichever you choose, keep promotions as reviewable Git changes.

Progressive delivery at the workload level is a different problem, handled by Argo Rollouts. Argo CD understands Rollout health and actions out of the box, so a canary that pauses shows as Suspended or Progressing rather than failing the sync. Keep the two concerns distinct: Argo CD decides which manifest version should exist; Rollouts decides how traffic moves to it.

Notifications close the loop. The notifications controller ships with Argo CD. Install the triggers and templates catalog, configure a service in argocd-notifications-cm, keep tokens in argocd-notifications-secret, and subscribe an application with an annotation:

kubectl apply -n argocd --server-side --force-conflicts \
  -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/notifications_catalog/install.yaml

metadata:
  annotations:
    notifications.argoproj.io/subscribe.on-sync-failed.slack: team-a-deploys
    notifications.argoproj.io/subscribe.on-health-degraded.slack: team-a-deploys

Test a template without waiting for a real event: argocd admin notifications template notify app-sync-succeeded guestbook.

Performance, scaling and cost

The problem: Argo CD works beautifully at twenty applications and fails in unexpected places at four hundred. To tune it you need to know which component is under pressure, and the answer is rarely "the UI".

The repo-server is the first thing to strain, because it forks helm and kustomize and clones repositories. Its symptoms are slow refreshes, Context deadline exceeded during reconciliation (manifest generation exceeding the controller's repo-server timeout, controller.repo.server.timeout.seconds, default 60), and out-of-memory kills. The levers are scaling it horizontally, limiting concurrency with reposerver.parallelism.limit (unset or 0 means unlimited, which is how it runs out of memory), mounting a real volume for /tmp when clones are large, and lengthening the exec timeout (ARGOCD_EXEC_TIMEOUT, default 90 seconds) only when you have a genuinely slow render.

The application controller is next. It scales by sharding clusters across StatefulSet replicas: raise replicas and set ARGOCD_CONTROLLER_REPLICAS to the same number. Pin a cluster to a shard with the shard field of its Secret. The default legacy algorithm hashes cluster UIDs and can distribute unevenly; round-robin and consistent-hashing are Alpha. Two constraints to internalise: sharding is per cluster, so one very large cluster cannot be split, and a controller's memory tracks the number of cached resources. argocd admin cluster shards and argocd admin cluster stats show how clusters landed.

Load on the controller is also reduced by defaults you should not undo without a reason: the default resource.exclusions, ignoreResourceUpdates and externally stored per-resource health (status.resourceHealthSource: appTree) were all introduced in 3.0 to cut controller work. If something on the exclusion list (Endpoints, Leases, policy reports) is not needed, leave it excluded; adding noisy kinds to a cluster cache costs memory and CPU for no benefit. Likewise, a namespaces entry on a cluster Secret makes the controller list and watch per namespace, which is expensive with many namespaces.

For large installations, other settings matter: ARGOCD_GRPC_MAX_SIZE_MB (default 200) above that for 3,000+ applications, ARGOCD_APPLICATION_TREE_SHARD_SIZE (suggested 100) to reduce Redis traffic, and ARGOCD_CLUSTER_CACHE_LIST_PAGE_BUFFER_SIZE if you see continue parameter is too old to display a consistent list result, which means a cluster list was slower than etcd's compaction window.

For high availability, use the HA manifests (manifests/ha/install.yaml), which need at least three nodes for pod anti-affinity. Run three or more argocd-server replicas and set ARGOCD_API_SERVER_REPLICAS to match so the concurrent-login limit divides correctly. Redis HA assumes exactly three servers and sentinels; Dex is in-memory, so keep it at one replica.

Cost. Argo CD is free, Apache-2.0 and a CNCF graduated project. What you pay for is compute and provider traffic: repo-server CPU and memory, controller memory, Redis memory, and Git API and egress load from polling. The levers that reduce it are the same ones that improve performance: webhooks with a longer poll interval, manifest-generate-paths, excluding high-churn kinds, capping repo-server parallelism, and, for single-admin clusters, the core install with no UI, API server or Dex. Where an instance serves teams in several regions, consider whether one management cluster per region keeps traffic and data local, at the price of more instances to upgrade.

Observability and debugging beyond the UI

The problem: the UI shows the final state but not the path to it. When an application misbehaves, you need to find which stage failed using logs, metrics and the CLI.

Each component exposes Prometheus metrics: the application controller on argocd-metrics:8082, the API server on argocd-server-metrics:8083, the repo-server on argocd-repo-server:8084, and the commit server on argocd-commit-server:8087. The metrics worth alerting on:

• argocd_app_info, with sync_status and health_status as labels. The older metrics argocd_app_sync_status, argocd_app_health_status and argocd_app_created_time were removed in 3.0, so dashboards from 2.x tutorials show nothing.
• argocd_app_sync_total, to count syncs and failures per app.
• argocd_cluster_connection_status, to detect a cluster Argo CD can no longer reach.
• argocd_git_request_total, split by ls-remote and fetch, to see what polling costs.
• argocd_app_reconcile, a histogram of comparison time.

Tracing is available with otlp.address (for example otel-collector:4317) in argocd-cmd-params-cm. Note that some OpenTelemetry metric names changed in 3.4, so check any dashboards built on them. Logs are JSON by default and can be switched with --logformat; per-component log levels are set in argocd-cmd-params-cm.

A debugging order that works for most problems:

A table of errors you will actually meet, with where to look:

Two more recoveries are worth having at your fingertips. If the admin password is lost, create a bcrypt hash with argocd account bcrypt --password X, patch it into argocd-secret as admin.password (and admin.passwordMtime), or delete both keys and restart argocd-server. And keep the CLI matched to the server, because since 3.0 Dex claim changes require the new CLI to obtain correct tokens.

Upgrades and the version traps

The problem: Argo CD ships a minor release every quarter (GA on the first Tuesday of February, May, August and November), and only the three most recent minor lines receive patches. On 30 September 2026 those are 3.5, 3.4 and 3.3. Standing still is not an option, and upgrading blind is how people discover the changes in this guide the hard way.

The rules from the upgrade overview: patch releases carry no breaking changes; minor releases "might introduce minor changes with a workaround"; majors break things. When you skip a minor, read every intermediate upgrade guide. Before any upgrade, take a backup:

argocd admin export -n argocd > backup.yaml

Always pass -n: the command does not fail if you point it at the wrong namespace, it just exports nothing useful. Then apply the full manifest set with --server-side --force-conflicts, not merely an image bump. --force-conflicts overwrites your local customisations of fields defined in the manifests (affinity, env, probes), so keep those in Kustomize patches or Helm values rather than editing the live objects. Upgrade the CLI with the server.

A compact list of what bites, by release:

• 3.3: server-side apply is mandatory for install and upgrade; the Source Hydrator moved to git notes and stopped cleaning paths; --self-heal-backoff-cooldown-seconds is deprecated. Helm 2 support is gone.
• 3.4: the cluster version format is vMajor.Minor.Patch (ApplicationSets selecting clusters by version must use argocd.argoproj.io/kubernetes-version in the new format); Missing semantics changed as described earlier; Dex 2.45 turns on ContinueOnConnectorFailure by default. Note that there is no v3.4.0 release: the first published 3.4 release is v3.4.1.
• 3.5: Helm 4 replaces Helm 3 in the image; the UI moved to React 19, so old UI extensions fail with Extension <name>.js failed to load until rebuilt with react/jsx-runtime externalised; impersonation now covers server operations; SSH host keys are validated against argocd-ssh-known-hosts-cm; event-listing gRPC methods return a new EventList type (REST is unchanged, and the CLI does not use them); --repo-server-strict-tls is deprecated in favour of --repo-server-ca-cert-path; GnuPG is replaced by Source Integrity. Optional repo-server mTLS turns on automatically when you create the Secret argocd-repo-server-mtls with client-ca.crt, client.crt and client.key.

From 2.x to 3.x, the table of default changes from earlier sections is your checklist. Test upgrades on a staging instance that holds a copy of your Applications, because the failures are almost all about semantics (tracking, RBAC, diffs), not about the install itself. A useful pre-flight is argocd admin settings validate -n argocd, which checks the settings objects for errors.

Putting it all together

Here is one coherent design that uses most of what you have read. A fintech in Riyadh runs a single hub Argo CD in a management cluster in its regional cloud, managing three workload clusters (dev, staging, prod) and keeping all data in-region.

1. Install and self-management. Install the HA manifests, pinned to v3.5.3 with --server-side --force-conflicts. A root Application manages Argo CD itself with ServerSideApply=true, and its source is a Kustomize base plus a patch directory, so local changes survive upgrades.
2. Identity. SSO with OIDC and url set. The local admin is disabled and argocd-initial-admin-secret is deleted. policy.default is empty; groups map to roles per team, each role includes logs, get; policies are tested in CI with argocd admin settings rbac validate.
3. Tenancy. One AppProject per team with explicit sourceRepos, destinations, and a narrow cluster resource whitelist. The default project is emptied. Applications live in team namespaces via sourceNamespaces. Impersonation maps each namespace to a per-team deployer service account.
4. Structure. A root app-of-apps installs platform ApplicationSets. A Cluster-generator ApplicationSet deploys the monitoring and policy addons to every cluster labelled by environment, with a create-update policy so a bad generator cannot delete applications. Application ApplicationSets use Git directory generators, one directory per service.
5. Delivery. CI builds the image, commits the sha-… tag to the staging overlay, and an argocd app wait step, using a project role token, confirms health. Promotion to prod is a pull request that copies the tag, requires two approvals via CODEOWNERS, and is protected by a deny sync window on Thursday afternoons (with manualSync for emergencies).
6. Safety. Automated sync with selfHeal; prune: true only in dev and staging, Prune=confirm for production namespaces. PreSync migration Jobs use BeforeHookCreation. Databases and namespaces carry Prune=false. Secrets come from External Secrets backed by an in-region vault.
7. Operations. Webhooks from the Git provider with timeout.reconciliation raised; manifest-generate-paths on every application in the monorepo; Prometheus alerts on argocd_app_info for Degraded and OutOfSync, on argocd_cluster_connection_status, and on repo-server memory. Notifications to a chat channel on sync failure and degraded health.
8. Change. A staging Argo CD instance rehearses each minor upgrade with a backup from production; the team reads every intermediate upgrade guide.

Every piece is a file in Git, and each decision has a reason you can now state. The exercise to try: write the AppProject, the RBAC policy and one ApplicationSet for a service of your own, and run the RBAC check and an argocd app diff --local against them before you commit.

Where you are now

You can describe how a commit becomes a live change, stage by stage, and say which cache to clear or which log to read when it does not. You can explain why an application is OutOfSync, silence phantom drift without hiding real drift, and choose between ignoreDifferences, server-side diff and IgnoreExtraneous. You can make sync safe with prune, self-heal, retry, options, hooks and waves; you can set boundaries with projects, tested RBAC, SSO and impersonation; and you can scale a pattern across clusters with ApplicationSets while keeping secrets out of Git and out of Redis. You can also tune a busy instance, read its metrics, and plan an upgrade with a backup and the right release notes.

What the Senior level adds is the platform view: the failure modes of each component under load, the security trust model in full, running Argo CD as a service for other teams with cost, upgrades and governance, disaster recovery playbooks, and the honest answer to when a different tool is the better fit. To keep building on this, the natural neighbours are Terraform for the clusters Argo CD deploys to, Kubernetes for the objects it applies, and Helm and Kustomize for the renderers that sit inside it.

Sources

• Argo CD documentation (stable)
• Core concepts
• Architecture
• Installation
• Declarative setup
• High availability
• Dynamic cluster distribution
• RBAC configuration
• User management and SSO
• Security
• Secret management
• Health checks
• Metrics
• Notifications
• Applications in any namespace
• Sync using impersonation
• ApplicationSet
• ApplicationSet generators
• Cluster bootstrapping
• Disaster recovery
• Upgrading overview
• Upgrading 3.4 to 3.5
• Upgrading 2.14 to 3.0
• Projects
• Automated sync policy
• Sync options
• Sync phases and waves
• Sync windows
• Diffing customization
• Diff strategies
• Resource tracking
• Multiple sources
• Source Hydrator
• Source Integrity
• Application deletion
• CI automation
• CLI reference
• FAQ
• Release process and cadence
• Argo CD v3.5.3 release

Part two of three. A cumulative review of Beginner and Mid-level material for Argo CD v3.5, organised as scenarios and trade-offs, in about thirty-five minutes. At this level an interviewer is not asking what Argo CD is. They are asking what you do when a sync hangs, how you would stop one team touching another team's namespace, and why you chose the option you chose. Senior reviews all three levels and adds system design.

Where this picks up

The thirty-second recap

Argo CD is a declarative GitOps controller for Kubernetes. It watches a Git, Helm or OCI source, renders the manifests, compares them with the live cluster, reports the difference as sync status, and optionally reconciles. It is pull-based: CI pushes commits, Argo CD pulls them, and CI needs no cluster credentials.

Keep four nouns ready. An Application is a source plus a destination plus a sync policy. A project is the tenancy boundary. A sync is the act of applying the target state. Health is whether the result is actually working. Everything else in this file hangs off one of those four.

How a change reaches the cluster

Interviewers like this one because the weak answer is "Argo CD checks Git." The strong answer names components and caches.

argocd-server is the stateless API and UI; it authenticates, enforces RBAC and receives webhooks. argocd-repo-server clones repositories and renders manifests (helm template, kustomize build, Jsonnet, plugins), and has no Kubernetes privileges. argocd-application-controller, a StatefulSet, keeps a watch-based cache of the live cluster, runs the comparison and executes syncs. Redis is a disposable cache; everything durable is in Kubernetes objects, so a lost Redis costs you warm caches and not data.

Refresh is triggered by polling (120 seconds plus up to 60 seconds of jitter by default) or a webhook to /api/webhook. A hard refresh also invalidates the repo-server's manifest cache. That is the tool for "I fixed the chart and it still shows the old error", because render failures are cached (Manifest generation error (cached)).

A strong answer adds: sync status and operation status are separate axes; the controller's memory scales with cached resources, not with application count; and the repo-server, not the UI, is usually the first thing to strain.

Scenario: "The app is OutOfSync and nothing changed"

Start by finding what is being compared. argocd app diff NAME shows the difference; its exit code is 0 for none, 1 for differences and 2 for an error. Then classify the diff.

• A controller owns the field. The classic case is an HPA changing spec.replicas. Fix with ignoreDifferences on /spec/replicas, and add RespectIgnoreDifferences=true so a later sync does not overwrite the field. Without that option the two tools fight, because ignoreDifferences only affects the comparison and not what a sync applies.
• An admission webhook or defaulting changes the object. Client-side diffing cannot predict this. Turn on Server-Side Diff (Stable since 3.1) with controller.diff.server.side: "true", or per app with the annotation argocd.argoproj.io/compare-options: ServerSideDiff=true.
• The manifest itself is malformed. Duplicate keys in a list, most often repeated env names, cause The order in patch list... doesn't match $setElementOrder list. Only a Git fix helps.
• Another tool writes app.kubernetes.io/instance. Under label tracking this causes fights. Since 3.0 the default is annotation tracking, argocd.argoproj.io/tracking-id, which avoids the collision.

A strong answer adds: what 3.0 changed. .status is ignored for all resources now, and ignoreDifferences also suppresses update events. A weak answer says "I'd click sync until it goes green", which hides the problem and, with self-heal on, may loop.

Scenario: "A sync deleted something it should not have"

Walk through the causes in order of likelihood.

1. Prune. With prune: true, anything no longer in the rendered output is deleted. A wrong path, a moved directory, or a renamed chart can make a whole namespace "no longer in Git". The default refuses an empty result (allowEmpty: false), which is the last line of defence.
2. The Application was deleted with a finalizer. resources-finalizer.argocd.argoproj.io makes deleting the Application cascade to its resources. Without it, deletion leaves the resources behind; with it, a tidy-up in the UI removes production.
3. An ApplicationSet generator returned nothing. A wrong Git path or a selector matching no clusters empties the parameter set, and the controller prunes every generated Application.
4. Replace=true. Issues a replace, or with Force a delete and recreate, which wipes state on anything with it.

Preventive measures, and the senior flourish: Prune=false on PVCs and namespaces as a resource annotation, Prune=confirm (approved with argocd app confirm-deletion) for production, PruneLast=true, and an ApplicationSet policy of create-update so the controller can never delete Applications. Argo CD's own orphaned-resource monitoring is a detective control, not a preventive one.

Scenario: "Deploys are slow, and the UI lags at 400 applications"

Reason about the pipeline stage by stage, using metrics before settings.

• Render time: argocd_app_reconcile and the repo-server's CPU and memory. Symptoms are Context deadline exceeded (the controller's repo-server timeout is 60 seconds by default), and out-of-memory kills. Scale the repo-server, cap concurrency with reposerver.parallelism.limit (unset means unlimited, which is how it runs out of memory), and give /tmp a proper volume for big clones.
• Git load: argocd_git_request_total by request type. Use webhooks plus a longer timeout.reconciliation, and webhook.refresh.jitter (3.5) to smooth bursts. In a monorepo, add argocd.argoproj.io/manifest-generate-paths so a commit only invalidates the applications it affects.
• Controller queues and memory: controller.status.processors (20) and controller.operation.processors (10); the docs suggest 50 and 25 for around a thousand apps. Shard the StatefulSet by cluster and set ARGOCD_CONTROLLER_REPLICAS to match. One huge cluster cannot be split.
• Don't undo the 3.0 defaults (resource exclusions, ignored resource updates, external health storage); they exist to cut controller work.

A strong answer starts with measurement and names the repo-server first. A weak answer says "add more replicas" without saying of what.

Scenario: "Team A can see and sync Team B's apps"

This is a tenancy question, and the answer has layers.

1. AppProject per team with explicit sourceRepos, destinations (server plus namespace glob), and a narrow clusterResourceWhitelist. Empty out the default project so that forgetting a project is refused.
2. RBAC in argocd-rbac-cm with policy.default empty, roles per team, and objects of the form <project>/<app>. Test with argocd admin settings rbac can role:team-a-dev sync applications 'team-b/x' --policy-file policy.csv.
3. SSO groups mapped to roles with g, <group>, <role>.
4. Impersonation (Beta) so that a sync into a team's namespace runs as a per-team service account, which in 3.5 also covers logs, delete, patch and actions.
5. Apps in any namespace with sourceNamespaces on the project if teams should own their Application objects.

Mention the 3.0 RBAC changes without being asked, because they are real traps: logs, get is now separate from applications, get; update and delete on an application no longer cascade to its resources; and policy.default access cannot be removed by a deny rule, so a default of role:readonly silently gives everyone read access.

A weak answer stops at "use projects". A strong answer distinguishes what Argo CD will apply, what a user may see or trigger, and what identity does the applying, and notes that a project is not a substitute for Git repository permissions.

Scenario: "Where do secrets go?"

The official position is to populate secrets on the destination cluster: Sealed Secrets, External Secrets Operator, the Secrets Store CSI Driver, or the Vault Secrets Operator. Argo CD then manages the reference object, not the value.

The docs strongly caution against injecting secrets at manifest generation time (for example a Vault plugin running as a config management plugin). Rendered manifests are cached in plaintext in Redis and served over the repo-server gRPC interface; secret rotation becomes coupled to syncs; and the approach is incompatible with the rendered-manifests pattern (the Source Hydrator).

A good follow-up to volunteer: Argo CD's own secrets (repo credentials, cluster credentials, argocd-secret) are Kubernetes Secrets, so etcd encryption and RBAC on the namespace matter. For regulated employers in the Gulf or Egypt, the detail worth adding is that the secret manager behind External Secrets can be in-region, and that choice is made in the store definition, not in Argo CD.

Scenario: "Design the path from merge to production"

A solid design has four beats.

1. CI builds and tests, pushes an immutable image tag (sha-…) or digest, then commits that tag to the deploy repository. CI has no kubectl and no kubeconfig.
2. A webhook refreshes Argo CD; polling is the backstop.
3. The staging Application syncs automatically. CI may block on argocd app wait NAME --health --sync using a token limited to one project.
4. Promotion to production is a reviewed Git change (pull request, CODEOWNERS), optionally gated by a sync window. Production uses Prune=confirm and manualSync for emergencies.

Trade-off on structure: directory-per-environment is simple and diffable, branch-per-environment causes merge pain, and the Source Hydrator (Beta since 3.5) renders manifests to a hydrated branch so reviewers see the exact YAML that will be applied. Name Argo CD Image Updater and GitOps Promoter as options, and say why you would still keep the final promotion as a pull request.

Trade-off questions

One shared Argo CD or one per team? One shared instance with a project per team is cheaper and gives a single view; it needs disciplined RBAC and gives a larger blast radius. An instance per team (namespace-install.yaml, which deploys only to external clusters and does not include CRDs) isolates failures and upgrades at the price of operating many instances. Several instances on one cluster need distinct installationID values and annotation tracking.

Automated or manual sync? Automated with selfHeal gives convergence and removes console drift; it also makes argocd app rollback unavailable, and any broken commit deploys immediately. Manual sync keeps a human in the loop at the cost of drift and slower delivery. Many teams automate dev and staging and keep production automated but gated by review, windows and Prune=confirm.

Helm or Kustomize as the renderer? Helm suits third-party charts and parameterised packages; it is rendered with helm template, so helm ls shows nothing. Kustomize suits in-house overlays with small per-environment patches. Argo CD supports both, plus multi-source apps that take a chart from one place and values from a Git repository.

Git polling or webhooks? Webhooks give near-instant refresh and lower Git load; polling is resilient when a webhook fails. Use both, with a longer polling interval.

Argo CD or Flux? Both are pull-based GitOps. Argo CD offers a rich UI, the Application and project model, and ApplicationSets; Flux is a set of composable controllers with a smaller footprint and no built-in UI. Compare the two in the Flux guide, and answer by team needs, not by preference.

What a strong answer covers that a weak one misses

• Names the component. "The repo-server renders, the controller compares", not "Argo CD checks".
• Separates the two status axes. Sync status versus operation status; sync status versus health.
• Distinguishes the comparison from the apply. ignoreDifferences changes one; RespectIgnoreDifferences changes the other.
• Uses the 3.x defaults. Annotation tracking, logs, get, server-side apply for install, Helm 4 in 3.5.
• Mentions safety controls before being asked. Prune settings, finalizers, allow-empty, generator policies.
• Keeps secrets out of Redis. Names the destination-cluster tools.
• Treats Git as the control plane. Reviews, CODEOWNERS and branch protection are part of the answer.

Commands to have ready

argocd app get NAME --hard-refresh                # invalidate the manifest cache
argocd app diff NAME --local ./path               # diff your working copy; exit 0/1/2
argocd app manifests NAME --source live           # what the cluster has
argocd app sync NAME --dry-run --prune            # see what a prune would do
argocd app wait NAME --health --sync --timeout 300
argocd app terminate-op NAME                      # cancel a stuck operation
argocd admin settings rbac can role:X sync applications 'proj/app' --policy-file policy.csv
argocd admin export -n argocd > backup.yaml       # always pass -n

Common interview questions

Q: What is the difference between refresh and hard refresh? A refresh re-compares the latest Git revision with the live state using the manifest cache. A hard refresh also invalidates the repo-server's cached manifests, so the application is re-rendered. Use it after fixing a cached render error or when a source the repo-server cannot track changes.

Q: Why does argocd app rollback fail? It is not allowed while automated sync is enabled, because automation would reapply Git straight away. Revert the commit in Git, which is the GitOps answer, or disable automation first.

Q: Why did the install fail with "Too long: may not be more than 262144 bytes"? Since 3.3 the ApplicationSet CRD is larger than the client-side apply annotation limit. The current command is kubectl apply -n argocd --server-side --force-conflicts -f <manifest>. A self-managed Argo CD needs ServerSideApply=true on its Application.

Q: What does self-heal do, and when is it a problem? It re-syncs when live state drifts without a Git change, reverting manual edits. It is a problem when another controller legitimately owns a field (an HPA, an operator), in which case you use ignoreDifferences plus RespectIgnoreDifferences=true, or when you need to hotfix in an incident; then you pause automation on the application.

Q: How do sync waves and hooks differ? Hooks attach a manifest, usually a Job, to a phase (PreSync, Sync, PostSync, SyncFail, plus PreDelete and PostDelete). Waves order resources within a phase by integer, with each wave needing to be healthy before the next. Waves are only as reliable as the health checks behind them.

Q: An Application is stuck Progressing. What do you check? Which resource is not healthy (argocd app get NAME). Usually it is a kind with no health logic or a legitimate wait, such as an Ingress without a load balancer address. Write a Lua check in resource.customizations.health.<group>_<kind>, and test it with argocd admin settings resource-overrides health.

Q: What changed in 3.0 that you would check first in an upgrade? Resource tracking moved from a label to an annotation (sync every app once afterwards); RBAC update and delete stopped inheriting to child resources; logs, get must be granted explicitly; resource.exclusions ships defaults; .status is ignored for all resources; and old metrics such as argocd_app_sync_status were removed.

Q: How do you give CI access without handing it the cluster? A project role or local account with apiKey capability, with a short-lived token limited to get and sync on one project. CI commits to Git and optionally waits on the application; it never holds a kubeconfig.

Q: How would you stop a deploy at certain times? A deny sync window on the project with a cron schedule and duration. Deny beats allow; selectors are OR-ed unless AND is selected; manualSync: true permits manual syncs; syncOverrun (3.5) lets a started sync finish. The application still reports OutOfSync during the window.

Q: What is new in 3.5 that you would mention? Helm 4 in the image (plain-HTTP OCI needs explicit opt-in, and the Helm version field is ignored), a React 19 UI that breaks old extensions, impersonation covering server operations, Source Integrity replacing GnuPG verification, optional repo-server mTLS, SSH host-key validation against argocd-ssh-known-hosts-cm, and the Source Hydrator moving to Beta.

Final self-test

Answer each in a sentence or two without looking back.

1. Which component renders manifests, and why does it have no Kubernetes privileges?
2. What does a hard refresh invalidate that a normal refresh does not?
3. Name two reasons an application is permanently OutOfSync and the fix for each.
4. Why does ignoreDifferences alone not stop a sync overwriting a field?
5. What are the three independent controls under automated?
6. What does BeforeHookCreation fix?
7. Why is policy.default: role:readonly more dangerous than it looks?
8. What is wrong with injecting secrets at render time?
9. What does the project's destinations list protect against?
10. Which command installs Argo CD 3.5 correctly, and what did it replace?
11. What are the two ways a generator can delete many Applications?
12. What is the difference between a sync window's deny and manualSync?

Part two of three. At this level your applications sync and your dashboards are green, so the problems change character. They are no longer "why won't this deploy" but "why does this application never stop being OutOfSync", "why did that sync delete a thing", and "why did the upgrade break our permissions". Start with the error table, then the practices, practice cards and recipes underneath it. Everything is written for Argo CD v3.5.

Common errors at this level

Cumulative: Beginner's errors still apply, and these are the ones that appear once things work.

The practices that pay off most

In rough order of leverage. The first three prevent most incidents, and the rest make the platform pleasant to live with.

1. Decide prune and finalizer behaviour per application, in writing. A short table in your platform repository (dev: prune on; prod: Prune=confirm; stateful namespaces: Prune=false) prevents the incident in which someone assumes the defaults. The defaults are safe only in that prune is off.
2. Build the project model before the second team arrives. Retrofitting restrictions onto applications that already deploy anywhere means breaking someone.
3. Put policies under test. RBAC, projects and ApplicationSets are all code.
4. Prefer one credential template per organisation. repo-creds Secrets with a GitHub App key mean one rotation, not dozens.
5. Pin everything. The Argo CD version in your install manifest, chart versions in multi-source apps, image tags as digests or sha- tags, and targetRevision for anything that is not meant to float.
6. Turn on Server-Side Diff early for clusters with admission webhooks, because it removes a whole class of phantom drift before anyone spends a day on it.
7. Alert on outcomes, not activity. Degraded, a failed sync, an unreachable cluster, and sustained OutOfSync are actionable; raw sync counts are not.
8. Rehearse upgrades. A staging instance plus argocd admin export turns each minor version into a small, dull event.

Practice cards

1. Watch the pipelineTail the repo-server log and run argocd app get NAME --hard-refresh. Identify the clone, the render and the cache write. Then break the path and see where the error appears.
2. Cause and cure phantom driftScale a Deployment with an HPA, observe OutOfSync, add ignoreDifferences, then force a sync and notice the overwrite. Add RespectIgnoreDifferences=true and repeat.
3. Try Server-Side DiffAnnotate an app with argocd.argoproj.io/compare-options: ServerSideDiff=true and compare argocd app diff before and after on a cluster with a mutating webhook.
4. Protect one resourceAnnotate a ConfigMap with argocd.argoproj.io/sync-options: Prune=false, remove it from Git, and confirm it is reported but not deleted.
5. Order a releaseWrite a PreSync migration Job with BeforeHookCreation and a CRD-then-resource pair in waves -1 and 0. Sync twice.
6. Write a health checkTake a custom resource, write a Lua script, and verify it with argocd admin settings resource-overrides health before pasting it into argocd-cm.
7. Lock down defaultEmpty its source repos and destinations, then try to create an app without a project. Read the refusal.
8. Test a policyWrite three roles, run rbac can for six subject and object pairs, and find the one you expected to be denied but was allowed.
9. Generate from clustersCreate an ApplicationSet with a Cluster generator, label a second cluster, and watch an Application appear. Break a parameter name with missingkey=error on.
10. Add a windowCreate a deny window covering now, push a change, and sync manually with manualSync allowed.

Making a safe production Application

Everything in this section is a recipe you can paste and adapt. The first is an Application that encodes safe defaults.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: payments-prod
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
  annotations:
    argocd.argoproj.io/manifest-generate-paths: .;../base
    notifications.argoproj.io/subscribe.on-sync-failed.slack: team-a-deploys
spec:
  project: team-a
  source:
    repoURL: https://github.com/acme/deploy-config.git
    targetRevision: main
    path: payments/overlays/prod
  destination:
    name: prod-eu
    namespace: team-a-payments
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    retry:
      limit: 5
      backoff:
        duration: 10s
        factor: 2
        maxDuration: 3m
    syncOptions:
      - CreateNamespace=true
      - PruneLast=true
      - ApplyOutOfSyncOnly=true
      - RespectIgnoreDifferences=true
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas

The reasoning behind each choice is what matters. The finalizer makes deletion cascade, which is what you want for an application that owns its namespace, but then restrict who may delete it. PruneLast means a replaced object stays until its successor is healthy. ApplyOutOfSyncOnly speeds up large applications, at the cost of needing one explicit full sync after tracking changes. RespectIgnoreDifferences pairs with the HPA exclusion. And the manifest-generate-paths annotation stops unrelated commits re-rendering this app.

To protect the pieces that must never be pruned, annotate them in Git:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: payments-data
  annotations:
    argocd.argoproj.io/sync-options: Delete=false,Prune=false
spec:
  accessModes: [ReadWriteOnce]
  resources:
    requests:
      storage: 50Gi

Debugging a failing sync, step by step

When a sync fails, avoid the urge to click around. Work the same way each time.

argocd app get payments-prod                     # conditions, last operation, per-resource status
argocd app history payments-prod                 # which revisions have been deployed
argocd app manifests payments-prod               # what the repo-server rendered
argocd app manifests payments-prod --source live # what is in the cluster
argocd app diff payments-prod                    # the difference; exit 1 means a diff
argocd app terminate-op payments-prod            # cancel a stuck operation before retrying

Then look in the right log. Render and Git errors are in the repo-server, sync and health errors in the application controller, and login or permission errors in the server:

kubectl logs -n argocd deploy/argocd-repo-server --tail=100
kubectl logs -n argocd statefulset/argocd-application-controller --tail=100
kubectl logs -n argocd deploy/argocd-server --tail=100

If the error is a permission failure on the target cluster rather than in Argo CD, check the events for the failing resource with kubectl describe, and, if you use impersonation, confirm the destination service account holds the verbs it needs. In 3.5 that includes get, patch, delete and list on events and get on pods/log, because server operations now run as that identity too.

A common final step is a stuck Terminating application. The cause is nearly always the cascade finalizer waiting for a resource that will not delete (a namespace with a stuck finalizer, a PVC still in use). Fix the underlying resource; removing the finalizer from the Application manually should be the last resort, because it orphans whatever was left.

Health checks that work

A custom health script that returns Progressing forever is worse than none, because it blocks waves and argocd app wait. Keep scripts defensive: check that obj.status exists, handle the "no conditions yet" case as Progressing, and return Degraded with the condition's message so the UI shows the reason.

data:
  resource.customizations.health.cert-manager.io_Certificate: |
    hs = {}
    if obj.status ~= nil and obj.status.conditions ~= nil then
      for _, c in ipairs(obj.status.conditions) do
        if c.type == "Ready" then
          if c.status == "True" then
            hs.status = "Healthy"
            hs.message = c.message
          else
            hs.status = "Progressing"
            hs.message = c.message
          end
          return hs
        end
      end
    end
    hs.status = "Progressing"
    hs.message = "Waiting for certificate"
    return hs

Test it offline against a saved object before it reaches a cluster:

kubectl get certificate api-tls -n team-a-payments -o yaml > cert.yaml
argocd admin settings resource-overrides health ./cert.yaml --argocd-cm-path ./argocd-cm.yaml

Two more health notes. Since 3.2, a CronJob whose last Job failed can make an application Degraded; if that is noise for you, annotate the CronJob with argocd.argoproj.io/ignore-healthcheck: "true". And since 3.4, an application is Missing only when all its resources are missing, so scripts that check for Missing should check OutOfSync instead.

RBAC recipes

Policies are easiest to keep correct if you start from a small, tested template and extend it. A developer role that can see, sync, read logs and restart deployments, but not edit or delete:

p, role:dev, applications, get, */*, allow
p, role:dev, applications, sync, */*, allow
p, role:dev, logs, get, */*, allow
p, role:dev, applications, action/apps/Deployment/restart, */*, allow
p, role:dev, projects, get, *, allow
g, acme-developers, role:dev

Scope it to a project by replacing */* with team-a/*. Validate and probe it before applying:

argocd admin settings rbac validate --policy-file policy.csv
argocd admin settings rbac can role:dev sync applications 'team-a/payments-prod' --policy-file policy.csv
argocd admin settings rbac can role:dev delete applications 'team-a/payments-prod' --policy-file policy.csv

Remember four things when writing these: every fine-grained object needs all four slash-separated parts because glob matching does not treat / as a separator; the scopes key in argocd-rbac-cm decides which token claim carries group names (default [groups], use '[groups, email]' to match on email); policy.matchMode can be switched from glob to regex; and policy.default access cannot be removed by a deny.

ApplicationSet recipes

A few patterns cover most needs. One application per directory of services:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: services
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  syncPolicy:
    applicationsSyncPolicy: create-update
  generators:
    - git:
        repoURL: https://github.com/acme/deploy-config.git
        revision: main
        directories:
          - path: services/*
  template:
    metadata:
      name: '{{.path.basename}}'
    spec:
      project: team-a
      source:
        repoURL: https://github.com/acme/deploy-config.git
        targetRevision: main
        path: '{{.path.path}}'
      destination:
        name: prod-eu
        namespace: 'team-a-{{.path.basename}}'
      syncPolicy:
        automated:
          selfHeal: true
        syncOptions:
          - CreateNamespace=true

Notice create-update: the controller may create and update Applications but never delete them, so an empty generator result cannot wipe the fleet. If you need removal, handle it as a reviewed step. Three habits keep ApplicationSets healthy. Change the ApplicationSet rather than editing a generated Application, since the controller overwrites direct edits. Use argocd appset get NAME to see what it produced and any errors. And treat a generator change as a production change, because it can create or remove many applications at once.

Keeping the platform healthy

Some recurring hygiene pays for itself.

• Watch the repo-server. Its CPU and memory are the first indicator of trouble. Put reposerver.parallelism.limit in place before you need it.
• Review the noisy kinds. If a resource kind changes constantly and nobody needs it in Argo CD (Endpoints, Leases, policy reports), keep the default resource.exclusions in place.
• Rotate tokens. Project role tokens have no expiry by default; always create them with -e. Local apiKey tokens from argocd account generate-token accept -e as well.
• Retire the admin. After SSO works, set admin.enabled: "false" and delete argocd-initial-admin-secret.
• Keep the CLI current. Since 3.0 the Dex claim changes need the new CLI for correct tokens.
• Back up. argocd admin export -n argocd > backup.yaml on a schedule, run with the image version that matches your server. The export does not fail against the wrong namespace, so always pass -n.
• Patch. Only three minor lines receive fixes. On 30 September 2026 those are 3.5, 3.4 and 3.3, and 3.6 GA is scheduled for 3 November 2026.

Upgrading without drama

A checklist that has worked on real instances:

1. Read every intermediate upgrade guide for the versions you are crossing, not only the last one.
2. Export a backup with argocd admin export -n argocd.
3. On a staging instance holding copies of your applications, apply the new manifests with --server-side --force-conflicts and watch for refusals, ComparisonError and permission failures.
4. Keep customisations (affinity, env, probes) in Kustomize patches or Helm values, because --force-conflicts overwrites local edits to fields the manifests define.
5. After a 2.x to 3.x move, sync every application once to complete the tracking-annotation migration, and use an explicit full sync for apps with ApplyOutOfSyncOnly=true.
6. Upgrade the CLI alongside the server.
7. Afterwards, run argocd admin settings validate -n argocd and look at dashboards for removed metrics.

For the 3.5 line specifically, inventory your Helm charts for Helm 4 compatibility, your UI extensions for React 19, your OCI registries for plain HTTP, your SSH repositories for host keys in argocd-ssh-known-hosts-cm, and any scripts that use argocd proj add-signature-key or --repo-server-strict-tls, which are deprecated.

Habits worth adopting now

• Read argocd app diff before you click sync, every time.
• Hard refresh before you debug a render error.
• Treat ignoreDifferences as a scalpel: one kind, one field, one reason in a comment.
• Give every Application a project, an owner annotation and a notification subscription.
• Put Applications, projects, RBAC and ApplicationSets in Git, reviewed like application code.
• Keep prune off in production until you have seen it dry-run, with argocd app sync NAME --dry-run --prune.
• Prefer immutable image references in the deploy repository, and never commit latest.
• Keep secrets out of the render path.
• Write down which choices you made and why; Argo CD has enough options that the next person cannot infer them.
• Point teammates at the neighbouring guides when their question is really about the renderer: Helm and Kustomize for templating, Kubernetes for what gets applied.

Checklist before you call an Argo CD setup production-ready

• Installed from a pinned manifest or chart version with server-side apply, in HA mode if it is shared.
• SSO working, url set, local admin disabled, initial admin secret deleted.
• policy.default empty; roles per team, including logs, get; policies tested in CI.
• One project per team; default project emptied.
• Repository credentials as templates; SSH host keys and TLS CAs registered.
• Prune, finalizers and ApplicationSet policies chosen deliberately and documented.
• Health checks for every custom kind that participates in sync waves.
• Secrets supplied by an on-cluster operator, never by render-time injection.
• Webhooks configured, manifest-generate-paths set for monorepos.
• Metrics scraped, alerts on Degraded, failed syncs and unreachable clusters.
• Backups scheduled; an upgrade rehearsal path exists.

This is part three of three, and it is written for the person who owns Argo CD as a platform that other teams depend on. The questions change. Earlier you asked how to make an Application sync; now you ask what happens to forty teams when the controller runs out of memory, who is allowed to deploy into which namespace, what an attacker gains by stealing one token, and how you upgrade a system that deploys everything else, including itself.

The stakes are unusual for a deployment tool. Argo CD holds credentials to every cluster it manages, it can create and delete anything in them, and it reads from repositories that many people can write to. A mistake in its configuration is rarely a failed deploy; it is a tenant deleting another tenant's namespace, or an outage that stops every team from shipping at the same time. This guide is about designing so those outcomes are hard to reach. It targets Argo CD v3.5.3, the current stable release at the time of writing (3.5 went GA in August 2026, and 3.6 is scheduled for November 2026, so treat 3.6 features as not yet stable).

Where this picks up

I begin with the architecture, because every failure mode in this guide is easier to predict once you know which process does which job and what state each one keeps.

Architecture and failure modes

Argo CD is several cooperating processes, and all of them run from one container image (quay.io/argoproj/argocd:v3.5.3). What differs is the command and the job. Knowing the job of each tells you what stops working when it dies.

argocd-server is the front door: a gRPC and REST API plus the web UI. It authenticates callers by JWT, enforces RBAC, holds no durable state, and receives webhooks. Because it is stateless you scale it by adding replicas. When it fails, people cannot log in or use the UI and CLI, and webhooks stop arriving, but deployments that are already automated keep reconciling, because the controller does not depend on the server. That distinction is worth saying to your tenants before an incident rather than during one.

argocd-repo-server clones repositories and renders manifests by running helm template, kustomize build, Jsonnet, or a Configuration Management Plugin sidecar. It has no Kubernetes privileges by design, which makes it the component that touches untrusted input (other people's charts and overlays) with the least authority. It is also the main consumer of CPU and memory, because it forks tool processes. It clones into /tmp (or TMPDIR), and caches rendered manifests in Redis for 24 hours by default. When it is slow or down, the controller cannot compute new desired state: apps show comparison errors, Context deadline exceeded appears in controller logs, and new commits stop taking effect, while the last rendered state stays live in the cluster.

argocd-application-controller is a StatefulSet, and it is the heart. It keeps a cache of the live resources in every cluster it manages, compares them with the rendered target state, updates the Application status, and performs syncs and hooks. It is the component that uses memory in proportion to how many resources you manage, and it is the one you shard. When it is down, nothing reconciles and nothing syncs, but workloads keep running, because Argo CD is not in the data path of your applications. An overloaded controller is subtler: reconciliation queues back up and status goes stale while everything looks alive.

argocd-applicationset-controller turns ApplicationSets into Applications. It is a separate Deployment, so an outage there stops generation and updates of generated Applications but leaves existing ones running. argocd-notifications-controller sends messages; its failure is silent unless you alert on it. argocd-dex-server is the bundled SSO broker, in memory, so you run a single replica and accept that a restart drops its sessions. argocd-redis is, per the docs, a cache and nothing more. The optional commit server, present only in the *-install-with-hydrator.yaml manifests, writes rendered manifests to Git for the Source Hydrator.

A useful exercise is to write down, for each component, three things: what it stores, what it talks to, and what your tenants notice when it is gone. The answers for the usual suspects:

The last row contains an important design property: Argo CD degrades to a static cluster, not a broken one. Design your incident communication around that. "Deploys are paused" is a very different message from "production is down".

One more architectural decision you will own is where Argo CD runs. The hub-and-spoke model puts one Argo CD in a dedicated management cluster that deploys to many workload clusters. It centralises credentials and audit and is the usual platform choice. The alternative is an instance per cluster or per team, which bounds blast radius at the price of operating many control planes. The namespace-install.yaml variant is built for the latter: it is namespace-scoped, deploys only to external clusters unless you grant more, and does not include the CRDs, which you must install separately with server-side apply. The decision is revisited in the multi-tenancy section; for now, note that hub-and-spoke concentrates risk in one place, and most of this guide is about managing that concentration.

Scaling: which component breaks first

Argo CD at fifty applications is a solved problem. At several thousand applications across dozens of clusters it is a capacity-planning exercise, and the components fail in a predictable order. The skill is knowing the order, because tuning the wrong component wastes a week.

The repo-server usually saturates first. Every refresh of an Application whose manifests are not in the cache requires a render. A commit to a monorepo can invalidate every Application that points into it, and each render forks Helm or Kustomize. The symptoms are high CPU and memory spikes on repo-server pods, OOM kills, and Context deadline exceeded in the controller, because manifest generation exceeds the controller's repo-server timeout (60 seconds by default, controller.repo.server.timeout.seconds). The levers, in order of how much they buy:

1. Scope what a commit invalidates. The annotation argocd.argoproj.io/manifest-generate-paths tells Argo CD that an Application's manifests depend only on certain paths, so a commit elsewhere in the monorepo does not trigger a re-render. Paths are semicolon-separated and . means relative to the source path. Two caveats from the docs: it does not help when you use shallow clones (depth > 0), and you should use fully qualified refs such as refs/heads/main in large monorepos so resolution is cheap.
2. Cap concurrency. reposerver.parallelism.limit (unset or 0 means unlimited) bounds simultaneous manifest generations. Without it, a burst of refreshes becomes a burst of forked processes and an OOM kill; with it, renders queue instead of exploding.
3. Add replicas and give /tmp a real volume. Many or large repositories need disk. Scaling repo-server horizontally spreads the work, at the cost of each replica holding its own clone cache.
4. Raise timeouts only after the above. ARGOCD_EXEC_TIMEOUT (90 seconds for Helm and Kustomize) and the controller's repo-server timeout hide a slow render rather than fixing it.

Two monorepo behaviours deserve a warning. Kustomize applications that use parameter overrides in the same repository serialize, and the docs list no workaround. Configuration Management Plugins need either the .argocd-allow-concurrency marker file or the sidecar pattern to render concurrently.

The application controller is next, and it is a memory problem first. Its cache holds the live state of every watched resource in every cluster it is responsible for, so its memory tracks the number of objects, not the number of Applications. Two things follow. First, a single huge cluster cannot be split: sharding divides work by cluster, never by application or by namespace within a cluster. Second, the defaults shipped in 3.0 (the default resource.exclusions for high-churn kinds such as Endpoints, EndpointSlice and Lease, the default ignoreResourceUpdates, and storing per-resource health outside the Application CR) exist specifically to lower controller load. Do not remove them without measuring.

Sharding is how you scale the controller horizontally. You raise the StatefulSet replica count and set ARGOCD_CONTROLLER_REPLICAS to the same number; clusters are distributed across replicas. The algorithm is set with controller.sharding.algorithm. The default legacy hashes the cluster UID and can distribute unevenly; round-robin and consistent-hashing are both marked Alpha in the fact base for 3.5. You can pin a cluster to a specific shard with the shard field in its cluster Secret, which is how you give one monster cluster a replica to itself. Dynamic cluster distribution (ARGOCD_ENABLE_DYNAMIC_CLUSTER_DISTRIBUTION) is also Alpha. You can inspect the result with argocd admin cluster shards and argocd admin cluster stats.

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cmd-params-cm
  namespace: argocd
  labels:
    app.kubernetes.io/name: argocd-cmd-params-cm
    app.kubernetes.io/part-of: argocd
data:
  controller.status.processors: "50"      # default 20; docs suggest ~50 for ~1000 apps
  controller.operation.processors: "25"   # default 10; docs suggest ~25 for ~1000 apps
  controller.sharding.algorithm: round-robin   # Alpha in 3.5; default is legacy
  reposerver.parallelism.limit: "10"      # unlimited by default; bound it to avoid OOM
  controller.repo.server.timeout.seconds: "120"

The two processor settings split the controller's work: status processors run reconciliation (comparing state and updating status) and operation processors run syncs. A deployment that feels "slow to notice Git changes" is starved of status processors; one where "sync takes ages to start" is starved of operation processors. Raise them together with CPU, since more goroutines on too little CPU just adds contention. Change a value, restart the affected workload (cmd-params are read at start), and compare argocd_app_reconcile histograms before and after.

The API server and Redis come later. The API server is stateless and scales with replicas; set ARGOCD_API_SERVER_REPLICAS to match, because that variable divides the concurrent-login limit. At several thousand applications you may need ARGOCD_GRPC_MAX_SIZE_MB above its default of 200, because the application list response grows with the estate. Redis traffic can be reduced with ARGOCD_APPLICATION_TREE_SHARD_SIZE=100. Redis HA (Sentinel plus HAProxy) assumes exactly three Redis servers and sentinels; do not run two.

Do not forget the Git provider and the Kubernetes API. Every poll is a Git request (visible in argocd_git_request_total, labelled ls-remote or fetch), and thousands of Applications polling every two to three minutes can hit provider rate limits long before Argo CD itself strains. The fix is structural: configure webhooks so Git pushes trigger refreshes, then raise timeout.reconciliation so polling becomes a safety net rather than the main mechanism. Since 3.5, webhook.refresh.jitter spreads the refresh burst that a webhook for a busy monorepo would otherwise cause. On the Kubernetes side, a cluster Secret with namespaces set triggers one list-and-watch per namespace, which is expensive with many namespaces; and in very large clusters the error continue parameter is too old to display a consistent list result means initial listing is slower than the etcd compaction window, so you raise ARGOCD_CLUSTER_CACHE_LIST_PAGE_BUFFER_SIZE.

The trust model

Security for Argo CD starts with an uncomfortable sentence: write access to a repository that Argo CD watches is deploy access to the clusters it targets. Everything else in this section is about narrowing what that sentence permits, and about protecting the credentials that make it true.

There are four trust relationships to keep straight.

1. Who can call Argo CD. Authentication is by JWT. The local admin account obtains a token from /api/v1/session; that token expires after 24 hours and is revoked when the password changes. SSO comes from the bundled Dex or from direct OIDC (oidc.config). Project roles issue tokens scoped to one project, and local accounts with the apiKey capability can hold API tokens for automation. After SSO works, disable the admin account with admin.enabled: "false" in argocd-cm and delete the argocd-initial-admin-secret. An admin login that nobody uses is a credential that nobody is watching.
2. What a caller may do. Authorization is Casbin RBAC in argocd-rbac-cm plus project roles. Set policy.default to the minimum, and use an empty string for no default access. A trap worth memorising: access granted by policy.default cannot be removed by a deny rule, so a generous default cannot be fixed by adding denies later.
3. What Argo CD may do to clusters. argocd cluster add installs a service account named argocd-manager in kube-system with a ClusterRole that is effectively cluster-admin. The docs say you can narrow create, update and delete, but Argo CD needs cluster-wide get, list and watch to build its cache. So by default the credential in the management cluster is a master key to every spoke.
4. What Argo CD will render and apply. The repo-server executes Helm, Kustomize and plugins against repository content. Kustomize remote bases and Helm dependencies can reach outside the repositories you allow. If you do not use a tool, turn it off (kustomize.enable, helm.enable, jsonnet.enable set to false).

The 3.x releases moved several of these defaults toward safety, and you should know them because old tutorials teach the opposite. Since 3.0, RBAC update and delete on an Application no longer inherit to its managed resources, so a developer who can update an app can no longer automatically delete its pods; fine-grained rights must be granted explicitly (for example delete/*/Pod/*/*, always with all four slashes, because glob matching does not treat / as a separator). Log access is also separate: p, role:x, logs, get, */*, allow must be granted explicitly. The web terminal stays off unless exec.enabled: "true" and exec, create RBAC are both set.

p, role:team-a-dev, applications, get,    team-a/*, allow
p, role:team-a-dev, applications, sync,   team-a/*, allow
p, role:team-a-dev, logs,         get,    team-a/*, allow
p, role:team-a-dev, applications, action/apps/Deployment/restart, team-a/*, allow
g, my-org:team-a-developers, role:team-a-dev

Test policy changes offline before they reach the cluster, which is the single best habit for RBAC:

argocd admin settings rbac validate --policy-file policy.csv
argocd admin settings rbac can role:team-a-dev sync applications 'team-a/guestbook' --policy-file policy.csv
argocd admin settings rbac can role:team-a-dev delete applications 'team-b/payments' --policy-file policy.csv

The first command checks syntax; the second should print Yes; the third should print No. A policy file that cannot demonstrate a refusal has not been tested.

Impersonation narrows the second relationship. Without it, a sync runs with the application controller's own (very broad) permissions, so the only barrier between tenant A's Application and tenant B's namespace is the AppProject. With application sync impersonation (still Beta), enabled by application.sync.impersonation.enabled: "true" in argocd-cm, each destination maps to a service account through AppProject.spec.destinationServiceAccounts, and Argo CD performs the sync as that service account. Kubernetes RBAC then enforces the boundary, independently of Argo CD's own checks. In 3.5 this grew: impersonation now covers server operations (logs, events, delete, patch, resource actions) and not only syncs, and application.sync.impersonation.enforced defaults to true. That means the tenant service accounts need extra permissions, such as get, list, patch and delete on the relevant resources, get on pods/log, and access to events. Forgetting this is a common 3.5 upgrade surprise: tenants suddenly cannot see logs or delete pods.

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: team-a
  namespace: argocd
spec:
  description: Team A workloads
  sourceRepos:
    - https://github.com/example-org/team-a-*
  destinations:
    - server: https://kubernetes.default.svc
      namespace: team-a-*
  clusterResourceWhitelist: []          # no cluster-scoped kinds
  namespaceResourceBlacklist:
    - group: ""
      kind: ResourceQuota
  destinationServiceAccounts:
    - server: https://kubernetes.default.svc
      namespace: team-a-*
      defaultServiceAccount: team-a-deployer
  sourceNamespaces:
    - team-a-apps

Transport security is the last piece. Components talk over TLS, and you can enforce a floor with --tlsminversion 1.2. Redis is password-protected but plaintext unless you configure otherwise. Since 3.5 you can turn on optional mutual TLS for the repo-server simply by creating a Secret named argocd-repo-server-mtls with client-ca.crt, client.crt and client.key; it is mounted at /app/config/reposerver/mtls and cannot be combined with --disable-tls. The old --repo-server-strict-tls flag is deprecated in favour of --repo-server-ca-cert-path; it still works in 3.5 but may disappear in 3.6 or later, so migrate now while it is a warning and not an outage.

Finally, patch cadence is a security control. Only the three most recent minor lines (3.5, 3.4 and 3.3 at the time of writing) receive patch releases, and the patches are frequent. A platform pinned to an EOL minor is a platform with no path to a fix. Watch the GitHub Security Advisories for argoproj/argo-cd and keep your budgeted upgrade capacity real.

Secrets and the supply chain

Two questions dominate here: where do application secrets live, and how do you know the thing Argo CD deployed is the thing someone reviewed. Both have an official answer in the docs, and both have popular alternatives that the docs explicitly warn against.

Application secrets belong on the destination cluster. The documented recommendation is to populate Secrets inside the target cluster using a tool built for it: Sealed Secrets, External Secrets Operator, the Secrets Store CSI Driver, or the Vault Secrets Operator. In that design Git contains either encrypted material or a reference (an ExternalSecret that names a path in a secret manager), and a controller inside the cluster resolves it. Argo CD only ever moves the reference.

The alternative is to have Argo CD fill in secrets while it renders manifests, typically with a plugin such as argocd-vault-plugin running as a CMP sidecar. The official secret-management page "strongly cautions" against this, and the reasons are architectural rather than stylistic:

• Rendered manifests are cached in Redis, so the secrets end up in plaintext in Redis, and they are exposed through the repo-server's gRPC API.
• It couples secret rotation to syncs: a rotated value does nothing until something re-renders.
• It is incompatible with the rendered-manifests pattern, because the Source Hydrator would commit the secret values to Git.

Argo CD's own secrets need the same discipline. Repository credentials, cluster credentials and argocd-secret (the admin password hash, the server signing key, webhook secrets and SSO client secrets) are ordinary Kubernetes Secrets. Enable etcd encryption at rest in the management cluster, restrict who can get secrets in the argocd namespace, and remember that a person with that permission holds every cluster credential. The setting resource.sensitive.mask.annotations hides annotations that may carry sensitive data in the UI and diffs, which is worth setting for any operator that stores material in annotations.

Git credentials deserve a least-privilege design. Use credential templates (repo-creds Secrets) that match a URL prefix, so one deploy identity with read-only access covers an organisation and is rotated in one place. For GitHub, a GitHub App (via argocd repocreds add ... --github-app-id ... --github-app-private-key-path) is preferable to a personal access token, because it is not tied to a person who will eventually leave. The hydrator needs a separate repository-write Secret, which you should scope to the single branch or repository it writes to.

The supply chain has four distinct links, and platform owners tend to secure only one.

1. The Argo CD release itself. Release assets are signed with cosign and carry SLSA Level 3 provenance (argocd-cli.intoto.jsonl), plus an SBOM and checksums. Verify the CLI you distribute to engineers with slsa-verifier against the provenance file, and pin the container image by digest in your own manifests.
2. The manifests Argo CD renders. Git review is your control. Protect the branches Argo CD tracks, require review, and treat the CODEOWNERS file of a deployment repository as a security boundary.
3. The sources themselves. Since 3.5 there is a first-class feature called Source Integrity, configured on the AppProject under spec.sourceIntegrity. It replaces the older GnuPG signature verification: spec.signatureKeys, argocd proj add-signature-key and remove-signature-key are deprecated and only kept for compatibility with a logged warning. With Source Integrity you attach policies to repositories by URL glob (a ! prefix negates a match) and require signed commits at a chosen strictness: gpg.mode of none, head or strict, with the trusted keys. When a criterion fails the Application cannot sync and reports a ResourceComparison error, and local syncs are blocked when it is enforced.
4. The images in the manifests. Argo CD does not verify image signatures. That is the job of an admission controller such as Kyverno or an OPA Gatekeeper policy in the workload cluster, so that a signed-image requirement holds no matter how the manifest arrived.

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: payments
  namespace: argocd
spec:
  sourceRepos:
    - https://github.com/example-org/payments-deploy
  destinations:
    - server: https://kubernetes.default.svc
      namespace: payments
  sourceIntegrity:
    git:
      policies:
        - repos:
            - url: https://github.com/example-org/*
          gpg:
            mode: head
            keys:
              - 4AEE18F83AFDEB23

Check the sourceIntegrity schema against the v3.5 source-integrity documentation before copying this: the feature is new in 3.5 and the key layout is the part most likely to be refined. Also be honest about what head mode proves: it verifies the signature of the commit at the head of the revision being deployed, not every commit in its history, so it answers "who approved this deploy" rather than "is every historical commit trusted". strict checks more.

A neighbouring threat is Git as an injection channel into the repo-server. A Helm chart with a dependency on an unapproved repository, or a Kustomize remote base pointing to an arbitrary URL, makes the repo-server fetch and execute content that your sourceRepos allow-list never saw. Since 3.5 Helm is Helm 4, and plain-HTTP OCI registries require explicit opt-in (insecureOCIForceHttp: "true" in the repo Secret, or --insecure-oci-force-http on argocd repo add), which is a small improvement: the insecure path is now a deliberate choice. Remember the documented edge case: setting both --insecure-skip-server-verification and --insecure-oci-force-http makes Helm 4 drop --plain-http, and you get http: server gave HTTP response to HTTPS client with no workaround.

Multi-tenancy

Multi-tenancy in Argo CD is a design you choose, and the choice has three shapes. Each trades isolation against operating cost.

The shared-instance model is the default for platform teams, and to make it safe you combine four mechanisms.

AppProject is the boundary. It restricts source repositories, destinations (cluster and namespace pairs), which cluster-scoped and namespace-scoped kinds may be deployed, which namespaces may host Applications (sourceNamespaces), sync windows, project roles and orphaned-resource monitoring. A team's project should be the smallest set that lets them do their job: their repositories, their namespaces, no cluster-scoped kinds except what they demonstrably need. Cluster-scoped kinds are the classic escalation route, because a tenant that can create a ClusterRoleBinding can grant itself anything; keep clusterResourceWhitelist empty for tenants and give cluster-scoped changes to the platform team's own project.

Apps in any namespace lets teams keep Application resources in their own namespaces instead of the argocd namespace. You enable it with application.namespaces in argocd-cmd-params-cm, list the permitted namespaces in each project's sourceNamespaces, and grant argocd-server the extra ClusterRole from the examples/k8s-rbac/argocd-server-applications example in the repository. Applications in other namespaces are addressed as <namespace>/<name>, and RBAC objects become <project>/<app-namespace>/<app>. The benefit is that Kubernetes RBAC on the team's namespace decides who can create Applications there. The documented error when you forget a step is application '<app>' in namespace '<ns>' is not permitted to use project '<p>', and its fix is almost always a missing sourceNamespaces entry or a namespace missing from application.namespaces.

SSO groups to project roles. Map groups from your identity provider to Argo CD roles rather than creating local accounts. Since 3.0 the Dex RBAC subject is federated_claims.user_id rather than sub, which silently breaks policies copied from old guides. Watch the cookie limit: a user in many groups can produce a JWT larger than the 4 KB cookie limit, producing invalid cookie, longer than max length 4093. Scope the groups your identity provider sends, or use the scopes setting in argocd-rbac-cm deliberately. Entra ID group overflow is handled through Microsoft Graph as of 3.5.

ApplicationSets need their own guardrails. An ApplicationSet is an Application factory, and in a shared instance it is a privilege amplifier, since one object can create hundreds of Applications in any project it names. Three rules:

1. Keep ApplicationSets in the platform team's namespace unless you have deliberately enabled them elsewhere (applicationsetcontroller.namespaces), and then pair that with an SCM provider allow-list (applicationsetcontroller.allowed.scm.providers), because a generator that calls an arbitrary URL is a server-side request forgery vector.
2. Let the template set project from a controlled source, never from tenant-writable input, or a tenant can nominate the default project or another team's.
3. Use the ApplicationSet's syncPolicy.applicationsSyncPolicy (for example create-only or create-update, so a removed element does not delete an Application) to stop an accidental generator change from mass-deleting workloads.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: tenant-baseline
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  syncPolicy:
    applicationsSyncPolicy: create-update   # never delete Applications on generator changes
  generators:
    - git:
        repoURL: https://github.com/example-org/platform-tenants
        revision: HEAD
        directories:
          - path: tenants/*
  template:
    metadata:
      name: '{{.path.basename}}-baseline'
    spec:
      project: '{{.path.basename}}'
      source:
        repoURL: https://github.com/example-org/platform-tenants
        targetRevision: HEAD
        path: '{{.path.path}}'
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{.path.basename}}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

Remember that changing syncPolicy.automated on a generated Application has no lasting effect: the ApplicationSet owns it, so change the template.

Progressive syncs (the ApplicationSet spec.strategy) let you roll a change across generated Applications in stages, for example a canary cluster group before the rest. The feature is Beta and off by default; enable it with applicationsetcontroller.enable.progressive.syncs. It is the right tool for fleet rollouts, but read its documented limits before relying on it for a safety-critical path.

The second and third tenancy shapes come into play when isolation has to be absolute. Running one instance per team with namespace-install.yaml means a compromise of one control plane yields only that team's clusters. When several instances manage one cluster, set a distinct installationID in argocd-cm and keep annotation-based resource tracking, so each instance recognises only its own resources.

Sync, drift and diffing at scale

Mid-level usage treats diffing as a display feature. At platform scale it is the main source of noise, load, and false alarms, and you own its configuration for everyone.

Argo CD decides OutOfSync by comparing the rendered target manifests to the live objects. There are three strategies. The legacy three-way diff uses the last-applied-configuration annotation and is the default. Structured-merge diff was discontinued. Server-Side Diff, Stable since 3.1, asks the Kubernetes API server to compute the result of applying the desired state through a dry run, so defaulting and mutating admission webhooks take part in the comparison. You enable it globally with controller.diff.server.side: "true" in argocd-cmd-params-cm, or per Application with the annotation argocd.argoproj.io/compare-options: ServerSideDiff=true. For platforms that run Kyverno or other mutating webhooks, it removes a whole class of permanently OutOfSync applications. Its documented caveat is the error field not declared in schema, which appears when Argo CD's static schema lacks a field, for example with managedFieldsManagers ignore rules or server-side apply without server-side diff; upgrading Argo CD or avoiding those combinations resolves it.

Resource tracking is the other shared setting with broad consequences. Since 3.0 the default is the annotation argocd.argoproj.io/tracking-id, which ends the well-known collisions with tools that also write app.kubernetes.io/instance (Helm charts do). If you are migrating from 2.x, sync every application once after switching, because the documented edge case is that a resource deleted during the first sync after the switch can be orphaned, and applications with ApplyOutOfSyncOnly=true need an explicit full sync.

Ignoring differences is a platform policy, not a per-team hack. A team that adds ignoreDifferences for /spec/replicas because an autoscaler owns it is doing the right thing; the pitfall is forgetting that ignoreDifferences only affects comparison. During a sync, Argo CD still applies the entire manifest, which would reset the replica count. Add the sync option RespectIgnoreDifferences=true so the ignored fields are also left alone when syncing. Since 3.0, ignoreDifferences also suppresses resource update events, and status is ignored in diffs for all resources, not just CRDs; both can be reverted through resource.compareoptions if you must, but the defaults reduce controller work.

data:
  resource.compareoptions: |
    ignoreAggregatedRoles: true
    ignoreDifferencesOnResourceUpdates: true
  resource.customizations.ignoreDifferences.apps_Deployment: |
    jqPathExpressions:
      - '.spec.template.spec.containers[] | select(.name == "istio-proxy")'
    jsonPointers:
      - /spec/replicas

Auto-sync policy is a platform decision as well. Automated sync with prune and selfHeal is the GitOps ideal, but each has a blast radius. Prune deletes live resources removed from Git: one accidental deletion in a manifest directory becomes a deletion in production. Protect critical resources with the annotation argocd.argoproj.io/sync-options: Prune=false, or require a human with Prune=confirm and then argocd app confirm-deletion. Self-heal reverts manual changes, which is correct until an incident responder scales a deployment by hand to stop an outage, and a few seconds later the controller reverts it. Self-heal pacing uses exponential backoff controlled by controller.self.heal.backoff.timeout.seconds (2), .factor (3) and .cap.seconds (300). Note that the auto-sync documentation still describes a fixed five-second timeout, which does not match the backoff settings; teach your responders the backoff behaviour and give them the toggle-auto-sync resource action (new in 3.5) or the sync-windows tool for a legitimate pause.

For change control across the whole platform, sync windows are the declarative lever. An AppProject can carry allow and deny windows with a cron schedule, a duration, and selectors for applications, namespaces or clusters. Deny overrides allow, manualSync: true lets a human through during a deny window, and 3.5 adds a syncOverrun option for syncs already in progress when a window closes. The errors Sync operation blocked by sync window are the expected result, not a fault.

Observability and cost

An Argo CD platform that you cannot see into will tell you it is unhealthy through your tenants. Build observability first, and then you can reason about cost with data.

Each component exposes Prometheus metrics: the application controller on argocd-metrics:8082, the API server on argocd-server-metrics:8083, the repo-server on port 8084, the commit server on 8087, the ApplicationSet controller on 8080 and the notifications controller on 9001. The metrics that answer platform questions are few:

Dashboards built for 2.x may break. Three metrics were removed in 3.0 (argocd_app_sync_status, argocd_app_health_status, argocd_app_created_time); their information now lives in labels on argocd_app_info. OpenTelemetry metric names and attributes also changed in 3.4 with the semantic-convention update, so check any OTEL-based panels after an upgrade. Tracing is available by setting otlp.address (for example otel-collector:4317) and, if needed, otlp.insecure and otlp.headers in argocd-cmd-params-cm. Logs are JSON by default (--logformat json|text) with per-component log levels.

Alert on symptoms tenants feel, not on every metric: a cluster connection that has been down for ten minutes, the controller's reconcile duration crossing a threshold, a rising count of Applications in Unknown or with error conditions (argocd_app_condition), repo-server restarts, and a sudden drop in Git request volume (which usually means polling or webhooks have broken). Pair them with notifications, so the owning team hears about their own failures. The bundled notifications controller reads services from argocd-notifications-cm (for example service.slack), keeps tokens in argocd-notifications-secret, and subscribes by annotation: notifications.argoproj.io/subscribe.<trigger>.<service>: <recipient>. Install the catalogue of ready-made triggers and templates with the server-side apply command from the docs, and test a template without sending anything using argocd admin notifications template notify.

Cost is almost entirely compute, since Argo CD is free software under Apache-2.0 and a graduated CNCF project. The bill comes from four places: repo-server CPU and memory (rendering), controller memory (proportional to cached objects), Redis memory, and Git provider API and egress traffic from polling. It follows that the cost levers are the same as the scaling levers:

• Webhooks plus a longer timeout.reconciliation cut Git requests and renders.
• manifest-generate-paths stops unrelated commits from rendering anything.
• resource.exclusions for high-churn kinds you never deploy shrinks the controller cache.
• reposerver.parallelism.limit lets you run smaller repo-server pods without OOM kills.
• A core install (core-install.yaml: no API server, UI or Dex) suits a single-administrator cluster that does not need the extras.

In the Middle East the cost conversation has a second axis: data residency. A hub in one region that reaches clusters in another sends cluster state and manifests across that boundary, and the Redis cache holds rendered manifests. If your employer has residency obligations, which is common in Gulf financial and government workloads, decide deliberately which region hosts the hub and whether separate hubs per jurisdiction are required. That argues for the per-region instance model, which also keeps an inter-region network failure from stopping deploys.

Hosted and enterprise distributions exist from several vendors. They are outside the upstream documentation, so evaluate them as products: what you buy is operational burden, and what you give up is some control over versions and configuration.

Upgrades and migrations

An upgrade of Argo CD is a change to the system that deploys all your other changes, so it deserves more ceremony than most. The good news is that the project has a disciplined release process: patch releases carry no breaking changes, minor releases (one per quarter, GA on the first Tuesday of February, May, August and November) may contain "minor changes with a workaround", and only the three latest minors are supported. The discipline is yours to supply.

The rules that prevent most upgrade incidents:

1. Read every intermediate upgrade guide. If you go from 3.2 to 3.5 you read 3.2-3.3, 3.3-3.4 and 3.4-3.5. Breaking notes hide in the ones you skip.
2. Upgrade one minor at a time in production, with a soak period in a lower environment that mirrors production's size and integrations.
3. Take a backup first with argocd admin export, running the image that matches your server version. The command does not fail when pointed at the wrong namespace, so always pass -n.
4. Apply the whole manifest set, not just the image tag: kubectl apply -n argocd --server-side --force-conflicts -f .../v3.5.3/manifests/install.yaml. Server-side apply has been mandatory since 3.3 because the ApplicationSet CRD exceeds client-side apply's annotation limit, and the failure message is metadata.annotations: Too long: may not be more than 262144 bytes. Helm users are unaffected.
5. Keep customisations out of the manifests. --force-conflicts overwrites your edits to fields defined in the upstream manifests (affinity, env, probes). Put them in Kustomize patches or Helm values, where an upgrade reapplies them.
6. Upgrade the CLI with the server. Since 3.0 the Dex claim change means an old CLI can obtain incorrect tokens.
7. Pin a version, including in any Kustomize remote base, rather than following stable.

What has actually broken in the 3.x line is the best guide to what to test. The 3.0 major changed the most, and a platform that started on 2.x carries its residue:

Several of these deserve a line of practical advice. For the 3.5 Helm 4 change, run your real charts through helm template with Helm 4 in CI before the upgrade; spec.source.helm.version: v3 is now ignored, because Helm 4 is always used, and plain-HTTP OCI registries need the explicit opt-in flag. For the React 19 change, UI extensions built for React 16 fail with Extension <name>.js failed to load; rebuild them with react/jsx-runtime externalised. For the known_hosts change, repositories without credentials now validate host keys against argocd-ssh-known-hosts-cm, so a repository that worked before can fail with knownhosts: key mismatch until you run argocd cert add-ssh --batch with the right keys. For the 3.4 health change, automation that waited for Missing should check OutOfSync instead.

Note the quirk in the release history: there is no v3.4.0 release. The first published 3.4 release is v3.4.1. Scripts that resolve v3.4.0 will fail, and the release note explains it is the first in the series although not named 3.4.0.

Migrations are the other change you own. The most common are moving from the 2.x defaults to 3.x ones, moving from a hand-managed estate to Argo CD, and moving from another GitOps tool. Whatever the source, use the same shape: import read-only first (Applications without automated sync so Argo CD reports drift but changes nothing), review the diffs with the teams, fix the noise with ignoreDifferences and health checks, and only then enable automation, one team at a time. Adopting existing resources requires that the tracking annotation appear on them at first sync, so turn prune off for the initial syncs. Replacing a previous Argo CD installation with a new hub is best done by exporting with argocd admin export, importing into the new instance, running both in read-only mode side by side, and moving the cluster credentials and then auto-sync one team at a time. Use a distinct installationID if both instances overlap on a cluster.

Governance and change control

Governance is how a platform stays safe after the founding engineers have moved on. It means policy that is written down, enforced by the system rather than by memory, and reviewable.

The enforcement points you have in Argo CD form a short list, and each is worth using on purpose:

• Who can change an Application: RBAC plus Git. If Applications live in Git (and they should), the review on the Application's manifest is the real change control. Restrict direct applications, create and update in the UI for production projects, so the UI becomes a read-and-sync tool and Git is the writer.
• What an Application may target: the AppProject. Its destinations, sources and kind restrictions are policy as data.
• When changes may happen: sync windows, with manualSync for break-glass.
• How changes are applied: sync options, hooks and waves, per resource.
• What changed and who did it: Kubernetes audit logs on the management cluster, Argo CD's own events (GET /api/v1/applications/{name}/events), Git history, and notifications.

Break-glass deserves a designed path because real outages will need one. A reasonable design is a dedicated role that can disable automated sync, sync outside a window, and run resource actions, bound to a group that is empty by default and populated through an approved request. Every use gets logged and reviewed. The mistake to avoid is an always-on admin login, since it nullifies every other control.

Policy as code extends this. Kyverno or OPA Gatekeeper in workload clusters enforces what the project boundary cannot: required labels, resource limits, signed images, forbidden registries. Two integration details matter. Kyverno's policy reports are excluded from Argo CD's view by default since 3.0, which avoids noise, and with server-side diff, admission webhooks participate in the comparison, so a mutating policy no longer produces a perpetual diff.

The Source Hydrator (Beta since 3.5) is a governance feature as much as a rendering one. It implements the rendered-manifests pattern: the drySource (your chart or overlay) is rendered by the commit server and committed to a syncSource branch, and Argo CD syncs the rendered output. The benefit is that reviewers can see the exact manifests that will be applied, and Git history records them. To enable it you install the *-install-with-hydrator.yaml manifests, set hydrator.enabled: "true", and provide a repository-write Secret. 3.5 adds syncSource.repoURL, which lets the hydrator write to a separate repository, which is usually the right design because it keeps write credentials away from the source repository. Plan for its caveats: since 3.3 it tracks state in git notes and no longer cleans the application path before writing, so stale files must be removed by hand, and syncSource.path may not be the repository root.

Incident playbooks

Good playbooks start from the symptom and narrow quickly. The five below cover the majority of platform incidents; each lists the first command to run, the most likely cause, and the repair.

Playbook 1: many applications show Unknown or comparison errors at once. The pattern of "many at once" points at a shared component, not at tenants. Check the repo-server first: kubectl get pods -n argocd, then kubectl logs deploy/argocd-repo-server -n argocd. OOM kills or restarts mean the render load exceeded its limits, so apply the parallelism limit and scale up. Manifest generation error (cached) means a failure was cached to prevent a retry storm; fix the cause, then hard-refresh with argocd app get NAME --hard-refresh. If the repo-server is fine, check Git reachability and credentials: repository not found or unable to resolve '<rev>' to a commit SHA implicates a rotated token or a deleted branch.

Playbook 2: status is stale but nothing is erroring. Look at the controller: argocd_app_reconcile latency, CPU throttling, and the work queues. Raise controller.status.processors, give the pod more CPU, and check whether one giant cluster is pinning a shard. If one cluster's cache is the problem, pin it to its own shard with the shard field.

Playbook 3: a bad change is live and automated sync keeps re-applying it. Stop the machine first: disable automated sync for the affected Application (argocd app set NAME --sync-policy none, or the toggle-auto-sync action in 3.5). Then fix forward by reverting the commit in Git, which is the GitOps-native repair. argocd app rollback is refused while automated sync is enabled, and it creates drift from Git that auto-sync would fight.

Playbook 4: an Application deletion cascaded further than intended. Prevention is a finalizer and a policy. The finalizer resources-finalizer.argocd.argoproj.io makes deleting the Application delete its resources, which is often what you want and sometimes a catastrophe. Protect critical resources with Delete=false, use Prune=confirm and Delete=confirm with argocd app confirm-deletion for human gates, and use argocd app delete NAME --cascade=false when you only want to remove Argo CD's record. For an ApplicationSet, set applicationsSyncPolicy to create-update or stricter so a generator glitch cannot delete Applications.

Playbook 5: SSO is broken and nobody can log in. This is when the local admin account pays for its existence. If you disabled it, the recovery path is to re-enable admin.enabled temporarily through argocd-cm (kubectl access needed), or use argocd login --core with cluster credentials. Keep that path documented. The usual causes are an expired client secret, an IdP redirect URI that does not include https://<argocd>/auth/callback, a missing url in argocd-cm, or, after upgrades, the Dex or go-oidc behaviour changes listed in the 3.4 notes. The error invalid cookie, longer than max length 4093 means a user belongs to too many groups.

Disaster recovery rests on the architecture: all durable state is in Kubernetes objects, so a recovery plan is about etcd and Git, not about Argo CD's internals. The official approach is argocd admin export -n argocd > backup.yaml and argocd admin import -n argocd - < backup.yaml, run with a CLI matching the server version. Take it on a schedule and store it encrypted, since it contains Secrets. Because your Applications are also in Git, the most resilient recovery is a bootstrap: install Argo CD from pinned manifests in a new cluster, apply a single root Application (an app of apps) that points at the repository of Applications and ApplicationSets, add cluster Secrets from your secret manager, and let the fleet reconstitute itself. Practise it. A bootstrap that has only been tried on paper fails on the first missing credential.

# restore drill on a scratch cluster
kubectl create namespace argocd
kubectl apply -n argocd --server-side --force-conflicts \
  -f https://raw.githubusercontent.com/argoproj/argo-cd/v3.5.3/manifests/ha/install.yaml
argocd admin import -n argocd - < backup.yaml
argocd admin settings validate -n argocd

Where Argo CD stops

Every tool has a boundary, and a platform owner must know where it is so that a real need is not forced into the wrong place.

Argo CD is a deployment reconciler, not a pipeline. It does not build images, run tests, or scan anything. CI builds and tests, then writes the result into Git (an image tag in a values file or a Kustomize images entry). The optional Argo CD Image Updater, from the argoproj-labs project, can write tags back to Git, but it is a separate component with its own maturity. If your need is "run these steps in this order with retries and artefacts", that is a workflow engine such as Argo Workflows, not a sync hook.

It is Kubernetes-only. Anything outside a Kubernetes API (cloud accounts, DNS, managed databases, SaaS) is out of reach unless you model it as Kubernetes resources through an operator such as Crossplane. For infrastructure that is not Kubernetes, a tool like Terraform with a review-and-apply workflow is the conventional answer, and the two coexist: Terraform builds the clusters, Argo CD fills them. See the Terraform guide and the Kubernetes guide for the neighbouring ground.

Progressive delivery is not built in. Argo CD applies manifests; it does not shift traffic gradually or analyse metrics during a rollout. Argo Rollouts provides canary and blue-green strategies, and Argo CD understands Rollouts' health. The GitOps Promoter project (in argoproj-labs) covers environment promotion, and the hydrator integrates with it.

Secret management is someone else's job, as covered above, and so is policy enforcement (admission controllers) and image signature verification.

It is not a general drift-detection or compliance scanner. It reports drift for what it manages. Resources created outside Argo CD are invisible unless you enable orphaned-resource monitoring on the project, and even then the feature only looks inside destination namespaces.

It is a pull-based system with a cache. If you need a synchronous, transactional deployment (a pipeline that must know the deploy finished before it continues), you use argocd app wait or the sync API, which are polling-based, and you design the pipeline to tolerate that. For most teams this is fine; for some release managers it is a culture change.

The honest comparison is with Flux, the other CNCF graduated GitOps tool. Both reconcile Git to clusters. Argo CD offers the application-centric model, a UI, and a central server and RBAC, which suits platforms serving many teams that want visibility. Flux is a set of controllers with no server or UI of its own, leaning on Kubernetes RBAC and composition, which suits platform teams that want everything to be a Kubernetes resource and have no central API surface to secure. See the Flux guide. Neither is universally better; choose by whether the platform's centre of gravity is a shared application view or a set of composable controllers. Helm and Kustomize are not alternatives but inputs: Argo CD renders with them (see the Helm guide and the Kustomize guide), and note that because it uses helm template, helm ls will show nothing for charts Argo CD deployed.

Running Argo CD as a platform

Everything above is mechanism. Running a platform means turning mechanism into a service that teams can adopt without talking to you, and that you can operate without heroics.

Define the contract. Write down what the platform promises and what it does not. A workable contract: onboarding takes a pull request and is merged within a stated time; syncs begin within a stated time of a merge (this is a function of webhooks and reconcile latency, so you can measure it); the platform team owns the control plane, teams own their manifests. And what it does not promise: it does not fix a team's broken chart, and it does not guarantee a deploy during a deny window.

Make the paved road the easy road. Provide a repository template, a tested ApplicationSet that produces the standard Application shape, and a default set of sync options and health checks. The "golden path" for a new service should be copy, edit three values, open a pull request. Teams that go off the path get support at a lower priority, which is a fair trade.

Provide self-service inside the guardrails. Through projects, apps in any namespace and ApplicationSets, teams create and change their own Applications without you. You review changes to the boundary (projects, RBAC, ApplicationSets), not changes within it.

Standardise the customisations once. Custom health checks for the CRDs your organisation uses (as Lua in resource.customizations.health.<group>_<kind>), custom resource actions, ignoreDifferences for common controllers, and resource.exclusions belong in the platform's configuration, tested with argocd admin settings resource-overrides health ./obj.yaml --argocd-cm-path ./argocd-cm.yaml, rather than copied into every team's Application. An Application stuck Progressing forever, for example on a SealedSecret or an Ingress that waits for a load balancer, is usually a missing health check, not a bug in the application.

Manage Argo CD's configuration as code, in the same way as everything else. The ConfigMaps (argocd-cm, argocd-cmd-params-cm, argocd-rbac-cm), projects, repository Secrets (from your secret manager), and cluster Secrets all belong in Git, reviewed and validated with argocd admin settings validate in CI. A setting changed in the UI and never committed is drift in your most important system.

Budget for upgrades as recurring work. One minor per quarter, with patch releases several times a month, means a support window of roughly nine months for a given minor. Put the upgrade in the calendar the week after each GA, and keep a known-good rollback.

Measure the platform like a product. Four numbers are worth publishing internally: time from merge to sync complete, the percentage of Applications Synced and Healthy, the change failure rate for syncs, and the time to onboard a new team. If they trend the right way, the platform is working, whatever the architecture looks like.

Tell the truth about the roadmap. Features marked Alpha or Beta (progressive syncs, impersonation, the hydrator, alternative sharding algorithms) can change. Offer them to tenants with that label attached, and do not build a mandatory dependency on an Alpha feature. The feature-maturity page in the operator manual is the authority.

The review checklist

Use this list when reviewing a change to the platform or an onboarding pull request. Each item names the failure it prevents.

Application and project shape

• Is the Application in a project other than default, and does the project restrict source repositories, destinations and cluster-scoped kinds? (A tenant reaching another tenant's namespace.)
• Does the Application have a finalizer only if cascade deletion is intended? (Orphaned resources, or an unexpectedly destructive delete.)
• Are the targetRevision and path explicit, and is the revision something reviewed (a protected branch or a tag)? (Deploying unreviewed code.)

Sync behaviour

• If automated, are prune and selfHeal deliberate, and is allowEmpty false? (A rendering bug deleting the workload.)
• Are irreversible resources (PVCs, data namespaces, CRDs) protected with Prune=false or Delete=false? (Data loss.)
• If ignoreDifferences is used, is RespectIgnoreDifferences=true set when a controller owns the field? (A sync that undoes an autoscaler.)
• Are hooks idempotent, with a hook-delete-policy, and are sync waves used instead of hoping for ordering? (Stuck syncs and hook pile-up.)

Security

• No secret values in Git, and none injected at render time? (Plaintext in Redis.)
• Do RBAC changes come with argocd admin settings rbac can evidence for one allowed and one refused action? (An untested policy.)
• Do tokens have an expiry, and is impersonation configured with service accounts that grant only what the team needs? (A long-lived broad credential.)

Scale and cost

• For monorepos: is manifest-generate-paths set? (Every commit re-rendering every app.)
• Does a new generator have a bounded number of outputs, and is its sync policy non-deleting? (An ApplicationSet explosion.)
• Does a new cluster registration use a narrowed ClusterRole where possible, and is its size known? (An unshardable hot cluster.)

Operations

• Is the change in the notifications and dashboards, so the owning team will hear about its failures?
• Is there a rollback that is a Git revert, and has someone read the upgrade guide if this is a version change?

The complete picture

Across this level, the same system looked different from each angle, and the angles connect.

Architecture told you that durable state is in etcd and Git, which is why recovery is a bootstrap. Scaling told you the order of failure: the repo-server, then the controller, then the API, with Git and the Kubernetes API as external ceilings. The trust model told you that repository write access is deploy access, and that three layers (Argo CD RBAC, project boundaries and Kubernetes RBAC through impersonation) are better than one. Secrets and supply chain told you to resolve secrets in the cluster and verify sources with Source Integrity. Multi-tenancy told you that an AppProject is the boundary and an ApplicationSet is an amplifier. Upgrades told you to read every guide, to use server-side apply, and to rehearse. Governance and playbooks told you that a rollback is a revert and that break-glass must be designed. The platform section tied it together as a product with a contract and measurements.

Where the series leaves you

The three levels took the same tool from "it makes my manifests appear in the cluster" to "it is a service I run for other people".

You should now be able to look at an Argo CD installation and read what it permits, where it will saturate, what it would take to recover it, and what an upgrade will change, without being told. The questions that start with "who can" and "what happens when" are the ones a platform owner is paid to answer before an incident does.

Carry the review instinct forward. The neighbouring guides pick up from here: the Kubernetes guide for the cluster you deploy into, the Helm guide and Kustomize guide for what Argo CD renders, the Terraform guide for what builds the clusters, and the Flux guide for the alternative you should be able to compare honestly. Most GitOps incidents are decided at review time, in a pull request, long before a controller acts on them.

Sources

• Argo CD documentation
• Architecture
• Installation
• High availability
• Dynamic cluster distribution
• Declarative setup
• argocd-cmd-params-cm reference
• RBAC configuration
• Security
• Secret management
• User management and SSO
• TLS configuration
• mTLS for the repo-server
• Metrics
• Notifications
• Applications in any namespace
• Sync using impersonation
• ApplicationSet
• Cluster bootstrapping
• Health checks
• Disaster recovery
• Troubleshooting
• Feature maturity
• Signed release assets
• Upgrading overview
• Upgrading 3.4 to 3.5
• Upgrading 3.3 to 3.4
• Upgrading 3.2 to 3.3
• Upgrading 2.14 to 3.0
• Projects
• Automated sync policy
• Sync options
• Sync windows
• Diff strategies
• Resource tracking
• Source Hydrator
• Source Integrity
• Application deletion
• CI automation
• FAQ
• Release process and cadence
• Argo CD v3.5.3 release

Part three of three. Senior interviews about Argo CD are rarely about commands. They give you a platform problem and watch how you reason: what you ask first, which component you suspect, what you refuse to trade away, and how you explain the decision to someone who pays for it. This file is organised as system-design prompts, incident questions, and trade-off arguments, with model answers and what separates a strong answer from a weak one. About fifty minutes to read, longer if you answer aloud first, which you should.

The thirty-second answer

Argo CD is a pull-based GitOps controller for Kubernetes: it continuously compares desired state in Git (or a Helm or OCI repository) with live state in clusters, reports the difference, and optionally reconciles it. As a platform owner, say the senior part unprompted: write access to a watched repository is deploy access, so the real work is bounding that with projects, RBAC and impersonation, scaling the repo-server and controller, keeping secrets out of the render path, and upgrading a system that deploys everything else, including itself.

Everything here targets Argo CD v3.5.3. Mention the version if asked, and mention that most tutorials still describe 2.x behaviour, which is itself a good signal that you have read the upgrade guides.

The facts to have ready

Interviewers use facts to test whether your design reasoning is grounded. Have these cold.

And the command set a platform owner reaches for:

argocd admin export -n argocd > backup.yaml
argocd admin settings rbac can role:dev sync applications 'team-a/app' --policy-file policy.csv
argocd admin settings resource-overrides health ./obj.yaml --argocd-cm-path ./argocd-cm.yaml
argocd admin cluster shards
argocd app get NAME --hard-refresh
argocd app set NAME --sync-policy none

System-design prompts

Design the GitOps platform for forty teams on six clusters

The prompt. A company has forty product teams, six Kubernetes clusters in two regions (say a Gulf region and a European one), and is moving from a push-based CI/CD to GitOps. You own the design. Walk through it.

A strong answer starts with questions, because the answer changes with them. How many Applications do you expect (hundreds or thousands)? Are any teams regulated or separately audited? Is there a data-residency requirement for where cluster state and rendered manifests may live? Who owns the clusters? Do teams need to deploy cluster-scoped resources? What are the availability expectations for deploys?

Then the structure. A hub-and-spoke model: one Argo CD control plane per region (or per jurisdiction, if residency forces it) in a dedicated management cluster, in its HA configuration, deploying to workload clusters as spokes. The argument for a hub is centralised credentials, audit and a single place to apply policy; the argument for a hub per region is that an inter-region failure should not stop deploys, and that rendered manifests in Redis and cluster state should stay in-region. I would say explicitly what I am trading: more control planes to upgrade in exchange for a smaller blast radius and cleaner residency.

Tenancy. One AppProject per team generated from a template, with sourceRepos limited to the team's repositories, destinations to their namespaces, and an empty clusterResourceWhitelist so tenants cannot create ClusterRoleBindings. The default project locked down. SSO groups mapped to project roles, policy.default set to no access, and the local admin disabled after SSO works. Apps in any namespace so that Kubernetes RBAC in a team's namespace governs who can create Applications there, with sourceNamespaces on each project.

Blast radius beyond the project. Enable sync impersonation so each project's syncs run as a per-team service account with only the rights that team needs; this puts Kubernetes RBAC behind Argo CD RBAC, so a bug in one layer is caught by the other. Mention that it is Beta, that 3.5 extends it to logs and deletes so the service accounts need those permissions, and that you would pilot it with one team first.

Onboarding and delivery. A platform repository with a directory per team, consumed by an ApplicationSet with a Git directory generator, so onboarding is a pull request. The generated Applications use automated sync with self-heal, prune on, and allowEmpty off; the ApplicationSet has applicationsSyncPolicy: create-update so a generator mistake cannot delete Applications. CI builds images and commits the tag to Git; webhooks trigger refreshes; timeout.reconciliation is raised so polling is a backstop.

Secrets and supply chain. External Secrets Operator (or another in-cluster resolver) so Git holds references, and nothing is injected at render time. Branch protection and CODEOWNERS on deployment repositories. Source Integrity on the production projects to require signed commits. An admission controller for image signatures, since Argo CD does not verify those.

Operations. Metrics into Prometheus, a dashboard for reconcile latency, Git request volume and cluster connection status, notifications routed to the owning team, Argo CD's own configuration in Git, and a documented restore drill using argocd admin export plus a bootstrap root Application.

What separates strong from weak. A weak answer names Argo CD and lists features. A strong answer asks the sizing and residency questions, makes the trade-offs explicit (hub per region versus one global hub), and names the failure modes it is designing against: cross-tenant deploys, a repo-server that falls over on a monorepo commit, secrets in Redis, a bad upgrade. The best answers also say what they would not do on day one: they would not turn on Alpha sharding algorithms or roll impersonation to everyone at once.

Make a 3,000-application installation fast

The prompt. Users complain that Argo CD is slow to notice commits and the UI times out. There are 3,000 Applications and eight clusters. What do you do?

Strong answer. Diagnose before tuning, and go in the order components actually break. First measure: argocd_app_reconcile percentiles, repo-server CPU and memory and restarts, Git request volume, and whether status is stale or whether renders are failing. Then:

1. If the repo-server is the bottleneck (OOM kills, Context deadline exceeded): set argocd.argoproj.io/manifest-generate-paths on monorepo Applications so a commit only invalidates what it touches; set reposerver.parallelism.limit to bound simultaneous renders; give /tmp a volume; scale replicas. Use refs/heads/main style refs in large monorepos.
2. If the controller is the bottleneck: raise controller.status.processors and controller.operation.processors (the docs suggest about 50 and 25 at roughly a thousand applications), give it CPU, and shard: raise StatefulSet replicas and ARGOCD_CONTROLLER_REPLICAS together. Check argocd admin cluster stats for one cluster that dominates, because sharding is per cluster and cannot split it; pin that cluster with the shard field.
3. If Git is the bottleneck: webhooks plus a longer timeout.reconciliation, and webhook.refresh.jitter so a webhook burst does not stampede.
4. If the API or UI times out: more server replicas with ARGOCD_API_SERVER_REPLICAS matched, ARGOCD_GRPC_MAX_SIZE_MB above 200 for this size, and ARGOCD_APPLICATION_TREE_SHARD_SIZE=100 to reduce Redis traffic.

The part that shows experience: do not remove the 3.0 defaults (resource.exclusions, ignoreResourceUpdates, externally stored health). They exist to reduce load. Change one thing at a time and keep the before-and-after metrics. And say what you would not try first: raising timeouts, which hides the problem.

Follow-up you should expect: "what if one cluster is 60 percent of the load?" Sharding cannot split a cluster. You pin it to its own shard and size that replica for it; beyond that, you reduce what is watched (resource.exclusions for high-churn kinds) or, in the longer term, split the cluster.

Design multi-region disaster recovery

The prompt. The management cluster is lost. What is the recovery plan and how do you know it works?

Strong answer. State the principle first: all durable state is in Kubernetes objects and Git, and Redis is a cache, so recovery is etcd and Git, not Argo CD internals. There are two recovery paths and a mature platform has both. The first is restore: argocd admin export -n argocd on a schedule (using an image matching the server version, always passing -n because the command does not fail on the wrong namespace), stored encrypted because it contains Secrets, and argocd admin import into a new instance. The second is bootstrap, which I prefer as the primary path because it is also how you build new environments: install Argo CD from pinned manifests with server-side apply, apply a root Application (app of apps) that points at the repository holding Applications, AppProjects and ApplicationSets, then reintroduce cluster and repository Secrets from the secret manager. The fleet then reconstitutes itself.

What makes it credible: the drill. A recovery plan that has never been rehearsed fails on the first missing credential. Mention that workloads keep running while Argo CD is down, so RTO is about restoring the ability to deploy rather than restoring service, and that this changes the urgency and the communication.

Trap to avoid: claiming Redis needs backing up. It does not.

Incident-response questions

Incident answers are scored on order of operations. Say what you check first and why that narrows the problem fastest.

"All Applications went Unknown at 09:10"

The word "all" means a shared component. Check recent changes first (an upgrade, a config change, a credential rotation), then the repo-server (kubectl get pods -n argocd, logs, restarts and OOM kills), then Git reachability and credentials. A rotated token produces repository not found or unable to resolve ... to a commit SHA everywhere at once; a repo-server OOM produces Context deadline exceeded. If it started after an upgrade, read the upgrade guide for that version. After a 3.5 upgrade, candidates include SSH repositories failing with knownhosts: key mismatch and plain-HTTP OCI registries failing with http: server gave HTTP response to HTTPS client.

Strong answer adds that you communicate early: workloads are still running, deploys are paused, and you will report when syncing resumes. That separates "deploy freeze" from "outage" for stakeholders.

"A bad config reached production and self-heal keeps restoring it"

Stop the loop before fixing. Disable automated sync on the affected Application (argocd app set NAME --sync-policy none, or the toggle-auto-sync action), or use a sync window for a wider pause. Then fix forward: revert the commit in Git and let the sync apply it. Do not use argocd app rollback as the repair. It is refused while automated sync is on, and even when allowed it puts the cluster out of step with Git.

What a strong answer adds: a question about why the bad change passed review, and a follow-up action (a policy, a validation step or a sync window) so it cannot recur.

"Someone deleted a production namespace through Argo CD"

Establish the mechanism before blame. Possibilities: a prune in an automated sync after a manifest was removed from Git; a cascade delete of an Application carrying the resources finalizer; an ApplicationSet that stopped generating an Application whose sync policy allowed deletion; or an RBAC grant that allowed a person to delete it. The audit trail is Git history, Argo CD's events, and the Kubernetes audit log. Recovery depends on data: a namespace holding a PersistentVolumeClaim is where Prune=false and Delete=false should have been, and is why irreversible resources get protection annotations. The preventive controls to name are Prune=confirm with argocd app confirm-deletion, applicationsSyncPolicy: create-update on ApplicationSets, a narrow namespaceResourceBlacklist for tenants, and RBAC that does not grant delete broadly (since 3.0 it no longer inherits from the Application).

"Nobody can log in"

Decide whether it is SSO or the server. The CLI with --core (using kubeconfig) bypasses the API server and works when the server is down. If the server is up but SSO fails, suspect the client secret, the redirect URI (the IdP must allow https://<argocd>/auth/callback), a missing url in argocd-cm, or the Dex and go-oidc changes from 3.4. If one user cannot log in, invalid cookie, longer than max length 4093 means too many groups. The break-glass path (re-enabling the local admin account through argocd-cm) must be documented and audited, because the tool you are fixing is the one that would normally authorise you.

"The upgrade to 3.5 went wrong: what do you look at?"

Note that a good answer begins with what you did before: read every intermediate guide, backed up with argocd admin export, rehearsed in staging. After the fact, go through the list of known changes. Charts and OCI sources (Helm 4, plain-HTTP registries, http: server gave HTTP response to HTTPS client); custom UI extensions (React 19, Extension ... failed to load); SSH repositories (knownhosts: key mismatch); tenants suddenly unable to see logs or delete pods (impersonation now covers server operations, so their service accounts need more permissions); deprecated flags (--repo-server-strict-tls). If the install itself failed with metadata.annotations: Too long, the command lacked --server-side --force-conflicts.

Trade-off arguments

These questions test whether you can argue rather than recite. A good structure: state the options, say what each optimises for, choose, and say what would change your mind.

Shared instance versus instance per team

The shared instance optimises for operating cost and consistency: one upgrade, one dashboard, one template for tenancy. Its cost is a bigger blast radius and a need to get multi-tenancy right. The per-team instance optimises for isolation: separate credentials, separate failures, separate upgrade timing. Its cost is operating many control planes, and a risk of drift between them. I would choose shared for teams in one organisation that broadly trust the platform team, with impersonation and projects as the boundary, and choose separate instances for regulated tenants or where production must be strictly separated. What would change my mind: an audit requirement that cannot be met by logical isolation, or evidence that one tenant's load is destabilising others despite limits.

Argo CD versus Flux

Both reconcile Git to clusters and both are CNCF graduated. Argo CD has an application-centric model, a UI and a central API with its own RBAC, which suits a platform serving many teams that want visibility and self-service. Flux is a set of controllers with no central server, leaning on Kubernetes RBAC and composition; it suits teams who want every part to be a Kubernetes resource and prefer not to secure an additional API surface. The honest senior answer refuses to declare a winner and names the deciding factor: whether the platform's centre of gravity is a shared application view or composable controllers. See the Flux guide for the other side.

Render-time secret injection versus in-cluster resolution

The injection approach is convenient, since secrets appear in manifests as they render. But the operator manual strongly cautions against it: rendered manifests are cached in Redis in plaintext and exposed through the repo-server API; rotation waits for a sync; and it does not work with the rendered-manifests pattern. In-cluster resolution with External Secrets Operator, Sealed Secrets, the Secrets Store CSI Driver or the Vault Secrets Operator keeps Argo CD out of the secret path entirely. The price is another component per cluster. I choose in-cluster, and I would only accept injection for a bounded, temporary migration with the risk written down.

Automated sync with prune in production

For: Git is the single source of truth and drift is corrected. Against: a mistaken deletion in Git becomes a deletion in production, and self-heal will undo an incident responder's manual fix. My position is to enable automation, with guardrails: allowEmpty off, Prune=false or Delete=false on irreversible resources, Prune=confirm for high-risk applications, sync windows for freeze periods, and a documented way to pause automation during an incident. The general principle is that automation is safe in proportion to the quality of its brakes.

Rendered manifests (Source Hydrator) versus rendering at sync time

The hydrator commits rendered output to Git, so reviewers see the exact manifests and history records them. The cost is an extra component (the commit server), extra write credentials, a Beta feature status in 3.5, and operational caveats such as stale files not being cleaned (since 3.3). Rendering at sync time is simpler and mature but leaves the final manifests invisible to review. I would adopt the hydrator where reviewability is a compliance need, use a separate write repository (syncSource.repoURL, new in 3.5), and keep it off the critical path for others until it is out of Beta.

Arguing to leadership

A senior owner must turn technical points into a decision a non-specialist can make. Three models follow.

"Why do we need a dedicated platform team for this?" Frame it as risk and throughput. Argo CD holds credentials to every cluster and can delete anything, so it is a high-consequence system; unowned, it becomes a tenant-versus-tenant outage waiting to happen. Owned, it gives every team a deploy path that takes a pull request rather than a ticket. Offer numbers: the four platform measures (merge to sync latency, percentage of Applications synced and healthy, change failure rate, onboarding time) before and after. Finish with what you are asking for: named ownership, an on-call, and an upgrade budget of one minor a quarter.

"Why can't we just upgrade once a year?" Say it plainly: only the three latest minors receive patches, and a minor ships each quarter, so a minor is supported for about nine months. Falling behind means running unpatched code in the most privileged system in the company and making the eventual jump larger, with more breaking notes to read at once. The 3.0 change list alone (tracking, RBAC, health, exclusions) shows why batching changes is riskier than taking them quarterly. The cost of staying current is a recurring, predictable, rehearsed upgrade; the cost of not staying current is an unplanned one.

"Can we let teams self-serve everything?" Yes, inside guardrails, and be specific about what the guardrails are: projects, namespace boundaries, impersonation, policy in admission control, and review of any change to the boundaries themselves. Self-service without guardrails is fast until the first incident, after which it is stopped entirely. With guardrails, the platform team reviews perhaps ten percent of changes (the ones that alter the boundary) and the teams own the rest.

Cost framing. Argo CD is free software; the cost is compute for the repo-server, controller, Redis and the Git provider traffic, plus engineering time. Levers that reduce compute (webhooks, path scoping, excluding high-churn kinds) are cheap and should be done first. If leadership asks about a vendor-hosted version, present it as buying operational burden and giving up some control over versions and configuration, and let them weigh that against headcount.

What a strong answer covers that a weak one misses

Common interview questions

What is the difference between sync status and health? Sync status is whether live state matches Git (Synced, OutOfSync, Unknown). Health is whether the resources are working (Healthy, Progressing, Degraded, Suspended, Missing, Unknown). An application can be Synced and Degraded. Note the 3.4 change: an app is Missing only when all its resources are missing.

Why is Argo CD pull-based and why does it matter? CI does not need credentials for the cluster; the cluster pulls. That shrinks the credential surface, but it moves the credential problem into Argo CD itself, which is why you harden it.

What is an AppProject and what does it not protect against? It restricts repositories, destinations, kinds and namespaces for Applications. It does not stop a person with Git write access from deploying anything the project permits, and on its own it does not limit what the sync can do in a cluster; that is what impersonation adds.

What happens if Redis loses its data? Nothing durable. It is a cache of manifests and trees. Expect a burst of re-renders and a slow UI while it refills.

Why did RBAC on Applications change in 3.0? Fine-grained RBAC no longer inherits: update and delete on an Application apply to the Application only, and resource-level rights must be granted (for example delete/*/Pod/*/*). logs, get must also be granted explicitly.

How do you stop a generator from deleting Applications? Set the ApplicationSet's applicationsSyncPolicy to create-only or create-update.

When would you use ignoreDifferences and what must you pair it with? When another controller legitimately owns a field (an autoscaler owning replicas). Pair it with RespectIgnoreDifferences=true so a sync does not reset the field.

How do you handle a cluster that is a large fraction of the load? Pin it to a dedicated shard with the shard field in its Secret, size that replica accordingly, and reduce watched resources with exclusions.

What is Server-Side Diff for? It lets the Kubernetes API server compute the diff through a dry run, so defaulting and mutating webhooks take part. It is Stable since 3.1 and removes many false OutOfSync states.

What does the Source Hydrator do and what is its status? It renders a dry source with the commit server, commits the result to a branch, and syncs that. It is Beta since 3.5 and needs the hydrator install manifests, hydrator.enabled, and a repository-write Secret.

Sixty-second self-test

Answer each aloud without looking back. If you hesitate, that topic is the one to revisit.

1. Name the six Argo CD components and what each stores.
2. What three things do tenants notice if the repo-server dies, and what keeps working?
3. In what order do components saturate, and what is the lever for each?
4. Why can sharding not fix one giant cluster?
5. Describe the three layers that stop one tenant reaching another.
6. Why is render-time secret injection discouraged?
7. What is the right repair for a bad deploy under automated sync, and what is the wrong one?
8. Which three changes from 3.0, 3.3 and 3.5 cause the most upgrade pain?
9. Describe the recovery plan if the management cluster is lost.
10. Name three things Argo CD is not, and what covers each.

If you can do all ten in a minute, you are ready for the senior conversation. The Kubernetes guide, the Helm guide and the Flux guide cover the neighbouring ground an interviewer is likely to wander into.

Part three of three. At this level the problems are not failed syncs; they are the things that only appear once forty teams, several thousand Applications and a real audit depend on the system. Start with the error table, then the ownership habits, verification recipes, and the review checklists for pull requests and design documents.

Production traps that only appear at scale

Cumulative: everything from Beginner and Mid still applies. These are the traps that cost platform teams days.

A few rows deserve elaboration.

The cached manifest error is a trap because it teaches the wrong lesson. Operators see an error persist after fixing the repository and conclude the fix did not work. The cache is working as designed, to avoid hammering a failing render. The discipline is to hard-refresh and read the repo-server logs for the real error, never the Application's message alone.

Sharding that "does nothing" is almost always a mismatch between the StatefulSet replica count and ARGOCD_CONTROLLER_REPLICAS. The controller distributes clusters according to the variable; if it still says one, the new replicas idle. After changing either, verify with argocd admin cluster shards.

The empty render is the quiet catastrophe of GitOps. A templating bug or a bad path produces zero manifests; a pruning automated sync faithfully deletes everything the Application owns. allowEmpty is off by default for this reason. Treat any Application whose resource count suddenly drops to zero as an incident, and protect stateful resources with Prune=false.

Platform-ownership habits

The habits below separate platforms that stay boring from platforms that generate incidents. They are ordered by how much they prevent.

1. Put Argo CD's own configuration in Git, and only change it there. argocd-cm, argocd-cmd-params-cm, argocd-rbac-cm, projects, and references to repository and cluster credentials should all be reviewed artefacts. A setting changed in the UI or with kubectl edit is drift in your most important system. Validate in CI with argocd admin settings validate.
2. Test every RBAC change before it lands. argocd admin settings rbac validate for syntax, and argocd admin settings rbac can for one allowed action and one refused action per role. Since policy.default access cannot be removed by a deny, be stingy with defaults.
3. Upgrade every quarter. The project releases a minor each quarter and supports three. Schedule the upgrade the week after each GA, read each intermediate guide, rehearse in staging, and keep the previous manifests ready. Patch releases are frequent, so a monthly patch routine is reasonable.
4. Pin everything. The Argo CD version (never stable in production), chart versions, and the CLI. Distribute a pinned CLI through your internal tooling so engineers do not drift.
5. Give every tenant a generated, identical shape. Projects, namespaces, service accounts and role bindings should come from one template. Hand-built tenants diverge, and divergence is where the security gaps live.
6. Keep a designed break-glass path. A role that can pause automation, sync outside a window and run resource actions, bound to a group that is empty by default, with every use reviewed. Delete the initial admin secret and disable the admin account once SSO works, and document how to re-enable it.
7. Expire every token. Use -e on argocd proj role create-token and argocd account generate-token. Better, have CI commit to Git and let a webhook trigger the refresh, so pipelines hold no Argo CD credential.
8. Publish four numbers. Merge-to-sync latency, the percentage of Applications Synced and Healthy, the change failure rate, and time to onboard a team. They turn arguments into data.
9. Rehearse recovery. Run the restore or bootstrap drill on a scratch cluster at least twice a year. Include the credentials you will need, since those are what a plan on paper forgets.
10. Label maturity honestly. Alpha and Beta features (progressive syncs, impersonation, the hydrator, non-default sharding algorithms) may change. Offer them with that label, do not make them mandatory, and check the feature-maturity page before each upgrade.

Verification recipes

A control that has never refused anything is decoration. These recipes make each control prove itself. Run them after setup and after every upgrade.

Verify the project boundary. Try to deploy where a tenant should not be able to.

argocd app create probe --project team-a \
  --repo https://github.com/example-org/team-b-repo.git --path . \
  --dest-server https://kubernetes.default.svc --dest-namespace team-b

Expect application repo ... is not permitted in project 'team-a' or the destination error. If the command succeeds, the boundary is open.

Verify RBAC offline.

argocd admin settings rbac validate --policy-file policy.csv
argocd admin settings rbac can role:team-a-dev sync applications 'team-a/guestbook' --policy-file policy.csv
argocd admin settings rbac can role:team-a-dev delete applications 'team-b/guestbook' --policy-file policy.csv

The second prints Yes and the third prints No. Add a line like this per role to your CI.

Verify impersonation. Create a project whose service account lacks rights to one resource kind, sync an Application that includes it, and read the failure. It should be a Kubernetes forbidden error naming the service account, which proves the inner boundary works independently of Argo CD.

Verify Source Integrity. Push an unsigned commit to a repository under a policy in head or strict mode and confirm that the Application reports a ResourceComparison error and will not sync. Then push a signed commit and confirm recovery.

Verify health and ignore rules offline. Before shipping a custom Lua health check or an ignoreDifferences rule, test it against a saved object:

argocd admin settings resource-overrides health ./obj.yaml --argocd-cm-path ./argocd-cm.yaml
argocd admin settings resource-overrides ignore-differences ./obj.yaml --argocd-cm-path ./argocd-cm.yaml

Verify sharding. After changing replicas, run argocd admin cluster shards and confirm clusters are spread. Watch memory per replica for a day, not for a minute.

Verify the backup. The only test that counts is a restore.

argocd admin export -n argocd > backup.yaml
grep -c 'kind: Application' backup.yaml
# on a scratch cluster, after installing the same version:
argocd admin import -n argocd - < backup.yaml
argocd app list

Verify notifications without sending. argocd admin notifications template notify app-sync-succeeded guestbook renders a template against an Application so you can see the message before subscribing a channel to it.

Verify the upgrade. After the change, run argocd version (client and server), argocd admin settings validate -n argocd, and argocd app list, and compare the count of OutOfSync, Degraded and Unknown with the snapshot you took before. A jump is your signal to open the upgrade guide.

Review checklist for pull requests

Use this on every pull request that touches the deployment repository, the platform repository or Argo CD configuration. The italic text says what the question prevents.

For an Application or ApplicationSet change

• Is project something other than default, and is it set by the template rather than tenant input? (A tenant nominating another team's project.)
• Is targetRevision a reviewed branch or an immutable tag, and is path explicit? (Deploying an unreviewed change.)
• If syncPolicy.automated is on, are prune and selfHeal deliberate, and is allowEmpty absent or false? (A bad render deleting workloads.)
• Do stateful or irreversible resources carry Prune=false or Delete=false? (Data loss.)
• If ignoreDifferences was added, is there a named owner controller for the field and RespectIgnoreDifferences=true? (A sync fighting an autoscaler.)
• For an ApplicationSet: is the output bounded, is applicationsSyncPolicy non-deleting, and are SCM or cluster generators restricted to an allow-list? (A generator explosion or a server-side request.)
• Are hooks idempotent with a hook-delete-policy, and is ordering done with sync waves rather than hope? (Stuck syncs.)

For a project, RBAC or credentials change

• Are sourceRepos, destinations and kinds the narrowest set the team needs? (Cross-tenant reach.)
• Is clusterResourceWhitelist empty for tenant projects? (Privilege escalation through cluster-scoped kinds.)
• Is there rbac can evidence for one allowed and one refused action? (An untested policy.)
• Do new tokens have an expiry, and is the service account for impersonation limited to what the team needs? (A broad credential with no end.)
• Did anyone widen policy.default? If so, why? (An unrevokable grant.)
• Are secrets references to in-cluster resolvers, never values, and never injected at render time? (Plaintext in Redis.)

For an Argo CD configuration change

• Is the change in argocd-cm, argocd-cmd-params-cm or argocd-rbac-cm in Git, and has it been validated with argocd admin settings validate? (Drift in the platform.)
• Does it restart a component that needs restarting, and is that component scheduled for a quiet time? (A surprise restart of the controller.)
• Does it remove one of the default exclusions or ignore rules from 3.0? If so, is there a measurement? (A silent increase in controller load.)

Review checklist for design documents

A design document is where the expensive mistakes are cheapest to catch. When reviewing a proposal that involves Argo CD, ask these and expect written answers.

1. What is the blast radius? If this component, credential or Application is compromised or wrong, what is the worst thing that can be deployed or deleted, in which clusters, and how would we know?
2. What is the tenancy model? Shared instance, instance per team, or several on one cluster, and why? What does each tenant's project look like, and how is it generated?
3. How big will it get, and what breaks first? Expected Applications, clusters, repositories, largest cluster. Which metric warns us, and what is the lever?
4. Where do secrets live? Confirm they are resolved in-cluster and that nothing passes through the render path.
5. What is the source of truth and who can change it? Branch protection, review requirements, CODEOWNERS and, where needed, signed commits under Source Integrity.
6. How does it fail and what do tenants see? For the repo-server, controller, server, Redis, Dex and the Git provider, what is the experience and the message?
7. How do we recover? The restore and bootstrap path, where the backup is stored, who holds the credentials to perform it, and when it was last rehearsed.
8. How do we upgrade? The version policy, who reads the guides, how it is rehearsed and how it is rolled back.
9. Which features are Alpha or Beta, and is anything mandatory that depends on them? Progressive syncs, impersonation, the hydrator and alternative sharding algorithms must be optional or explicitly accepted.
10. What does it cost? Compute for the repo-server, controller and Redis, Git provider load, and the engineering time to operate it. If residency matters, where does cluster state and rendered content live?
11. What are the exit and alternatives? What would make us move to another tool, and how portable are the manifests? (They are plain Kubernetes, which is the point.)
12. What do we measure? Merge-to-sync latency, the synced-and-healthy percentage, change failure rate, onboarding time.

Short checklist

Keep this beside you when you take over an installation or review a platform.

• Version pinned, inside the supported three minors, upgrade scheduled
• Installed and upgraded with --server-side --force-conflicts
• HA manifests or chart values in production, on at least three nodes
• Configuration, projects and roles in Git, validated in CI
• default project locked down; every tenant project narrow and generated
• policy.default minimal; RBAC tested with rbac can
• SSO in place; local admin disabled; break-glass role designed and tested
• Tokens expire; CI holds no Argo CD credential if it can avoid it
• Impersonation piloted for tenant syncs; service accounts reviewed
• Secrets resolved in-cluster, nothing injected at render time
• Source Integrity enabled where provenance matters; admission policy for image signatures
• Webhooks configured; timeout.reconciliation raised; manifest-generate-paths on monorepos
• reposerver.parallelism.limit set; processors and sharding sized from measurements
• Dashboards, alerts and notifications for cluster connection, reconcile latency and repo-server health
• allowEmpty off; irreversible resources protected from prune and delete
• ApplicationSets use non-deleting sync policies and restricted generators
• Backup scheduled and stored encrypted; a restore or bootstrap drill done this half-year
• Four platform numbers published

Practice cards

Use these to rehearse out loud, alone or with a colleague. Each is a two-minute exercise.

Card 1: the repo-server dies at noon. List what tenants notice, what keeps working, and the first two commands you run. Then say what you would change afterwards so it does not recur.

Card 2: a team wants to deploy a ClusterRole. Explain why their project does not permit it, what the legitimate route is (a platform-owned project or a reviewed exception), and how you would implement the exception without widening their project.

Card 3: the Git provider is rate-limiting you. Describe the structural fix, in order, and the metric that confirms it worked.

Card 4: an engineer asks for the admin password. Answer with what you will give them instead (an SSO group mapped to a role scoped to their project) and how you would demonstrate it works.

Card 5: upgrading from 3.3 to 3.5. List the guides you would read, the three changes most likely to affect a typical estate, and the order in which you would roll it through environments.

Card 6: a generator change would remove fifty Applications. Say which setting prevents it, where it lives, and how you would prove it in a test before production.

Card 7: leadership asks why Argo CD and not just CI deploys. Give the answer in three sentences: pull versus push, drift correction and audit, and what it costs you to run.

The surrounding guides reward the same habits: the Kubernetes guide for the clusters you deploy into, the Helm guide and Kustomize guide for what Argo CD renders, and the Flux guide for the alternative you should be able to compare honestly. As with most platform work, the incidents you avoid are decided in review, long before a controller acts on the change.