The Complete Jenkins Guide

This is part three of three. Beginner taught you to run a pipeline, and Mid taught you to write good ones. This level is about the thing underneath: a Jenkins controller that thirty teams depend on, that holds every deploy credential the company owns, and that has no standby replica. When it is slow, nobody ships. When it is compromised, every system it can deploy to is compromised with it.

That combination is what makes Jenkins a platform problem rather than a tooling problem. The software is free and flexible, and the flexibility is exactly the risk: anyone who can edit a Pipeline can run code on your agents, and the controller is a single stateful Java process with a file-based data store. By the end of this guide you should be able to explain where that process breaks first, how to split it, how to secure it, how to upgrade it without a bad weekend, and, just as important, when to stop adding to it and move the work elsewhere. This guide targets Jenkins LTS 2.580.1 on Java 21 (Java 25 is also supported).

Where this picks up

I am starting with architecture, because every decision below follows from the shape of the controller.

Architecture and failure modes

A Jenkins controller is one Java process (served by Winstone, which embeds Jetty) that owns three things at once: the scheduler that decides which build runs where, the web tier that serves the UI and the REST API, and the state, which lives as files in JENKINS_HOME. There is no database server. Job configuration is XML, build records are files under each job's builds/ directory, credentials are encrypted blobs in XML, and plugins are archives unpacked into plugins/. The core is not active-active and not highly available. The open-source answer to "what if it dies" is to restart it quickly from persistent storage, and everything in a sound design is arranged to make that restart fast and safe.

The practical failure modes fall into a small number of families, and learning to recognise which family you are in shortens every incident.

Heap exhaustion. The controller keeps job configuration, the build queue, plugin state and the in-flight state of every running Pipeline in memory. Huge console logs, unbounded build history, large objects held in Pipeline variables and big JsonSlurper payloads all sit in that heap. The symptom is java.lang.OutOfMemoryError: Java heap space, or before it arrives, garbage-collection thrash where the UI is unresponsive and builds still appear to run. Raising -Xmx buys time, but the durable fix is to keep less in memory: buildDiscarder everywhere, logic moved to agents, and heap dumps enabled with -XX:+HeapDumpOnOutOfMemoryError so the next occurrence is diagnosable rather than mysterious.

CPU starvation from Pipeline Groovy. This one surprises people, and it is covered in its own section below. Pipeline code executes on the controller, not on the agent. One hundred concurrent pipelines each doing string manipulation in Groovy are one hundred tenants sharing the controller's cores.

Disk I/O. Pipelines write their state to disk as they run, and logs stream to disk continuously. On slow or network-attached storage the controller spends its time waiting on fsync. The Jenkins documentation's sizing advice puts storage size ahead of speed for the controller, but for Pipeline-heavy workloads SSD matters, and the durability setting described later trades safety for less I/O.

Thread and connection pressure. Every agent connection, every webhook, every API poller and every browser tab holds resources. A dashboard that polls /api/json with depth=3 every few seconds for hundreds of jobs can do more damage than a hundred builds. The fix is webhooks instead of polling and ?tree= filters on every API call you control.

The plugin blast radius. Nearly all functionality is a plugin, plugins share the controller's JVM and class loader space, and a plugin that misbehaves can degrade or crash the whole process. This is why a test controller for plugin upgrades is an architectural requirement, not a luxury.

State damage and state leaks. A corrupted disk, a bad upgrade that rewrites configuration, or an operator deleting the wrong folder. And, worse, the leak: JENKINS_HOME contains the keys (secrets/) that decrypt every credential in it, so a stolen backup is a stolen credential store.

Two architectural consequences follow. First, the built-in node should have zero executors in production (Manage Jenkins, Nodes, Built-In Node, Configure). A build on the built-in node has the same access to the controller's file system as the Jenkins process itself, which means any Pipeline author can read secrets/ and every other team's job configuration. Second, because the controller cannot be made redundant, you reduce its blast radius by running several of them, each smaller, each owned by a clear group, and each reconstructible from code.

What breaks first when you scale

It helps to think of scale on three axes that do not move together: the number of jobs (configuration and scan load), the number of concurrent builds (scheduler, Groovy CPU and log I/O), and the number of users and API clients (web tier). The documentation's planning heuristics are deliberately rough, and worth quoting as orders of magnitude rather than promises: roughly 500 jobs per controller, and executors in the region of the job count multiplied by 0.03. They are a prompt to measure, not a formula.

In practice the order in which things break is usually this.

1. The API and UI, from polling. External tools that ask Jenkins "what is the status of everything?" on a timer are the most common self-inflicted load. Replace them with webhooks and event notifications, and cap the depth of every query.
2. Branch indexing and SCM polling. A Multibranch Pipeline or Organization Folder scans repositories to find branches and pull requests. With hundreds of repositories, scheduled scans collide. Webhooks drive scans on demand, and the periodic scan becomes a slow safety net rather than the main trigger.
3. Pipeline Groovy CPU. Covered in detail in the next section. It is the limit that does not show up in any "number of executors" arithmetic.
4. The queue and executor supply. This is the good problem, because ephemeral agents (next section but one) absorb it. The controller's scheduler, not the agent fleet, is what eventually saturates.
5. Heap and disk. Long retention, giant logs and thousands of builds per job push these. Retention policy is capacity planning.
6. Plugin interactions. Past a certain plugin count, the upgrade matrix itself becomes the scaling limit. You can measure it by how long it takes to get a plugin update through staging.

The response is not a bigger machine beyond a point. The documentation's own guidance on horizontal scaling is to split controllers by product line, organisation or environment, and its sizing advice is to "invest in a larger machine for your controller over a faster one", meaning that capacity (memory, cores) is worth more than raw clock speed. A controller per business unit, each with its own agents and its own credentials, is also the only strong multi-tenancy Jenkins offers, which is a point I return to in the multi-tenancy section.

Measure before deciding. The Prometheus plugin exposes the signals that tell you which axis you are on, and the queue is the honest one: a queue that is long while executors are idle means a scheduling or label problem, while a long queue with every executor busy means you need more agents. A controller that is slow while the queue is empty means you are CPU-, heap- or disk-bound on the controller itself.

# Is the controller the bottleneck, or are we short of agents?
# Queue length and how long items have waited
curl -s -u "$JENKINS_USER_ID:$JENKINS_API_TOKEN" \
  "https://ci.example.com/queue/api/json?tree=items[why,inQueueSince,task[name]]"

# Busy versus total executors across all nodes
curl -s -u "$JENKINS_USER_ID:$JENKINS_API_TOKEN" \
  "https://ci.example.com/computer/api/json?tree=busyExecutors,totalExecutors"

The why field in the queue output is gold. It says things like Waiting for next available executor on ‘linux’ or There are no nodes with the label ‘linux’, which tell you in a sentence whether the problem is capacity, a missing label, or a cloud cap.

Pipeline execution, CPS and durability

The single most useful fact about Jenkins performance is that Pipeline Groovy runs on the controller. The steps that do real work, such as sh and bat, run on the agent. But the Groovy that decides what to do next, loops over lists, concatenates strings and calls your shared library runs in the controller's JVM, under the Continuation-Passing Style (CPS) interpreter.

CPS exists so that a Pipeline can be serialised to disk and resumed after a restart. Every time execution reaches a step, the interpreter can capture its whole state. That is a wonderful property and it has a price: CPS-transformed Groovy is much slower than ordinary Groovy, every local variable that is live across a step must be serialisable, and the cost scales with how much Groovy you write. A Pipeline that parses a 20 MB JSON file with JsonSlurper in Groovy is doing a heavy job on the shared controller. The same job in jq inside an sh step costs the controller nothing.

The rules that follow are the ones a platform team enforces in review:

• Groovy is glue. Real work belongs in sh steps on agents, or in tools invoked by them.
• Combine many tiny sh steps into one script. Each step has fixed overhead on the controller.
• Avoid controller-side HTTP calls and big parsing in Pipeline code. Use readJSON (from Pipeline Utility Steps) for small data and jq or a script for large data.
• Keep shared libraries lean and do not make them load enormous data at startup.
• Never call steps from @NonCPS methods, and never hold non-serialisable objects across step boundaries.

The error messages teach the model. java.io.NotSerializableException: java.util.regex.Matcher means a non-serialisable object was still live when CPS tried to persist state, so compute inside a @NonCPS method that returns plain data. And expected to call java.util.ArrayList.toSorted but wound up catching org.jenkinsci.plugins.workflow.cps.CpsClosure2.call means non-CPS code (a Java or GDK method) tried to call a CPS closure, so do not pass Pipeline closures into GDK methods.

// Bad: heavy parsing runs in CPS on the controller
def data = new groovy.json.JsonSlurperClassic().parseText(readFile('report.json'))

// Better: let the agent do the work and hand back one small value
def failures = sh(script: "jq '.failures | length' report.json", returnStdout: true).trim()
echo "failures: ${failures}"

Durability: what you trade for speed

Because Pipelines persist their state as they run, how often and how carefully they do it is a tunable. Jenkins offers three durability hints. MAX_SURVIVABILITY is the original behaviour and the default: the slowest, with the strongest guarantees. SURVIVABLE_NONATOMIC writes less carefully. PERFORMANCE_OPTIMIZED does much less disk I/O, but a dirty shutdown (a crash or power loss, as opposed to a graceful restart) can lose the state of running Pipelines.

That is a genuine trade-off and a good interview topic. A fleet of short CI builds on network storage benefits enormously from PERFORMANCE_OPTIMIZED, because a lost running build is simply re-run. A release Pipeline that takes six hours and waits on human approval should stay at the safe setting. You can set it globally (Manage Jenkins, System, Pipeline Speed/Durability Settings), per job, or in the Jenkinsfile:

pipeline {
  agent { label 'linux' }
  options {
    durabilityHint('PERFORMANCE_OPTIMIZED')   // short CI run: a lost build is just re-run
    timeout(time: 30, unit: 'MINUTES')
    buildDiscarder(logRotator(numToKeepStr: '20'))
  }
  stages {
    stage('Build') { steps { sh 'make build' } }
  }
}

A related option for long-lived platforms is disableResume(). By default a Pipeline tries to resume after a controller restart, which is what you want until an agent has vanished and the build sits in Resuming build ... Waiting to resume part of ... : ‘agent-x’ is offline. For ephemeral Kubernetes agents, resuming a build whose pod no longer exists is pointless, and the Kubernetes plugin's retry(count: 2, conditions: [kubernetesAgent(), nonresumable()]) pattern gives you a cleaner answer.

Agents as a fleet

At the platform level agents stop being machines you care about and become fungible capacity. The documentation's advice is to standardise them so any agent can run any job, which means jobs declare what they need (a label, or a pod spec) and never depend on a particular host.

There are three ways agents attach to the controller. SSH agents are the ones the controller connects out to: the documentation treats them as the most stable, they need the SSH Build Agents plugin and an "SSH Username with private key" credential, and host key verification matters (choose Known hosts file or a manually provided key, and avoid "Non verifying"). Inbound agents connect in from the agent side, which is firewall-friendly, and with -webSocket they need no extra port and pass through an ordinary HTTP reverse proxy. Without -webSocket they use the inbound TCP port (50000 in the container images), which is disabled by default (-1) since Jenkins 2.0. Clouds create agents on demand and delete them afterwards.

# An inbound agent over WebSocket: no TCP 50000, works through a reverse proxy
curl -sO https://ci.example.com/jnlpJars/agent.jar
java -jar agent.jar \
  -url https://ci.example.com/ \
  -secret @secret-file \
  -name "linux-01" \
  -webSocket \
  -workDir "/home/jenkins/agent"

The security model of the connection is worth stating. Agent-to-controller security has been on since 2.326 and cannot be switched off in the UI, so an agent cannot ask the controller to read arbitrary files. That does not make agents safe to share between trust levels: an agent runs whatever the build tells it to, and anything one build leaves behind (a workspace, a Docker socket, a cached credential) is visible to the next build on the same machine. Ephemeral agents solve this by construction.

The Kubernetes cloud

For most platform teams the answer is the Kubernetes plugin, which creates a pod per build and deletes it afterwards. The agent container is named jnlp (image jenkins/inbound-agent), and Pipelines describe their own toolchain as extra containers.

pipeline {
  agent {
    kubernetes {
      defaultContainer 'maven'
      yaml '''
        apiVersion: v1
        kind: Pod
        spec:
          containers:
          - name: maven
            image: maven:3.9.9-eclipse-temurin-21
            command: ["sleep"]
            args: ["infinity"]
      '''
    }
  }
  stages { stage('build') { steps { sh 'mvn -B verify' } } }
}

The settings a platform owner actually tunes are few but consequential. The container cap is a concurrency limit on the cloud: it stops a burst of builds from asking the cluster for more pods than it can hold, and when it is reached builds queue with a "waiting" reason rather than failing. idleMinutes keeps a pod alive between builds to save start-up time, at the price of idle cost and some state leakage. podRetention (never(), onFailure(), always(), evicted()) controls whether finished pods remain for debugging. activeDeadlineSeconds kills runaway pods. Use WebSocket when the controller is reachable only over HTTP(S). When containers in one pod must share the workspace, they need the same UID through securityContext.runAsUser, or you will chase permission errors.

The typical fleet failures are all visible from kubectl. Pods stuck in Pending mean the cluster has no capacity or the requests are too large; ImagePullBackOff means a bad image or missing pull secret; repeated pod creation failures in the controller log mean the service account lacks rights in the namespace. If you run the cluster yourself, the Kubernetes guide covers the scheduling side, and the controller itself is commonly installed with the Helm chart, whose values are the place to set controller.installPlugins, controller.JCasC.configScripts and persistence.storageClass.

A tour of the interface

Before building anything, spend five minutes learning where things are. Jenkins' interface has been redesigned several times, and recent LTS lines also include experimental redesigned pages, so a screenshot in an older tutorial may not match exactly. The landmarks below are stable.

The dashboard is the home page. It lists your jobs with their last status, and on the left has New Item (create a job), Build History and Manage Jenkins. The build queue and the executor status sit in a side panel, so you can see at a glance whether work is waiting.

Manage Jenkins is the administrator's control room. You will visit four places here again and again. System holds global settings such as the Jenkins URL. Plugins lets you install and update plugins. Nodes lists the machines that run builds, including the built-in node. Credentials stores secrets. There is also a Tools page for configuring things like a JDK or Maven by name, and a System Log page that shows what Jenkins itself is saying. When something odd happens, Manage Jenkins often shows a yellow or red banner explaining it, so read the banners before you search the internet.

A job page shows the job's name, a Build Now button, its build history and, for pipelines, a graphical view of the stages. Click a build number such as #3 and you reach the build page, which has Console Output, the full text log of everything that ran. Console Output is the most useful page in Jenkins. When a build fails, the answer is almost always in there, a few lines above the red text.

Pipeline Syntax, at /pipeline-syntax on your server, is a helper called the Snippet Generator. You choose a step from a dropdown, fill in a form, and it writes the exact code for the plugins you actually have installed. Use it constantly; it is more reliable than memory and more current than a blog post. A sibling tool, the Declarative Directive Generator, does the same for blocks such as environment and options.

Your first pipeline

Time to build something. The plan is to create a pipeline job whose script is typed directly into Jenkins. That is a teaching shortcut; a few sections from now you will move the script into a repository, which is how real projects do it.

1. On the dashboard click New Item.
2. Type a name, hello-pipeline, choose Pipeline and click OK.
3. Scroll to the Pipeline section. Leave Definition as "Pipeline script".
4. Paste the script below into the box and click Save.

pipeline {
  agent any
  stages {
    stage('Greet') {
      steps {
        echo 'Hello from Jenkins'
      }
    }
    stage('Look around') {
      steps {
        sh 'whoami'
        sh 'pwd'
        sh 'uname -a'
      }
    }
  }
}

Now click Build Now. A new entry, #1, appears in the build history. Click it, then Console Output. You should see something close to this:

Started by user admin
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/jenkins_home/workspace/hello-pipeline
[Pipeline] stage
[Pipeline] { (Greet)
[Pipeline] echo
Hello from Jenkins
[Pipeline] }
[Pipeline] stage
[Pipeline] { (Look around)
[Pipeline] sh
+ whoami
jenkins
[Pipeline] sh
+ pwd
/var/jenkins_home/workspace/hello-pipeline
[Pipeline] sh
+ uname -a
Linux 6.x ... x86_64 GNU/Linux
[Pipeline] End of Pipeline
Finished: SUCCESS

Read this log the way you will read every log. The lines starting [Pipeline] are Jenkins narrating its own progress. Running on Jenkins in ... tells you which node ran the build and which workspace directory it used; "Jenkins" is the name of the built-in node on a fresh install. A line starting with + is a shell command that the sh step ran, and the output follows it. The last line, Finished: SUCCESS, is the build's result.

If your log says whoami printed jenkins, that teaches something useful: the build ran as the operating-system user that runs the Jenkins service, which is why giving Jenkins broad privileges is dangerous.

What each part of the script does

pipeline { ... } is the outer block that marks this as a Declarative Pipeline. Everything lives inside it.

agent any says "run this on any available node". agent is required; Jenkins refuses a Declarative script without one. You will meet stricter choices in a later section.

stages { ... } contains one or more stage blocks, which are required too. Each stage has a name, which is what you see in the graphical view, and a steps { ... } block holding the actions. echo prints a message. sh runs a shell command and fails the stage if the command exits with a non-zero status. On a Windows agent you would use bat or powershell instead of sh.

The last point deserves emphasis because it is how Jenkins decides success and failure. A step fails when something goes wrong, and a failed step fails its stage and then the build. For shell commands, "goes wrong" means a non-zero exit status, which is the universal convention for "this command did not succeed". That is why test commands such as pytest or mvn test can be used directly: if a test fails, the tool exits non-zero, and Jenkins marks the build red.

Now break it on purpose. Edit the pipeline (Configure), change sh 'uname -a' to sh 'false', save and build again. The console ends with:

+ false
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // stage
[Pipeline] End of Pipeline
ERROR: script returned exit code 1
Finished: FAILURE

ERROR: script returned exit code 1 is the most common failure message in Jenkins, and it is not an error in Jenkins. It means the command you ran returned a failure status, so look above this line for that command's own output. The command false is a real program that does nothing and always fails, which makes it handy for tests like this one.

Writing a real Jenkinsfile

The script in the box is easy, but it has a serious flaw: it lives inside Jenkins, where it cannot be reviewed, versioned or restored. The standard practice is to store the pipeline as a file named Jenkinsfile at the root of the project repository. That gives you the benefits you already take for granted with code: history, code review, branches, and the ability to rebuild an old version exactly as it was.

Here is a slightly more realistic Jenkinsfile. It builds and tests a generic project that uses make, which stands in for whatever your language uses.

pipeline {
  agent any
  options {
    timeout(time: 30, unit: 'MINUTES')
    buildDiscarder(logRotator(numToKeepStr: '20'))
  }
  environment {
    APP = 'demo'
  }
  stages {
    stage('Build') {
      steps {
        sh 'make build'
      }
    }
    stage('Test') {
      steps {
        sh 'make test'
      }
      post {
        always {
          junit 'reports/**/*.xml'
        }
      }
    }
  }
  post {
    success {
      archiveArtifacts artifacts: 'dist/**', fingerprint: true
    }
    failure {
      echo 'The build failed'
    }
    cleanup {
      cleanWs()
    }
  }
}

Go through the blocks one at a time.

options sets behaviour for the whole pipeline. timeout(time: 30, unit: 'MINUTES') aborts the build if it runs for more than half an hour. Without a timeout, a hung test can occupy an executor for ever. buildDiscarder(logRotator(numToKeepStr: '20')) keeps only the last twenty builds' records. Without it, build history grows until the disk fills, and this is one of the most common causes of a stuck Jenkins. Put both options in every Jenkinsfile from day one.

environment defines environment variables for every step. Inside a shell step, use them as $APP. Inside Groovy code, read them as env.APP.

post holds actions that run after the stages finish, depending on the outcome. The conditions available are always, changed, fixed, regression, aborted, failure, success, unstable, unsuccessful and cleanup, evaluated in that order, with cleanup last so that it runs whatever happened. A post block can sit at the pipeline level, as here, or inside a single stage. The test stage uses always to publish test reports even when tests fail. That matters: the moment tests fail is the moment you most want the report.

junit 'reports/**/*.xml' reads test results in the JUnit XML format, which almost every test tool can produce, and gives you a test-results page with trends. It needs the JUnit plugin. Note the quoting: the path is a pattern, and the ** means "any number of directories".

archiveArtifacts saves files from the workspace as downloadable results of the build. fingerprint: true records a checksum so Jenkins can tell where a particular file came from.

cleanWs() deletes the workspace. It belongs to the Workspace Cleanup plugin, so it fails with "No such DSL method" if that plugin is missing.

A few practical points about shell steps. Use single quotes around shell commands: sh 'echo $APP' lets the shell expand $APP, which is what you want. Groovy treats double-quoted strings specially, expanding $APP itself before the shell ever sees it. For multi-line commands use triple single quotes:

steps {
  sh '''
    set -e
    echo "Building $APP build number $BUILD_NUMBER"
    make build
  '''
}

Jenkins runs sh with -xe, printing each command and stopping at the first failure. Writing set -e yourself is harmless and makes the intent clear.

Validating before you commit

Declarative scripts are checked before any step runs, so syntax mistakes produce errors like Undefined section "foo" or Missing required section "agent" straight away. You can check a file without running it, using the Declarative Linter built into the server:

curl -X POST -F "jenkinsfile=<Jenkinsfile" -u user:token \
  http://localhost:8080/pipeline-model-converter/validate

A valid file returns Jenkinsfile successfully validated. This saves you from commit-push-wait-fail loops. (The user:token part is an API token; you will create one in the section on the command line and REST API.)

Getting the code: Pipeline from SCM

A build needs the source code. There are two ways to give it to Jenkins, and the difference is important.

The first is a Pipeline job that reads its script from source control. You tell Jenkins the repository address and the path of the Jenkinsfile, and Jenkins checks the repository out and runs what it finds. Create it like this: New Item, name it, choose Pipeline, and in the Pipeline section change Definition to Pipeline script from SCM. Choose Git as the SCM, paste the repository URL, pick credentials if the repository is private (next section), set the branch, for example */main, and keep Script Path as Jenkinsfile.

In a job like this, the checkout of the repository happens automatically before your first stage, which is why the simple Jenkinsfile above never mentions git. If you ever need to disable that automatic checkout, use options { skipDefaultCheckout() }, and to check out again explicitly use the step checkout scm.

The second is a Multibranch Pipeline, which watches a whole repository and creates a child job for every branch (and, with the right plugin, every pull request) that contains a Jenkinsfile. You push a new branch, Jenkins notices, and a job for it appears; you delete the branch and the job goes away. To use it you need a branch-source plugin such as GitHub Branch Source, Bitbucket Branch Source or GitLab Branch Source. For a beginner the single-branch job is enough, and you can move to Multibranch when the team starts working with feature branches. Inside a Multibranch job, extra variables such as BRANCH_NAME and CHANGE_ID tell your script what it is building, and a stage can be limited to pull requests with when { changeRequest() }.

For a local experiment you can point Jenkins at a repository on the same machine, but with Docker the container cannot see your laptop's folders. Use a hosted repository (public repositories on GitHub or GitLab need no credentials) for your first tries.

Polling versus webhooks: how a build starts by itself

So far you clicked Build Now. The real value of CI is that nobody has to click. There are three common triggers.

Polling means Jenkins asks the repository every so often "has anything changed?". It is simple and works everywhere, but it wastes effort and adds delay. A webhook is the reverse: the repository sends Jenkins a message the moment a push happens, so the build starts within seconds and nothing is wasted. For GitHub, the webhook address is your Jenkins URL followed by /github-webhook/. Prefer webhooks whenever your Jenkins is reachable from the hosting provider. A schedule runs the job at fixed times regardless of commits, which suits nightly jobs.

You declare triggers in the Jenkinsfile:

pipeline {
  agent any
  triggers {
    cron('H 2 * * 1-5')
    pollSCM('H/15 * * * *')
  }
  stages {
    stage('Build') { steps { echo 'building' } }
  }
}

The schedule syntax has five fields: minute, hour, day of the month, month, day of the week. H 2 * * 1-5 means "once between 02:00 and 02:59, on weekdays". The letter H stands for "hash": Jenkins picks a stable, job-specific value so that fifty jobs all scheduled for "2 am" do not all start at the same second and overload the server. Prefer H to a fixed minute such as 0. H/15 * * * * means "every fifteen minutes, at a per-job offset". Aliases such as @daily and @hourly exist for the common cases.

A third trigger, upstream(upstreamProjects: 'job1,job2', threshold: hudson.model.Result.SUCCESS), runs this job when other jobs finish successfully.

Environment variables, parameters and choices

A pipeline that does exactly the same thing every time is a good start, but real pipelines need input. Jenkins offers three sources: variables Jenkins provides, variables you define, and parameters that a person supplies when they start a build.

Variables Jenkins gives you

Every build has built-in variables. The ones you will use most are BUILD_NUMBER (for example 7), BUILD_ID, BUILD_TAG, BUILD_URL (the address of this build's page), JOB_NAME, JENKINS_URL, NODE_NAME, WORKSPACE (the directory the build is using) and EXECUTOR_NUMBER. Multibranch jobs add BRANCH_NAME, and the Git plugin adds GIT_COMMIT. You can see all of them for your own server by opening http://localhost:8080/pipeline-syntax/globals.

There are two ways to read a variable, and mixing them up is a classic beginner error.

steps {
  echo "Build number is ${env.BUILD_NUMBER}"
  sh 'echo "Build number is $BUILD_NUMBER"'
}

In the first line, the double-quoted string is Groovy, which reads env.BUILD_NUMBER itself. In the second, the single quotes hand the text to the shell untouched, and the shell expands $BUILD_NUMBER. Both print the same thing, but they work by different mechanisms, and when you write "$FOO" in a Groovy string for a variable that only exists in the shell, you get groovy.lang.MissingPropertyException: No such property: FOO. Rule of thumb: single quotes for shell commands, and only use double quotes when you want Groovy to substitute something.

Parameters

A parameter turns a job into a small application with a form. Declare them in a parameters block:

pipeline {
  agent any
  parameters {
    string(name: 'TARGET_BRANCH', defaultValue: 'main', description: 'Branch to build')
    choice(name: 'ENVIRONMENT', choices: ['dev', 'staging'], description: 'Where to deploy')
    booleanParam(name: 'RUN_SLOW_TESTS', defaultValue: false, description: 'Include the slow suite')
  }
  stages {
    stage('Show') {
      steps {
        echo "Branch ${params.TARGET_BRANCH}, environment ${params.ENVIRONMENT}"
        echo "Slow tests: ${params.RUN_SLOW_TESTS}"
      }
    }
  }
}

Read parameters as params.NAME. The other types are text (multi-line), and password (hidden, but prefer proper credentials, below). After a first build the job's button changes from Build Now to Build with Parameters, and shows the form.

The catch: parameters do not exist until the pipeline has run once, because Jenkins learns about them by running the parameters block. The first build of a new Jenkinsfile therefore runs with no form, and that first run may fail with a "No such property" error if a stage uses a parameter directly. Run it once, and from then on the form appears.

Credentials: handling secrets safely

Builds need secrets: a token to pull a private repository, a password for a registry, an API key for a cloud. The worst way to provide them is to type them into the Jenkinsfile, because the file is committed and everyone with read access to the repository can see it, forever, in its history. Jenkins has a dedicated store for this, and learning to use it is the most important security habit in this guide.

Credentials are secrets stored encrypted on the controller and referred to by an ID that you choose. To add one, go to Manage Jenkins, then Credentials, pick the System store and the Global domain, and click Add Credentials. You choose a Kind: Secret text for a single token, Username with password, Secret file for something like a kubeconfig, or SSH Username with private key. Give it a short, meaningful ID such as registry-login, because the Jenkinsfile will refer to that ID and never to the secret itself.

The two scopes you will see are Global, usable by jobs, and System, meant for the controller's own use such as agent connections. Use Global for anything a pipeline needs.

In a pipeline there are two ways to use a credential. The compact one is credentials() in the environment block:

pipeline {
  agent any
  environment {
    API_TOKEN = credentials('api-token-id')
    REG = credentials('registry-login')
  }
  stages {
    stage('Call the API') {
      steps {
        sh 'curl -s -H "Authorization: Bearer $API_TOKEN" https://example.com/api/status'
        sh 'echo "Logging in as $REG_USR"'
      }
    }
  }
}

For a Secret text credential, API_TOKEN holds the value. For a Username with password credential, Jenkins sets three variables: REG (as user:password), REG_USR and REG_PSW. The second way is withCredentials, which limits the secret to a block, and is the better choice when only one step needs it:

steps {
  withCredentials([usernamePassword(credentialsId: 'registry-login',
                                    usernameVariable: 'REG_USER',
                                    passwordVariable: 'REG_PASS')]) {
    sh 'echo "$REG_PASS" | docker login -u "$REG_USER" --password-stdin registry.example.com'
  }
}

Other binding types include string(credentialsId:, variable:) for secret text, file(credentialsId:, variable:) for secret files, and sshUserPrivateKey(credentialsId:, keyFileVariable:, usernameVariable:) for SSH keys. Use the Snippet Generator to fill the form rather than memorising the shape.

If you write the same thing with double quotes, you will see a warning and have created a real leak.

One more thing to know about the SCM side. When Jenkins clones a Git repository over SSH, it checks the server's host key. If you see Host key verification failed or Permission denied (publickey), the credential selected in the job is wrong or missing, or Jenkins has not been told to trust the host. The trust setting is under Manage Jenkins, Security, Git Host Key Verification Configuration.

Agents, executors and labels

Everything so far used agent any, and on a fresh install it ran on the built-in node, the controller's own machine. That is fine for learning and wrong for anything shared. A build is arbitrary code written by someone. If it runs on the controller, it runs with the same access to the controller's files as Jenkins itself, including the secrets directory. The standard guidance is to set the built-in node's executors to zero (Manage Jenkins, Nodes, Built-In Node, Configure) and run builds on agents.

There are three ways to get an agent.

1. A permanent agent over SSH. The controller connects out to a machine you own using an SSH key. This is the most stable and the recommended style for static machines. It needs the SSH Build Agents plugin and an "SSH Username with private key" credential.
2. An inbound agent. The agent connects in to the controller. You create the node in the UI, get a secret, and run java -jar agent.jar -url ... -secret ... -name ... -webSocket on the machine. The -webSocket option means no extra port is needed and it works through ordinary HTTPS proxies.
3. A cloud agent, created on demand (a Kubernetes pod, a cloud virtual machine, a Docker container) and deleted afterwards. This is the modern approach because every build starts from a clean machine.

To create a permanent node, go to Manage Jenkins, Nodes, New Node, name it, choose Permanent Agent, and fill in the Remote root directory (the folder on the agent where workspaces will live), the number of executors, the labels, and the launch method.

Labels connect jobs to nodes. If you give a node the labels linux docker, a pipeline can ask for it:

pipeline {
  agent { label 'linux && docker' }
  stages {
    stage('Where am I') {
      steps { sh 'hostname' }
    }
  }
}

A label expression can use && (and), || (or) and ! (not). If no online node matches, the build sits in the queue with a message such as Waiting for next available executor on ‘linux’ or There are no nodes with the label ‘linux’. That is not a crash; it is Jenkins telling you that nothing can run the work. Check that the agent is online and the label is spelled exactly.

You can also ask for a fresh container as the agent, using the Docker Pipeline plugin (docker-workflow), provided Docker is installed on the node that runs it:

pipeline {
  agent {
    docker { image 'maven:3.9-eclipse-temurin-21' }
  }
  stages {
    stage('Build') { steps { sh 'mvn -v' } }
  }
}

The build runs inside that image, so the tools (here Maven and a JDK) come with it and nothing needs installing on the node. This pattern is very popular, because it is the same idea as the Docker guide, applied to builds: the environment is part of the pipeline. Other agent forms are agent none (declare no global agent and set one per stage), dockerfile (build an image from a Dockerfile in the repository) and kubernetes (run in a pod, covered in the Mid-level guide).

Results, test reports and artifacts

A build that only prints text is half a build. The other half is what it leaves behind: test results you can browse, and files you can download.

Build results. A build is SUCCESS if everything passed and FAILURE if a step failed. It is UNSTABLE when it ran to the end but something soft went wrong, such as failing tests that the junit step reported. ABORTED means a person or a timeout stopped it, and NOT_BUILT means it was skipped. Colours follow: blue or green for success, yellow for unstable, red for failure, grey for aborted or not built.

Test reports. Most test tools can emit JUnit-style XML. Point the junit step at those files, and Jenkins shows the number of passed and failed tests, lists failures with their messages, and draws a trend graph across builds. Put it in a post { always { ... } } block as in the earlier Jenkinsfile, so reports are collected even when tests fail.

Artifacts. archiveArtifacts artifacts: 'dist/**', fingerprint: true stores the matching files from the workspace with the build, and they appear as downloads on the build page. Artifacts are for modest outputs that belong with a build. Large binaries and container images should go to a proper artifact repository or registry, not to JENKINS_HOME, which is the same directory that holds your configuration.

Timestamps. Adding timestamps() to options prefixes every log line with a time, which makes slow steps obvious. It needs the Timestamper plugin.

Finally, decide what to do when a step fails but the pipeline should continue. By default the first failure stops the build. post and catchError let you react:

stage('Optional lint') {
  steps {
    catchError(buildResult: 'SUCCESS', stageResult: 'FAILURE') {
      sh 'make lint'
    }
  }
}

Here a failing lint marks the stage red but lets the build go on and finish successful. Use this sparingly; a pipeline that ignores its own failures is worse than none.

Plugins: what they are and how to treat them

Jenkins' power and its most common headaches both come from plugins. Each is a package (.hpi or .jpi file) that adds a feature, installed from the Update Center. You manage them at Manage Jenkins, Plugins, which has tabs for Updates, Available plugins, Installed plugins and Advanced settings (where you configure an HTTP proxy or a different update site).

The plugins a beginner meets, with the ID you would use to install them in a file:

Good habits with plugins begin on day one. Install only what you need, because every plugin is code running with Jenkins' authority and every one is something to keep updated. Read the update list before clicking Download now and install after restart, since an update can change behaviour. Take a backup before you update on a server that matters. And treat security advisories seriously: Jenkins publishes them at jenkins.io/security/advisories/, and a vulnerable plugin is one of the most common ways Jenkins servers are attacked.

A final piece of history matters for upgrades. In LTS 2.580.1, nine plugins that used to be bundled inside jenkins.war are no longer included (among them JAXB, JavaMail API, SSH server and Instance Identity). A normal online server downloads them automatically when they are needed. An offline server must fetch the .hpi files and place them in $JENKINS_HOME/plugins before starting the new version, or some configuration will not load.

The command line and the REST API

Clicking is fine for learning, but automation needs to talk to Jenkins from scripts. There are two supported ways: the command-line client (a jar you download from your own server) and the HTTP API. Both need you to identify yourself, and the right way is an API token.

Create a token. Click your name in the top right, then Security, then Add new token under API Token. Copy it immediately; Jenkins shows it only once. Since LTS 2.555.1 a token can have an expiration date, which is a good practice for anything unattended. A token is a password equivalent for scripts, so treat it like one, and use a different token for each script so you can revoke one without breaking the others.

Trigger a build with curl:

curl -X POST -u admin:YOUR_TOKEN http://localhost:8080/job/hello-pipeline/build

For a job with parameters use buildWithParameters:

curl -X POST -u admin:YOUR_TOKEN \
  http://localhost:8080/job/hello-pipeline/buildWithParameters --data ENVIRONMENT=staging

A successful request returns HTTP 201 and a Location: header pointing at /queue/item/N/. That is the queue entry, not the build; Jenkins starts the build when an executor is free. Jobs inside folders have longer paths: /job/folder/job/name/.

Read information. Add /api/json to almost any Jenkins page address and you get a machine-readable version of it. The tree parameter limits the output to what you need:

curl -s -u admin:YOUR_TOKEN \
  'http://localhost:8080/api/json?tree=jobs[name,color,lastBuild[number,result]]'
curl -s -u admin:YOUR_TOKEN http://localhost:8080/job/hello-pipeline/lastBuild/api/json
curl -s -u admin:YOUR_TOKEN http://localhost:8080/job/hello-pipeline/3/consoleText

The last one prints the full console log of build 3 as plain text, which is handy for feeding into other tools.

About the CSRF crumb. Jenkins protects against cross-site request forgery (a malicious web page making your browser send a request), by requiring a special value called a crumb on state-changing requests. When you authenticate with an API token, you do not need one, which is one more reason to use tokens. If you authenticate with a password or a browser session and get HTTP ERROR 403 No valid crumb was included in the request, switch to a token. Never switch CSRF protection off to make the error go away.

The command-line client.

curl -O http://localhost:8080/jnlpJars/jenkins-cli.jar
export JENKINS_USER_ID=admin
export JENKINS_API_TOKEN=YOUR_TOKEN
java -jar jenkins-cli.jar -s http://localhost:8080/ who-am-i
java -jar jenkins-cli.jar -s http://localhost:8080/ build hello-pipeline -f -v

who-am-i confirms your identity. build hello-pipeline -f -v starts the job, waits for it to finish (-f) and streams the console (-v). Running help lists every command. The client connects over WebSocket by default (since 2.391), and also supports -http and -ssh. The old -remoting mode no longer exists; if an old script uses it, rewrite it.

Configuration, JENKINS_HOME, backups and upgrades

A lot of beginners treat Jenkins as a black box until the day it will not start. A little knowledge of how it stores things removes most of the fear.

What lives in JENKINS_HOME. Open the directory (remember the default locations from the install section) and look around. config.xml is the main configuration. jobs/ has a folder per job with its config.xml and its builds. plugins/ holds installed plugins. secrets/ has the keys that encrypt your credentials. users/ has accounts, and nodes/ has agent definitions. Every setting you change in the interface ends up as XML in this tree.

Backups. Back up config.xml, jobs/, plugins/, secrets/, users/, nodes/ and credentials.xml. You can skip war/, cache/, tools/ and workspaces, since Jenkins recreates them. Two points catch people out. A backup that lacks secrets/ cannot decrypt the credentials it contains, and a backup that includes secrets/ is itself sensitive, so store it encrypted. On a server, filesystem or volume snapshots are the simplest approach; the ThinBackup plugin is another. Test restoring at least once, on a scratch machine.

Keeping disk under control. Use buildDiscarder in every Jenkinsfile. Node monitors mark a machine offline when free space drops below about 1 GiB, with a message such as Disk space is too low. Only 0.9GB left on /var/lib/jenkins.; if you see it, clean old builds and workspaces.

Upgrading safely. Stay on the LTS line, and read the upgrade guide for the version you are moving to (for example jenkins.io/doc/upgrade-guide/2.580/) together with the changelog. The routine is: back up JENKINS_HOME, check that your Java version is still supported, update plugins, then upgrade Jenkins, then check the banners in Manage Jenkins. On Docker, change the image tag and keep the same volume. For graceful restarts, safe-restart in the CLI waits for running builds to finish first, instead of killing them.

Configuration as code. Clicking through settings does not scale and cannot be reviewed. The Configuration as Code plugin (JCasC) lets you write controller settings in a YAML file. You will use it at the Mid-level; for now, know it exists and that a setting changed in the interface may be overwritten by JCasC on the next restart if you use both.

Common errors and how to read them

Jenkins errors look intimidating because they are long Java stack traces, but almost all of them reduce to a short list of causes. The method is always the same: read the console from the top, find the first line that says ERROR or shows an exception, and ignore the dozens of lines that follow.

Two of these deserve a longer word. The Java message is the classic "I followed an old tutorial" failure: you installed the operating system's default Java, which might be 17 or even 11, and Jenkins refuses it with an exact message listing the supported versions. The fix is to install Java 21 and tell Jenkins which Java to use. On an agent, the equivalent message is UnsupportedClassVersionError ... class file version 65.0, which means the agent's Java is older than the controller's; upgrade it.

And the "Scripts not permitted" message deserves caution. Jenkins runs untrusted pipeline code inside a sandbox that only allows a safe list of operations. When it blocks something, an administrator can approve the call on the In-process Script Approval page, but some approvals (for example anything that reaches the Jenkins internals directly) give away administrator power. Before approving, ask whether a normal step would do the job.

Putting it all together

Here is one small project that uses nearly everything above: a Python service with tests, built on an agent, with a timeout, log retention, a parameter, a secret, test reports, an archived artifact and a nightly schedule. Put this in a repository with a tiny app.py and test_app.py, then create a Pipeline script from SCM job.

pipeline {
  agent {
    docker { image 'python:3.12-slim' }
  }
  options {
    timeout(time: 20, unit: 'MINUTES')
    buildDiscarder(logRotator(numToKeepStr: '15'))
    timestamps()
  }
  triggers {
    cron('H 2 * * 1-5')
  }
  parameters {
    booleanParam(name: 'PUBLISH', defaultValue: false, description: 'Upload the package')
  }
  environment {
    HOME = "${WORKSPACE}"
  }
  stages {
    stage('Install') {
      steps {
        sh 'pip install --user pytest'
      }
    }
    stage('Test') {
      steps {
        sh '~/.local/bin/pytest --junitxml=reports/results.xml'
      }
      post {
        always {
          junit 'reports/results.xml'
        }
      }
    }
    stage('Package') {
      steps {
        sh 'mkdir -p dist && tar czf dist/app-${BUILD_NUMBER}.tgz app.py'
      }
    }
    stage('Publish') {
      when { expression { return params.PUBLISH } }
      steps {
        withCredentials([string(credentialsId: 'package-token', variable: 'TOKEN')]) {
          sh 'echo "would upload with a token of length ${#TOKEN}"'
        }
      }
    }
  }
  post {
    success {
      archiveArtifacts artifacts: 'dist/**', fingerprint: true
    }
    cleanup {
      cleanWs()
    }
  }
}

Walk through it in order. The agent is a python:3.12-slim container, so nothing has to be installed on the node apart from Docker itself. The environment block sets HOME into the workspace, because the container runs as a user that may have no writable home directory; that is what lets pip install --user work. The three options give you a timeout, bounded history and timestamps. The Test stage publishes reports in always, so a failure still produces a readable report. The Package stage builds a file whose name includes BUILD_NUMBER, making every artifact traceable to a build. The Publish stage only runs when a person ticks the box, through a when condition, and uses a credential the safe way. Finally, post archives the artifact on success and cleans the workspace whatever happened.

Run it three times: once normally, once with the PUBLISH box ticked, and once after you break a test on purpose. Compare the three build pages. The broken run should show a red Test stage, a test report listing the failure, no Package stage and no archived artifact, because the pipeline stopped where it should.

When you are comfortable with this, connect a webhook so every push builds, and then read how the same pipeline would deploy to a cluster in the Kubernetes guide.

What you can now do, and what comes next

You can install Jenkins on the platform of your choice and confirm the version and the Java it runs on. You can explain the controller, agents, executors, jobs, builds and workspaces, and say why the built-in node should run nothing in production. You can write a Declarative Jenkinsfile with stages, options, environment variables, parameters, triggers, post actions, test reports and artifacts. You can keep secrets out of your code and out of your logs, run builds on labelled agents or in containers, and start and inspect jobs from curl and the command-line client. And you can read a red build: find the first real error, classify it, and fix it.

What you cannot do yet, and where the next levels go. The Mid-level guide covers the parts that turn a working pipeline into a good one: parallel stages, when conditions and matrix builds, approval gates, shared libraries so ten repositories do not copy one Jenkinsfile, Kubernetes-based agents, how Pipeline code actually executes (and why some Groovy fails mysteriously), and proper multibranch setups. The Senior guide covers running Jenkins as a platform: securing it with real authorization, isolating teams, scaling, observability, upgrades and backups at scale, and when to choose something else.

Neighbouring guides in this catalogue are the natural next stops. Docker deepens the container skills your pipelines depend on. Kubernetes and Helm cover where many teams deploy to and how they package what they deploy. Terraform is the usual way to create the machines that Jenkins runs on. And Argo CD is what many teams pair with Jenkins: Jenkins builds and tests and pushes an image, and Argo CD handles the deployment by watching a Git repository.

A closing piece of advice. The people who get the most from Jenkins treat the Jenkinsfile as production code: small, reviewed, tested by running, and boring. The people who struggle let pipelines grow into hundreds of lines of clever Groovy, click changes into the interface that nobody records, and install plugins without reading them. Choose to be the first kind from your first build.

Sources

• Jenkins documentation
• Jenkins LTS changelog
• Upgrade guide for 2.580
• Java support policy
• Installing Jenkins on Linux
• Installing Jenkins with Docker
• Installing Jenkins on macOS
• Installing Jenkins on Windows
• Running Jenkins from the WAR file
• Installing Jenkins on Kubernetes
• Pipeline syntax
• Using a Jenkinsfile
• Using credentials
• Using agents
• Managing nodes
• Jenkins CLI
• Remote access API
• CSRF protection
• Script approval
• Blue Ocean status
• Security advisories

Part one of three. A twenty-five-minute review of the Beginner Detailed track for Jenkins LTS 2.580.1, then the questions interviewers actually ask at this level. Mid-level reviews this plus its own material; Senior reviews all three.

Interviewers for junior DevOps and MLOps roles rarely want Groovy trivia. They want to hear that you understand what a CI server is for, that you can read a failed build, and that you know which habits keep Jenkins safe. Answer in short, concrete sentences, and back each claim with a command or a file name.

The thirty-second answer

Jenkins is an open-source automation server. A central controller stores configuration, schedules work and serves the web interface, while agents run the builds. You describe the work as a pipeline in a Jenkinsfile kept in the repository, split into stages of steps, and Jenkins runs it on every push, on a schedule or on demand, keeping the log, test results and artifacts of every run.

Then add the line that shows you have used it: "the controller should not run builds itself, because build code would then share the controller's files and secrets. Agents do the work, and the built-in node is set to zero executors."

The model

Interviewers often ask you to draw it. Draw five boxes and three arrows and say the nouns aloud.

The controller is the Java web application that holds configuration and decides what runs. An agent is a machine or container running agent.jar that connects to it. A node is any machine able to run builds, including the controller's built-in node. An executor is one build slot on a node. A label is a tag on a node that jobs use to choose where to run. A job is the configured recipe, and a build is one numbered run of it with a result of SUCCESS, UNSTABLE, FAILURE, ABORTED or NOT_BUILT. A pipeline is a job defined as code, in a Jenkinsfile, made of stages and steps. A workspace is the directory on the agent where a build works.

Everything lives in JENKINS_HOME: configuration, jobs, build records, plugins and the keys that encrypt credentials. That is what you back up and protect.

Declarative versus Scripted

A classic opener. There are two Pipeline syntaxes.

Declarative starts with pipeline { ... } and has a fixed structure: agent, stages, stage, steps, post, plus directives such as options, environment, parameters, triggers and when. It is validated before it runs, which gives readable errors early, and it is the recommended syntax to start with. Scripted starts with node { ... } and is plain Groovy with loops, conditionals and functions. It is more flexible and easier to get wrong.

A good answer adds: "Declarative has a script { } block as an escape hatch into Scripted Groovy when I need a variable or a loop, and I keep those blocks small."

The commands and snippets to have ready

You will be asked to write something on a whiteboard or a shared screen. Have these shapes in your fingers.

A minimal Declarative pipeline:

pipeline {
  agent any
  options {
    timeout(time: 30, unit: 'MINUTES')
    buildDiscarder(logRotator(numToKeepStr: '20'))
  }
  stages {
    stage('Build') { steps { sh 'make build' } }
    stage('Test') {
      steps { sh 'make test' }
      post { always { junit 'reports/**/*.xml' } }
    }
  }
  post {
    success { archiveArtifacts artifacts: 'dist/**', fingerprint: true }
    cleanup { cleanWs() }
  }
}

Triggering a build from a script, with an API token:

curl -X POST -u alice:TOKEN https://ci.example.com/job/my-job/build
curl -X POST -u alice:TOKEN https://ci.example.com/job/my-job/buildWithParameters --data BRANCH=main

Reading status as JSON:

curl -s -u alice:TOKEN 'https://ci.example.com/job/my-job/lastBuild/api/json?tree=number,result'

The command-line client:

curl -O https://ci.example.com/jnlpJars/jenkins-cli.jar
java -jar jenkins-cli.jar -s https://ci.example.com/ -auth alice:TOKEN who-am-i
java -jar jenkins-cli.jar -s https://ci.example.com/ -auth alice:TOKEN build my-job -f -v

Install and verify, on a Linux package install:

sudo systemctl status jenkins
sudo cat /var/lib/jenkins/secrets/initialAdminPassword
journalctl -u jenkins.service
curl -sI http://localhost:8080/login | grep -i X-Jenkins

Docker quick start:

docker run -p 8080:8080 -p 50000:50000 --restart=on-failure \
  -v jenkins_home:/var/jenkins_home jenkins/jenkins:lts-jdk21

Be ready to explain every part of that last command: the two published ports, why there is a named volume, and why a bind mount of a host folder often fails (the container runs as user 1000).

Which version, which Java

Expect one version question, because it separates people who followed a current tutorial from people who copied an old one. The current long-term-support release is 2.580.1, released on 30 September 2026. It runs on Java 21 or Java 25. Java 17 stopped being supported with LTS 2.555.1, and the container images for it were removed. The Java requirement covers the controller, every agent and the CLI client.

If they ask about release lines: LTS gets a new baseline about every twelve weeks and is then patched as .1, .2, .3 roughly every four weeks, and weekly releases ship every week. Production should use LTS.

Follow-up facts that impress: Jenkins tests on HotSpot-based JVMs (OpenJDK, Temurin, Corretto) and not on OpenJ9; the Linux packages use a repository key named jenkins.io-2026.key; and Blue Ocean has been deprecated since July 2026, so you use Pipeline Graph View instead.

Build results and what "unstable" means

SUCCESS means every step passed. FAILURE means a step failed. UNSTABLE means the build ran to the end but something soft went wrong, typically failing tests that the junit step recorded. ABORTED means a person or a timeout stopped it. NOT_BUILT means it was skipped.

The mechanism behind failure: a shell step fails when the command exits non-zero. That is why pytest or mvn test can be called directly, with no special integration. If a pipeline must continue after a failing step, catchError(buildResult: 'SUCCESS', stageResult: 'FAILURE') records the failure without stopping the build. Use it sparingly.

Agents, executors and labels

Interviewers like this topic because it tests whether you understand where code runs.

There are three ways to connect an agent. SSH: the controller connects out to a machine you own; it is stable and is the usual choice for static machines. Inbound: the agent connects in, using java -jar agent.jar -url ... -secret ... -name ... -webSocket. WebSocket means no extra port is needed and it works through HTTP proxies. Cloud: Kubernetes, EC2 or Docker creates an agent per build and deletes it afterwards.

Why not run builds on the controller? A build is arbitrary code, and on the built-in node it has "the same level of access to the controller file system as the Jenkins process". That includes the secrets/ directory. So set the built-in node's executors to zero and use agents.

Why one executor per node? Two builds on one machine share CPU, disk and ports and produce failures that are hard to explain. The documentation calls one executor per node the safest configuration.

A label expression can use &&, || and !: agent { label 'linux && docker' }. If no online node matches, the build waits in the queue with Waiting for next available executor or There are no nodes with the label. That is not a bug; it is a capacity or naming problem.

A Docker agent, with its dependency:

agent { docker { image 'maven:3.9-eclipse-temurin-21' } }

It needs the Docker Pipeline plugin and Docker on the node that runs the build.

Triggers: polling, webhooks and schedules

Three ways a build starts without a click. Polling (pollSCM('H/15 * * * *')) makes Jenkins ask the repository whether anything changed. Webhooks make the repository tell Jenkins, so builds start within seconds and nothing is wasted; for GitHub the address is JENKINS_URL/github-webhook/. Schedules (cron('H 2 * * 1-5')) run at set times.

Explain the cron syntax: minute, hour, day of month, month, day of week. Explain H: it hashes a value per job so that many jobs scheduled for "2 am" do not all start in the same second. Mention the aliases @daily, @hourly, @weekly. And the practical catch: a triggers block is only registered after the pipeline has run once.

The preferred answer to "polling or webhooks?" is webhooks, with polling as a fallback when Jenkins is not reachable from the repository host.

Credentials

This is the most important section for the "safe habits" question.

Secrets are stored encrypted on the controller and referenced by ID. Kinds include secret text, username with password, secret file and SSH key. Scopes are Global (usable by jobs) and System (for the controller's own use, such as agent connections). In a pipeline you use them with credentials('id') in environment (which sets VAR, plus VAR_USR and VAR_PSW for username/password) or with withCredentials([...]) to limit the secret to a block.

The trap, which interviewers love: never use a secret inside a Groovy double-quoted string. sh "curl -u $USER:$PASS ..." makes Groovy expand the secret before the shell runs, and it becomes part of the process arguments, visible to other users via ps. Jenkins prints Warning: A secret was passed to "sh" using Groovy String interpolation, which is insecure. The fix is single quotes: sh 'curl -u "$USER:$PASS" ...', so the shell expands the variable.

Add the second point: masking is best-effort. Jenkins replaces the exact secret with **** in the log, but a base64-encoded or split value is not recognised. Masking is a courtesy, not a security boundary.

Plugins

Nearly all functionality comes from plugins installed from the Update Center. Name a few with their purpose: Pipeline (workflow-aggregator), Git, Credentials Binding, Pipeline Graph View, Workspace Cleanup (ws-cleanup, for cleanWs), JUnit, Timestamper (for timestamps()), Docker Pipeline (docker-workflow).

The habit to state: install only what you need, pin and review updates on servers that matter, and watch the security advisories, because every plugin is code running with Jenkins' authority. The standard symptom of a missing plugin is No such DSL method 'xyz' found among steps.

A recent fact worth having: LTS 2.580.1 stopped bundling nine plugins (including JAXB, JavaMail API and SSH server) inside jenkins.war. Online installs fetch them on demand; offline installs must place the .hpi files in $JENKINS_HOME/plugins before upgrading.

The HTTP API and the CSRF crumb

Any page address plus /api/json returns JSON, and ?tree= limits the fields. Authentication is HTTP Basic with a username and an API token, which you create under your user menu, Security, API Token. Since 2.555.1 a token may carry an expiration date.

Why the crumb exists: Jenkins requires a crumb on state-changing requests to stop cross-site request forgery, where a malicious page makes your logged-in browser send a request. Requests authenticated with an API token are exempt from crumbs. If you see HTTP ERROR 403 No valid crumb was included in the request, you used a password or a session without the crumb. Use a token. If you must use a session, fetch /crumbIssuer/api/json with a cookie jar and send the Jenkins-Crumb header together with the same cookie. Never disable CSRF protection to make the error go away.

A successful build request returns HTTP 201 with a Location: header pointing to /queue/item/N/, which is the queue item, not the build.

Reading a failed build

Interviewers often hand you a log and ask what went wrong. Use a fixed routine.

1. Go to Console OutputOpen the failed build and read the text log. The graphical view tells you which stage failed; the log tells you why.
2. Find the first real errorStart at the bottom, then scroll up until you meet the first command output that looks wrong. Ignore the long stack trace that follows it.
3. Classify the linescript returned exit code N means a shell command failed. No such DSL method means a missing plugin. Expected a step means Groovy outside script. Waiting for next available executor means no node matches.
4. Read the exit code1 is a general failure, 127 is command not found, 137 is killed (often out of memory) and 143 is a signal or timeout.
5. Reproduce it by handRun the same command on the agent, or in the same container image. Most failures are environment differences.
6. Fix and rebuildChange the Jenkinsfile, commit, and let the next build prove the fix.

The traps, and why they share one cause

Four traps account for most beginner failures, and they share one cause: the build does not run where, or how, you assume.

The wrong quotes. "$FOO" in a Groovy string is expanded by Groovy, not by the shell, which produces MissingPropertyException for shell-only variables and leaks secrets. Single quotes hand the text to the shell.

The wrong Java. Jenkins 2.555.1 and later needs Java 21 or 25, and the operating system's default Java is often older. The controller refuses to start with a message listing the supported versions. On an agent, UnsupportedClassVersionError ... class file version 65.0 says the agent's Java is too old.

The missing plugin. A step that does not exist on this server, because its plugin is not installed.

The wrong place. The build ran on an agent without the tool it needs, the workspace had old files in it, or a Docker-based agent had no access to the Docker socket. The console's Running on ... line tells you which node ran it.

Add two operational traps that show you have run a server: a full disk caused by missing buildDiscarder (the node goes offline with Disk space is too low), and a Docker container with no persistent volume, which loses every job when the container is removed.

Common interview questions

1. What is Jenkins and why use it?An automation server that runs builds, tests and deployments automatically and keeps the record. It makes the steps explicit, repeatable and early: a failing test found minutes after a commit is cheap to fix. It also works anywhere, including inside a private network, which matters when source code or data must stay in a particular country or company.
2. What is the difference between a controller and an agent?The controller stores configuration, schedules builds, serves the UI and API. Agents run the builds. In production the built-in node has zero executors so that build code never runs next to the controller's secrets.
3. What is a Jenkinsfile and why keep it in the repository?A text file defining the pipeline. Stored with the code it is versioned, reviewed, branched and restored like any other source, so an old commit can be rebuilt exactly as it was.
4. Declarative or Scripted?Declarative first: structured, validated early, easier to read. Scripted when I need full Groovy, and inside Declarative I use a small script block for that.
5. Explain the stages of a typical pipeline.Checkout (automatic in Pipeline-from-SCM), build, test, package or publish, and deploy, with test reports published in post { always } and artifacts archived on success.
6. How do you handle secrets?Store them as credentials, reference them by ID, bind them with credentials() or withCredentials, use single quotes so the shell expands them, and never echo them. Masking is best-effort, so I also restrict who can use each credential.
7. What does post do and what are its conditions?It runs after the stages. Conditions are always, changed, fixed, regression, aborted, failure, success, unstable, unsuccessful and cleanup, with cleanup running last.
8. How do you start a build automatically?A webhook from the repository host, which is fastest; pollSCM as a fallback; or cron for schedules. I use H in schedules to spread load.
9. What is H in a cron expression?A per-job hash that picks a stable minute or hour within the range, so many jobs with the same nominal schedule do not start simultaneously.
10. How do you pass input to a pipeline?Parameters: string, choice, booleanParam, text and password, read as params.NAME. They appear after the pipeline has run once, and the button becomes Build with Parameters.
11. How would you run a build in a clean environment?A Docker agent such as docker { image 'python:3.12-slim' } (needs the Docker Pipeline plugin and Docker on the node), or an ephemeral cloud agent. The tools come with the image, so nothing is installed on the node.
12. A build says "Waiting for next available executor". What do you check?Whether an online node has the requested label, whether the built-in node has zero executors with only agent any available, whether all executors are busy, and whether a cloud cap has been reached.
13. How do you trigger Jenkins from a script?curl -X POST -u user:TOKEN .../job/NAME/build or buildWithParameters; or the CLI with build NAME -f -v. An API token avoids the crumb requirement.
14. What do you back up?JENKINS_HOME: config.xml, jobs/, plugins/, secrets/, users/, nodes/ and credentials.xml; not war/, cache/, tools/ or workspaces. The backup contains secret keys, so it must be stored encrypted.
15. How do you keep Jenkins from filling its disk?buildDiscarder(logRotator(numToKeepStr: '20')) in every pipeline, cleanWs() in post { cleanup }, large artifacts pushed to a repository instead of kept in JENKINS_HOME, and monitoring of free space.
16. What changed recently that an older tutorial would get wrong?Java 17 is no longer supported (21 or 25 only), the Debian and RPM signing key is jenkins.io-2026.key, Blue Ocean is deprecated, the old CLI -remoting mode is gone, and the built-in node and agents replaced the words master and slave.
17. What is the Snippet Generator?A page at /pipeline-syntax that builds the exact step syntax for the plugins installed on this server. I use it instead of memorising arguments.
18. Jenkins or GitHub Actions?Actions is simplest when the code is on GitHub and hosted runners are acceptable. Jenkins is chosen when builds must run inside a private network or on special hardware, when one system must serve many source hosts, or when a large estate of jobs already exists. The cost of Jenkins is that you run and patch it yourself.
19. How do you deploy from Jenkins?At this level: a stage that runs the deployment tool (kubectl, helm, terraform or a script) from an agent that has the credentials. A common pattern is for Jenkins to build, test and push an image, and for a GitOps tool such as Argo CD to do the deployment.
20. What does ERROR: script returned exit code 1 tell you?Not that Jenkins is broken. A shell command failed. The real error is above that line.

A worked scenario

Interviewers often finish with a scenario. Here is one, with the answer path.

"Your pipeline passed yesterday. Today the same commit fails at the Test stage with pytest: command not found. What do you do?"

Start by reading the console rather than guessing. Exit code 127 says the command does not exist on the node that ran it. Check the Running on line, which shows whether a different agent picked up the job today. If it did, the old agent had pytest installed and the new one does not, which means the environment lives on the machine instead of in the pipeline. The durable fix is to put the environment in the pipeline: use a Docker agent with an image that contains the tool, or an install step inside the pipeline. Then explain what you would change so the problem cannot come back: the same image every time, and a label that only selects agents prepared for this job.

That answer shows the habit interviewers look for: read first, identify where the code ran, and move the environment into code.

Sixty-second self-test

• Say in two sentences what Jenkins is, and what the controller and agents each do.
• Explain why the built-in node should have zero executors in production.
• Name the core nouns (controller, agent, node, executor, job, build) and say what a label is.
• Write a Declarative pipeline with agent, stages, steps and post from memory.
• Explain why sh "...$SECRET..." is unsafe and what to write instead.
• Name the post conditions and say which runs last.
• Explain H in H 2 * * 1-5.
• Say why webhooks beat polling.
• Explain when a parameters form first appears, and why.
• Give the current LTS and its Java requirement, and say what is wrong with Java 17 advice.
• Explain the CSRF crumb and why an API token avoids it.
• Say what exit codes 127, 137 and 143 mean.
• List what to back up, and what makes a backup sensitive.
• Give the three ways to connect an agent.
• Name what replaced Blue Ocean.

Part one of three. Most Jenkins trouble for beginners has three sources: the build runs somewhere or as someone you did not expect, Groovy and the shell are mixed up, or the server was set up by clicking and nobody wrote down how. Start with the table, then the habits and the recipes.

Common errors at this level

Mistakes beginners actually make

The table lists symptoms. The sections below follow the ones that cost the most time, each with what you see, why, and what to do.

Building on the controller

Symptom. Everything works, because agent any runs on the built-in node. Then a teammate's test writes into /tmp, another build fills the disk, or a script does something to the machine that Jenkins itself runs on.

Cause. Builds are arbitrary code. On the built-in node they run with "the same level of access to the controller file system as the Jenkins process", including the secrets/ directory that decrypts every credential.

Fix. Set the Built-In Node's executors to zero (Manage Jenkins, Nodes, Built-In Node, Configure) as soon as you have one real agent, even a container. For learning on a single machine this is unnecessary, but write down that it must change before anyone else uses the server.

Mixing up quotes

Symptom. A variable prints empty, or MissingPropertyException, or a warning that a secret was passed using Groovy String interpolation.

Cause. In Groovy, double-quoted strings substitute $name themselves; single-quoted strings do not. A shell variable written in double quotes is looked up in Groovy before the shell ever runs.

Fix. Default to single quotes for sh steps. Reach for double quotes only when you want Groovy to insert a value, for example sh "echo ${params.TARGET}" for a non-secret parameter, and never for a secret.

Typing secrets into the Jenkinsfile or a job

Symptom. A token sits in a committed file, or a password parameter holds a real password that appears in build records.

Cause. It was the quickest way to make the build pass.

Fix. Create a credential, refer to it by ID, and rotate anything that has ever been committed. A secret in Git history is compromised even after you delete the line. A password parameter is hidden in the form, but it is not a substitute for the credentials store.

Forgetting buildDiscarder and timeout

Symptom. Weeks later the disk is full and every node reports Disk space is too low, or a single hung test holds an executor for a day.

Cause. Jenkins keeps every build by default and waits as long as a build runs.

Fix. These two lines belong in every Jenkinsfile from day one:

options {
  timeout(time: 30, unit: 'MINUTES')
  buildDiscarder(logRotator(numToKeepStr: '20'))
}

Treating a Docker volume as optional

Symptom. You upgrade or recreate the Jenkins container and find a blank server, the unlock screen included.

Cause. Without a volume, /var/jenkins_home lived in the container's writable layer.

Fix. Always mount a named volume: -v jenkins_home:/var/jenkins_home. To upgrade, change the image tag and keep the same volume.

Following an old tutorial

Symptom. Java 17 instructions, redhat-stable URLs, the 2023 repository key, Blue Ocean, the -remoting CLI mode.

Cause. Jenkins is old and the internet remembers everything.

Fix. Check the date of any tutorial. Current facts for this guide: Java 21 or 25, rpm-stable, jenkins.io-2026.key, Pipeline Graph View instead of Blue Ocean, and CLI modes -webSocket, -http and -ssh. When in doubt, read the official documentation for your version at jenkins.io/doc/.

Expecting parameters and triggers to work on the first run

Symptom. A new job has no Build with Parameters button, its cron never fires, and the first run fails on params.X.

Cause. Jenkins learns about parameters and triggers by running the pipeline. The first run is what registers them.

Fix. Run a new Jenkinsfile once, then test the parameters and schedule. Give parameters default values so the first run can succeed.

Turning off safety features to silence an error

Symptom. A 403 crumb error leads someone to disable CSRF protection; a sandbox rejection leads to approving every script.

Cause. The error appears when you are trying to get something working, and the quickest route is to remove the check.

Fix. Use an API token, which does not need a crumb. For sandbox errors, prefer a plugin step, and approve a method signature only after you understand what it can do. Some approvals, such as direct access to the Jenkins object, give administrator-level power.

Installing every plugin that looks useful

Symptom. Slow startup, warnings on the Manage Jenkins page, a plugin update that breaks a job.

Cause. Each plugin is extra code and extra maintenance.

Fix. Keep a list of the plugins you have a reason for. Put it in a plugins.txt file so a new server can be rebuilt from it, and review the list every few months.

The habits that pay off most

Adopt these in your first week. They cost minutes and save days.

1. Put the Jenkinsfile in the repository. Never leave the only copy of a pipeline in the web form.
2. Start every pipeline with timeout and buildDiscarder. They prevent the two slowest-growing failures.
3. Single quotes for shell, always. Switch to double quotes only on purpose.
4. Secrets only through credentials, bound in the smallest block that needs them.
5. Read Console Output from the bottom up, and find the first real error.
6. Use the Snippet Generator at /pipeline-syntax rather than copying old syntax.
7. Put the environment in the pipeline, with Docker agents or explicit install steps, so a new agent does not break builds.
8. Publish test reports in post { always }, so failures are visible.
9. Prefer webhooks to polling, and use H in schedules.
10. Use an API token for every script, one per script, with an expiry where possible.
11. Back up JENKINS_HOME, secrets included, and store the backup encrypted. Try a restore once.
12. Stay on LTS, and read the upgrade guide before each upgrade.

Practice cards

Short exercises. Do one at a time, and write down what you expected before you run it.

Debugging a build that will not start

Sometimes the problem is that nothing runs at all. Work through these in order.

1. Is the build in the queue?The left panel shows the queue. A build waiting there has not reached an executor, so the problem is capacity or labels, not your code. Hover over the item; Jenkins gives the reason.
2. Does any node match the label?Manage Jenkins, Nodes. Is the agent online, and is the label spelled exactly as the pipeline says? Labels are case-sensitive text.
3. Is the node offline for a reason?Click the node. A red icon and a message such as Disk space is too low or a lost connection explains it.
4. Are executors zero?If only the built-in node exists and it has zero executors, agent any has nowhere to run.
5. Did the Jenkinsfile parse?Run the Declarative Linter on it. Syntax errors stop a build before any stage.
6. Is the trigger registered?If an automatic build never starts, check that the job has run once, that the webhook reached Jenkins (look at the delivery log on the repository host), and that the Jenkins URL in system settings is reachable from outside.

Debugging a build that starts and fails

Once it runs, the routine changes.

1. Open Console Output and find the first error, not the last.
2. Find the Running on ... line so you know which node and which workspace.
3. Copy the failing command and run it by hand on that node, or in the same container image.
4. Compare versions. Ask "what differs from the machine where it works?" Usually it is a tool version, an environment variable or a missing file.
5. Add sh 'env | sort' temporarily, but be aware it can print variables you bound; never do it in a step that has credentials in scope.
6. Re-run with a minimal pipeline to isolate the stage.

Writing pipelines that are easy to read

A pipeline is read far more often than written. Keep these conventions.

One job per stage. A stage should do one thing you can name: Build, Test, Package. If you cannot name it, split it.

Put real work in scripts. A five-line sh block is fine. A fifty-line one belongs in a script such as ci/build.sh in the repository, which you can also run on your laptop. The Jenkinsfile then says what happens, and the script says how.

Fail fast. Put quick checks (lint, unit tests) before slow ones, so a typo fails in one minute and not twenty.

Prefer steps to shell where they exist. junit, archiveArtifacts and checkout scm understand Jenkins and report better than a shell imitation.

Keep Groovy to glue. Pipeline Groovy runs on the controller, so heavy computation there slows everyone down. Run heavy work in sh on agents.

pipeline {
  agent { label 'linux' }
  options {
    timeout(time: 20, unit: 'MINUTES')
    buildDiscarder(logRotator(numToKeepStr: '15'))
    timestamps()
  }
  stages {
    stage('Lint')  { steps { sh 'ci/lint.sh' } }
    stage('Test')  {
      steps { sh 'ci/test.sh' }
      post { always { junit 'reports/**/*.xml' } }
    }
    stage('Package') { steps { sh 'ci/package.sh' } }
  }
  post {
    success { archiveArtifacts artifacts: 'dist/**', fingerprint: true }
    cleanup { cleanWs() }
  }
}

Platform notes

Windows agents. Use bat or powershell instead of sh, or install a shell and be consistent. Paths use backslashes in bat, but forward slashes work in most Jenkins steps. The service account needs the "Log on as a service" right, and a dedicated account is better than LocalSystem.

macOS. The Homebrew package is maintained by third parties and may lag the official release. Jenkins data lives in ~/.jenkins. If you run builds that need Docker on a Mac, remember Docker runs in a virtual machine, so bind mounts of host folders can be slow.

Linux. Edit service settings with sudo systemctl edit jenkins, read logs with journalctl -u jenkins.service, and choose the Java with update-alternatives or JENKINS_JAVA_CMD.

Behind a company proxy. If plugin downloads fail, set the HTTP proxy under Manage Jenkins, Plugins, Advanced settings. A PKIX path building failed message means a TLS-inspecting proxy; import its certificate authority into the JVM trust store rather than disabling verification.

Behind a reverse proxy. The banner It appears that your reverse proxy set up is broken means the proxy is not passing the original host and protocol headers, or the Jenkins URL in system configuration differs from the address in your browser. Fix the URL first.

Data residency. If your employer requires that source code and build logs stay in a particular country, keep the controller and agents in a region that satisfies the rule, and remember that logs and artifacts are stored in JENKINS_HOME on the controller. Regional cloud providers in the Gulf and Egypt are a common choice for this.

Keeping a learning instance tidy

A few habits keep a practice server fast.

• Delete old test jobs. Each holds history.
• Run Manage Jenkins, Plugins, Updates monthly, read what changes, and restart in a quiet moment with the safe-restart CLI command or the "restart when no jobs are running" option.
• Watch the banners on Manage Jenkins. They report reverse-proxy problems, old Java, and pending security fixes.
• When you are done with a Docker-based instance, stop the container and remove the volume deliberately: docker rm and docker volume rm jenkins_home. Removing the volume is irreversible.

Small things to do from day one

• Give every job a meaningful name without spaces (web-app-ci, not New Job 1). Folder paths and URLs are easier to script.
• Describe each job in one sentence.
• Keep one Jenkinsfile per repository, at the root, named exactly Jenkinsfile.
• Add timestamps() so you can see which step is slow.
• Put the tool versions you need in the pipeline (for example the Docker image tag) and pin them, rather than relying on whatever the node has today.
• Write a short README in the repository saying how to run the same commands without Jenkins. A pipeline you cannot reproduce by hand is hard to debug.
• Create a second, non-admin user and log in as them once. Seeing what a normal user can and cannot do teaches more about permissions than reading about them.

The beginner checklist

Before you call a Jenkins setup finished, tick these.

• Jenkins runs on Java 21 or 25 and shows version 2.580.1 (or the current LTS).
• The Jenkins home is on a volume or disk that survives restarts.
• An admin user exists with a strong password, and the server is not exposed to the open internet.
• The Built-In Node has zero executors, or you have consciously decided it does not matter for this learning instance.
• The Jenkinsfile is in the repository and starts with timeout and buildDiscarder.
• Shell steps use single quotes, and secrets come from credentials.
• Test reports publish in post { always }.
• Builds start from a webhook or a schedule, not from someone remembering to click.
• A script can trigger a build using an API token.
• JENKINS_HOME is backed up, including secrets/, and the backup is encrypted.
• You can read a failed build and name the first real error within a minute.

When every box is ticked, you have a Jenkins that a small team could depend on. The next level shows how to make it faster, safer and easier to maintain as it grows, and the Docker guide and Kubernetes guide cover the platforms it will run on.

This is part two of three. It assumes you already run Jenkins jobs at work, and it takes the parts you touch every day and explains what is really happening underneath, so that you can predict behaviour instead of discovering it during an outage. It targets Jenkins LTS 2.580.1 (released 30 September 2026) on Java 21 (Java 25 is also supported). If a tutorial you are following says "Java 17 or 21", it is out of date: Java 17 stopped being supported with LTS 2.555.1 in April 2026.

Where this picks up

Each section starts with the problem it solves and ends with a Try it that you can run against a real controller in a few minutes. Where a neighbouring tool in this catalogue is the natural next step, the text links to it: for example the Docker guide for images, the Kubernetes guide for clusters, the Helm guide for charts, and the Argo CD guide for delivery.

How a build really runs

The problem: when a build is slow, stuck or strange, the console log tells you what your script did, but not what Jenkins did around it. Knowing the machinery lets you ask the right question first.

Start with the roles. The controller is one Java process, a web application served by Winstone and Jetty. It holds configuration, schedules work, serves the UI and the API, and stores every plugin and credential. All of its state lives in one directory, JENKINS_HOME (for example /var/lib/jenkins for Linux packages or /var/jenkins_home in the container image), as plain files: XML for configuration, one directory per job, one per build. There is no database. That is why backups are file backups, and why the disk underneath the controller matters more than almost anything else.

A node is any machine or container that can run builds. The controller has a node object of its own, the built-in node (it used to be called "master"; the rename to controller and built-in node landed in 2.307 weekly and 2.319.1 LTS, and old property names such as JENKINS_SLAVE_AGENT_PORT still carry the word "slave"). Builds that run on the built-in node have the same access to the controller file system as the Jenkins process itself, which means a build can read secrets/ and rewrite configuration. For that reason, production controllers set the built-in node to 0 executors (Manage Jenkins, Nodes, Built-In Node, Configure) and run everything on agents.

An agent is a small Java client, agent.jar, built on the Remoting library. It connects a node to the controller in one of three ways:

• SSH: the controller connects out to the machine and starts the agent. Stable and simple when the controller can reach the host.
• Inbound: the agent connects in to the controller, either over WebSocket or over the inbound TCP port. Friendly to firewalls that only allow outbound traffic.
• Cloud: a cloud plugin (Kubernetes, EC2, Docker) creates the agent on demand and throws it away afterwards.

Each node has a number of executors, and an executor is one build slot. The official advice is that one executor per node is the safest configuration, because two builds sharing a machine share its CPU, memory, ports and Docker daemon. Labels tag nodes so a build can ask for a kind of machine instead of a specific one, and label expressions support &&, || and !, as in agent { label 'linux && docker' }.

Now follow one build through the system:

A trigger creates a queue item. The queue holds items until a node whose labels match has a free executor. Only then does a build start, and only then does a workspace exist: a directory on that agent where the checkout and your shell commands happen. If a job runs twice concurrently on the same node, the second gets name@2; the @ separator comes from the hudson.slaves.WorkspaceList system property. A build ends as SUCCESS, UNSTABLE, FAILURE, NOT_BUILT or ABORTED, and the controller records the result, the log and any archived artifacts in JENKINS_HOME.

Two practical consequences follow. First, "my build is not starting" is a queue problem and not a build problem. The queue item tells you why it is waiting, and the messages differ in meaning: Waiting for next available executor on 'linux' means a matching node exists but is busy, while There are no nodes with the label 'linux' means no online node carries that label at all, often because an agent went offline or the built-in node has zero executors and agent any has nowhere to go. Second, state you rely on in the workspace is not guaranteed to be there next time, because the next build may land on a different agent, or on a brand new pod. Treat the workspace as scratch space and keep anything that matters in an artifact store, a cache you control, or the SCM.

You can see the controller's view of all of this over HTTP, which is handy when the UI is slow:

curl -s -u alice:API_TOKEN "https://ci.example.com/queue/api/json?tree=items[why,task[name]]"
curl -s -u alice:API_TOKEN "https://ci.example.com/computer/api/json?tree=computer[displayName,offline,numExecutors]"

The first shows every waiting item with its why text; the second lists each node, whether it is offline, and how many executors it has.

Pipeline execution: CPS, serialisation and durability

The problem: Pipeline Groovy is not ordinary Groovy, and treating it as ordinary is the source of the strangest errors at this level (NotSerializableException, expected to call ... but wound up catching ..., builds that resume after a restart in a half-finished state). One idea explains all of them.

A Pipeline can run for hours or days (waiting for an approval, for example), and the controller may restart in the middle. To make that possible, Jenkins runs your Groovy using Continuation-Passing Style (CPS). The interpreter records where it is after each step, so the whole program state can be written to disk and restored later. When the controller comes back, the build resumes from the last recorded point instead of starting over. The price is three rules you need to know.

Rule one: anything alive across a step must be serialisable. When a step such as sh or sleep runs, Jenkins may persist your local variables. If one of them holds a java.util.regex.Matcher, a groovy.json.internal.LazyMap or a LinkedHashMap$Entry, persisting fails with java.io.NotSerializableException. The classic trigger is a regular expression match held in a variable while a later step runs:

// Fails: the Matcher is still in scope when sh runs
def m = (env.GIT_BRANCH =~ /release\/(\d+)/)
def version = m[0][1]
sh "echo ${version}"

The fix is to do the non-serialisable work inside a method marked @NonCPS, which runs as plain Groovy without the recording machinery, and return only simple data:

@NonCPS
String releaseVersion(String branch) {
  def m = (branch =~ /release\/(\d+)/)
  return m.find() ? m.group(1) : null
}

def version = releaseVersion(env.BRANCH_NAME)
sh "echo ${version}"

Rule two: @NonCPS code must not call steps, and non-CPS code must not call CPS code. A @NonCPS method has no recording, so it cannot call sh, echo or anything else from the Pipeline step library. The reverse direction causes the odd message expected to call java.util.ArrayList.toSorted but wound up catching org.jenkinsci.plugins.workflow.cps.CpsClosure2.call. It means a Java or Groovy library method (here a sort) was handed a Pipeline closure and called it, but the library method is not CPS-aware. The fix is to keep closures out of GDK methods: put the logic in a @NonCPS method that takes plain data, or use a simple for loop.

Rule three: Pipeline Groovy runs on the controller. Your Jenkinsfile logic executes in the controller JVM, on a thread shared with everything else. A shell command runs on the agent, but the for loop that decides which commands to run runs on the controller. Loops over thousands of items, JsonSlurper on a large payload and HTTP calls made from Groovy all consume controller CPU and heap. The guidance from the Jenkins project is to keep Groovy as glue and push heavy work into a single sh step that runs on the agent.

Durability and resuming

How often Jenkins writes that recorded state is the durability setting. There are three values:

You can set it globally (Manage Jenkins, System, Pipeline Speed/Durability Settings), per job, or in the file:

pipeline {
  agent { label 'linux' }
  options { durabilityHint('PERFORMANCE_OPTIMIZED') }
  stages {
    stage('Build') { steps { sh 'make build' } }
  }
}

PERFORMANCE_OPTIMIZED is worth a serious look when JENKINS_HOME sits on slow or networked storage such as NFS, because disk writes by running Pipelines are often the hidden bottleneck. The trade is clear: you accept that a hard crash may lose in-flight builds. Since those builds would usually be rerun anyway, many teams take that trade for short CI jobs and keep MAX_SURVIVABILITY for long release pipelines.

Resumption has a flip side. After a restart you may see Resuming build at ... after Jenkins restart followed by Waiting to resume part of ...: 'agent-x' is offline. The build is waiting for the agent it was using. If you would rather fail fast than wait, options { disableResume() } stops a build from resuming at all, and for Kubernetes agents there is a retry condition covered later.

When you need to restart the controller yourself, do it gracefully. quiet-down stops new builds from starting, and safe-restart waits for running builds to finish before restarting (you can do both from the UI, from the CLI, or with POST /quietDown and POST /safeRestart). A plain restart interrupts them and relies on resume.

Declarative Pipeline in depth

The problem: the beginner pipeline has a handful of stages, but real ones need conditional stages, approvals, parallel work and sensible failure handling. Declarative syntax has a directive for each, and each has precise semantics that are easy to get slightly wrong.

Options

options changes how the whole pipeline (or, for a few, a single stage) behaves. The ones that pay for themselves:

options {
  buildDiscarder(logRotator(numToKeepStr: '20'))   // keep 20 builds, stop JENKINS_HOME growing forever
  timeout(time: 30, unit: 'MINUTES')               // a hung build cannot hold an executor all night
  timestamps()                                     // needs the Timestamper plugin
  disableConcurrentBuilds(abortPrevious: true)     // a newer commit cancels the older build
  skipDefaultCheckout()                            // you will check out yourself
  skipStagesAfterUnstable()                        // stop early once a stage goes UNSTABLE
  parallelsAlwaysFailFast()                        // every parallel block stops at the first failure
  retry(2)                                         // rerun the whole pipeline on failure; use sparingly
}

disableConcurrentBuilds(abortPrevious: true) deserves a note: on a busy branch it is an enormous saving, because a developer pushing three commits in a minute would otherwise run three full builds. Use it for CI builds, not for deploy jobs where you want every build to finish. Stage-level options are narrower: a stage can set skipDefaultCheckout, timeout, retry and timestamps, which lets you give one slow stage a longer timeout without loosening the rest.

when: skipping stages, and the beforeAgent trap

A when block decides whether a stage runs. The conditions you will use are branch, tag, buildingTag(), changeRequest, changeset, changelog, environment, equals, expression, triggeredBy, and the combinators not, allOf and anyOf:

stage('Deploy to staging') {
  when {
    allOf {
      branch 'main'
      not { changeRequest() }
      changeset 'services/api/**'
    }
  }
  steps { sh './deploy.sh staging' }
}

branch accepts a comparator of EQUALS, GLOB or REGEXP, for example branch pattern: 'release/*', comparator: 'GLOB'. changeset is how you make a monorepo build only what changed.

The trap is ordering. By default Jenkins allocates the stage's agent first and evaluates when afterwards. For a stage with a Docker or Kubernetes agent, that means you pay to start a container (or a whole pod) just to learn that the stage is skipped. Set beforeAgent true and Jenkins evaluates the condition first:

stage('Publish') {
  agent { label 'linux' }
  when { beforeAgent true; branch 'main' }
  steps { sh './publish.sh' }
}

There are two sibling flags, beforeInput true and beforeOptions true. The precedence is options, then input, then agent, so beforeOptions runs the condition earliest of all.

Approvals that do not hold an executor

An input step pauses the pipeline until a person answers. The mistake is to pause while holding an agent: the executor sits idle, possibly for days, blocking other builds. The stage-level input directive runs before the stage's agent is allocated, so it does not hold one, provided the pipeline-level agent is none:

pipeline {
  agent none
  stages {
    stage('Build') {
      agent { label 'linux' }
      steps { sh 'make dist' }
    }
    stage('Approve') {
      options { timeout(time: 2, unit: 'DAYS') }
      input {
        message 'Deploy to production?'
        ok 'Deploy'
        submitter 'release-managers'
        submitterParameter 'APPROVER'
      }
      steps { echo "Approved by ${APPROVER}" }
    }
    stage('Deploy') {
      agent { label 'linux' }
      steps { sh './deploy.sh production' }
    }
  }
}

Always wrap a human wait in a timeout, otherwise an abandoned approval keeps a build "running" forever. submitter limits who may approve (a user name, a comma-separated list, or a group depending on your security realm), and submitterParameter records who actually did, which is the audit trail you want. The approver needs the right permission on the job, so test with a second account rather than as an administrator.

Parallel, matrix and failure semantics

Parallel stages run simultaneously, each on its own agent allocation if it declares one. Add failFast true to stop the siblings when one fails:

stage('Tests') {
  failFast true
  parallel {
    stage('Unit')        { steps { sh 'make test-unit' } }
    stage('Integration') { steps { sh 'make test-integration' } }
    stage('Lint')        { steps { sh 'make lint' } }
  }
}

When the same stages must run across a grid of values, matrix saves you from copy and paste. Each combination of axis values becomes its own parallel cell, and excludes removes impossible combinations:

stage('Compatibility') {
  matrix {
    axes {
      axis { name 'JDK';  values '21', '25' }
      axis { name 'DB';   values 'postgres', 'mysql' }
    }
    excludes {
      exclude {
        axis { name 'JDK'; values '25' }
        axis { name 'DB';  values 'mysql' }
      }
    }
    stages {
      stage('Test') { steps { sh "make test JDK=${JDK} DB=${DB}" } }
    }
  }
}

Every axis value becomes an environment variable inside its cell, which is why JDK and DB are usable as written.

post ordering

post conditions run in a fixed order, regardless of the order you write them: always, changed, fixed, regression, aborted, failure, success, unstable, unsuccessful, cleanup. Two of them confuse people. changed and fixed compare with the previous build: changed fires when the status differs, and fixed when a failing build turns successful. cleanup always runs last, after every other condition, which makes it the right home for cleanWs() (from the Workspace Cleanup plugin): nothing you put in post can still need the workspace once it has run.

Handling failure without failing everything

Sometimes a stage should not stop the pipeline, for example a flaky optional scanner. catchError marks the stage and the build separately:

stage('Optional scan') {
  steps {
    catchError(buildResult: 'SUCCESS', stageResult: 'FAILURE') {
      sh './scan.sh'
    }
  }
}

The stage shows red in the visualisation while the build stays green. Use unstable('message') when you want a yellow build, and error('message') to fail one deliberately. Remember sh itself fails the step on a non-zero exit code. If you want the code instead, use sh(script: './check.sh', returnStatus: true), and read the conventional meanings when it fails: 127 is command not found, 137 is a kill (often the out-of-memory killer), 143 is a SIGTERM such as a timeout.

Agents that fit the work

The problem: a single long-lived agent accumulates state. Tools drift, workspaces fill up, one build leaves a process running and the next one fails mysteriously. The remedy is to make agents fungible (interchangeable and rebuildable) and, where you can, ephemeral (created for one build and discarded).

Choosing how agents connect

Three connection styles, and the situations where each wins:

An inbound agent is a single command:

curl -sO https://ci.example.com/jnlpJars/agent.jar
java -jar agent.jar \
  -url https://ci.example.com/ \
  -secret @secret-file \
  -name "linux-01" \
  -webSocket \
  -workDir "/home/jenkins/agent"

-webSocket is the detail that matters. Without it, an inbound agent uses the inbound TCP port (conventionally 50000), which is disabled by default since Jenkins 2.0 and which you would have to open in every firewall and load balancer between the agent and the controller. With it, the agent speaks over the same HTTP(S) port users already reach, so it passes through ordinary reverse proxies. Keepalives on that connection are governed by jenkins.websocket.pingInterval (30 seconds) and jenkins.websocket.idleTimeout (60 seconds), which is where to look if a proxy drops idle connections. -workDir turns on the remoting work directory that stores agent logs and a jar cache. The secret comes from the node's page in the UI; pass it from a file with @ so it does not appear in the process list.

Recent versions tighten what agents may be: the minimum supported Remoting jar is 3176.v207ec082a_8c0 from LTS 2.555.1, and weekly 2.584 drops support for Remoting versions released before August 2024. Since the agent downloads its jar from the controller, agents that start with agent.jar fetched fresh stay current on their own; agents with a jar baked into an old image are the ones that break.

The Java requirement applies to agents too, not just the controller. If an agent still runs Java 17, the failure looks like this:

java.lang.UnsupportedClassVersionError: hudson/remoting/Launcher has been compiled by a more recent version of the Java Runtime (class file version 65.0), this version of the Java Runtime only recognizes class file versions up to 61.0

Read the numbers: class file version 65 is Java 21, and 61 is Java 17. The message tells you the agent's JVM is too old. The Versions Node Monitors plugin shows every agent's Java version in one table, so you can find stragglers before an upgrade and not after.

Docker agents

If your agents have Docker installed and the Docker Pipeline plugin (docker-workflow) is present, a stage can run inside a container image of your choosing:

pipeline {
  agent none
  stages {
    stage('Build') {
      agent {
        docker {
          image 'maven:3.9-eclipse-temurin-21'
          args '-v $HOME/.m2:/root/.m2'
          reuseNode true
        }
      }
      steps { sh 'mvn -B verify' }
    }
  }
}

The image is the tool chain, so the build does not depend on what is installed on the host. args passes extra docker run arguments (here a shared Maven cache). reuseNode true runs the container on the node already allocated for the pipeline instead of allocating a second one, which matters when a pipeline-level agent has already claimed a workspace. alwaysPull, registryUrl and registryCredentialsId cover private registries. If the image must be built from a file in the repository, use the dockerfile { filename 'Dockerfile.ci'; dir 'ci' } agent instead. For everything about the images themselves, see the Docker guide.

Kubernetes agents

The Kubernetes plugin creates an agent pod for each build and removes it afterwards. That removes drift entirely, and it scales to zero when nothing is building. A pipeline describes the pod it needs:

pipeline {
  agent {
    kubernetes {
      defaultContainer 'maven'
      yaml '''
        apiVersion: v1
        kind: Pod
        spec:
          containers:
          - name: maven
            image: maven:3.9.9-eclipse-temurin-21
            command: ["sleep"]
            args: ["infinity"]
            resources:
              requests: { cpu: "1", memory: 2Gi }
              limits:   { memory: 2Gi }
      '''
    }
  }
  stages {
    stage('Build') { steps { sh 'mvn -B verify' } }
  }
}

Understand what the plugin adds to that pod. A container named jnlp (image jenkins/inbound-agent) is injected, and it is the piece that connects back to the controller and executes your steps. Your own containers must therefore keep running (hence sleep infinity) so that sh steps can be executed inside them. By default steps run in the jnlp container, so defaultContainer 'maven' makes the maven container the target. When a stage needs another tool, switch with container('name') { ... }:

steps {
  container('kaniko') {
    sh '/kaniko/executor --context "$WORKSPACE" --destination "registry.example.com/app:$BUILD_NUMBER"'
  }
}

All containers in the pod share the workspace volume, and they need to run as the same user to read each other's files, which is what securityContext.runAsUser in the pod spec is for. A mismatch shows up as permission-denied on the workspace.

Settings that matter for cost and stability, configured on the cloud (Manage Jenkins, Clouds) or in the pod template:

• idleMinutes keeps a pod alive for a few minutes after a build so the next one starts instantly. A small value saves startup time on busy queues and costs a little in idle resources.
• podRetention (never(), onFailure(), always(), evicted()) decides whether to keep pods for debugging. onFailure() is a useful temporary setting when a build dies in a pod you need to inspect.
• activeDeadlineSeconds ends pods that run too long.
• The container cap limits how many agents the cloud may run at once. When this is reached, builds queue with a message that looks like an ordinary wait, which is easy to misdiagnose.
• WebSocket should be enabled in the cloud settings when the controller is only reachable over HTTP(S), for example behind an ingress.

Nodes vanish, especially on spot or preemptible capacity. Declare what should happen, so a lost pod retries instead of failing the build:

retry(count: 2, conditions: [kubernetesAgent(), nonresumable()]) {
  node(POD_LABEL) {
    sh 'make test'
  }
}

kubernetesAgent() matches pod-level failures, nonresumable() matches builds whose agent vanished, and POD_LABEL is the label of the pod template the plugin generated. The Helm chart for Jenkins (helm install jenkins jenkinsci/jenkins) configures this cloud for you when Jenkins itself runs in the cluster. Keep the rest of the cluster side in the Kubernetes guide and the Helm guide.

Credentials and secrets in pipelines

The problem: a pipeline needs to push an image, call an API or reach a cluster, and that takes a secret. Jenkins stores credentials encrypted on the controller and lets pipelines reference them by ID, so the secret never appears in the Jenkinsfile. How you bind them decides whether they stay secret.

Credentials live in stores with a scope. Global credentials are usable by jobs. System credentials are usable only by Jenkins itself (for example the private key it uses to launch an SSH agent), and are invisible to pipelines. Folders can hold their own credentials, visible only to jobs beneath them, which is how you stop the payments team's secrets being usable by everyone. The encryption keys sit in $JENKINS_HOME/secrets/, so anyone who can copy that directory together with the configuration can decrypt every credential, and backups must be encrypted for the same reason.

There are two ways to bind a credential. The environment block is compact:

environment {
  API_TOKEN = credentials('api-token-id')        // secret text: the value itself
  REG       = credentials('registry-userpass')   // username/password: sets REG, REG_USR, REG_PSW
  KUBECFG   = credentials('kubeconfig-file')     // secret file: a path to a temporary file
}

withCredentials limits the lifetime to a block and supports more types:

withCredentials([
  usernamePassword(credentialsId: 'registry', usernameVariable: 'REG_USER', passwordVariable: 'REG_PASS'),
  string(credentialsId: 'api-token-id', variable: 'API_TOKEN'),
  file(credentialsId: 'kubeconfig-file', variable: 'KUBECONFIG'),
  sshUserPrivateKey(credentialsId: 'deploy-key', keyFileVariable: 'KEY', usernameVariable: 'SSH_USER')
]) {
  sh '''
    echo "$REG_PASS" | docker login registry.example.com -u "$REG_USER" --password-stdin
    kubectl --kubeconfig "$KUBECONFIG" get pods
  '''
}

Notice the quotes. In a Groovy double-quoted string, $VAR is expanded by Groovy on the controller before the shell ever sees it. That means sh "curl -u $U:$P https://..." puts the real password into the command line, where ps on the agent can read it, and Jenkins prints a warning saying that a secret was passed to sh using Groovy string interpolation, which is insecure. With single quotes, or a triple-single-quoted block as above, Groovy leaves the text alone and the shell expands the variable from the environment, so the secret never travels as an argument.

Jenkins masks bound secret values in the console log, replacing them with ****. That is useful and worth having, but it is best effort. A value that the script transforms (base64-encoded, split across lines, reversed) is not recognised and prints in clear. Treat masking as a safety net for accidents, never as the security boundary. Notably, anyone with Job/Configure on a Pipeline can write a step that prints the secret in a transformed way, which is why that permission must be given carefully.

Wherever you can, avoid long-lived secrets altogether. Credential provider plugins (HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, GCP Secret Manager, the Kubernetes credentials provider) fetch values at the moment of use, and short-lived identity from the platform (workload identity on a cloud, OIDC where the plugin supports it) means there is nothing permanent to steal. The Terraform guide covers the infrastructure side of issuing those identities.

Shared Libraries

The problem: your organisation has forty repositories, each with a Jenkinsfile, and thirty of them contain the same sixty lines to build, scan and push an image. A fix to that block means forty pull requests. A Shared Library is reusable Pipeline code held in its own repository, loaded by version at the start of each build.

The repository has a fixed layout:

(root)
├── vars/
│   ├── buildImage.groovy       # a custom step: buildImage(...)
│   └── buildImage.txt          # optional help text for the step
├── src/
│   └── org/example/Version.groovy   # ordinary classes
└── resources/
    └── org/example/scan.sh     # files read with libraryResource

A file in vars/ whose name is buildImage.groovy defines a global variable, and a call method makes it usable as a step:

def call(Map config = [:]) {
  String image = config.image ?: error('buildImage: image is required')
  String tag   = config.tag   ?: env.BUILD_NUMBER
  sh "docker build -t ${image}:${tag} ."
  if (config.push) {
    withCredentials([usernamePassword(credentialsId: config.credentialsId,
                                      usernameVariable: 'U', passwordVariable: 'P')]) {
      sh 'echo "$P" | docker login -u "$U" --password-stdin'
      sh "docker push ${image}:${tag}"
    }
  }
}

(Notice that the credential is used with single quotes, and only the harmless image name is interpolated.) The consuming Jenkinsfile is now tiny:

@Library('ci-library@v1.4.0') _

pipeline {
  agent { label 'docker' }
  stages {
    stage('Image') {
      steps { buildImage(image: 'registry.example.com/team/api', push: true, credentialsId: 'registry') }
    }
  }
}

Three details are worth remembering:

• Version pinning. @Library('ci-library@v1.4.0') loads a tag; @Library('ci-library@main') loads the branch tip, and so does the bare @Library('ci-library') if the library's default version is a branch. Pin to tags in production, so that a change to the library cannot break every pipeline at once, and roll new versions out by bumping the tag one repository at a time. The alternative, library 'ci-library@main', loads a library dynamically in the middle of a script, which is useful for selecting a version at runtime.
• The trailing underscore. @Library(...) is an annotation, and an annotation must attach to something. If you do not import a class from the library, the _ gives it a harmless target. Forgetting it is a syntax error.
• Multiple libraries. @Library(['lib1', 'lib2@v2']) _ loads several.

Trusted and untrusted libraries

A library defined globally (Manage Jenkins, System, Global Trusted Pipeline Libraries) is trusted: its code runs outside the Groovy sandbox, so it can call any API, including ones that can read and change the whole controller. A library defined on a folder is always untrusted, and runs in the sandbox like any Jenkinsfile. That gives you a clean policy. A small, reviewed, platform-owned library is trusted and exposes safe steps; everything else is untrusted. Write access to a trusted library's repository is therefore equivalent to administrator access to the controller, and the repository's branch protections should reflect that.

You can go further and define the whole pipeline in the library, so that repositories contain almost nothing:

def call(Map cfg) {
  pipeline {
    agent { label cfg.label ?: 'linux' }
    stages {
      stage('Build') { steps { sh cfg.buildCommand } }
      stage('Test')  { steps { sh cfg.testCommand } }
    }
  }
}

Only one pipeline block may run per build, and the approach trades flexibility for consistency: a team that needs something unusual must either extend the library or leave the standard. Choose it when consistency matters more than freedom, for example for a platform team that also owns compliance checks.

Libraries are code, so test them like code. Keep the logic in src/ classes or plain scripts that run without Jenkins, put a small Jenkins job in the library's own repository that exercises each step on a real agent, and record breaking changes in tags, so consumers can move deliberately.

Multibranch pipelines and webhooks

The problem: a Jenkinsfile per repository is only useful if a job exists for every branch and pull request, and if builds start when code changes and not when someone clicks. A Multibranch Pipeline creates and deletes child jobs automatically, and webhooks replace polling.

A Multibranch job (or an Organization Folder, which scans every repository in a GitHub organisation or Bitbucket project) is configured with a branch source: the GitHub Branch Source, Bitbucket Branch Source or GitLab Branch Source plugin. Jenkins performs branch indexing: it lists the branches and pull requests, and for each one that contains a Jenkinsfile it creates a child job. Branches that disappear are removed after the retention period you configure. Inside a child build, a handful of variables tell you where you are: BRANCH_NAME, CHANGE_ID (the pull request number, set only for PRs), CHANGE_TARGET, and TAG_NAME for tag builds. The Git plugin adds GIT_COMMIT.

Use those facts to give branches and pull requests different behaviour from the same file:

pipeline {
  agent { label 'linux' }
  options { buildDiscarder(logRotator(numToKeepStr: '15')) }
  stages {
    stage('Build and test') { steps { sh 'make ci' } }
    stage('Preview environment') {
      when { changeRequest target: 'main' }
      steps { sh './deploy-preview.sh "PR-${CHANGE_ID}"' }
    }
    stage('Release image') {
      when { beforeAgent true; anyOf { branch 'main'; buildingTag() } }
      steps { sh './push-image.sh' }
    }
  }
}

The preview stage runs only on pull requests targeting main, and the release stage only on main or on a tag. Jobs on feature branches run just the first stage, which is what you want: fast feedback for everyone, expensive work only where it matters.

Why webhooks beat polling

pollSCM('H/15 * * * *') asks the repository "anything new?" every fifteen minutes. Multiply that by hundreds of jobs and the controller spends its time polling, builds start up to fifteen minutes late, and the Git server sees constant traffic. A webhook reverses it: the Git server sends an HTTP request to Jenkins the moment something happens, and Jenkins starts exactly the builds that are affected. For GitHub the endpoint is https://ci.example.com/github-webhook/ (the trailing slash matters), and the branch source plugins for GitLab and Bitbucket have equivalents described in their documentation.

The H in cron expressions is worth knowing for the triggers you keep, such as nightly jobs. It hashes the job name to a value within the range, so a hundred jobs asking for H 2 * * * spread across the 2 o'clock hour instead of all starting at 2:00. The field order is minute, hour, day of month, month, day of week, and @daily, @hourly and similar aliases are accepted.

Three operational notes:

• Jenkins must be reachable by the Git host. A controller behind a corporate firewall cannot receive webhooks from a cloud Git service. Options are an allow-listed ingress, a relay, or accepting a periodic scan as a fallback (Multibranch jobs can periodically re-index, which covers missed webhooks).
• Set the Jenkins URL correctly under Manage Jenkins, System. Links in notifications and the commit status checks use it, and a wrong value also triggers the reverse-proxy warning covered later.
• Stop re-indexing storms. An Organization Folder over a large organisation can run thousands of API calls per scan. Keep the scan interval long and rely on webhooks for freshness.

To stop an irrelevant trigger from running a stage, use triggeredBy 'TimerTrigger' in a when block, which lets you run a long nightly suite from a schedule while pull requests run only the fast subset. overrideIndexTriggers(false) in options prevents branch indexing from resetting trigger settings on a branch.

Testing and validating pipelines

The problem: a typo in a Jenkinsfile is found by running the pipeline, which means a commit, a push and a wait, for each attempt. That loop is slow, noisy in history, and uses real agents. There are faster loops, and the discipline is to use them in the right order.

1. Validate syntax without running anything. The Declarative Linter checks a Jenkinsfile against the controller's installed plugins and gives errors like WorkflowScript: 12: Expected a step @ line 12, column 9. or Missing required section "agent":

curl -s -X POST -u alice:API_TOKEN \
  -F "jenkinsfile=<Jenkinsfile" \
  https://ci.example.com/pipeline-model-converter/validate

java -jar jenkins-cli.jar -s https://ci.example.com/ -auth alice:API_TOKEN declarative-linter < Jenkinsfile

The response is Jenkinsfile successfully validated. or a list of errors with line numbers. Most editors can call this on save; wire it to a pre-commit hook and a whole class of "fix typo" commits disappears. The linter validates Declarative only, so a Scripted pipeline passes through without checks.

2. Use the Snippet Generator and Directive Generator. /pipeline-syntax on your controller builds the exact step syntax for the plugins you have installed, including every parameter with its documentation. It is the authoritative reference because it reflects your versions; copying a snippet from a blog post written for a different plugin version is a common cause of No such DSL method errors. That error also appears, plainly, when the plugin that provides a step is missing (for instance cleanWs without Workspace Cleanup, readJSON without Pipeline Utility Steps, or docker without Docker Pipeline).

3. Replay. On a finished Pipeline build, Replay opens the script (and any loaded library code) in an editor, lets you change it, and runs a new build with your edit without committing anything. It is the fastest way to experiment on a real agent with real credentials. Two cautions: the change is not saved to Git, so copy a working result back into the repository, and the Run/Replay permission amounts to running arbitrary Pipeline code (inside the sandbox), so grant it accordingly.

4. Move logic out of Groovy into testable scripts. The best-tested part of a pipeline is the part that does not live in the Jenkinsfile. A make ci target or a scripts/deploy.sh can be run and unit-tested on a laptop and in a container, produces the same result locally as in CI, and is unaffected by CPS and the sandbox. The Jenkinsfile should say when and where to run each script, and little else.

5. Test the library changes on a branch. @Library('ci-library@my-branch') _ in one consumer repository lets you try a library change in a real pipeline before you merge and tag it. Combine it with Replay when you need to iterate quickly.

6. Publish test results properly. Emit JUnit XML and publish it with the junit step, so Jenkins shows per-test history, trends and flaky tests. Put it in a post { always { ... } } block so results are recorded even when tests fail:

stage('Test') {
  steps { sh 'make test' }
  post { always { junit 'reports/**/*.xml' } }
}

For results that cross agents, stash name: 'bin', includes: 'target/*.jar' and unstash 'bin' copy small files from one agent to another through the controller. They are not a way to move large artifacts, because everything passes through controller storage. Use an artifact repository (Nexus, Artifactory, S3) for anything large, and archiveArtifacts only for small outputs you genuinely want attached to the build.

Performance and controller health

The problem: Jenkins rarely gets slow because of one thing. It gets slow because the controller, the single stateful component, is asked to do more work than it was sized for, and it does so quietly until a queue backs up. Knowing what loads the controller tells you where to look first.

The controller does four jobs: serving the UI and API, scheduling the queue, running Pipeline Groovy (CPS), and reading and writing JENKINS_HOME. Agents only do build work. So a slow Jenkins nearly always means one of those four.

The actions with the best return, roughly in order:

1. Always set buildDiscarder. Old builds and their logs grow JENKINS_HOME without bound. A logRotator(numToKeepStr: '20') default, enforced through a shared library or a template, is the cheapest performance fix there is. Also move big artifacts out of Jenkins to an artifact store.
2. Keep Groovy light. Combine many small sh steps into one script. Avoid JsonSlurper on large payloads and HTTP calls from Groovy; use readJSON (Pipeline Utility Steps) on small files, or do the parsing in a script that runs on the agent.
3. Choose durability deliberately (see earlier), especially on networked storage.
4. Replace API polling with webhooks and event-driven triggers. External dashboards that poll /api/json every few seconds with depth=3 can cost more than the builds themselves. Use tree= to ask only for the fields you need.
5. Size the heap properly and capture evidence. Set -Xmx through JAVA_OPTS (in a systemd override for package installs) and add -XX:+HeapDumpOnOutOfMemoryError so a crash leaves a heap dump to analyse instead of a mystery.
6. Scale out agents before you scale up the controller. Executors are cheap on ephemeral agents. When the controller itself is the limit, the official guidance is to split by product line, team or environment into several controllers, since the open-source controller is not active-active. The documentation's sizing heuristics are around 500 jobs per controller and roughly one core per 100 jobs, but treat those as a starting point and measure.
7. Protect disk headroom on agents. Node monitors take an agent offline when free space drops below about 1 GiB (you will see Disk space is too low. Only 0.9GB left). Clean workspaces with cleanWs() in post { cleanup { ... } }, and prune Docker images on long-lived agents.

For a systemd install, set JVM options in an override file, not in the ignored legacy defaults file:

sudo systemctl edit jenkins
# add:
# [Service]
# Environment="JAVA_OPTS=-Xmx4g -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/jenkins/dumps"
sudo systemctl restart jenkins
journalctl -u jenkins.service -n 50

Editing /etc/default/jenkins or /etc/sysconfig/jenkins does nothing on current packages, which have moved to systemd overrides, and this is a frequent source of "I changed the setting and nothing happened".

Caching is where build time is won on the agent side. Mount dependency caches (Maven's ~/.m2, npm's cache, Gradle's home) as persistent volumes or shared directories for long-lived agents; for ephemeral Kubernetes pods, use a persistent volume claim or a remote build cache, since a fresh pod otherwise downloads the world on every run. Sharing one cache between concurrent builds requires the tool to tolerate it, so read the tool's documentation before mounting a cache read-write in many pods at once.

Configuration as code and plugin management

The problem: a Jenkins configured by clicking is a snowflake. Nobody knows what changed, a rebuild after a disk failure is a day of archaeology, and a staging controller never quite matches production. Two tools fix this: Configuration as Code (JCasC) for settings, and a pinned plugin list for what is installed.

JCasC is the configuration-as-code plugin. It reads YAML and applies it to the controller on startup (and on demand). The default file is $JENKINS_HOME/jenkins.yaml, and you can point elsewhere with the environment variable CASC_JENKINS_CONFIG or the Java property -Dcasc.jenkins.config, giving it a file, a directory (every *.yml and *.yaml inside, recursively), an https:// URL, or a comma-separated list.

jenkins:
  systemMessage: "Managed by Configuration as Code. UI edits will be overwritten."
  numExecutors: 0
  securityRealm:
    local:
      allowsSignup: false
      users:
        - id: admin
          password: ${ADMIN_PASSWORD}
  authorizationStrategy:
    globalMatrix:
      entries:
        - user:
            name: admin
            permissions:
              - Overall/Administer
        - group:
            name: authenticated
            permissions:
              - Overall/Read
credentials:
  system:
    domainCredentials:
      - credentials:
          - string:
              scope: GLOBAL
              id: api-token
              secret: "${API_TOKEN}"
unclassified:
  location:
    url: https://ci.example.com/

The four root keys you will use constantly are jenkins (core settings, security, nodes), tool (tool installations), unclassified (the many plugin settings that appear on the System page) and credentials. The shape under authorizationStrategy depends on the version of the Matrix Authorization plugin, so after installing, open Manage Jenkins, Configuration as Code, View Configuration to see the exact YAML your controller would export, and adapt it. That export page is the best way to bootstrap a file: configure something once in the UI, then copy how JCasC represents it.

Understand these behaviours before relying on it:

• Secrets are not written into the file. Values like ${ADMIN_PASSWORD} are resolved from environment variables, from files in /run/secrets (the default secrets directory, which suits Kubernetes and Docker secrets), or through extra plugins from Vault and cloud secret stores. To write a literal dollar-brace, escape it as ^${VAR}.
• Multiple files must be supplementary. They are merged, and two files setting the same value conflict instead of one overriding the other. Split by area (security, agents, tools), not by environment, and vary values with variables.
• JCasC does not install plugins. If the YAML mentions a configuration element whose plugin is missing, startup fails with errors like No configurator for the following root elements or Invalid configuration elements for type. That almost always means a missing plugin or a misspelled key.
• UI edits are overwritten on the next restart or reload. Decide which mechanism owns each setting, and tell the team. A systemMessage saying so, as above, prevents surprises.
• Deprecated and unknown keys are configurable. Under configuration-as-code: you can set deprecated: warn or unknown: warn instead of rejecting. This matters concretely for LTS 2.555.1 and later: the CSRF crumb no longer includes the client IP address, so the crumbIssuer section became deprecated, and a file that still contains it cancels startup unless you remove it or set deprecated: warn.

Reloading without a restart is possible from the UI (Manage Jenkins, Configuration as Code, Reload existing configuration) or by an authorised POST to /configuration-as-code/reload with a token set in CASC_RELOAD_TOKEN. Keep the YAML in Git, review changes as pull requests, and apply them through the same pipeline that manages the rest of your infrastructure, as described in the Terraform guide and the Argo CD guide.

Plugins as a pinned list

Nearly all of Jenkins is plugins, and plugin upgrades are the number one cause of "it worked yesterday". Treat them like dependencies. The jenkins-plugin-cli tool installs a list from a file:

# Pipeline and SCM
workflow-aggregator:latest
git:latest
pipeline-graph-view:latest
# Builds and credentials
credentials-binding:latest
ws-cleanup:latest
timestamper:latest
junit:latest
# Configuration as code
configuration-as-code:latest

Replace latest with exact versions when you freeze them (git:5.7.0 style, one line per plugin), and commit the file. The container image applies it at build time:

FROM jenkins/jenkins:2.580.1-jdk21
COPY --chown=jenkins:jenkins plugins.txt /usr/share/jenkins/ref/plugins.txt
RUN jenkins-plugin-cli -f /usr/share/jenkins/ref/plugins.txt

Useful flags: --available-updates --output txt lists what could be upgraded, --list shows what is installed, and --latest false installs the minimum required version of dependencies, which avoids surprise upgrades. Files placed under /usr/share/jenkins/ref/ are copied into JENKINS_HOME on first start, a .override suffix forces the copy on every start, and init.groovy.d/ holds startup scripts. For the Helm chart the same idea is controller.installPlugins and controller.JCasC.configScripts in values.yaml.

Note that the Docker installation page of the official documentation still shows blueocean in its example Dockerfile. Blue Ocean has been deprecated since July 2026 and receives no more updates or security fixes. Do not install it. Use Pipeline Graph View (pipeline-graph-view), which is actively maintained, together with the Stage View plugin if you want that style.

A healthy plugin routine has four habits: prune plugins you do not use (every plugin is attack surface and an upgrade risk), read the plugin's own changelog before bumping, test plugin upgrades on a staging controller that has the same plugin set, and subscribe to the security advisories.

Automating Jenkins: the CLI and the REST API

The problem: sooner or later you need to do something to Jenkins from a script: trigger a build from another system, read a result, list jobs, restart during a change window. Jenkins has a command-line client and an HTTP API, and the two share the same authentication model.

Authentication and the CSRF crumb

Use an API token, created under your user menu, Security, API Token, Add new token. From LTS 2.555.1 a token can carry an expiration date, which is a good habit for tokens given to automation. Requests authenticated with an API token as the Basic-auth password are exempt from CSRF crumbs. That is why scripts should always use a token and not a password.

If you authenticate with a password or a browser session, Jenkins insists on a crumb, and you will see HTTP ERROR 403 No valid crumb was included in the request. The crumb must go along with the same session cookie that was used to fetch it:

COOKIES=$(mktemp)
CRUMB_JSON=$(curl -s -c "$COOKIES" -u alice:PASSWORD https://ci.example.com/crumbIssuer/api/json)
CRUMB=$(echo "$CRUMB_JSON" | jq -r .crumb)
curl -s -b "$COOKIES" -u alice:PASSWORD -H "Jenkins-Crumb: $CRUMB" -X POST https://ci.example.com/job/demo/build

The response from the issuer looks like {"crumbRequestField":"Jenkins-Crumb","crumb":"..."}. A missing cookie is the most common reason a correct-looking crumb is rejected. Do not "solve" the error by disabling CSRF protection (DISABLE_CSRF_PROTECTION): use a token.

The Remote Access API

Append /api/json (or /api/xml) to almost any Jenkins URL to get a machine-readable version of that page. The control that matters is tree, which selects fields and keeps responses small:

curl -s -u alice:API_TOKEN \
  "https://ci.example.com/job/team/job/api/api/json?tree=lastBuild[number,result,url],color"
curl -s -u alice:API_TOKEN "https://ci.example.com/job/team/job/api/42/consoleText" | tail -50
curl -s -u alice:API_TOKEN https://ci.example.com/whoAmI/api/json

Jobs inside folders are addressed as /job/folder/job/name/, which surprises people who expect /job/folder/name. To start a build:

curl -s -i -X POST -u alice:API_TOKEN \
  "https://ci.example.com/job/team/job/api/buildWithParameters" \
  --data BRANCH=main --data DEPLOY=false

A successful request returns 201 Created with a Location: header that points at a queue item, such as https://ci.example.com/queue/item/317/. The build does not exist yet. Poll /queue/item/317/api/json until it contains an executable object with the build's url, then poll that build's api/json until building is false and read result. Scripts that assume the build starts immediately race against the queue.

You can read and write job definitions too: GET /job/NAME/config.xml returns the XML, POST /job/NAME/config.xml replaces it, and POST /createItem?name=NEW with Content-Type: application/xml creates one. That is useful for a one-off bulk change, but prefer Multibranch jobs plus Jenkinsfiles or the Job DSL plugin for anything you will repeat, since raw XML is fragile across plugin versions.

The version of a running controller is in the X-Jenkins response header:

curl -sI https://ci.example.com/login | grep -i '^X-Jenkins'

The CLI

The CLI is a jar downloaded from the controller itself, which keeps the client and the server in step:

curl -O https://ci.example.com/jnlpJars/jenkins-cli.jar
export JENKINS_USER_ID=alice
export JENKINS_API_TOKEN=11ab...
java -jar jenkins-cli.jar -s https://ci.example.com/ who-am-i
java -jar jenkins-cli.jar -s https://ci.example.com/ build team/api -f -v -p BRANCH=main
java -jar jenkins-cli.jar -s https://ci.example.com/ console team/api
java -jar jenkins-cli.jar -s https://ci.example.com/ list-plugins

build -f -v waits for the build to finish (-f follows, -v shows the log) and exits with a non-zero status when it fails, which makes it ideal for wiring Jenkins into another script. -p passes a parameter. The transport is WebSocket by default (since 2.391), with -http and -ssh as alternatives; the SSH mode also supports ECDSA and Ed25519 keys from weekly 2.570. The old -remoting mode was removed long ago, and any guide that uses it is outdated and unsafe. The SSH variant needs the SSH server plugin, which from LTS 2.580.1 is no longer bundled and must be installed explicitly.

Administrative commands deserve care: quiet-down and cancel-quiet-down, safe-restart, reload-configuration, offline-node and online-node, install-plugin ID -deploy, and groovy = (run a script from standard input, administrator only). The Script Console at /script and POST /scriptText do the same thing through the web. They are powerful and dangerous in exactly the same measure: anyone who can reach them can run arbitrary code as the Jenkins process, so keep them limited to administrators, and keep those scripts in Git when they are part of a procedure.

Debugging beyond the console log

The problem: the console output shows the last few lines before the failure, and the real cause is often elsewhere: an agent that died, a plugin that threw, a controller under memory pressure, a proxy that rewrote a header. A fixed order of checks finds it faster than guessing.

Several errors show up again and again, and each has a characteristic meaning:

The script-security row deserves an explanation. Pipeline code runs in a sandbox that only allows a vetted set of methods. When you call something outside it, the build fails and an administrator can approve the signature in Manage Jenkins, In-process Script Approval. Be careful: approving a signature such as Jenkins.get() or ItemGroup.getItems() effectively hands that code administrator-level power. A better habit is to stop needing the approval at all, by using a step from a plugin, or moving the work to a script on the agent.

The reverse-proxy message has a built-in test that tells you what Jenkins sees:

curl -iL -e https://ci.example.com/manage \
  https://ci.example.com/administrativeMonitor/hudson.diagnosis.ReverseProxySetupMonitor/test

A 200 means the proxy is fine; a redirect or an error shows what the proxy is rewriting. The usual culprits are a Jenkins URL in System settings that differs from the address in the browser, and a proxy that strips X-Forwarded-Proto. For WebSocket agents and the CLI, nginx additionally needs the Upgrade and Connection headers forwarded.

Observability, so you know before users do

For a service that every team depends on, logs are not enough. The Prometheus plugin exposes metrics at /prometheus/ (the trailing slash is required), ready for Grafana, and is built on the Metrics plugin, which also offers health checks. The OpenTelemetry plugin goes further: it exports each pipeline and stage as a trace, and metrics such as ci.pipeline.run.* and jenkins.queue.*, over OTLP to any compatible backend, and it is configured in JCasC under unclassified: openTelemetry:. A trace of a slow pipeline shows which stage and which step consumed the time, which is far faster than reading timestamps. Alerts worth having: queue length and wait time, number of offline agents, controller heap and disk, and build duration regressions for your key pipelines. A liveness check against /login (which returns 200 when the application is up) is the common choice for a load balancer or Kubernetes probe.

Upgrades, backups and security hygiene

The problem: the controller is stateful and central. A careless upgrade or a missing backup costs every team their CI at once. The discipline here is boring on purpose.

Follow the LTS line, read the upgrade guide, and test first. The LTS line produces a new baseline about every 12 weeks, then patches it at roughly four-week intervals (2.580.1, then 2.580.2 and so on). Before each baseline jump read the upgrade guide on jenkins.io, which lists exactly what breaks. Recent ones matter more than usual:

• 2.555.1 (April 2026) requires Java 21 or newer on the controller and every agent. Upgrade Java first, use the Versions Node Monitors plugin to check agents, and update plugins both before and after. Java 17 container images are gone. It also removed the client IP from the CSRF crumb (see the JCasC note above), and made the Overall/Manage permission enabled by default.
• 2.541.1 (January 2026) introduced the unified RPM repository https://pkg.jenkins.io/rpm-stable/, a new package signing key (jenkins.io-2026.key) that administrators must accept, and a built-in Content-Security-Policy that is off by default. Once you upgrade on RPM systems you cannot downgrade without pointing back to the legacy repository.
• 2.580.1 (30 September 2026) stops bundling nine detached plugins in jenkins.war: Bouncycastle API, Command Agent Launcher, Oracle Java SE Development Kit Installer, JAXB, Trilead API, SSH server, JavaBeans Activation Framework API, JavaMail API and Instance Identity. Online controllers download them when needed. Offline controllers must place the .hpi files in $JENKINS_HOME/plugins before starting the new version, or configuration that depends on them fails to load. If you upgrade directly from 1.577 or earlier, five more are no longer bundled (JUnit, Mailer, Matrix Authorization Strategy, Matrix Project and OWASP Markup Formatter), so install them first or go through an intermediate LTS.
• Security releases. The September 2026 advisory (2026-09-02, fixed in LTS 2.568.3) covered thirteen core issues including deserialisation, session fixation and stored XSS. Core fixes ship in the weekly and the next LTS patch on the same day, so subscribe to https://www.jenkins.io/security/advisories/ and its RSS feed and plan to apply patch releases promptly.

A good upgrade sequence: back up JENKINS_HOME, put the controller in quiet mode so running builds finish, update plugins, upgrade Java if required, then upgrade Jenkins itself, then update plugins again and check the administrative monitors on the Manage Jenkins page. In a container, bump the image tag and keep the volume; with Helm, helm upgrade with the chart and plugin versions pinned. Rehearse on a staging controller that is restored from a recent backup, which doubles as your backup test.

Back up the right things. That is config.xml, the jobs/ directory (configuration, and build history if you need it), plugins/, secrets/, identity.key*, users/, nodes/ and the top-level XML files such as credentials.xml. Exclude war/, cache/, tools/ and workspaces, which are rebuilt. Use filesystem or volume snapshots, or the ThinBackup plugin. Two points are frequently missed. Without secrets/ the encrypted values in a restored configuration cannot be decrypted, so a backup without it is worth much less than it looks. And because that directory holds the decryption keys, the backup itself must be stored encrypted and with restricted access. JCasC plus Git protects your configuration, but build history still needs a file backup.

Permissions, briefly. The safest default is a matrix or role-based authorisation strategy with least privilege and never "Anyone can do anything". Know the dangerous ones: Overall/Administer is everything, including the Script Console; Job/Configure on a Pipeline lets a user run code on agents and use any credential in scope; Agent/Configure is effectively root on agents; and Run/Replay is arbitrary Pipeline code. Overall/Manage (on by default since 2.555.1) gives partial administration without the ability to run code, which is useful for operators. Senior material covers the trust model in depth; at this level the task is to notice when a convenient permission is also a code-execution permission.

Jenkins in the wider stack

The problem: Jenkins is rarely the whole pipeline. It builds, tests and publishes, and then something else deploys. Being clear about the boundary prevents both gaps and overlap.

A common and sturdy pattern is Jenkins for CI, GitOps for CD. Jenkins builds and tests the code, publishes an immutable image (tagged with the commit, see the Docker guide), and then opens a change against a separate configuration repository that records the new image tag. A GitOps controller, Argo CD or Flux, watches that repository and rolls the change into the cluster. The pipeline never needs cluster credentials, every deployment is a reviewed Git commit, and rollback is a revert. The Jenkins side of it is small:

stage('Promote to staging') {
  when { branch 'main' }
  steps {
    withCredentials([sshUserPrivateKey(credentialsId: 'gitops-key', keyFileVariable: 'KEY')]) {
      sh '''
        GIT_SSH_COMMAND="ssh -i $KEY -o StrictHostKeyChecking=yes" \
          git clone git@git.example.com:platform/deploy-config.git
        cd deploy-config
        sed -i "s|tag: .*|tag: ${GIT_COMMIT}|" envs/staging/values.yaml
        git commit -am "api: deploy ${GIT_COMMIT}" && git push
      '''
    }
  }
}

Other integrations follow the same rule of keeping each tool to its strength. Infrastructure changes with Terraform or Ansible run in a container inside a stage, with credentials bound only for that stage (see the Terraform guide). Helm charts can be linted and packaged in CI and deployed by the GitOps controller (see the Helm guide). Kubernetes provides elastic agents (see the Kubernetes guide). For quality gates, the SonarQube Scanner plugin offers withSonarQubeEnv and waitForQualityGate, and image scanners such as Trivy run as a plain sh step. Notifications go out through Slack, Email Extension or Microsoft Teams plugins. For a job that generates other jobs, the Job DSL plugin (and the jobs: key in JCasC) keeps them in code.

For teams in the Gulf and Egypt, the practical regional point is data residency. Builds handle source code, credentials and sometimes test data, so run the controller and its agents in a region that satisfies your employer's rules (most major cloud providers offer Middle East regions), keep artifact storage and the credentials provider in the same jurisdiction, and check that any SaaS notification or log-forwarding integration does not move build logs out of it.

It is also honest to know when you might move. Teams that live entirely in GitHub or GitLab often find GitHub Actions or GitLab CI simpler, because the runner, secrets and triggers are part of the platform. Jenkins earns its place when you need its plugin ecosystem, complex or long-running pipelines, on-premises or air-gapped control, or years of existing pipelines you cannot afford to rewrite. Senior material goes deeper on that decision.

Putting it all together

Here is one pipeline that uses most of this level: a pinned shared library, a Kubernetes agent, a timeout and build retention, a gated deploy that does not hold an executor while waiting, single-quoted secrets, published test results, and a hand-off to GitOps.

@Library('ci-library@v1.4.0') _

pipeline {
  agent none
  options {
    buildDiscarder(logRotator(numToKeepStr: '20'))
    timeout(time: 45, unit: 'MINUTES')
    timestamps()
    disableConcurrentBuilds(abortPrevious: true)
    durabilityHint('PERFORMANCE_OPTIMIZED')
  }
  environment { IMAGE = 'registry.example.com/team/api' }

  stages {
    stage('Build and test') {
      agent {
        kubernetes {
          defaultContainer 'maven'
          yaml '''
            apiVersion: v1
            kind: Pod
            spec:
              containers:
              - name: maven
                image: maven:3.9.9-eclipse-temurin-21
                command: ["sleep"]
                args: ["infinity"]
                resources:
                  requests: { cpu: "1", memory: 2Gi }
                  limits:   { memory: 2Gi }
          '''
        }
      }
      steps { sh 'mvn -B verify' }
      post { always { junit 'target/surefire-reports/*.xml' } }
    }

    stage('Image') {
      agent { label 'docker' }
      when { beforeAgent true; branch 'main' }
      steps { buildImage(image: env.IMAGE, tag: env.GIT_COMMIT, push: true, credentialsId: 'registry') }
    }

    stage('Approve production') {
      when { beforeInput true; branch 'main' }
      options { timeout(time: 1, unit: 'DAYS') }
      input { message 'Promote to production?'; submitter 'release-managers'; submitterParameter 'APPROVER' }
      steps { echo "Approved by ${APPROVER}" }
    }

    stage('Promote') {
      agent { label 'linux' }
      when { beforeAgent true; branch 'main' }
      steps {
        withCredentials([sshUserPrivateKey(credentialsId: 'gitops-key', keyFileVariable: 'KEY')]) {
          sh './scripts/bump-config.sh production "$GIT_COMMIT"'
        }
      }
    }
  }
  post {
    failure { echo "Build ${env.BUILD_URL} failed" }
    cleanup { echo 'done' }
  }
}

Read it against the sections. Nothing runs on the built-in node, and the agent is ephemeral. The approval happens at pipeline level with agent none, so a day's wait costs no executor. Nothing interpolates a secret in double quotes: buildImage uses single quotes internally, and the promotion script reads KEY from the environment. Test results are recorded even when tests fail. The behaviour that used to live in sixty repeated lines is one pinned library call. And the deployment itself is a commit to a Git repository that a GitOps controller watches.

The same discipline applies to the controller around it. Its settings live in jenkins.yaml, its plugins in a pinned plugins.txt baked into an image, its backups include secrets/ and have been restored at least once, and its upgrades follow the LTS upgrade guide on a staging controller first.

Where you are now

You can predict how a build is scheduled and where it runs, explain why Pipeline Groovy behaves differently from ordinary Groovy and fix serialisation errors, write Declarative pipelines with conditional stages, approvals that do not hold executors and sensible failure handling, run builds on ephemeral Docker and Kubernetes agents, bind credentials without leaking them, share code through pinned libraries, use Multibranch jobs and webhooks, validate and replay pipelines before committing, keep the controller healthy, manage configuration and plugins as code, automate Jenkins through its CLI and API, debug in a fixed order, and upgrade and back up with a plan.

Senior takes every one of these further, with a scale, security or ownership dimension. Which component breaks first as a controller grows, and when to split controllers. The trust model in detail: who can run code where, and how sandbox, trusted libraries and permissions combine. Multi-tenancy with folders, credentials and separate controllers. Secrets with short-lived identity instead of stored tokens. The supply chain of plugins and base images. Upgrade and migration planning across many baselines, incident playbooks, cost control, and running Jenkins as a platform for other teams, including knowing when to choose a different tool.

Sources

• Jenkins documentation
• Stable (LTS) changelog
• Upgrade guide
• Upgrade guide: 2.580
• Upgrade guide: 2.555
• Upgrade guide: 2.541
• Java support policy
• Pipeline syntax reference
• Using a Jenkinsfile
• Extending with Shared Libraries
• Pipeline best practices
• Scaling Pipelines
• CPS method mismatches
• Managing nodes
• Using agents
• Using credentials
• Jenkins CLI
• Remote access API
• Configuration as Code
• Script approval
• System properties
• Managing security
• CSRF protection
• Permissions
• Architecting for scale
• Monitoring Jenkins
• Reverse proxy troubleshooting
• Installing Jenkins with Docker
• Security advisories
• Plugin Installation Manager Tool
• Jenkins Docker images
• Kubernetes plugin
• Prometheus plugin
• OpenTelemetry plugin

This is the mid-level interview guide for Jenkins. At this level an interviewer is no longer checking that you can write a Jenkinsfile. They are checking whether you understand why it behaves as it does, whether you can diagnose a failing or slow system in a sensible order, and whether you can argue a trade-off. The answers below are written the way a strong candidate would say them aloud: a direct answer first, then the reasoning, then the caveat that shows experience. The guide targets Jenkins LTS 2.580.1 on Java 21.

Where this picks up

Beginner interviews ask what a pipeline, stage, agent and credential are. This level assumes all of that and asks about the machinery behind it.

The thirty-second version of the model

If you are asked to explain Jenkins in half a minute, this is a shape that works. Jenkins is a controller: one stateful Java process that stores configuration and plugins in a directory (JENKINS_HOME), schedules a queue, and serves the UI and API. Builds should run on agents, which connect by SSH, inbound WebSocket, or are created on demand by a cloud such as Kubernetes. A Pipeline is code in a Jenkinsfile, run by the controller in a special serialisable style so it can survive restarts, with the heavy work done by shell steps on the agent. Nearly everything else is a plugin. The consequence is that the controller is the thing to protect and keep light.

Scenario questions

"Builds are waiting in the queue and nothing is running. What do you check?"

A strong answer starts with the queue item, because Jenkins tells you why it is waiting. Hover over the item or call /queue/api/json?tree=items[why]. The text distinguishes the cases. Waiting for next available executor on 'linux' means matching nodes exist but are busy, so you look at capacity: executor counts, whether builds are stuck, and whether a cloud has hit its container cap. There are no nodes with the label 'linux' means no online node carries the label. Then the cause is an agent that went offline, a label typo, or a pipeline that uses agent any on a controller whose built-in node has zero executors and no agents at all.

Next, go to the agents: Manage Jenkins, Nodes, look for offline ones, and read why (a disk-space monitor, a failed connection, a Java mismatch after an upgrade). If the agents are Kubernetes pods, the queue will not tell you the pod failed to schedule or pull its image, so you check kubectl describe pod in the agent namespace. Finally, check for a quiet-down: if an administrator left the controller in "preparing for shutdown" mode, nothing new starts.

A weak answer says "restart Jenkins" or "add more executors" without looking at the reason text.

"A pipeline that used to take ten minutes now takes forty. How do you investigate?"

Separate waiting from running, then find the expensive stage. First, is the time spent in the queue before the build starts, or inside the build? If it is queue time, the earlier answer applies. If it is inside the build, look at the stage timings in Pipeline Graph View, or better, traces from the OpenTelemetry plugin if you have it, because a trace shows stage and step durations without reading timestamps.

Then ask what changed: a dependency added, a test suite that grew, a base image that changed, a cache that stopped being hit (a fresh Kubernetes pod with no cache mounted will download the world every time), or an agent on a slower or busier machine. Compare a fast old build and a slow new one, stage by stage.

If the slowness is between steps, not within them, suspect the controller: Pipeline state being written to slow or network storage (consider the PERFORMANCE_OPTIMIZED durability setting, understanding that it risks losing in-flight builds on a hard crash), heavy Groovy running on the controller, or a controller short of heap or CPU. Mention that Pipeline Groovy runs on the controller, so a loop that parses a large JSON file in Groovy costs controller time, while the same work in a script on the agent does not.

What sets a strong answer apart: it distinguishes queue time from run time, it names caching for ephemeral agents, and it knows where the Groovy runs.

"Jenkins keeps running out of memory. What now?"

Do not start by raising -Xmx. A larger heap delays the next failure. First capture evidence: make sure the JVM runs with -XX:+HeapDumpOnOutOfMemoryError (set through JAVA_OPTS in a systemd override) so the next crash leaves a heap dump. Then look for the usual causes: thousands of retained builds and large logs (fix with buildDiscarder), Pipelines doing heavy work in Groovy on the controller (JsonSlurper on large payloads, loops over big collections), a plugin leaking memory (check what was updated recently), or simply a controller carrying too many jobs, where the documented answer is to split into several controllers.

State the order: stop the bleeding (restart, raise heap as a temporary measure), find the cause (heap dump, recent changes), then fix the class of problem (retention policy in a shared template, moving logic into scripts on agents). Mention that editing /etc/default/jenkins no longer works on current packages; JVM options belong in a systemctl edit jenkins override.

"Someone's build printed a secret in the console. What do you do?"

First, treat the secret as compromised and rotate it, because logs are readable by many people and may have been forwarded elsewhere. Then fix the cause. Masking replaces the exact bound value with ****, but it is best effort: if the script transformed the value (base64, split, reversed), masking cannot recognise it. Look for double-quoted sh "..." with a credential variable, where Groovy expands the secret before the shell sees it and the value appears in the process list. The fix is single quotes so the shell expands the variable.

Then address the system: who had Job/Configure on that pipeline (that permission lets a user write a step that prints a secret in any form), whether the credential should have been scoped to a folder rather than global, and whether a short-lived credential from Vault or workload identity would have limited the damage. Finally, delete or restrict the build log, noting that deleting it does not undo exposure.

"The controller was restarted and a long build is stuck. What happened?"

Pipelines are serialised with CPS so they can resume after a restart. A build that was running resumes from its last recorded point, but it is waiting for the agent it was using. If that agent was ephemeral (a Kubernetes pod) and is gone, the log says something like Waiting to resume part of ...: 'agent-x' is offline and it waits until the node reappears, which may be never. The options are to abort it, to have designed for this with options { disableResume() }, or, for Kubernetes agents, to wrap the work in retry(count: 2, conditions: [kubernetesAgent(), nonresumable()]) so a lost pod is retried on a new one.

Follow with the prevention: restart gracefully with quiet-down then safe-restart, which waits for running builds, instead of a hard restart.

"A pipeline fails with NotSerializableException. Explain and fix."

A local variable holding a non-serialisable object (a regex Matcher, a LazyMap from JsonSlurper) is alive when a step runs, and Jenkins tries to persist program state at steps. The fix is to do that work inside a @NonCPS method that returns only plain, serialisable data, or to use a step such as readJSON. Mention the two rules that go with it: @NonCPS methods cannot call Pipeline steps, and Pipeline closures must not be passed into non-CPS methods, which produces the expected to call ... but wound up catching ... error.

"A teammate changed the shared library and broke forty pipelines. How do you prevent this?"

Pin consumers to a tag (@Library('ci-library@v1.4.0') _) rather than a branch, so a change on main reaches nobody until a consumer opts in. Add a test job in the library's repository that runs each step on a real agent, test changes from a branch (@Library('ci-library@my-branch') _) in one consumer first, use semantic version tags and a changelog that names breaking changes, and protect the library's default branch with review. If the library is trusted (global), emphasise that write access to it is effectively administrator access to the controller, since it runs outside the sandbox.

"We upgraded to the new LTS and the controller will not start. Where do you look?"

Read the log first: journalctl -u jenkins.service on a package install. The likely culprits at this point in time map to specific changes:

• A message that Java 17 is older than the minimum means the LTS requires Java 21 or newer (from 2.555.1). Install Java 21 and point JENKINS_JAVA_CMD, JAVA_HOME or update-alternatives at it.
• A Configuration as Code failure mentioning crumbIssuer means a deprecated key remained in jenkins.yaml. Remove it, or set configuration-as-code: deprecated: warn.
• On an offline controller upgrading to 2.580.1, configuration that depends on SSH server, JAXB or another of the nine detached plugins fails to load, because they are no longer bundled. Place their .hpi files in $JENKINS_HOME/plugins and start again.
• From 1.577 or earlier, five more plugins (JUnit, Mailer, Matrix Authorization Strategy, Matrix Project, OWASP Markup Formatter) must be installed first.

Then say what you would have done to avoid it: read the upgrade guide, rehearse on a staging controller restored from backup, and upgrade Java and plugins in the documented order.

Trade-off questions

These have no single correct answer. A strong response names the options, the deciding factor, and what you would choose in a stated situation.

Declarative or Scripted Pipeline?

Declarative is structured and validated: the syntax is checked before running (the Declarative Linter), it has first-class options, when, post, input and matrix, and it is easier for a team to read and review. It limits you on purpose. Scripted is imperative Groovy with full freedom, which means more power and more pitfalls (CPS, serialisation, sandbox). The mature answer: Declarative by default, with script { } blocks for small escape hatches, and real logic moved out into scripts or a shared library. Scripted is justified for unusual dynamic behaviour, but it is the exception, and a whole-pipeline library (vars/standardPipeline.groovy) gives you both consistency and Declarative structure.

SSH agents, inbound agents or Kubernetes pods?

Choose ephemeral pods when you already run Kubernetes and want consistency and cost control. Choose SSH for stable, special hardware or a small fleet. Choose inbound WebSocket when the agent is behind NAT or a strict firewall. Note that -webSocket avoids opening the TCP 50000 inbound port.

Polling or webhooks?

Webhooks give near-instant builds and far less load, because events replace repeated polling. They require that the Git host can reach Jenkins, which is not always possible on private networks. Polling works anywhere but costs load, delays builds by the interval, and scales poorly. A good answer uses webhooks as the main trigger, with a periodic branch re-index in Multibranch jobs as a safety net for missed events.

One big controller or several?

The open-source controller is a single stateful JVM with no active-active mode, so the official scaling advice is to add agents first and then split by team, product or environment. A single controller is simpler to run and gives one place to look; several controllers limit the blast radius of a bad plugin, a runaway job or a compromised administrator, allow different plugin sets and upgrade schedules, and give stronger tenant isolation, at the cost of more to maintain and more configuration to keep consistent (which is where JCasC comes in). The documentation's rough heuristic is about 500 jobs per controller. The deciding factors are trust between teams and how painful a shared outage is.

Shared Library or copy the logic into each Jenkinsfile?

A library removes duplication and lets a platform team fix something once, at the cost of a coupling point: a bad release affects everyone. Copying is isolated but drifts. The answer: use a library for genuinely shared and stable behaviour, version it with tags, keep it small, and keep project-specific logic in the repository. Mention that a trusted library is powerful and must be treated with administrator-level care.

Jenkins, GitHub Actions or GitLab CI?

If the source lives on GitHub or GitLab, their built-in CI is often less to operate: runners, secrets and triggers come with the platform. Jenkins is stronger where you need its plugin ecosystem, control over where builds run (on-premises or air-gapped, or strict data-residency requirements common for Gulf and Egyptian employers), complex or very long pipelines, or a large existing investment. A sensible position is to avoid a rewrite for its own sake, and to migrate when the operational cost of running Jenkins exceeds the value of what only Jenkins provides.

Jenkins for CD, or GitOps?

Having Jenkins run kubectl apply or helm upgrade works, but the pipeline must hold cluster credentials and the record of what is deployed lives in build history. The GitOps pattern has Jenkins build, test and push an image, then commit the new tag to a configuration repository, and lets Argo CD or Flux apply it. You get reviewed, auditable deployments, simple rollback through a revert, and no cluster credentials in CI. The cost is another repository and another system to run.

What a strong answer covers that a weak one misses

Commands and snippets to have ready

You will often be asked to write something small on a whiteboard or in a shared editor. These are the ones worth memorising.

A declarative pipeline with a timeout, retention and a post block:

pipeline {
  agent { label 'linux' }
  options {
    timeout(time: 30, unit: 'MINUTES')
    buildDiscarder(logRotator(numToKeepStr: '20'))
    disableConcurrentBuilds(abortPrevious: true)
  }
  stages {
    stage('Test') {
      steps { sh 'make test' }
      post { always { junit 'reports/**/*.xml' } }
    }
  }
  post { cleanup { cleanWs() } }
}

Safe use of a credential:

withCredentials([usernamePassword(credentialsId: 'registry', usernameVariable: 'U', passwordVariable: 'P')]) {
  sh 'echo "$P" | docker login registry.example.com -u "$U" --password-stdin'
}

A @NonCPS helper:

@NonCPS
String releaseVersion(String branch) {
  def m = (branch =~ /release\/(\d+)/)
  return m.find() ? m.group(1) : null
}

Starting a build over HTTP and following the queue:

curl -s -i -X POST -u alice:API_TOKEN \
  "https://ci.example.com/job/team/job/api/buildWithParameters" --data BRANCH=main
# 201 Created, Location: https://ci.example.com/queue/item/317/
curl -s -u alice:API_TOKEN https://ci.example.com/queue/item/317/api/json

Validating a Jenkinsfile and reading the version:

curl -s -X POST -u alice:API_TOKEN -F "jenkinsfile=<Jenkinsfile" \
  https://ci.example.com/pipeline-model-converter/validate
curl -sI https://ci.example.com/login | grep -i '^X-Jenkins'

Graceful restart:

java -jar jenkins-cli.jar -s https://ci.example.com/ quiet-down
java -jar jenkins-cli.jar -s https://ci.example.com/ safe-restart

Common interview questions

Why should the built-in node have zero executors? Builds on it have the same access to the controller's file system as the Jenkins process. A build could read the encryption keys in secrets/ or alter configuration. Agents isolate that.

What is the difference between stash and archiveArtifacts? stash passes small files between stages (possibly on different agents) within one build and is discarded afterwards; archiveArtifacts attaches files to the build record for later download. Neither is for large artifacts: both pass through the controller, so use an artifact repository such as Nexus, Artifactory or S3.

How do you make a stage run only for pull requests? when { changeRequest() }, optionally with target: 'main'. Variables like CHANGE_ID and CHANGE_TARGET are set only in pull-request builds of a Multibranch job.

What does post { cleanup { } } do differently from always? cleanup runs after every other post condition, so it is the place for things that must happen last, such as deleting the workspace.

How do you stop a newer commit from waiting behind an older one? disableConcurrentBuilds(abortPrevious: true) aborts the older build when a newer one starts.

How do you run a stage in a container? A docker agent (needs Docker Pipeline and Docker on the agent), or a kubernetes agent with a pod spec. reuseNode true keeps the container on the node already allocated.

What is the Snippet Generator for? It generates exact step syntax for the plugins installed on your controller, at /pipeline-syntax, which avoids version-mismatch mistakes.

Why does No such DSL method 'cleanWs' found happen? The plugin that provides the step (Workspace Cleanup) is not installed, or the name is wrong.

What changed in the recent LTS lines that you must know? Java 21 or 25 required from 2.555.1, Java 17 gone. A new package signing key and unified RPM repository in 2.541.1. Nine plugins un-bundled in 2.580.1. Blue Ocean deprecated since July 2026, with Pipeline Graph View as the replacement.

How do you authenticate a script to Jenkins? An API token used as the Basic-auth password. Token-authenticated requests are exempt from CSRF crumbs. Password or session authentication needs a crumb from /crumbIssuer/api/json plus the same session cookie.

What is JCasC, and what does it not do? Configuration as Code declares controller settings in YAML (jenkins.yaml), applied at startup or reload. It does not install plugins, and it overwrites manual UI changes on reload.

How would you reduce cost on Kubernetes agents? Ephemeral pods so nothing idles, a sensible idleMinutes, resource requests and limits that match reality, persistent or remote caches so builds do not re-download dependencies, and retention rules so artifacts do not pile up on the controller.

Sixty-second self-test

Answer each aloud without looking back.

1. Give the two different queue messages and what each means.
2. State the three rules of CPS in one sentence each.
3. Why does sh "curl -u $U:$P ..." leak, and what is the fix?
4. What does beforeAgent true save you?
5. How does a stage-level input avoid holding an executor?
6. Why pin a shared library to a tag, and what is special about a trusted library?
7. Why do webhooks beat polling, and what is the fallback?
8. Name three things that commonly break an LTS upgrade, from the last two years.
9. What does masking protect against, and what does it not?
10. Where would you look first if a Kubernetes-agent build never started?

This is the practical companion to the detailed guide: the errors you will actually meet when running Jenkins at work, the habits that pay back the most, cards to practise on, and recipes for the operational tasks that come up every few weeks. It targets Jenkins LTS 2.580.1 on Java 21. Each error is given as symptom, cause and fix, because the symptom is what you will have in front of you.

Common errors at this level

Symptom: NotSerializableException halfway through a build

java.io.NotSerializableException: java.util.regex.Matcher

Cause. Pipeline Groovy runs in continuation-passing style so a build can survive a restart, and Jenkins persists your local variables at step boundaries. A Matcher, a LazyMap from JsonSlurper or a map entry cannot be persisted.

Fix. Do the work inside a @NonCPS method that returns plain data, and null out any leftover variable before the next step. Or use a step that returns serialisable data, such as readJSON from Pipeline Utility Steps.

@NonCPS
List<String> changedModules(String raw) {
  return raw.readLines().findAll { it.startsWith('modules/') }.collect { it.split('/')[1] }.unique()
}

Symptom: expected to call X but wound up catching CpsClosure2.call

Cause. Non-CPS code (a Java method, some Groovy collection methods, a @NonCPS method) was handed a Pipeline closure or step and called it. The non-CPS code cannot cooperate with the continuation machinery.

Fix. Keep closures and steps out of library methods: use a plain for loop, or move the logic into a @NonCPS method that takes and returns only data. Never call steps from @NonCPS.

Symptom: Scripts not permitted to use method ...

org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException: Scripts not permitted to use method groovy.json.JsonSlurperClassic parseText java.lang.String

Cause. The sandbox allows a fixed list of methods, and your code called one outside it.

Fix. First ask whether you need it: use readJSON, or call jq from a shell step. If you truly need it, an administrator approves the signature in Manage Jenkins, In-process Script Approval, or the code moves to a trusted global library. Be wary of approvals such as Jenkins.get(), which effectively give that code administrator power.

Symptom: a secret shows in ps, or Jenkins warns about Groovy interpolation

Warning: A secret was passed to "sh" using Groovy String interpolation, which is insecure.

Cause. A credential variable inside a double-quoted Groovy string. Groovy expands it on the controller and passes the plain value as a command-line argument.

Fix. Single quotes, so the shell expands it:

withCredentials([string(credentialsId: 'api-token-id', variable: 'API_TOKEN')]) {
  sh 'curl -H "Authorization: Bearer $API_TOKEN" https://api.example.com/deploy'
}

Symptom: No such DSL method 'xyz' found among steps

Cause. The plugin that supplies the step is not installed on this controller, or the name is misspelled. Common cases are cleanWs (Workspace Cleanup), readJSON and readYaml (Pipeline Utility Steps), docker agents (Docker Pipeline), lock (Lockable Resources) and timestamps (Timestamper).

Fix. Install the plugin and add it to plugins.txt. Use the Snippet Generator at /pipeline-syntax to confirm the exact step and parameter names for your versions.

Symptom: Expected a step, Undefined section, Missing required section "agent"

Cause. A Declarative syntax error, most often a Groovy statement placed directly in a stage or steps block instead of inside script { }.

Fix. Wrap the statement in script { }, and run the linter before pushing:

curl -s -X POST -u alice:API_TOKEN -F "jenkinsfile=<Jenkinsfile" \
  https://ci.example.com/pipeline-model-converter/validate

Symptom: MissingPropertyException: No such property: FOO for class: groovy.lang.Binding

Cause. Either the variable is undefined, or you wrote $FOO inside a Groovy double-quoted string meaning to reach a shell variable, and Groovy tried to resolve it itself. A parameter also does not exist until a first build has registered it.

Fix. Use params.FOO or env.FOO in Groovy, and escape shell variables as \$FOO in double-quoted strings, or use single quotes.

Symptom: the build waits forever in the queue

Cause. The queue's reason text is the answer. No node has the label, every matching node is busy, a cloud has reached its container cap, or the agent pods cannot start. On a hardened controller with zero built-in executors, agent any with no agents waits forever.

Fix. Read the why text, check Manage Jenkins, Nodes for offline agents, and for Kubernetes agents run kubectl describe pod on the pending pod.

Symptom: Git checkout fails with Permission denied (publickey) or Host key verification failed

hudson.plugins.git.GitException: Command "git fetch --tags --force --progress -- git@github.com:org/repo.git +refs/heads/*:refs/remotes/origin/*" returned status code 128

Cause. The job has no SSH credential, the wrong one, or the agent does not trust the Git host's key.

Fix. Select a credential of type "SSH Username with private key" in the job or branch source, and configure Manage Jenkins, Security, Git Host Key Verification Configuration (known hosts file, accept first connection, or manually provided keys). Do not choose the "non verifying" option in production.

Symptom: HTTP ERROR 403 No valid crumb was included in the request

Cause. A script used a password or session login for a POST, without the crumb header or without the cookie the crumb belongs to.

Fix. Use an API token as the password, which is exempt from crumbs. If you must use a session, fetch /crumbIssuer/api/json with a cookie jar and send both the Jenkins-Crumb header and the cookie. Never disable CSRF protection to make the error go away.

Symptom: the controller refuses to start after an upgrade

Cause, one of:

• Java older than the new minimum: Running with Java 17 ... which is older than the minimum required version (Java 21).
• A deprecated crumbIssuer key left in jenkins.yaml after 2.555.1.
• An offline upgrade to 2.580.1 without the nine detached plugins placed in $JENKINS_HOME/plugins.

Fix. Install Java 21 and point JENKINS_JAVA_CMD or update-alternatives at it; remove the crumbIssuer key or set configuration-as-code: deprecated: warn; pre-place the .hpi files from the update center. Read journalctl -u jenkins.service first, since it names the cause.

Symptom: an agent will not connect, or drops repeatedly

Cause, one of: the agent JVM is older than Java 21 (you see UnsupportedClassVersionError ... class file version 65.0), the agent.jar is older than the supported Remoting version, a reverse proxy drops idle WebSocket connections, or TCP 50000 is blocked for an agent that was started without -webSocket.

Fix. Upgrade the agent's Java, fetch a fresh agent.jar from /jnlpJars/agent.jar, start with -webSocket, and check the proxy's idle timeout against jenkins.websocket.idleTimeout (60 seconds by default).

Symptom: Disk space is too low. Only 0.9GB left

Cause. The node monitor takes an agent offline when free space falls below about 1 GiB. Workspaces, Docker images and caches accumulate on long-lived agents.

Fix. cleanWs() in post { cleanup { } }, buildDiscarder on jobs, a scheduled docker system prune on agents, and ephemeral agents where you can.

Symptom: PKIX path building failed when downloading plugins

Cause. A TLS-inspecting proxy or a private certificate authority that the Jenkins JVM does not trust. If your laptop also shows certificate errors on local builds, the same kind of proxy is usually the reason.

Fix. Import the authority's certificate into the JVM truststore with keytool -importcert -cacerts ..., or point the JVM at your own store with -Djavax.net.ssl.trustStore. Configure the HTTP proxy under Manage Jenkins, Plugins, Advanced settings. Do not turn verification off.

The practices that pay off most

Ordered by leverage, meaning the ratio of effort to reduced pain.

Three of these deserve a longer word.

Pipelines should be short and boring. The best Jenkinsfile is a readable list of stages that call scripts or library steps. When you feel the urge to write a clever Groovy loop, write a script and call it. Scripts run on the agent, which keeps the controller light, and they can be tested outside Jenkins.

Make failures cheap to reproduce. A build in a container image or a Kubernetes pod spec defined in the repository can be run the same way on a laptop. The fastest debugging loop is "run the same image with the same command". If you cannot do that, your pipeline depends on hidden state on the agent, and the next agent will surprise you.

Treat the controller like a production service. It has a backup with a tested restore, monitors and alerts (queue length, offline agents, heap, disk), a staging copy for plugin upgrades, and an owner. Teams that do this have boring upgrades. Teams that do not discover what they lacked during the incident.

Practice cards

1. Read a queue reasonStart a build with a label nobody has, then with a label that is busy. Compare the two messages in the queue and explain each to a colleague.
2. Cause and cure a serialisation errorHold a Matcher across a sh step, read the exception, and fix it with @NonCPS returning a string.
3. Prove the leakWrite sh "echo $T" with a bound secret, run ps on the agent during a sleep, and then fix it with single quotes and check again.
4. Skip without payingAdd a Docker agent and a when that is false. Compare the run with and without beforeAgent true and count the containers started.
5. Approve without holding a slotBuild the agent none plus stage-level input pipeline, leave it waiting, and confirm no executor is busy in the nodes view.
6. Pin a libraryTag a library at v1, change main, and confirm a consumer pinned to v1 is unaffected while one on main changes.
7. Lint before pushBreak a Jenkinsfile on purpose, run the linter from the command line, and wire it into a pre-commit hook.
8. Round-trip the configurationExport settings with Configuration as Code, View Configuration, trim it to a jenkins.yaml, apply it to a fresh container, and diff the two controllers' settings pages.
9. Restore a backupBack up a test controller including secrets/, restore it elsewhere, and run a job that uses a credential.
10. Follow the queue over HTTPTrigger a parameterised build with curl, take the Location header, and poll until you have the build URL and result.

Operational recipes

Recipe: upgrade the LTS baseline

# 1. Read the upgrade guide for the target baseline and note what applies to you.
# 2. Back up JENKINS_HOME (including secrets/), stored encrypted.
# 3. Stop new builds and let running ones finish.
java -jar jenkins-cli.jar -s https://ci.example.com/ quiet-down
# 4. Update plugins, and Java first if the baseline needs it (21 or 25).
# 5. Upgrade Jenkins (package, image tag, or chart), then start it.
sudo apt update && sudo apt install jenkins        # Debian and Ubuntu packages
# 6. Update plugins again, then check Manage Jenkins administrative monitors.
java -jar jenkins-cli.jar -s https://ci.example.com/ cancel-quiet-down
curl -sI https://ci.example.com/login | grep -i '^X-Jenkins'   # confirm the new version

Rehearse on a staging controller restored from the last backup. For containers, change the image tag and keep the volume (jenkins/jenkins:2.580.1-jdk21); for Helm, run helm upgrade with the chart and plugin versions pinned. On Debian and Ubuntu systems, make sure the repository uses the current signing key jenkins.io-2026.key, since tutorials that mention jenkins.io-2023.key are out of date, and on RPM systems use https://pkg.jenkins.io/rpm-stable/.

Recipe: find out which Java everything runs on

Before any upgrade that changes the Java requirement, list the JVMs in use. The Versions Node Monitors plugin shows each agent's Java version in the nodes table, and the controller's is on Manage Jenkins, System Information. On a package install, check and switch the JVM with:

java -version
sudo update-alternatives --config java
sudo systemctl edit jenkins     # add: Environment="JENKINS_JAVA_CMD=/usr/lib/jvm/java-21-openjdk-amd64/bin/java"
sudo systemctl restart jenkins

Recipe: read a hung build

1. Open the build's console and find the last line printed, and its timestamp if you use timestamps().
2. In the Pipeline Graph View, see which stage is running, and whether it waits on an input.
3. On the node, look at the process tree for the workspace (ps aux, filtered by the job's workspace path) to see whether a child process is really running or has died.
4. If the agent is a pod, kubectl logs and kubectl describe pod show OOM kills (exit 137) and evictions.
5. Abort the build, and add a timeout so this case cannot hold an executor again.

Recipe: a nightly job that does not stampede

Use H in cron expressions so jobs spread across the hour instead of all starting at once:

pipeline {
  agent { label 'linux' }
  triggers { cron('H 2 * * 1-5') }      // some minute in the 2 o'clock hour, weekdays
  stages { stage('Nightly') { steps { sh 'make nightly' } } }
}

The five fields are minute, hour, day of month, month and day of week, and aliases such as @daily work. Combine with when { triggeredBy 'TimerTrigger' } to run the expensive suite only on schedule.

Recipe: copy a job's configuration between controllers

java -jar jenkins-cli.jar -s https://old-ci.example.com/ get-job team/api > config.xml
java -jar jenkins-cli.jar -s https://new-ci.example.com/ create-job team/api < config.xml

Useful for a one-off migration. For anything you will do twice, prefer Multibranch jobs with Jenkinsfiles, or the Job DSL plugin, so the definition lives in Git.

Recipe: a service-account token for automation

1. Create a dedicated user (for example ci-bot) with only the permissions the automation needs: typically Job/Build and Job/Read on one folder.
2. Sign in as that user, open Security, API Token, and add a token with an expiration date.
3. Store the token in your secret manager, and expose it to the automation as JENKINS_API_TOKEN alongside JENKINS_USER_ID=ci-bot.
4. Put a reminder to rotate the token before it expires, since an expired token fails with an authentication error that looks like a wrong password.

Recipe: diagnose a reverse-proxy warning

When Manage Jenkins shows It appears that your reverse proxy set up is broken, run the built-in test and then check the three usual causes:

curl -iL -e https://ci.example.com/manage \
  https://ci.example.com/administrativeMonitor/hudson.diagnosis.ReverseProxySetupMonitor/test

• The Jenkins URL in Manage Jenkins, System must exactly match the address users type.
• The proxy must forward X-Forwarded-Host, X-Forwarded-Port and X-Forwarded-Proto.
• If Jenkins runs under a path, the context path (--prefix=/jenkins or JENKINS_PREFIX) must match the proxy's.

For WebSocket agents and the CLI, the proxy also has to pass the Upgrade and Connection headers.

Recipe: a Kubernetes agent that hands you its logs on failure

Keep the pod around when the build fails, so you can look at it:

agent {
  kubernetes {
    defaultContainer 'maven'
    podRetention onFailure()
    activeDeadlineSeconds 3600
    yaml '''
      apiVersion: v1
      kind: Pod
      spec:
        containers:
        - name: maven
          image: maven:3.9.9-eclipse-temurin-21
          command: ["sleep"]
          args: ["infinity"]
    '''
  }
}

Use this as a temporary measure while debugging. A retained pod is a running pod, so remove the setting once you have the answer, or finished pods will accumulate and use cluster capacity.

Compare the habits

Habits worth adopting now

Put an orientation stage at the top of every pipeline. It prints NODE_NAME, WORKSPACE, the Java version and the versions of the main tools. When a build that worked yesterday fails today, this is the first thing you compare.

Fail fast and loud. Use failFast true on parallel stages, timeout on everything, and skipStagesAfterUnstable() where later stages make no sense after an unstable result. A fast red build is worth more than a slow green one.

Keep the workspace disposable. Do not depend on files surviving between builds. Cache on purpose, in a place you control, and clean the workspace in post { cleanup { cleanWs() } }.

Read the changelog and advisory feed. The stable changelog and the security advisories tell you what is about to affect you. The September 2026 advisory (2026-09-02) fixed thirteen core issues, including deserialisation and session fixation, and core fixes ship in the weekly and the next LTS patch on the same day. Patch releases are not optional.

Name one owner. Whoever runs the controller should be a named person or team, with a channel, an on-call arrangement and a written upgrade routine. Most "Jenkins is broken" incidents are ownership gaps first and technical failures second.

Prefer Pipeline Graph View, and do not install Blue Ocean. Blue Ocean has been deprecated since July 2026 and receives no more security fixes. Some older install instructions, including an example in the official Docker page, still list it. Use pipeline-graph-view.

Watch for region and residency. If your employer is in the Gulf or Egypt and handles regulated data, put the controller, agents, artifact store and credential provider in a region that satisfies its rules, and check that log or notification integrations do not send build output out of it.

A short checklist

Use this for a review of a Jenkinsfile or of the controller itself.

• The built-in node has 0 executors, and builds run on agents (ideally ephemeral).
• Every pipeline has timeout and buildDiscarder.
• No secret appears inside a double-quoted sh; credentials are bound only around the steps that need them.
• Approvals use agent none or a stage-level input, wrapped in a timeout.
• Stages that start expensive agents use when { beforeAgent true ... }.
• Shared libraries are pinned to tags, and trusted libraries have protected branches.
• Webhooks trigger builds; polling is a fallback.
• plugins.txt and jenkins.yaml are in Git, with versions pinned.
• Agents run Java 21 or 25, and the controller and agents are checked before every baseline upgrade.
• Backups include secrets/, are stored encrypted, and have been restored in a test.
• Someone reads the security advisories and the upgrade guide for each LTS baseline.

This is part three of three. Beginner taught you to run a pipeline, and Mid taught you to write good ones. This level is about the thing underneath: a Jenkins controller that thirty teams depend on, that holds every deploy credential the company owns, and that has no standby replica. When it is slow, nobody ships. When it is compromised, every system it can deploy to is compromised with it.

That combination is what makes Jenkins a platform problem rather than a tooling problem. The software is free and flexible, and the flexibility is exactly the risk: anyone who can edit a Pipeline can run code on your agents, and the controller is a single stateful Java process with a file-based data store. By the end of this guide you should be able to explain where that process breaks first, how to split it, how to secure it, how to upgrade it without a bad weekend, and, just as important, when to stop adding to it and move the work elsewhere. This guide targets Jenkins LTS 2.580.1 on Java 21 (Java 25 is also supported).

Where this picks up

I am starting with architecture, because every decision below follows from the shape of the controller.

Architecture and failure modes

A Jenkins controller is one Java process (served by Winstone, which embeds Jetty) that owns three things at once: the scheduler that decides which build runs where, the web tier that serves the UI and the REST API, and the state, which lives as files in JENKINS_HOME. There is no database server. Job configuration is XML, build records are files under each job's builds/ directory, credentials are encrypted blobs in XML, and plugins are archives unpacked into plugins/. The core is not active-active and not highly available. The open-source answer to "what if it dies" is to restart it quickly from persistent storage, and everything in a sound design is arranged to make that restart fast and safe.

The practical failure modes fall into a small number of families, and learning to recognise which family you are in shortens every incident.

Heap exhaustion. The controller keeps job configuration, the build queue, plugin state and the in-flight state of every running Pipeline in memory. Huge console logs, unbounded build history, large objects held in Pipeline variables and big JsonSlurper payloads all sit in that heap. The symptom is java.lang.OutOfMemoryError: Java heap space, or before it arrives, garbage-collection thrash where the UI is unresponsive and builds still appear to run. Raising -Xmx buys time, but the durable fix is to keep less in memory: buildDiscarder everywhere, logic moved to agents, and heap dumps enabled with -XX:+HeapDumpOnOutOfMemoryError so the next occurrence is diagnosable rather than mysterious.

CPU starvation from Pipeline Groovy. This one surprises people, and it is covered in its own section below. Pipeline code executes on the controller, not on the agent. One hundred concurrent pipelines each doing string manipulation in Groovy are one hundred tenants sharing the controller's cores.

Disk I/O. Pipelines write their state to disk as they run, and logs stream to disk continuously. On slow or network-attached storage the controller spends its time waiting on fsync. The Jenkins documentation's sizing advice puts storage size ahead of speed for the controller, but for Pipeline-heavy workloads SSD matters, and the durability setting described later trades safety for less I/O.

Thread and connection pressure. Every agent connection, every webhook, every API poller and every browser tab holds resources. A dashboard that polls /api/json with depth=3 every few seconds for hundreds of jobs can do more damage than a hundred builds. The fix is webhooks instead of polling and ?tree= filters on every API call you control.

The plugin blast radius. Nearly all functionality is a plugin, plugins share the controller's JVM and class loader space, and a plugin that misbehaves can degrade or crash the whole process. This is why a test controller for plugin upgrades is an architectural requirement, not a luxury.

State damage and state leaks. A corrupted disk, a bad upgrade that rewrites configuration, or an operator deleting the wrong folder. And, worse, the leak: JENKINS_HOME contains the keys (secrets/) that decrypt every credential in it, so a stolen backup is a stolen credential store.

Two architectural consequences follow. First, the built-in node should have zero executors in production (Manage Jenkins, Nodes, Built-In Node, Configure). A build on the built-in node has the same access to the controller's file system as the Jenkins process itself, which means any Pipeline author can read secrets/ and every other team's job configuration. Second, because the controller cannot be made redundant, you reduce its blast radius by running several of them, each smaller, each owned by a clear group, and each reconstructible from code.

What breaks first when you scale

It helps to think of scale on three axes that do not move together: the number of jobs (configuration and scan load), the number of concurrent builds (scheduler, Groovy CPU and log I/O), and the number of users and API clients (web tier). The documentation's planning heuristics are deliberately rough, and worth quoting as orders of magnitude rather than promises: roughly 500 jobs per controller, and executors in the region of the job count multiplied by 0.03. They are a prompt to measure, not a formula.

In practice the order in which things break is usually this.

1. The API and UI, from polling. External tools that ask Jenkins "what is the status of everything?" on a timer are the most common self-inflicted load. Replace them with webhooks and event notifications, and cap the depth of every query.
2. Branch indexing and SCM polling. A Multibranch Pipeline or Organization Folder scans repositories to find branches and pull requests. With hundreds of repositories, scheduled scans collide. Webhooks drive scans on demand, and the periodic scan becomes a slow safety net rather than the main trigger.
3. Pipeline Groovy CPU. Covered in detail in the next section. It is the limit that does not show up in any "number of executors" arithmetic.
4. The queue and executor supply. This is the good problem, because ephemeral agents (next section but one) absorb it. The controller's scheduler, not the agent fleet, is what eventually saturates.
5. Heap and disk. Long retention, giant logs and thousands of builds per job push these. Retention policy is capacity planning.
6. Plugin interactions. Past a certain plugin count, the upgrade matrix itself becomes the scaling limit. You can measure it by how long it takes to get a plugin update through staging.

The response is not a bigger machine beyond a point. The documentation's own guidance on horizontal scaling is to split controllers by product line, organisation or environment, and its sizing advice is to "invest in a larger machine for your controller over a faster one", meaning that capacity (memory, cores) is worth more than raw clock speed. A controller per business unit, each with its own agents and its own credentials, is also the only strong multi-tenancy Jenkins offers, which is a point I return to in the multi-tenancy section.

Measure before deciding. The Prometheus plugin exposes the signals that tell you which axis you are on, and the queue is the honest one: a queue that is long while executors are idle means a scheduling or label problem, while a long queue with every executor busy means you need more agents. A controller that is slow while the queue is empty means you are CPU-, heap- or disk-bound on the controller itself.

# Is the controller the bottleneck, or are we short of agents?
# Queue length and how long items have waited
curl -s -u "$JENKINS_USER_ID:$JENKINS_API_TOKEN" \
  "https://ci.example.com/queue/api/json?tree=items[why,inQueueSince,task[name]]"

# Busy versus total executors across all nodes
curl -s -u "$JENKINS_USER_ID:$JENKINS_API_TOKEN" \
  "https://ci.example.com/computer/api/json?tree=busyExecutors,totalExecutors"

The why field in the queue output is gold. It says things like Waiting for next available executor on ‘linux’ or There are no nodes with the label ‘linux’, which tell you in a sentence whether the problem is capacity, a missing label, or a cloud cap.

Pipeline execution, CPS and durability

The single most useful fact about Jenkins performance is that Pipeline Groovy runs on the controller. The steps that do real work, such as sh and bat, run on the agent. But the Groovy that decides what to do next, loops over lists, concatenates strings and calls your shared library runs in the controller's JVM, under the Continuation-Passing Style (CPS) interpreter.

CPS exists so that a Pipeline can be serialised to disk and resumed after a restart. Every time execution reaches a step, the interpreter can capture its whole state. That is a wonderful property and it has a price: CPS-transformed Groovy is much slower than ordinary Groovy, every local variable that is live across a step must be serialisable, and the cost scales with how much Groovy you write. A Pipeline that parses a 20 MB JSON file with JsonSlurper in Groovy is doing a heavy job on the shared controller. The same job in jq inside an sh step costs the controller nothing.

The rules that follow are the ones a platform team enforces in review:

• Groovy is glue. Real work belongs in sh steps on agents, or in tools invoked by them.
• Combine many tiny sh steps into one script. Each step has fixed overhead on the controller.
• Avoid controller-side HTTP calls and big parsing in Pipeline code. Use readJSON (from Pipeline Utility Steps) for small data and jq or a script for large data.
• Keep shared libraries lean and do not make them load enormous data at startup.
• Never call steps from @NonCPS methods, and never hold non-serialisable objects across step boundaries.

The error messages teach the model. java.io.NotSerializableException: java.util.regex.Matcher means a non-serialisable object was still live when CPS tried to persist state, so compute inside a @NonCPS method that returns plain data. And expected to call java.util.ArrayList.toSorted but wound up catching org.jenkinsci.plugins.workflow.cps.CpsClosure2.call means non-CPS code (a Java or GDK method) tried to call a CPS closure, so do not pass Pipeline closures into GDK methods.

// Bad: heavy parsing runs in CPS on the controller
def data = new groovy.json.JsonSlurperClassic().parseText(readFile('report.json'))

// Better: let the agent do the work and hand back one small value
def failures = sh(script: "jq '.failures | length' report.json", returnStdout: true).trim()
echo "failures: ${failures}"

Durability: what you trade for speed

Because Pipelines persist their state as they run, how often and how carefully they do it is a tunable. Jenkins offers three durability hints. MAX_SURVIVABILITY is the original behaviour and the default: the slowest, with the strongest guarantees. SURVIVABLE_NONATOMIC writes less carefully. PERFORMANCE_OPTIMIZED does much less disk I/O, but a dirty shutdown (a crash or power loss, as opposed to a graceful restart) can lose the state of running Pipelines.

That is a genuine trade-off and a good interview topic. A fleet of short CI builds on network storage benefits enormously from PERFORMANCE_OPTIMIZED, because a lost running build is simply re-run. A release Pipeline that takes six hours and waits on human approval should stay at the safe setting. You can set it globally (Manage Jenkins, System, Pipeline Speed/Durability Settings), per job, or in the Jenkinsfile:

pipeline {
  agent { label 'linux' }
  options {
    durabilityHint('PERFORMANCE_OPTIMIZED')   // short CI run: a lost build is just re-run
    timeout(time: 30, unit: 'MINUTES')
    buildDiscarder(logRotator(numToKeepStr: '20'))
  }
  stages {
    stage('Build') { steps { sh 'make build' } }
  }
}

A related option for long-lived platforms is disableResume(). By default a Pipeline tries to resume after a controller restart, which is what you want until an agent has vanished and the build sits in Resuming build ... Waiting to resume part of ... : ‘agent-x’ is offline. For ephemeral Kubernetes agents, resuming a build whose pod no longer exists is pointless, and the Kubernetes plugin's retry(count: 2, conditions: [kubernetesAgent(), nonresumable()]) pattern gives you a cleaner answer.

Agents as a fleet

At the platform level agents stop being machines you care about and become fungible capacity. The documentation's advice is to standardise them so any agent can run any job, which means jobs declare what they need (a label, or a pod spec) and never depend on a particular host.

There are three ways agents attach to the controller. SSH agents are the ones the controller connects out to: the documentation treats them as the most stable, they need the SSH Build Agents plugin and an "SSH Username with private key" credential, and host key verification matters (choose Known hosts file or a manually provided key, and avoid "Non verifying"). Inbound agents connect in from the agent side, which is firewall-friendly, and with -webSocket they need no extra port and pass through an ordinary HTTP reverse proxy. Without -webSocket they use the inbound TCP port (50000 in the container images), which is disabled by default (-1) since Jenkins 2.0. Clouds create agents on demand and delete them afterwards.

# An inbound agent over WebSocket: no TCP 50000, works through a reverse proxy
curl -sO https://ci.example.com/jnlpJars/agent.jar
java -jar agent.jar \
  -url https://ci.example.com/ \
  -secret @secret-file \
  -name "linux-01" \
  -webSocket \
  -workDir "/home/jenkins/agent"

The security model of the connection is worth stating. Agent-to-controller security has been on since 2.326 and cannot be switched off in the UI, so an agent cannot ask the controller to read arbitrary files. That does not make agents safe to share between trust levels: an agent runs whatever the build tells it to, and anything one build leaves behind (a workspace, a Docker socket, a cached credential) is visible to the next build on the same machine. Ephemeral agents solve this by construction.

The Kubernetes cloud

For most platform teams the answer is the Kubernetes plugin, which creates a pod per build and deletes it afterwards. The agent container is named jnlp (image jenkins/inbound-agent), and Pipelines describe their own toolchain as extra containers.

pipeline {
  agent {
    kubernetes {
      defaultContainer 'maven'
      yaml '''
        apiVersion: v1
        kind: Pod
        spec:
          containers:
          - name: maven
            image: maven:3.9.9-eclipse-temurin-21
            command: ["sleep"]
            args: ["infinity"]
      '''
    }
  }
  stages { stage('build') { steps { sh 'mvn -B verify' } } }
}

The settings a platform owner actually tunes are few but consequential. The container cap is a concurrency limit on the cloud: it stops a burst of builds from asking the cluster for more pods than it can hold, and when it is reached builds queue with a "waiting" reason rather than failing. idleMinutes keeps a pod alive between builds to save start-up time, at the price of idle cost and some state leakage. podRetention (never(), onFailure(), always(), evicted()) controls whether finished pods remain for debugging. activeDeadlineSeconds kills runaway pods. Use WebSocket when the controller is reachable only over HTTP(S). When containers in one pod must share the workspace, they need the same UID through securityContext.runAsUser, or you will chase permission errors.

The typical fleet failures are all visible from kubectl. Pods stuck in Pending mean the cluster has no capacity or the requests are too large; ImagePullBackOff means a bad image or missing pull secret; repeated pod creation failures in the controller log mean the service account lacks rights in the namespace. If you run the cluster yourself, the Kubernetes guide covers the scheduling side, and the controller itself is commonly installed with the Helm chart, whose values are the place to set controller.installPlugins, controller.JCasC.configScripts and persistence.storageClass.

The trust model

Security on a CI server starts from an uncomfortable premise: the whole point of the product is to run other people's code, with access to deployment credentials. So the question is never "can a user run code?" but "what can that code reach?". Answering it means understanding what each permission really grants, because several of them are far more powerful than their names suggest.

The authorisation strategy should be Matrix-based, Project-based Matrix, or Role-Based Strategy, never "Anyone can do anything". "Logged-in users can do anything" is acceptable only for a tiny and completely trusted setup. The security realm (who you are) should be your identity provider: LDAP or Active Directory, or SAML, OIDC or Entra ID through plugins, so that offboarding happens in one place. Jenkins' own user database is the default, and it becomes a liability the moment you have people who come and go.

A caution about the Overall/Manage permission, because the documentation is inconsistent in a way that has caught people: it is enabled by default from 2.555.1, and the system-properties reference still lists jenkins.security.ManagePermission as defaulting to false. The changelog and the permissions page are the authoritative sources. Verify on your own controller what a "manager" role can do, rather than relying on whichever page you read last.

The Script Console is a root shell

The Script Console (/script, and POST /scriptText) runs Groovy with full controller privileges. It is invaluable for incident response and it is also the first thing an attacker looks for. Restrict Overall/Administer to a handful of people, log who uses it, and prefer auditable changes (JCasC through a pull request) over console snippets. Script approval has the same shape: approving a signature such as Jenkins.get() or ItemGroup.getItems() effectively grants administrator-level power to every sandboxed Pipeline, so a platform team should approve signatures rarely and only after asking whether a plugin step or an agent-side tool could do the job.

Trusted and untrusted code

Pipeline code runs in a Groovy sandbox that restricts it to an allowlist of methods. Anything outside the allowlist fails with RejectedAccessException: Scripts not permitted to use method ... until an administrator approves it in In-process Script Approval. The important distinction is for shared libraries. A global library (Manage Jenkins, System, Global Trusted Pipeline Libraries) can be marked trusted and run outside the sandbox, with full access to the controller. A folder-level library is always untrusted and sandboxed. So the rule is: trusted libraries are code you review like production software, changed only by the platform team, and team-owned libraries live in folders where the sandbox contains them.

// Load a pinned, reviewed version of the platform library. A branch name here
// means every team's pipeline changes the moment someone merges to that branch.
@Library('platform-lib@v3.4.0') _

pipeline {
  agent { label 'linux' }
  stages {
    stage('Deploy') {
      steps { deployApp(env: 'staging') }   // vars/deployApp.groovy in the library
    }
  }
}

Pinning library versions by tag is a platform habit worth enforcing. @Library('platform-lib@main') means a bad merge to the library breaks every pipeline in the company at once; a tag means teams adopt changes on their own schedule, and you can measure adoption.

Hardening settings that are easy to forget

• CSRF protection stays on. Requests authenticated with an API token are exempt from crumbs, so automation should use tokens, and never a disabled protection. The property hudson.security.csrf.GlobalCrumbIssuerConfiguration.DISABLE_CSRF_PROTECTION exists and should never be set. Since 2.555.1 the crumb no longer includes the client IP address, and a leftover crumbIssuer block in JCasC cancels startup unless deprecated: warn is configured.
• API tokens can carry an expiration date from 2.555.1. Require it in your platform's conventions, and keep the defaults that stop administrators seeing other users' tokens (jenkins.security.ApiTokenProperty.showTokenToAdmins is false).
• Served user content. Files in workspaces and archived artifacts are served by Jenkins itself, so a malicious HTML file could run script in your controller's origin. The default hudson.model.DirectoryBrowserSupport.CSP is restrictive. Rather than loosening it when a team complains that their HTML report does not render, configure a Resource Root URL, a separate domain for user content, which is the documented preferred answer.
• Built-in Content-Security-Policy for the Jenkins UI arrived in 2.541.1 and is disabled by default for backward compatibility. Evaluate enabling it on a test controller, and if you use the older CSP plugin, disable it or move to its 2.x release. CSP headers can now be disabled entirely as of 2.555.1.
• The markup formatter stays on "Safe HTML" or plain text, because a permissive formatter turns every job description into a cross-site-scripting vector.
• Deserialization. Weekly 2.572 made deserializing fields of type Object prohibited by default. It is a hardening change, and it can break an old plugin, which is a good reason to treat plugin age as a risk indicator.

The advisory stream is part of the trust model. Jenkins publishes security advisories at https://www.jenkins.io/security/advisories/ (with an RSS feed at /security/advisories/rss.xml), and the core advisories in 2026 fell on 02-18, 03-18, 06-10, 08-05 and 09-02. Core fixes ship in the weekly release and the next LTS patch the same day. The 2026-09-02 advisory, fixed in LTS 2.568.3, covered thirteen core issues including several deserialization problems, session fixation (SECURITY-4016), cross-origin exposure of the CSRF token (SECURITY-3878) and stored XSS in the system log viewer (SECURITY-3967). Somebody on the platform team must own reading these, and your patch SLA should be written down: for example, "security releases applied to staging within two days and production within a week". That service-level target is an example of the kind of commitment to make, not a number from the Jenkins project.

Secrets and the supply chain

There are two separate questions under this heading: how credentials are stored and handed to builds, and how you know that the code running on your controller is what you think it is.

How credentials live

Credentials are stored encrypted in credentials.xml and in job configuration, using keys held in $JENKINS_HOME/secrets/ (notably master.key and hudson.util.Secret). Two conclusions follow directly. The keys and the ciphertext sit together, so anyone who copies JENKINS_HOME holds everything needed to decrypt the credentials. Backups of JENKINS_HOME must therefore be treated as secrets and stored encrypted with tight access. And the secrets/ directory must be included in backups, because without it the encrypted values are unrecoverable. Both statements are true at once.

Credential scope and domains limit reach. A Global credential is usable by any job. A System credential is usable only by the controller itself, which is where agent-launch keys belong. Folder-level credential stores scope a credential to one team's folder, and this is the building block of multi-tenancy. Jobs reference credentials by ID, so IDs become part of your platform's contract: name them predictably (team-a-registry-push) and never reuse an ID for a different secret.

Do not keep long-lived secrets in Jenkins if you can avoid it

The best Jenkins secret is one that is not there. The higher-maturity options, in rough order, are:

1. Short-lived credentials obtained at run time through workload identity or OIDC from the build to the cloud, where the plugins support it, so there is nothing stored to steal.
2. An external secret store as the source of truth: HashiCorp Vault, AWS Secrets Manager Credentials Provider, Azure Key Vault, GCP Secret Manager, or the Kubernetes Credentials Provider, so Jenkins reads secrets at use rather than holding a copy.
3. JCasC interpolation (${VAR}) reading from environment variables or files in the secrets directory, so the YAML in Git never contains a value.
4. Stored Jenkins credentials, scoped to a folder, rotated on a schedule.

Whatever the source, the usage rule never changes: single-quote shell strings that use secrets. Masking replaces the value with **** in the console log, but masking is best-effort and is not a security boundary. A value that has been transformed (base64-encoded, split across lines) is not masked.

steps {
  withCredentials([usernamePassword(credentialsId: 'team-a-registry-push',
                                    usernameVariable: 'REG_USER',
                                    passwordVariable: 'REG_PASS')]) {
    // Single quotes: the shell expands $REG_PASS on the agent.
    sh 'echo "$REG_PASS" | docker login registry.example.com -u "$REG_USER" --password-stdin'
    // Double quotes would make Groovy interpolate the secret on the controller,
    // and Jenkins warns: "A secret was passed to "sh" using Groovy String interpolation".
  }
}

The plugin supply chain

Every plugin is code that runs inside the controller's JVM with the controller's privileges. A platform team therefore has a supply-chain surface of several hundred packages that it did not write. The controls are practical ones:

• Keep the list short. Every plugin you do not install is one you never patch. Review the list twice a year and remove what is unused.
• Pin versions. plugins.txt should list versions explicitly, and jenkins-plugin-cli --latest false takes the minimum versions rather than floating to the newest, avoiding surprise upgrades.
• Change by pull request. An update to the pinned list goes through review, then through the staging controller, then to production.
• Watch for abandonment. Plugins can be marked as unmaintained or deprecated on their pages, and Blue Ocean is the current example: it has been deprecated since July 2026 and receives no further security fixes or updates. Replace it with Pipeline Graph View (pipeline-graph-view) and Pipeline: Stage View. Note that the official Docker installation page's example Dockerfile still installs blueocean, so do not copy it without editing.
• Pin what agents run. Container images used in Kubernetes pod specs are part of the supply chain too. Pin by digest where you can, and see the Docker guide for the signing and provenance work.
• Protect the Jenkinsfile's source. A Jenkinsfile in a repository is executable configuration. Branch protection and required review on the pipeline definition (and on the shared library) matter as much as any Jenkins setting.

An important related case is the pull-request build. A Multibranch job that builds pull requests from forks runs the contributor's Jenkinsfile with your agents. Configure the branch source so that untrusted contributors' changes to the Jenkinsfile are not used, and never expose production credentials to pull-request builds. Use a separate folder, label and credential set for untrusted builds.

Multi-tenancy, honestly

Jenkins is weakly multi-tenant. It gives you tools for organisation (folders, per-folder permissions and credential stores) and for soft isolation (labels that restrict where jobs run, libraries scoped to a folder). It does not give you a hard wall between tenants sharing a controller, because of the fact from the previous section: anyone with Job/Configure on a Pipeline runs code that can reach any agent it can label and any credential in scope.

A workable design has tiers, and being able to explain the tiers is the heart of a platform answer.

Soft isolation is right when the teams trust each other and the goal is tidiness and accident prevention. Each team gets a folder, with permissions granted on the folder (Project-based Matrix or Role-Based Strategy), credentials in a folder-scoped store, and a folder-level library for team-specific code. Restrict which agents a folder may use, with the Job Restrictions or Label restrictions plugins, so one team's Pipelines cannot land on another team's production-deploy agents.

Firmer isolation adds the Authorize Project plugin so a build runs as a named user with that user's permissions, instead of the all-powerful SYSTEM identity, and uses ephemeral agents in a per-tenant Kubernetes namespace with its own service account and quotas. This limits the damage of a mistake and of a curious engineer, though it is not a guarantee against a determined one.

Hard isolation is one controller per tenant. It is the documentation's horizontal-scaling advice, it is the only approach that survives a hostile tenant, and it is cheaper than it sounds when the controllers are cattle: created from the same Helm chart, the same JCasC base and the same pinned plugin list, differing only in a tenant-specific overlay. The cost is operational (more controllers to patch), and the answer to that cost is automation, which is the subject of the next section.

# Helm values: one controller per tenant, all built from the same base
controller:
  numExecutors: 0
  installPlugins:
    - kubernetes:latest        # pin exact versions in real use
    - workflow-aggregator:latest
    - configuration-as-code:latest
    - pipeline-graph-view:latest
  JCasC:
    configScripts:
      tenant: |
        jenkins:
          systemMessage: "Tenant: payments. Managed by the platform team."
persistence:
  storageClass: fast-ssd

Regional and legal factors show up here. Employers in the Gulf and Egypt often have data-residency rules for source code and build artifacts, and a per-tenant or per-region controller makes that simple to satisfy, because the controller, its agents and its JENKINS_HOME can sit in the required cloud region while shared templates stay identical. A single global controller cannot easily make that promise.

Configuration as code and plugin management

If the controller is going to be rebuilt from code on demand, the code has to describe everything: the image and plugins, the configuration, and the jobs. Three artifacts do that. plugins.txt and jenkins-plugin-cli define which plugins are installed. JCasC (the configuration-as-code plugin) describes configuration in YAML. Job DSL (the jobs: key, with the job-dsl plugin) or Organization Folders create the jobs. JCasC does not install plugins, which is the single most common misunderstanding: it can only configure what is already present.

jenkins:
  systemMessage: "Managed by JCasC. Do not edit in the UI."
  numExecutors: 0
  securityRealm:
    local:
      allowsSignup: false
      users:
        - id: admin
          password: ${ADMIN_PASSWORD}
  authorizationStrategy:
    globalMatrix:          # key and format depend on the matrix-auth plugin version; verify
      entries:
        - user: { name: admin, permissions: [Overall/Administer] }
        - group: { name: authenticated, permissions: [Overall/Read] }
credentials:
  system:
    domainCredentials:
      - credentials:
          - string: { scope: GLOBAL, id: api-token, secret: "${API_TOKEN}" }
unclassified:
  location:
    url: https://ci.example.com/

The behaviours a platform owner must know:

• The config source is the environment variable CASC_JENKINS_CONFIG or the Java property -Dcasc.jenkins.config. It can be a file, a folder (every *.yml and *.yaml recursively), an https:// URL, or a comma-separated list. The default is $JENKINS_HOME/jenkins.yaml.
• Multiple files must be supplementary. They merge, and they cannot override each other's values. So a base layer and a tenant overlay must each own different keys, and this shapes how you split files.
• Secrets use ${VAR} interpolation from environment variables, files in a secrets directory (/run/secrets by default), or Vault and AWS Systems Manager through extra plugins. A literal ${ is written ^${VAR}.
• You can reload without restart from Manage Jenkins, Configuration as Code, or with POST /configuration-as-code/reload using CASC_RELOAD_TOKEN. The "View Configuration" export page helps bootstrap YAML from a hand-configured controller.
• Do not mix UI edits with JCasC. The next reload or restart overwrites them, and people will be angry. Put a systemMessage on the controller saying so, and remove the permissions that let people edit what JCasC owns.
• Unknown and deprecated keys are configurable: configuration-as-code: deprecated: reject|warn and unknown: reject|warn. This is what turned the 2.555.1 crumb change into a startup failure: a leftover crumbIssuer block is deprecated, and with reject Jenkins refuses to start.

Failures read like this: io.jenkins.plugins.casc.ConfiguratorException: Invalid configuration elements for type ... or No configurator for the following root elements ... both mean that a key is misspelled or the plugin that owns it is not installed. The fix is almost always in plugins.txt, not in the YAML.

The image, end to end

FROM jenkins/jenkins:2.580.1-jdk21
COPY --chown=jenkins:jenkins plugins.txt /usr/share/jenkins/ref/plugins.txt
RUN jenkins-plugin-cli -f /usr/share/jenkins/ref/plugins.txt
COPY --chown=jenkins:jenkins casc/ /usr/share/jenkins/casc/
ENV CASC_JENKINS_CONFIG=/usr/share/jenkins/casc
ENV JAVA_OPTS="-Djenkins.install.runSetupWizard=false"

Skipping the setup wizard is only safe because JCasC configures the security realm and authorisation in the same image. Without that, runSetupWizard=false leaves an unsecured controller on the network, which is one of the classic ways a Jenkins server gets exposed. The plugins.txt format allows id, id:1.2.3, id:latest, id:experimental, an explicit URL (id::https://url/plugin.hpi), and comments with #. In regulated environments, jenkins-plugin-cli can also check for updates (--available-updates --output txt), which is a useful step in a scheduled job that opens a pull request for plugin bumps.

Upgrades and migrations

Staying current is the cheapest security control Jenkins has, and it is also the operation most likely to hurt you, because the controller has state and plugins and no rollback button. The discipline is to stay on the LTS line (a new baseline roughly every twelve weeks, then .1, .2 and .3 patches about four weeks apart), read the upgrade guide for each baseline, and rehearse on a staging controller restored from a production backup.

The recent baselines each carried a change that broke somebody, and reading them as a sequence is the best way to learn what "breaking" looks like in this product.

The 2.580.1 detail that catches offline installs

The nine plugins no longer bundled are Bouncycastle API, Command Agent Launcher, Oracle Java SE Development Kit Installer, JAXB, Trilead API, SSH server, JavaBeans Activation Framework (JAF) API, JavaMail API and Instance Identity. An online instance downloads them automatically when something needs them. An offline or air-gapped instance does not, so a configuration that depended on, for example, the SSH server (the SSH CLI) or JAXB fails to load. The fix is to download the .hpi files from the update center, put them in $JENKINS_HOME/plugins, and only then start the new version.

If you jump straight from 1.577 or earlier, five more are unbundled (JUnit, Mailer, Matrix Authorization Strategy, Matrix Project and OWASP Markup Formatter), and you must install them before starting, or matrix security, multi-configuration jobs, JUnit reports, email and markup formatting cannot load. The alternative is to upgrade through an intermediate LTS. A version gap that large is rare, but old controllers are precisely where platform teams inherit them. One more detail avoids a wrong claim in your runbook: Apache Commons Lang 2 was dropped from the weekly line at 2.579, but LTS 2.580.1 restored Commons Lang 2.6 for compatibility, so do not announce its removal for the LTS. Plugins should still stop relying on core to supply it.

A procedure worth writing down

1. Read the upgrade guide for the target baseline (/doc/upgrade-guide/2.580/) and every baseline between here and there, plus the changelog.
2. Back up JENKINS_HOME (next section), including secrets/.
3. Fix the Java first if the baseline demands it. Use the Versions Node Monitors plugin to audit agents, and on Linux select the JVM with update-alternatives or set JENKINS_JAVA_CMD in a systemd override.
4. Update plugins on the current version first, on staging, so the combination that arrives in production has already run.
5. Stage by restoring the production backup to a test controller and upgrading there. Run a few representative Pipelines on real agents, and watch the Manage Jenkins administrative monitors for warnings.
6. Drain production with quiet-down (no new builds start), wait for running builds, then safe-restart. Both preserve Pipeline state better than a hard stop.
7. Upgrade by changing the image tag or package and keeping the volume. On Helm, helm upgrade with the chart and plugin versions pinned.
8. Verify with the X-Jenkins response header (curl -sI https://ci.example.com/login | grep -i x-jenkins), the administrative monitors, a smoke Pipeline and the agent list.
9. Keep the rollback ready: the previous image tag and the pre-upgrade backup. Note that some changes are one-way, so after 2.541.1 a package downgrade requires pointing back at the legacy RPM repository.

# Graceful upgrade on a Linux package install
java -jar jenkins-cli.jar -s https://ci.example.com/ -auth @token-file quiet-down
# ...wait for the executors to drain (watch /computer/api/json?tree=busyExecutors)...
sudo systemctl stop jenkins
sudo apt install --only-upgrade jenkins     # or: sudo dnf upgrade jenkins
sudo systemctl start jenkins
curl -sI https://ci.example.com/login | grep -i x-jenkins

For package installs, two earlier changes still bite. Do not edit /etc/default/jenkins or /etc/sysconfig/jenkins, which have been ignored since the systemd migration; use systemctl edit jenkins for overrides. And the package signing key changed, so an old jenkins.io-2023.key in a tutorial is wrong, and the repository must trust jenkins.io-2026.key.

Migrations

The other migration you will actually run is leaving an old controller: from a snowflake VM to a container, from one cloud to another, or from one controller to several. Copying JENKINS_HOME works, but it copies every unknown as well. The better path for a large estate is to rebuild from code (image, plugins.txt, JCasC, Job DSL) and migrate only what must be migrated (job history for a few key jobs, credentials through an export/re-import into the external store). Migrate a tenant at a time, run old and new in parallel for a release cycle, and use the migration as the moment to delete jobs nobody has run in a year. The jobs you delete are the cheapest ones to maintain.

Backup, restore and disaster recovery

A backup you have not restored is not a backup. Jenkins makes this unusually concrete because the state is just a directory, which means a restore test is cheap. What must be in the backup: config.xml, jobs/ (configuration, and optionally build history), plugins/ (the .hpi and .jpi files, so you can reproduce the exact set), secrets/, identity.key*, credentials.xml, users/, nodes/ and the other top-level XML files. What to exclude: war/, cache/, tools/ and workspaces, which are regenerated and large.

There are three workable mechanisms and they complement each other. Volume or filesystem snapshots are fast and complete, and they are the right first line on cloud block storage (take them while the controller is quiet, or accept crash-consistent state). The ThinBackup plugin writes configuration backups on a schedule. And JCasC plus Git makes configuration reproducible without any backup at all, leaving only build history and secrets as the things that need file-level protection.

Decide what you actually promise, in two numbers. The recovery point is how much work you can afford to lose: for configuration under version control it is zero, and for build history it may be a day. The recovery time is how long until developers can build again, and for a code-defined controller with persistent storage it is the time to start a container and reattach a volume. State them, test them quarterly, and record the test.

# Restore rehearsal: does yesterday's backup actually start?
mkdir -p /srv/jenkins-restore-test
tar -xzf /backups/jenkins-home-latest.tar.gz -C /srv/jenkins-restore-test
docker run --rm -p 18080:8080 \
  -v /srv/jenkins-restore-test:/var/jenkins_home \
  jenkins/jenkins:2.580.1-jdk21
# Then: log in, list jobs, open a credential (without revealing it), run a smoke Pipeline.

Observability and cost

You cannot run a platform on user complaints. Observability for Jenkins means answering five questions continuously: is it up, is it slow, is the queue healthy, is the fleet healthy, and is it getting more expensive?

The Prometheus plugin exposes metrics at /prometheus/ (the trailing slash is required), built on the Metrics plugin, which also provides health checks. Its behaviour is tuned through environment variables including PROMETHEUS_NAMESPACE, PROMETHEUS_ENDPOINT, COLLECTING_METRICS_PERIOD_IN_SECONDS (120 by default) and COLLECT_DISK_USAGE. The OpenTelemetry plugin exports builds and Pipelines as traces, plus metrics such as ci.pipeline.run.* and jenkins.queue.*, over OTLP. It requires Jenkins 2.516.3 or later, it is configured in JCasC under unclassified: openTelemetry:, and it can pass OTEL settings to builds as environment variables and store Pipeline logs in Elastic or Loki. Traces answer a question metrics cannot: which stage of which Pipeline got slower last week, and what did it wait for?

Built-in signals are worth keeping in mind too: the controller-to-agent ping thread with its four-minute default timeout (which is why an agent that vanishes takes a while to be noticed), the node monitors for disk, temporary space, swap, clock skew and response time, the System Log and Log Recorders, /manage/systemInfo, and the Support Core plugin for support bundles. The node monitor that takes an agent offline when free disk falls below the threshold (Disk space is too low. Only 0.9GB left on /var/lib/jenkins.) is a common hidden cause of mysterious capacity loss. Put it on a dashboard before it happens.

A practical dashboard has six panels: queue length and oldest queued item's wait; executor utilisation by label; build duration percentiles per tenant; controller heap, GC pause and CPU; agent count and pod start-up time; and failure rate split by infrastructure failures (lost agents, pods that never started) from test failures. The last split is the one that turns complaints into data: developers blame Jenkins for red builds, and the platform team needs to show which ones are actually its fault.

Where the money goes

Jenkins costs nothing to license (it is MIT-licensed), so the bill is infrastructure plus the time of the people who run it, and the people are usually the larger part. The levers that move infrastructure cost are few:

• Ephemeral agents that scale to zero overnight, using the Kubernetes or EC2 plugins, and spot or preemptible capacity for build fleets that tolerate retries.
• idleMinutes tuning: pods kept warm cost money, and pods started cold cost developer minutes. Measure both before choosing.
• Caching: mount Maven or Gradle caches or use a remote build cache, since dependency downloads are often the largest share of build time.
• Artifact retention: push artifacts to Nexus, Artifactory or object storage rather than keeping them in JENKINS_HOME, and use buildDiscarder so build records do not grow without bound.
• Log size: verbose logs cost disk, I/O and heap.

Attribute cost by tenant. A build's folder, label or namespace gives you a key for charging back agent time, and a number next to a team's name changes behaviour faster than any guideline. If you need commercial support and a real high-availability offering, that comes from CloudBees CI, a separate product and not open-source Jenkins. That purchase is a legitimate answer for a large estate, and it belongs in the options you present to leadership.

Incident playbooks

Incidents on a CI platform have a particular rhythm: many people are blocked at once, they all ask the same question, and the real cause is usually one of a few. The skill is ordering your checks so you rule out the cheap, common ones first. The pattern below works for most "Jenkins is broken" reports.

Playbook: the UI is crawling. Take a thread dump and heap statistics before restarting anything, since a restart destroys the evidence. Look for GC thrash (memory near the maximum, the collector running continuously), a blocked thread pool, or a single hot thread. The usual culprits are an API poller, a runaway Pipeline doing heavy Groovy, or a huge log. Stop the offender, raise the heap only as a short-term measure, and start the heap-dump flag if it was not already on.

Playbook: nothing starts, the queue is growing. Read the why reasons. ‘Jenkins’ doesn’t have label ‘linux’ or There are no nodes with the label ‘linux’ means no agent carries the label (a relabelled or disconnected pool). Waiting for next available executor with all executors busy is load. For a Kubernetes cloud, check the controller log for pod creation errors and run kubectl describe pod on any pod stuck in Pending, and confirm the container cap has not been reached. If an identity provider outage locks everyone out, use the break-glass administrator defined in your JCasC, which is why you keep exactly one local emergency account with a strong secret in the vault.

Playbook: a bad plugin update or a failed upgrade. Stop the controller, restore the previous image tag (or the plugins/ directory from the pre-change backup), and start it. This is fast only because you pinned versions and kept the backup. If the controller will not start because of JCasC, the log names the key (Invalid configuration elements); fix the YAML or the plugin list, and do not edit the running instance by hand.

Playbook: a credential may have leaked. Treat it as real until proven otherwise. Identify which credential IDs and scopes were reachable, rotate the secrets at their source, revoke and reissue API tokens, and review the build logs and Pipeline history around the time for use of Replay and the Script Console. If JENKINS_HOME or a backup left your control, rotate everything it held. Then fix the path: how did an untrusted Pipeline reach that credential?

Playbook: a security advisory with a fix in the next LTS patch. Decide on severity using the advisory's own description (is it reachable without authentication? does it need Job/Configure?), then run the upgrade procedure above on an accelerated schedule. Because the fix ships in both the weekly and the LTS patch on the same day, there is no reason to sit on a known hole.

Playbook: disk full. Symptoms are an agent taken offline with Disk space is too low, or builds failing oddly on the controller. Find the growth: old build records and artifacts under jobs/, workspaces on an agent, Docker images on a build host. Then fix the policy (buildDiscarder, cleanWs, artifact offload, image pruning), not just the instance.

Where Jenkins stops

Jenkins is the right answer more often than its critics admit, and the wrong answer in specific, nameable situations. Learn to name them, because a senior engineer who can say "not here, and this is why" earns more trust than one who defends the tool on principle.

The clearest boundary is CI versus CD. Jenkins is excellent at building, testing and producing an artifact. It is a poor place to hold the desired state of production, because that state then lives in a process with write access to everything and no reconciliation loop. The common and healthy pattern is for Jenkins to build and test, push an image, and open a pull request that bumps the image tag in a configuration repository, after which Argo CD or Flux reconciles the cluster to match. Jenkins then holds no cluster credentials at all, which removes the most dangerous secret from the most exposed system. Provisioning follows the same logic: run Terraform or Ansible through a container in a Pipeline if you must, but prefer a dedicated plan-and-apply workflow with its own approvals.

The other boundaries are operational. If your team spends more time keeping Jenkins alive than the pipelines are worth, a hosted runner is cheaper. If you need real HA, you are looking at CloudBees CI or at another system altogether. If your builds are a thin layer over a container, GitLab CI or GitHub Actions, where the pipeline definition lives next to the repository's permissions, remove a whole class of administration. Tekton is the usual Kubernetes-native comparison. None of those is free of its own operational burden; the point of the comparison is to be honest about which burden you are choosing.

A sound migration away from Jenkins is incremental: move one team, one pipeline type at a time, run both for a cycle, and use the shared library as the map of what you have to replicate. The library is also where the lock-in lives, so keep it small and keep logic in scripts in the repositories rather than in Groovy.

Running Jenkins as a platform

Everything so far has been technique. Running Jenkins as a platform for other teams is mostly a matter of product thinking: your users are engineers, your product is "a build that works", and your job is to make the easy path the safe path.

Offer a paved road. A shared library step such as standardPipeline(...) that gives a team build, test, scan, publish and notify in a few lines is worth more than any guideline document. Make it versioned (tags), documented and backward compatible, and let teams leave it if they must, since the escape hatch keeps the road honest. The library is also where you enforce policy invisibly: the scan step that cannot be skipped, the timeout that is always set, the buildDiscarder that is always applied.

// A paved road: timeout, retention, scan and notification are not optional
def call(Map cfg = [:]) {
  pipeline {
    agent { label cfg.get('label', 'linux') }
    options {
      timeout(time: cfg.get('timeoutMinutes', 30), unit: 'MINUTES')
      buildDiscarder(logRotator(numToKeepStr: '20'))
      disableConcurrentBuilds(abortPrevious: true)
    }
    stages {
      stage('Build') { steps { sh cfg.get('buildCmd', 'make build') } }
      stage('Test')  { steps { sh cfg.get('testCmd', 'make test') } }
      stage('Scan')  { steps { sh 'trivy fs --exit-code 1 --severity HIGH,CRITICAL .' } }
    }
    post { failure { echo 'Notify the owning team here (Slack or Email Extension).' } }
  }
}

Write the contract. Users should know what you promise and what you do not: supported Java and tool versions, the maintenance window, the retention policy, the support channel and its response times, what happens to jobs that have not run in twelve months, and how a team onboards. The onboarding should be a pull request that adds a folder, a role binding and a credential namespace, and never a ticket that someone handles by hand in the UI.

Govern by code review. Everything that changes the platform goes through pull requests to the repository holding plugins.txt, JCasC and the library: a new plugin needs a justification and an owner, a new global credential needs a reason it cannot be folder-scoped, and a new permission grant names who asked. This gives you an audit trail and a place to say no politely. Define a plugin policy explicitly: a short approved list, a request process for additions, and a rule that unmaintained plugins are removed on a schedule.

Measure the platform as a product. Track build queue time, build success rate separated by infrastructure versus test failures, time to onboard a team, the age of your controller and plugins relative to current, and the number of open advisories affecting you. Report these to leadership monthly. They make the case for headcount, for splitting controllers and for the day you propose migrating off.

Deprecate deliberately. Freestyle jobs configured through the UI, old Blue Ocean dependencies, scripted pipelines that pre-date the library, pinned agents: each needs a deprecation notice, a date and a migration path, or it lives forever. The Declarative Linter (java -jar jenkins-cli.jar declarative-linter < Jenkinsfile, or a POST to /pipeline-model-converter/validate) can gate new Jenkinsfiles in pull requests, which stops the problem getting worse while you fix the backlog.

Keep a second pair of hands. The bus factor of a Jenkins platform is notoriously one. Document the restore, rotate on-call, and make sure more than one person can read the heap dump and the JCasC.

The review checklist

Use this for a design review, a pull request to the platform repository, or an audit of an inherited controller. Each item is a question with an observable answer.

Architecture and scale

• Does the built-in node have zero executors?
• Is JENKINS_HOME on persistent storage that reattaches quickly, with a liveness probe on /login?
• Is the heap sized deliberately, with a heap dump on out-of-memory enabled?
• Does every job set buildDiscarder and a timeout?
• Are external systems notified by webhook instead of polling, and do API calls use ?tree=?
• Is there a plan, with a trigger number, for splitting the controller?

Pipelines

• Is heavy work done in sh on agents and not in Groovy on the controller?
• Is durabilityHint chosen deliberately, and disableResume() considered for ephemeral agents?
• Are shared libraries pinned by tag, and is the trusted library limited to the platform team?

Security

• Is the security realm your identity provider, and is authorisation matrix-based or role-based?
• Who holds Overall/Administer, and is that list short and reviewed?
• Who holds Job/Configure, Agent/Configure and Credentials/*, and can each give a reason?
• Are secrets single-quoted in every sh, scoped to folders, and sourced from an external store where possible?
• Is CSRF protection on, API tokens expiring, and user content served from a Resource Root URL?
• Do pull-request builds from forks run without production credentials?

Supply chain and config

• Is the plugin list pinned, minimal and changed only by pull request? Is Blue Ocean gone?
• Is configuration in JCasC with no UI drift, and does the image build from code?

Operations

• Has a restore been tested in the last quarter, and does the backup include and encrypt secrets/?
• Are Java versions on all agents supported, and has the next baseline's upgrade guide been read?
• Is there a named owner watching security advisories, and a written patch target?
• Are metrics, traces and an alert on queue wait and disk in place?

The complete picture

Put the series together and the model is this. Jenkins is a single stateful controller that schedules work, runs your Pipeline Groovy, and holds your credentials. Its real limits are the controller's heap, CPU and disk, not the number of agents, and its real risk is that anyone who can edit a Pipeline can run code with whatever that Pipeline can reach. You deal with the limits by keeping Groovy thin, using ephemeral agents, retaining less, and splitting controllers along ownership lines. You deal with the risk through least-privilege authorisation, folder-scoped and external secrets, trusted code kept minimal, pinned plugins, and isolation that matches how much you trust the tenant. You deal with change by describing the controller in code and treating upgrades as a rehearsed procedure. And you deal with the boundaries of the tool by sending delivery to GitOps and by being honest about when a hosted CI is a better use of your team.

The version-specific facts that change your runbooks today are these: LTS 2.580.1 needs Java 21 or 25 (Java 17 ended at 2.555.1); nine plugins are no longer bundled, so offline instances must pre-stage them; JCasC must not contain the old crumbIssuer block; the package signing key and the RPM repository moved in 2.541.1; Blue Ocean is deprecated and should be replaced by Pipeline Graph View; and Windows Server 2019 container images are gone.

Where the series leaves you

You started this series able to click "Build Now". You can now reason about a Jenkins controller as a system: what it is made of, which part fails first under load, what each permission truly grants, how a secret travels from a store to a shell and where it can leak, how to rebuild the whole thing from a repository, and how to upgrade it without fear. You can also argue for the tool or against it with specifics.

The next steps are practical. Take one controller you own and run the review checklist honestly. Build the image from code and rehearse a restore. Move one team's deploys to a GitOps pull request with Argo CD or Flux, and read the Kubernetes guide and the Helm guide for the cluster side of your agent fleet. For the container hygiene that underpins every pipeline, return to the Docker guide. The skill that separates a platform owner from a pipeline author is not knowing more plugins. It is being able to say what happens when this fails, and having already prepared the answer.

Sources

• Jenkins documentation
• LTS changelog
• Changelog 2.555.1
• Changelog 2.541.1
• Changelog 2.568.1
• Upgrade guide
• Upgrade guide 2.580
• Upgrade guide 2.555
• Java support policy
• Installing on Docker
• Installing on Kubernetes
• Installing on Linux
• Pipeline syntax
• Shared libraries
• Pipeline best practices
• Scaling Pipelines
• CPS method mismatches
• Configuration as Code
• Managing nodes
• Script approval
• System properties
• Jenkins CLI
• Remote access API
• Using credentials
• Managing security
• CSRF protection
• Controller isolation
• Permissions
• Architecting for scale
• Monitoring Jenkins
• Reverse proxy troubleshooting
• Blue Ocean (deprecated)
• Security advisories
• Advisory 2026-09-02
• Configuration as Code plugin
• Plugin Installation Manager Tool
• Jenkins Docker image
• Kubernetes plugin
• Prometheus plugin
• OpenTelemetry plugin

Part three of three. A senior Jenkins interview rarely asks you to write a Jenkinsfile. It hands you a company, a constraint and a failure, and watches how you reason. This file is organised around three kinds of prompt: system-design questions, incident-response questions, and the harder skill of arguing a trade-off to people who do not care about Jenkins. About fifty minutes of reading, and the most value comes from saying the answers aloud.

The thirty-second framing

Keep one paragraph ready that frames everything else. Jenkins is a single stateful controller, a Java process with a file-based home directory, that schedules builds, runs Pipeline Groovy, serves the UI and API, and holds every credential. Builds run on agents. The controller is not highly available, so you design for fast, rehearsed recovery and for a small blast radius. Anyone who can edit a Pipeline can run code on agents with the credentials in scope, so security is about who can write programs and what those programs can reach.

If your first two minutes contain that, the rest of the interview is you filling in detail. Interviewers at this level are checking that you think in failure modes and trade-offs, and not in plugin names.

Facts to have ready

Some numbers and names come up often enough that fumbling them costs you credibility. This guide targets LTS 2.580.1, released on 2026-09-30, on Java 21 or 25. Java 17 support ended with 2.555.1. The LTS line gets a new baseline roughly every twelve weeks and .1, .2, .3 patches about every four weeks. "Master" is now controller, the controller's own node is the built-in node (set its executors to zero), and "slave" is agent.

• The inbound TCP agent port is disabled by default (-1) and WebSocket agents need no extra port. Agent-to-controller security has been on since 2.326.
• Overall/Administer is everything. Overall/Manage is partial administration without code execution, on by default since 2.555.1. Job/Configure on a Pipeline is code execution on any agent the user can label.
• Pipeline durability hints: MAX_SURVIVABILITY (default), SURVIVABLE_NONATOMIC, PERFORMANCE_OPTIMIZED.
• Blue Ocean has been deprecated since July 2026, with no further security fixes. Use Pipeline Graph View.
• 2.580.1 unbundled nine detached plugins, which matters for offline installs. 2.555.1 broke JCasC files that still contain crumbIssuer.
• The JCasC source is CASC_JENKINS_CONFIG; multiple files must be supplementary and cannot override each other.
• Prometheus metrics live at /prometheus/ with the trailing slash.

Part one: system-design prompts

Each of these has a model answer below. Notice the shape: clarify requirements, state the constraint, propose a design, name what you are giving up, and say how you would know it is working.

Design a Jenkins platform for thirty teams

Clarify first. How many builds a day, and how bursty? Are all teams equally trusted, and is any regulated? Which clouds or regions, and is there a data-residency requirement? What do builds need to reach (registries, clusters, production)? Who will operate it? Saying that you would ask these before drawing anything is part of the answer.

A strong answer. Start from the constraint that one controller has no standby and weak multi-tenancy. I would not put thirty teams on one controller. I would group teams by trust and ownership (for instance by business unit or by regulatory zone) into perhaps four to six controllers, each created from the same Helm chart, the same pinned plugins.txt and the same JCasC base layer, differing only in a tenant overlay. Each controller has zero built-in executors and builds on ephemeral Kubernetes pods in a per-controller namespace with its own service account and resource quota. Identity comes from the company's provider by SAML or OIDC; authorisation is role-based, granted per folder; credentials are folder-scoped, with an external store (Vault or a cloud secret manager) as the source of truth and short-lived cloud credentials through workload identity where possible. A paved-road shared library, pinned by tag, carries policy: timeouts, retention, scanning. Deployment is not done by Jenkins: Jenkins builds and pushes an image and opens a pull request, and Argo CD or Flux deploys, so Jenkins holds no cluster credentials.

The trade-offs to name. More controllers means more to patch, and I answer that with automation, because they are cattle. A paved road constrains creativity, so I keep an escape hatch. Ephemeral agents cost start-up time, and I measure it and tune idleMinutes. A regional controller may be required by data-residency rules for source code and artifacts, which is a real consideration for employers in the Gulf and Egypt, and is easier to satisfy per region than with one global controller.

What makes it weak. Naming plugins without reasons. Proposing a single giant controller "for simplicity". Not mentioning recovery at all. Putting production deploy credentials in Jenkins without comment.

The controller is slow. How do you find out why?

Do not guess. Separate the three possibilities: the controller is starved, the agents are starved, or something external is slow. Start with the queue. The why text on queued items and the ratio of busy to total executors tell you whether work is waiting for capacity or not being scheduled at all. If the queue is empty and the UI is slow, the controller itself is the problem, so look at heap usage and garbage-collection pauses, CPU, and disk wait. Take a thread dump before any restart. Then ask what is doing work on the controller: Pipeline Groovy (CPS), an API poller, branch indexing for a large organisation, or a giant console log. The remedies are specific: move Groovy to sh on agents, replace polling with webhooks and ?tree= filters, cap build history and log size, and only then raise the heap or split the controller.

A memorable line is that the number of executors does not measure controller load, because Pipeline code runs on the controller regardless of how many agents exist.

Design secret management for Jenkins

The best secret is one that is not in Jenkins. Prefer short-lived credentials obtained at run time by workload identity or OIDC. Next best is an external store read at use (Vault, AWS Secrets Manager, Azure Key Vault, GCP Secret Manager, the Kubernetes Credentials Provider). Next, JCasC ${VAR} interpolation from environment or files so the YAML in Git never has a value. Last, Jenkins-stored credentials, folder-scoped and rotated. In every case, single-quote shell strings that use them, because Groovy interpolation puts the value into a process argument on the controller, and masking is best-effort and not a boundary. I would also say what I do about JENKINS_HOME: the secrets/ directory decrypts everything, so backups are encrypted and access-controlled, and a leak means rotating every credential.

Design the migration from one giant controller to several

Inventory first: jobs by owner, last run, credentials in use, plugins in use. Delete what nobody runs. Then build the target from code (image, plugins, JCasC, Job DSL or organisation folders) and migrate tenant by tenant, starting with a willing, low-risk team. Run old and new in parallel for a release cycle. Move credentials by re-provisioning them from the external store rather than copying secrets/. Keep build history only where someone truly needs it, since history is what makes JENKINS_HOME large. Decide the stop condition up front, and set a date for switching off the old controller and stick to it.

How would you make Jenkins highly available?

The honest answer has two parts. Open-source Jenkins has no active-active mode, so "HA" means fast recovery: persistent storage that reattaches, a liveness probe, agents that reconnect by themselves, and a rehearsed restart, with running Pipelines surviving through their on-disk state. For real HA or vendor support, CloudBees CI is a separate commercial product. Then reduce the need for HA by reducing blast radius: several smaller controllers mean that one failure stops one business unit, not the company. Quoting recovery time and recovery point numbers, and saying when you last tested them, is what elevates this answer.

Part two: incident-response prompts

Incident questions test order of operations. The right order is almost always: scope, stabilise, diagnose with evidence, fix, then prevent. Say why you take each step, because an interviewer is listening for the reasoning and a list of commands alone does not show it.

"Nobody can build. The queue is 400 items."

First establish the scope: everyone or one team, and since when. Then read the why on a few queued items. If they say There are no nodes with the label ‘linux’, a pool was relabelled or disconnected. If they say Waiting for next available executor, check utilisation: busy equals total means load, and busy below total with a queue means something prevents scheduling (a cloud cap reached, pods failing to start). For a Kubernetes cloud, look at pods in Pending and kubectl describe pod for the scheduler's reason, at ImagePullBackOff, and at the controller log for pod-creation errors. Then ask what changed: a plugin update, a JCasC reload, a cluster change or an expired credential for the cloud. Mitigation comes before root cause: if the cap is the limit, raise it carefully or add capacity, and communicate status to users while you work.

"After the upgrade the controller will not start."

Read the log, not the UI. The common shapes are a Java version error (Running with Java 17 ... which is older than the minimum required version (Java 21)), which means fix the JVM with JENKINS_JAVA_CMD or update-alternatives; a JCasC failure, where ConfiguratorException or a leftover deprecated crumbIssuer names the offending key; or, after 2.580.1 on an offline instance, unbundled plugins that are missing from $JENKINS_HOME/plugins. The recovery path is to roll back to the previous image tag or restore the pre-change backup, since you took one and pinned versions. Then reproduce on staging, fix, and retry. The prevention is an upgrade procedure that included staging, plugin pre-updates and a Java audit of agents.

"We think a credential leaked through a build."

Treat it as a real incident. Identify which credential IDs were in scope for the Pipeline in question and which agents it ran on. Rotate at the source of truth (the cloud, the registry, the vault), not in Jenkins, because changing it in Jenkins does not invalidate the old value. Revoke API tokens and sessions for any implicated user. Review who had Job/Configure, Run/Replay or Script Console access, and check the logs around the time. Then establish the path: was the secret echoed in a double-quoted sh, did a fork's pull-request build reach a production credential, or did someone approve a dangerous script signature? Fix the path with a control (folder-scoped credentials, untrusted-build isolation, a library that forbids the pattern), not with a reminder email.

"An old Pipeline is taking the controller down."

Find it by evidence. A thread dump shows the hot thread and the Pipeline name; the heap dump shows what is retained. The fix has two parts: stop the immediate harm (abort the build, disable the job) and fix the class of problem (move heavy Groovy to an agent, cap history, add a timeout). Mention CPS specifically: serialisation and Groovy cost live on the controller, so a loop over thousands of items in Groovy is an infrastructure problem even though it looks like application code.

"We got a security advisory for the core."

Read it for reachability: does exploitation need authentication, which permission, and does a plugin or configuration of ours make us exposed? Core fixes ship in the weekly release and the next LTS patch on the same day, so the path is the normal upgrade procedure on an accelerated timeline, with staging and the Java and plugin checks still performed. Apply a documented patch target, for example staging within days and production within a week, which is a policy you write down and not a Jenkins number. Keep interim mitigations in mind if you cannot patch immediately, such as restricting network access to the controller.

Part three: arguing a trade-off to leadership

Leadership does not want to hear about CPS. They want cost, risk and time, in words they can act on. Your job is to translate the technical trade-off into those terms and end with a recommendation.

"Why do we need four controllers? One is cheaper."

Weak answer: "Jenkins does not scale and it is best practice." Strong answer: one controller is cheaper on the infrastructure invoice and more expensive in risk and in people. A single controller is one failure domain, so an outage or a bad plugin upgrade stops every team at once, and each upgrade has to satisfy thirty teams' timelines. It is also one trust domain: any of those teams can reach any other team's credentials, which fails an audit for regulated workloads. Four controllers, created from the same code, cost some extra infrastructure and need automation to patch, but they limit an outage to one business unit, let teams upgrade on their own schedule, and make a data-residency promise possible. Present the numbers: expected cost of an outage per hour times the probability, against the additional fleet cost.

"Should we move everything to GitHub Actions or GitLab CI?"

Give a real answer, not a defence of your tool. The case for moving: less to operate, pipeline definitions next to repository permissions, no controller to patch, and faster onboarding. The case for staying: a large estate of working pipelines and shared libraries, builds that need on-premises or special hardware, data that must stay in your network, and plugins nothing else matches. A strong recommendation is usually incremental: new services start on the hosted system, and existing pipelines migrate as they are touched, with a date to retire the old controller. Show that you have counted the migration cost honestly, including the shared library you would have to replicate. The GitLab CI guide is a good basis for the comparison.

"Why are we spending engineer time on Jenkins upgrades?"

Because the alternative is a growing, hidden liability. Each deferred upgrade widens the gap: Java 17 to 21 is a required step, plugins become unmaintained, and security advisories accumulate on a server that holds every deploy credential. Show the trend: age of the controller, number of open advisories, the number of plugins past their maintainers. Then show the cost of doing it routinely (a rehearsed half-day a quarter) against an emergency upgrade under pressure. The right framing is risk reduction with a known cost.

"Can we give developers admin so they stop filing tickets?"

No, and the answer should be constructive. Explain that administrator access on the controller means arbitrary code execution and access to every credential, so it is a production-level privilege. Then fix the actual pain: self-service through a pull request for new jobs and folders, folder-level ownership so teams manage their own Pipelines, a clear plugin request process with a response time, and the Overall/Manage permission for the operations that do not need code execution. The goal is to remove the reason people ask.

"Is it worth paying for CloudBees CI?"

It is a legitimate option, and a good answer does not treat it as heresy. The commercial product adds vendor support and a real high-availability and multi-controller management story that open-source Jenkins does not have. Whether it is worth it depends on how much an outage costs, how many controllers you run, and how much platform-engineering time you would otherwise spend building and operating that yourself. Put those three numbers on a page and let the decision-makers choose.

What a strong answer covers that a weak one misses

Common interview questions

What is the built-in node, and why set its executors to zero? It is the node object for the controller itself. A build on it has the same file-system access as the Jenkins process, so any Pipeline author can read secrets/ and other teams' configuration. Build only on agents.

Why does Pipeline Groovy matter for scaling? It runs on the controller under CPS, which makes it slower than ordinary Groovy and serialises state to disk. Heavy Groovy on many concurrent builds starves the controller. Keep Groovy as glue and do real work in sh on agents.

What is the difference between a trusted and an untrusted shared library? A global library can be trusted and runs outside the sandbox with controller access, so only the platform team should change it. A folder-level library is always sandboxed. Pin versions by tag.

What does PERFORMANCE_OPTIMIZED durability trade? Much less disk I/O in exchange for possibly losing the state of running Pipelines on a dirty shutdown. Good for short CI runs, wrong for a long release with approvals.

How do you keep JCasC and the UI from fighting? Treat JCasC as the source of truth, remove the ability to edit what it owns, and put the warning in the system message, because reloads and restarts overwrite UI edits.

A JCasC reload fails after an upgrade. What do you check? The message. Invalid configuration elements or No configurator for the following root elements usually means a missing plugin or a misspelled key. After 2.555.1, a leftover crumbIssuer cancels startup unless deprecated: warn is set.

Why are pull-request builds from forks dangerous? They run contributor-controlled Pipeline code on your agents. Do not use the contributor's changes to the Jenkinsfile, keep production credentials out of reach, and use a separate folder, label and credential set.

How do you provide isolation between tenants? Folders, folder credentials and roles for soft isolation; Authorize Project and per-tenant namespaces for firmer; separate controllers for hard. Say which you would choose for which trust level.

Why should Jenkins not deploy to production directly? It then holds the most powerful credentials in the most exposed system, with no reconciliation loop. Build and push in Jenkins, and let Argo CD or Flux reconcile from a configuration repository.

What is the first thing you do during an upgrade? Read the upgrade guide and audit Java on the controller and agents. Since 2.555.1 they need Java 21 or 25, and the Versions Node Monitors plugin shows every agent.

How do you know your backups work? You restore them on a schedule, into a clean environment, and run a smoke Pipeline and a credential check. Record the time it takes as your recovery time.

What would make you leave Jenkins? Operating it costing more than the value it delivers, a need for HA or strict tenancy without running many controllers, or an estate that fits a hosted CI better. The move is incremental, never a big bang.

Final self-test

Answer each aloud in under a minute, then check against this guide.

1. Why is the controller, and not the agent count, the scaling limit?
2. Name three things that live in JENKINS_HOME and say which must be encrypted in a backup.
3. What does Job/Configure on a Pipeline effectively grant?
4. What three durability hints exist and what does the fastest one risk?
5. Why are two JCasC files unable to override each other, and how does that shape your layout?
6. What changed in 2.580.1 for an offline instance?
7. Why is masking not a security boundary? What should you do instead?
8. Give the soft, firmer and hard isolation options for tenants.
9. Where does Jenkins stop and GitOps begin?
10. You are paged: the queue is growing and executors are idle. What are the first three things you check?

If you can answer eight of ten without looking, you are ready. For the rest, reread the matching section of the senior detailed guide and its checklist.

Part three of three. The mistakes at this level are not syntax errors. They are decisions that work for months and then fail when the estate grows, when an upgrade lands, or when someone with Job/Configure does something you did not imagine. This file collects the production traps that only appear at scale, the habits that keep a platform healthy, and review checklists you can paste into pull request templates.

Production traps that only appear at scale

Each trap below has a symptom, a cause and a fix. Read them as a catalogue of "what I would have wanted to be told before the third team onboarded".

Polling that looks cheap and is not

Symptom: the UI is sluggish during office hours and calm at night, with no matching rise in builds. Cause: dashboards, bots and scripts calling the REST API on a timer, often with ?depth=3 or no filter, and Multibranch jobs scanning hundreds of repositories on a schedule. Each call walks objects held in controller memory. Fix: replace polling with webhooks and event notifications, add ?tree= to every call you own (for example ?tree=jobs[name,color,lastBuild[number,result]]), and make the scheduled scan a slow safety net while webhooks drive the real triggering. Audit who polls by reading the access log for the heaviest client addresses.

Groovy on the controller

Symptom: controller CPU is high while executors are mostly idle, or builds slow down together. Cause: Pipeline Groovy runs under CPS on the controller, so parsing large JSON with JsonSlurper, looping over thousands of files, or making HTTP calls from Pipeline code costs shared CPU. Fix: treat Groovy as glue. Do the heavy work in sh with jq, a script or a proper tool, and pass one small value back with returnStdout: true. Combine chains of tiny sh steps into one script. Put a review comment on every @NonCPS method that does real work.

Unbounded history

Symptom: heap grows week after week; restarts get slower; JENKINS_HOME fills a disk. Cause: no buildDiscarder, archived artifacts kept forever, huge logs. Fix: make retention a platform default through the shared library or JCasC, push artifacts to Nexus, Artifactory or object storage, and set a global cap. Retention is capacity planning: a number of builds times average size times jobs, written down.

The plugin that nobody owns

Symptom: a quarterly upgrade takes two weeks because something in the list breaks. Cause: dozens of plugins installed for one-off needs, with no owner and no justification. Fix: a plugin register with a named owner and a reason per plugin, reviewed twice a year, plus a rule that unused and unmaintained plugins are removed. The fastest upgrade is the one with fewer moving parts.

Trusting masking

Symptom: a secret appears in a console log despite withCredentials. Cause: masking is best-effort, and a value that was base64-encoded, split or transformed is not masked. Another cause is double-quoted Groovy strings, which produce the warning A secret was passed to "sh" using Groovy String interpolation, which is insecure. Fix: single-quote every shell string that uses a credential, never print credentials for debugging, and scan logs for secrets as part of the platform's routine.

@Library('name@main')

Symptom: every pipeline in the company breaks at once after a merge to the shared library. Cause: an unpinned library means every build picks up the latest commit of that branch. Fix: pin by tag (@Library('platform-lib@v3.4.0') _), release with semantic versions, keep the previous two versions working, and publish a changelog. Load main only in a canary folder that tests the library before release.

The agent that is nobody's fault

Symptom: a build passes on one run and fails on the next with missing tools or stale files. Cause: long-lived agents with hand-installed tools, several executors sharing one workspace area and a Docker socket, and state left behind by the previous build. Fix: ephemeral agents with one executor each, toolchains declared in the pod spec or agent image, and cleanWs() in post { cleanup { } } where agents persist.

Cloud cap, pods pending, builds waiting

Symptom: the queue grows and the reason says the build is waiting for an executor, although the cluster has room. Cause: the Kubernetes cloud's container cap has been reached, the pod requests are larger than any node, or pod creation is failing quietly. Fix: check the cloud's logs and kubectl describe pod, size requests to real usage, raise the cap deliberately, and alert on the queue's oldest wait rather than on its length.

Upgrade surprises from unpinned inputs

Symptom: the controller works on Tuesday and misbehaves on Wednesday with no deliberate change. Cause: an image tag like lts or latest, plugin versions set to latest, or a package repository moving. Fix: pin the controller tag (jenkins/jenkins:2.580.1-jdk21), pin every plugin version, and take a change through staging. jenkins-plugin-cli --latest false takes minimum versions and avoids surprise upgrades.

Reverse proxy half-configured

Symptom: the banner It appears that your reverse proxy set up is broken, or WebSocket agents disconnecting behind a proxy. Cause: missing X-Forwarded-Host, X-Forwarded-Port or X-Forwarded-Proto headers, a Jenkins URL in System configuration that differs from the browser URL, a context path mismatch with --prefix, or no Upgrade and Connection headers for WebSocket. Fix: set the headers, make the Jenkins URL match, and test with curl -iL -e http://proxy/jenkins/manage http://proxy/jenkins/administrativeMonitor/hudson.diagnosis.ReverseProxySetupMonitor/test.

Certificates and proxies on the plugin path

Symptom: plugin installs fail with PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException. Cause: a TLS-inspecting corporate proxy or a private certificate authority that the JVM does not trust. Fix: import the authority into the JVM truststore with keytool -importcert -cacerts ... (or pass -Djavax.net.ssl.trustStore), and set the HTTP proxy under Manage Jenkins, Plugins, Advanced settings. Do not turn verification off to make the error go away.

Platform-ownership habits

The habits below separate a platform that stays healthy from one that decays into an incident generator. They are ranked roughly by leverage, the first being the ones that pay back fastest.

Pin, then bump on a schedule. Pinned versions are not a way to avoid change; they are a way to make change deliberate. Put a recurring job in the platform team's calendar (a scheduled Pipeline that runs jenkins-plugin-cli -f plugins.txt --available-updates --output txt and opens a pull request is enough) so that the pinned list never goes stale.

Keep the release train visible. LTS has a baseline every twelve weeks and patches about every four. Know which baseline you are on, which is next, and read its upgrade guide when it is announced rather than when it is urgent. The recent ones each needed something: Java 21 for 2.555.1, a new signing key and repository for 2.541.1, unbundled plugins for 2.580.1.

Write the contract. Publish what you support (Java and tool versions, plugin policy), when you patch, how long builds are retained, how to get help, and what is out of scope. Every ambiguity in a contract turns into a ticket.

Own the advisories. One named person reads the security advisory feed (https://www.jenkins.io/security/advisories/) and the plugin advisories that affect your list, and posts a triage note each time. The habit is what matters: advisories arrive whether or not anyone is watching.

Delete aggressively. Jobs that have not run in a year, credentials that no job references, agents nobody uses, plugins nobody can explain. Every item you keep is a patch surface, a migration cost and an attack path.

Make "why" visible in configuration. A comment beside a system property, a JCasC key or a plugin pin saying why it is there is worth a great deal two years later. Tie it to a ticket where you can.

Practice cards

Short drills you can run on a test controller in fifteen minutes each. Do them before you need them.

Card 1: the noisy neighbour. Write two Pipelines: one that loops 100,000 times in Groovy and one that does the same work inside sh. Run ten of the first at once and watch controller CPU and the UI's response time. Expected: the UI degrades, although the executors are barely busy. Lesson: executors do not measure controller load.

Card 2: the pinned library. Create a shared library with two tags. Load the first in a Pipeline by tag, merge a breaking change to the branch, and confirm the tagged Pipeline is unaffected while one using @main breaks. Lesson: pin by tag.

Card 3: the leaking secret. Print a credential with double quotes and read the warning. Then print its base64 and see the masking fail. Finally use single quotes and --password-stdin. Lesson: masking is not a boundary.

Card 4: the restore. Back up a controller (including secrets/, excluding war/, cache/, tools/ and workspaces), restore it to a fresh container, and open a credential. If it does not decrypt, you missed secrets/.

Card 5: the drifting UI. Put the security realm in JCasC, change it in the UI, reload, and watch the change vanish. Write the sentence you would put in systemMessage.

Card 6: the offline upgrade. Start a controller with no network access on the new version without the unbundled .hpi files and read what fails. Add the files and retry.

Card 7: the permission audit. Create a role with Job/Configure on one folder and a Pipeline that tries to use a credential from another folder and another label. Record which attempts succeed, then tighten the configuration until none do.

Operational recipes

Copy-paste material for the platform team's runbook. Adjust the host and credentials to your environment.

Check the version and the Java in one go

curl -sI https://ci.example.com/login | grep -i x-jenkins
java -jar jenkins-cli.jar -s https://ci.example.com/ -auth @token-file version
# Java of each node, from the Versions Node Monitors plugin, is on /manage/computer/

Graceful drain and restart

java -jar jenkins-cli.jar -s https://ci.example.com/ -auth @token-file quiet-down
# wait until busy executors reach zero
curl -s -u "$JENKINS_USER_ID:$JENKINS_API_TOKEN" \
  "https://ci.example.com/computer/api/json?tree=busyExecutors"
java -jar jenkins-cli.jar -s https://ci.example.com/ -auth @token-file safe-restart
# if you change your mind before restarting:
java -jar jenkins-cli.jar -s https://ci.example.com/ -auth @token-file cancel-quiet-down

Validate a Jenkinsfile before it merges

curl -s -X POST -u "$JENKINS_USER_ID:$JENKINS_API_TOKEN" \
  -F "jenkinsfile=<Jenkinsfile" \
  https://ci.example.com/pipeline-model-converter/validate

A successful response says the Jenkinsfile validated successfully. Put this behind a pull-request check, so a syntax error such as Expected a step @ line 12, column 9 is caught before it reaches the controller.

Trigger a parameterised build from automation

curl -s -i -X POST -u "$JENKINS_USER_ID:$JENKINS_API_TOKEN" \
  "https://ci.example.com/job/team-a/job/deploy/buildWithParameters" \
  --data BRANCH=main
# 201 with a Location header pointing at /queue/item/N/ ; poll that URL's api/json for executable.url

API-token requests are exempt from CSRF crumbs. If you see No valid crumb was included in the request, you are using a password or a session, and you should switch to a token.

Reload JCasC without a restart

curl -s -X POST \
  "https://ci.example.com/configuration-as-code/reload?casc-reload-token=$CASC_RELOAD_TOKEN"

Set CASC_RELOAD_TOKEN on the controller first. Check the System Log afterwards for configurator errors, because a partial reload is worse than a failed one.

Find what is growing on disk

sudo du -sh /var/lib/jenkins/jobs/* | sort -h | tail -20      # which jobs hold the history
sudo du -sh /var/lib/jenkins/{plugins,secrets,caches,war} 2>/dev/null
df -h /var/lib/jenkins

A plugin update job

jenkins-plugin-cli -f plugins.txt --available-updates --output txt
# Turn the output into a pull request. Never apply it straight to production.

Review checklists

For a pull request to a Jenkinsfile or a shared library

• Does every pipeline set a timeout and a buildDiscarder?
• Does it ask for an agent by label or pod spec, never by name, and never run on the built-in node?
• Is heavy work done in sh, not in Groovy on the controller? Are there new @NonCPS methods, and do they call steps?
• Are credentials used through withCredentials or credentials(), with single-quoted shell strings?
• Are approvals outside an agent and wrapped in a timeout, so they do not hold an executor?
• Is the library loaded by tag, and is a new step backward compatible with earlier tags?
• Does it validate with the Declarative Linter?
• Could a pull request from a fork reach anything it should not?

For a pull request to the platform repository

• Does a plugin addition name its owner, its reason and its alternatives? Is the version pinned?
• Does a JCasC change avoid keys that overlap another file (supplementary files cannot override each other)?
• Is a new permission grant scoped to a folder and a role, with a named requester?
• Is a new global credential justified, and could it be folder-scoped or replaced by short-lived identity?
• Does the change build into an image that passes the smoke test on staging?
• Is the rollback the previous image tag and the same volume?

For a design document

• What is the blast radius of the worst failure, and how long is recovery?
• Which components are single points of failure, and what is the honest answer for each?
• What does the trust model say about who can run code where, and with which credentials?
• How does this scale to ten times the load, and which component breaks first?
• What does it cost per month, and how is it attributed to tenants?
• How will an upgrade work, including Java and plugins, and what is one-way?
• What are we explicitly not building, and when would we choose another tool?
• Where does Jenkins end and deployment begin? Is there a GitOps hand-off?

The checklist to run before calling the platform ready

• Built-in node at zero executors, agents ephemeral with one executor each.
• Security realm from the company identity provider, matrix or role-based authorisation, no "Anyone can do anything".
• CSRF protection on, API tokens set to expire, user content on a Resource Root URL, markup formatter safe.
• Secrets external or folder-scoped, never double-quoted in sh, backups encrypted and containing secrets/.
• Plugins pinned and minimal, Blue Ocean removed in favour of Pipeline Graph View.
• Image, plugins.txt and JCasC in Git, and no UI drift.
• Prometheus or OpenTelemetry on, with alerts on queue wait, disk and controller health.
• Restore rehearsed this quarter, upgrade guide read for the next baseline, advisories owned.
• The contract published, onboarding through a pull request, and a second person who can run the recovery.

What to carry forward

Three ideas cover most of what goes wrong at this level. The controller is one process with one copy of state, so the work is recovery and blast radius and not availability. A Pipeline is a program, so security is about who may write programs and what those programs can reach. And a platform is a product, so its quality shows in how easy the safe path is. When you next review a Jenkins change, ask those three questions first. The Kubernetes guide and Argo CD guide cover the neighbouring systems that let you keep Jenkins small.