The Complete GitLab CI/CD Guide

This is part one of three. It covers everything you need to build real pipelines with GitLab CI/CD, not a teaser. By the end you can write a .gitlab-ci.yml from a blank file, run tests on every push, pass build output from one job to the next, speed up slow jobs with a cache, keep a password out of your repository, put a manual approval button in front of a deploy, and read a red pipeline well enough to fix it in minutes. Mid-level and Senior take the same topics further; nothing you learn here is thrown away.

Each section ends with a Try it task. Do them as you go. They take a few minutes each, and pipelines only start to make sense once you have watched your own job fail, read its log, and fix it.

One note on versions before we start. This guide was checked against GitLab 19.4, released on 17 September 2026, with GitLab Runner 19.4.1. GitLab ships a minor release every month and a major release every May, so a few things in older tutorials are now wrong. Wherever that matters, this guide says so and shows the current form.

What GitLab CI/CD is, and the problem it solves

CI/CD stands for continuous integration and continuous delivery (or deployment). The words sound heavier than the idea. Continuous integration means that every time someone pushes code, a machine automatically builds it and runs the tests, so that a broken change is noticed in minutes rather than at the end of the month. Continuous delivery means that the steps between "the tests passed" and "it is running for users" are also automated, so releasing is a button press rather than a ritual.

GitLab CI/CD is the part of GitLab that does this. You describe what should happen in a text file called .gitlab-ci.yml, at the root of your repository. GitLab reads that file on every push, creates a pipeline, and hands the individual steps to machines that execute them. The results (green ticks, red crosses, logs, test reports, downloadable files) appear in the same web page where you review the code.

To see why this exists, picture the world before it. A team of five shares a repository. Each developer runs the tests on their own laptop, more or less carefully, before pushing. Some forget. Some have a different Python version. Someone has a slightly stale database schema. A change that passes on one laptop fails on another, and a change that passes on all of them fails in production because production runs something else again. Releases are done by whoever remembers the steps, typing commands from a wiki page, and the wiki page is two versions out of date.

The first generation of fix was a dedicated build server, usually Jenkins, that watched the repository and ran a script. That worked, but the build server was a separate system with its own login, its own plug-ins, its own configuration living in a web interface, and its own person who understood it. When the server broke, nobody could release. When the configuration changed, nobody could say who changed it or why.

GitLab's approach is to make the pipeline part of the repository. The recipe lives in .gitlab-ci.yml, committed next to the code it builds. It is versioned, reviewed in merge requests, and diffed like any other file. When a pipeline breaks, git log on that file tells you who changed it and when. A new team member gets the whole automation setup by cloning the project. This is the same idea as keeping a Dockerfile next to your source, applied to the whole delivery process, and it is the single biggest reason teams adopt GitLab CI/CD.

What people use it for:

For MLOps work specifically, the same machinery runs your data-validation checks, unit tests on feature code, model evaluation gates, and image builds for serving. A pipeline is simply an automated checklist that never gets tired and never skips a step.

You need very little to follow along. A free GitLab.com account, a web browser, and either a terminal with Git or the web editor that GitLab provides. If your employer runs a self-managed GitLab (common in regulated organisations in the Gulf and in Egypt, where data residency rules push teams to host their own), everything here applies to it as well, and the setup section explains the one difference.

The mental model: pipelines, stages, jobs and runners

GitLab CI/CD has four nouns. Everything else in this guide is a detail of one of them, so spend a few minutes on these before writing any YAML.

A job is the smallest unit. It is one named task with a list of shell commands to run, such as "install the dependencies and run the tests". A job has a name you choose (test, build-image, lint) and, at minimum, a script key listing the commands.

A stage is a named group of jobs that belong together and run in parallel. Stages run in order. By default the stages are build, then test, then deploy: all the jobs in build run at the same time, and only when they all succeed does GitLab start the jobs in test. If any job in a stage fails, the later stages do not start (you can change that, but it is the default, and it is usually what you want: there is no point deploying code whose tests failed).

A pipeline is the whole run: every stage and every job created for one event, such as a push to a branch or a merge request being opened. A pipeline belongs to one specific commit. You see it in the project under Build > Pipelines as a row with a status, and clicking it shows the stages as columns and the jobs as boxes inside them.

A runner is the machine (or container, or Kubernetes pod) that actually executes a job's commands. GitLab itself never runs your scripts. It decides what should run and when, then a program called GitLab Runner asks GitLab "is there a job for me?", receives the commands, runs them, streams the log back, and reports success or failure. This split matters, and it explains many beginner surprises: the job runs on a runner's machine, not on GitLab's web server and not on your laptop.

Two facts about how these pieces communicate will save you confusion later. First, the runner polls GitLab: it makes outbound HTTPS requests asking for work. GitLab never connects into the runner. That is why a runner can sit on a laptop, behind a corporate firewall, or inside a private network and still work, because it needs only outbound access. Second, GitLab reads your .gitlab-ci.yml and freezes the configuration when the pipeline is created. If you click "retry" on a failed job after fixing the YAML, the retry still uses the old configuration, because it belongs to the old pipeline. To test a config change you need a new pipeline, which a new commit gives you.

Jobs also start from a clean slate. Each job runs in a fresh environment, typically a fresh container, with your repository checked out at the pipeline's commit. Nothing installed or created by one job is visible to the next unless you explicitly pass it along with artifacts (files kept for later jobs, covered in their own section) or speed it up with a cache. Most "it worked in the previous job" confusion comes from forgetting this.

Here is the vocabulary in one table, to come back to:

Getting set up: an account, a project, and a runner

You do not install GitLab to learn GitLab CI/CD. The fastest route is a project on GitLab.com, the hosted service, where machines to run your jobs are already provided.

Create an account at gitlab.com, then create a new blank project (the New project button, then Create blank project). Give it a name such as ci-playground, leave it private, and tick the option to initialise the repository with a README so that it has a first commit.

Now the machines. GitLab.com offers GitLab-hosted runners: a fleet of virtual machines that GitLab operates and that every project can use without any setup. Each job gets a brand-new virtual machine, which is deleted when the job ends. A few facts worth knowing on day one:

• Jobs that do not ask for a specific kind of runner go to a small Linux machine with two virtual CPUs and 8 GB of memory (the runner tag is saas-linux-small-amd64). Larger sizes exist on paid plans.
• If your .gitlab-ci.yml does not name an image, the job runs in a default container image (ruby:3.1). You will almost always name your own image, which is one of the first things the first pipeline does.
• A single job can run for at most three hours on hosted runners.
• Hosted runners spend compute minutes, a monthly allowance that depends on your plan. A pipeline that runs for four minutes across two parallel jobs uses the minutes of both jobs, with a multiplier for larger machines.
• On the Free tier, the first time you run a pipeline on GitLab.com you may be asked for identity verification, a one-time check that exists to stop abuse of the free compute. Without it, jobs fail with Identity verification is required in order to run CI jobs. Complete the verification the page asks for and re-run the pipeline.

If your organisation uses a self-managed GitLab (installed on its own servers), there is no hosted fleet. Someone has to provide runners, and that may be you. Runner installation is short and worth seeing once, even if you never need it, because it makes the word "runner" concrete. The modern order of operations is:

1. In the project, open Settings > CI/CD > Runners and choose Create project runner. Give it a description and tags (tags are covered later). GitLab shows a runner authentication token that starts with glrt-. Copy it; it is shown once.
2. Install the gitlab-runner program on a machine that is not the GitLab server itself.
3. Register it, which links that machine to the runner you created in step 1.
4. Start the service.

On a Debian or Ubuntu machine that looks like this:

curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" -o script.deb.sh
less script.deb.sh          # read what you are about to run as root
sudo bash script.deb.sh
sudo apt install gitlab-runner
sudo gitlab-runner register --url https://gitlab.com --token glrt-YOUR_TOKEN_HERE

The register command asks which executor to use. The executor is how the runner runs a job. For a first runner, choose docker, which runs each job in a fresh container, and give a default image such as alpine:latest. The answers are written to a file called config.toml (at /etc/gitlab-runner/config.toml when run as root). Then confirm that everything is alive:

gitlab-runner --version
sudo gitlab-runner status
sudo gitlab-runner verify

verify should print Verifying runner... is alive, and in the GitLab web page the runner shows a green Online status. If you install on a Mac, Windows machine, inside Docker, or through Helm on Kubernetes, the commands differ slightly, and the official install pages (linked in Sources) cover each of them.

Two more rules. Keep the runner's version close to your GitLab's version: the docs recommend matching the major and minor numbers, and this guide's runner is 19.4.1 to match GitLab 19.4. And never install the runner on the GitLab server itself, because a job can run arbitrary commands, and you do not want that happening on the machine that stores your code.

Your first pipeline

The smallest possible pipeline is three lines. Create a file named .gitlab-ci.yml at the root of your project (the leading dot matters, and the name must match exactly) with this content:

hello:
  script:
    - echo "Hello from $CI_COMMIT_SHORT_SHA on $CI_COMMIT_BRANCH"

You can create it in the web editor by choosing New file from the repository page, or on your laptop with git add, git commit and git push. The moment the commit lands on the project, GitLab creates a pipeline. Open Build > Pipelines and you will see a row with a status badge moving from pending (waiting for a runner to pick it up) to running to passed. Click the badge, click the hello job, and read its log.

Every job log has the same shape, and learning to read it is the most useful early skill. From the top you will see lines about the runner starting, about the image being pulled and the repository being cloned (the job's setup), then a green $ echo "Hello from ..." line for each command of your script, followed by the command's output, and finally Job succeeded or ERROR: Job failed: exit code 1.

Let us read the file, since three lines hide a lot:

• hello: is the job name. Any top-level key that is not a reserved keyword is a job. You choose the name.
• script: is the only required key of a job. It is a list, and each item is one shell command, run in order.
• $CI_COMMIT_SHORT_SHA and $CI_COMMIT_BRANCH are predefined variables. GitLab fills in dozens of these for every job (the commit, the branch, the project name, the pipeline id) without you declaring anything. They are how a pipeline knows what it is building.

Notice what you did not write: no stage, no image, no runner selection. GitLab used the default stage (test), the default image on hosted runners, and any available runner. Defaults get you started, but real pipelines state these things explicitly, so let us do that. Replace the file with a real pipeline for a small Python project. First the project files; create these at the root of the repository:

def greet(name: str) -> str:
    """Return a friendly greeting."""
    if not name:
        raise ValueError("name must not be empty")
    return f"Hello, {name}!"

import pytest

from app import greet


def test_greet_returns_greeting():
    assert greet("Aya") == "Hello, Aya!"


def test_greet_rejects_empty_name():
    with pytest.raises(ValueError):
        greet("")

pytest>=8,<9

And now the pipeline that tests it:

default:
  image: python:3.12

stages:
  - test

unit-tests:
  stage: test
  script:
    - python --version
    - pip install -r requirements.txt
    - pytest -v

Commit and push. The default: block sets values that apply to every job; here it makes every job run inside the official python:3.12 container image, so you get a known Python regardless of what the runner's own machine has. stages: declares the stage names in order. The job joins the test stage, installs the dependencies, and runs pytest. In the log you will see pytest's own output, 2 passed, and a green job.

Now break it on purpose, because you will meet red pipelines constantly and it is better to meet your first one calmly. Edit app.py so that greet returns f"Hi, {name}!", commit, and push. The new pipeline fails: the job turns red, and the log ends with pytest's assertion diff and ERROR: Job failed: exit code 1. The reason is simple and worth memorising: a job fails when any command in its script exits with a non-zero status. Pytest exits with 1 when a test fails, the runner sees it, stops the job, and marks the pipeline failed. Nothing magical is happening: the same rule applies to every shell command, which is why a script that ends with a failing grep or a curl that returns 404 can also fail a job.

Stages, parallel jobs, and what happens when one fails

A single job is useful; several jobs in stages are where pipelines earn their keep. Extend the project so it checks code style and builds something, not only tests:

default:
  image: python:3.12

stages:
  - build
  - test
  - deploy

build-package:
  stage: build
  script:
    - mkdir -p dist
    - cp app.py dist/
    - echo "$CI_COMMIT_SHA" > dist/VERSION

lint:
  stage: test
  script:
    - pip install ruff
    - ruff check .

unit-tests:
  stage: test
  script:
    - pip install -r requirements.txt
    - pytest -v

The stages list defines the order: build, then test, then deploy. Every job names its stage with stage:. A job with no stage: lands in test, which is why the very first pipeline worked without one. Notice that nothing in this file uses the deploy stage yet. A stage with no jobs is simply skipped.

Open the pipeline graph after pushing. You will see three columns. build-package is alone in the first. lint and unit-tests sit together in the second, and they run at the same time, on separate runners or separate slots of a runner, because two jobs in one stage are independent by design. That parallelism is free speed: if linting takes one minute and tests take three, the stage takes three, not four. The third column is empty, so it does not appear.

The waiting rule is the part to internalise. The test stage starts only after every job in build has succeeded. If build-package fails, lint and unit-tests never run, and they show as grey "skipped" or not at all. This is what you want for a chain where later steps need the earlier ones to have worked. It is not what you want for a check that is merely informational, and there are two keywords for that.

allow_failure: true lets a job fail without failing the pipeline. The job shows an orange warning icon instead of a red cross, and later stages still run. Use it for checks you are introducing gradually, such as a new linter that currently complains about old code:

lint:
  stage: test
  allow_failure: true
  script:
    - pip install ruff
    - ruff check .

when: controls whether a job runs based on what happened before it. Its values are on_success (the default: run only if everything earlier passed), on_failure (run only if something earlier failed, useful for a notification job), always (run regardless), manual (wait for a human to press a play button), delayed (run after a wait you specify with start_in), and never. A common use is cleanup that must happen even after a failure:

notify-failure:
  stage: .post
  when: on_failure
  script:
    - echo "The pipeline for $CI_COMMIT_SHORT_SHA failed. Check $CI_PIPELINE_URL"

That job uses a special stage, .post, which always runs last. Its sibling .pre always runs first. You do not declare them in stages; GitLab adds them. They are handy for one-off setup and teardown that does not belong to any of your own stages.

Two other job-level keys you will meet early. timeout: sets how long a job may run before GitLab kills it (for example timeout: 15 minutes); the project default is 60 minutes, and a job that hangs forever otherwise burns your compute allowance. retry: re-runs a job automatically when it fails, for example retry: 2, which is useful for flaky infrastructure problems (a package mirror timing out) and dangerous as a cover for flaky tests, because a retried failure that later passes hides a real bug.

Scripts, images and services: what a job really runs

A job is three things: an environment, some preparation, and commands. This section covers how you control each one.

The environment comes from the image: key. On a Docker-executor runner (the usual case, including hosted runners), every job runs inside a fresh container started from the image you name. image: python:3.12 means a Linux container with Python 3.12 installed, pulled from Docker Hub. This is what makes a pipeline reproducible: the job does not depend on what happens to be installed on the runner. Pick the image that has the tools your script needs. A Node project uses node:22, a Go project golang:1.23, a job that only runs shell commands can use a tiny alpine:3.20. If your script says command not found for a tool you expected, nine times in ten the image is wrong.

You can set image: per job or once under default:. A per-job value overrides the default, which is how a Python pipeline can still have one job that uses a Node image for a front-end check:

default:
  image: python:3.12

frontend-lint:
  image: node:22
  script:
    - npm ci
    - npm run lint

The preparation is what before_script is for. Commands listed there run before the script of each job, in the same shell, so anything they set up (an activated virtual environment, an exported variable) is visible to the script. A typical use is installing dependencies once for every job:

default:
  image: python:3.12
  before_script:
    - python --version
    - pip install -r requirements.txt

unit-tests:
  script:
    - pytest -v

after_script is its counterpart for cleanup. It runs after the main script even when that script failed, but in a separate shell, so it does not see variables you exported in script. Use it for things like printing diagnostics or stopping a helper process, not for passing values.

One sharp edge is how a job decides success. The runner runs your script lines in a shell that stops at the first command returning non-zero. That is the behaviour you want, and it is also why a script that looks fine can fail: a grep that finds nothing returns 1, and a test -f somefile for a missing file returns 1. If a command is allowed to fail, handle it explicitly in the shell (grep pattern file || true) rather than hoping.

For anything longer than a line or two, resist the urge to write a mini-program in YAML. A multi-line shell block is written with |:

report:
  script:
    - |
      echo "Branch: $CI_COMMIT_BRANCH"
      if [ -f dist/VERSION ]; then
        echo "Version: $(cat dist/VERSION)"
      else
        echo "No version file"
      fi

Once a script grows past a few lines, move it to a file in the repository (scripts/report.sh), make it executable, and call it from the job. Files can be linted and run on your laptop; YAML cannot.

Services are the third piece. A services: entry starts an extra container next to your job, reachable by hostname, which is how a test job gets a database without installing one:

integration-tests:
  image: python:3.12
  services:
    - name: postgres:16
      alias: db
  variables:
    POSTGRES_PASSWORD: example-only-not-a-real-secret
  script:
    - pip install -r requirements.txt
    - pytest tests/integration -v

Inside the job, the database is at host db, port 5432. The service container is started before the script and thrown away after. A service that does not come up is one of the classic confusing failures, so if a job cannot reach its database, read the top of the log for the service's own startup messages.

Variables: passing values into a pipeline

Hard-coding values into scripts is the fastest way to write a pipeline nobody can reuse. CI/CD variables are environment variables that GitLab gives to each job, and they come from several places. Knowing which place to use for what is most of the skill.

Predefined variables come from GitLab and describe the context. You have met a few. These are the ones worth memorising:

YAML variables you declare yourself in the file, either for all jobs or for one:

variables:
  APP_ENV: "staging"
  PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"

deploy-preview:
  stage: deploy
  variables:
    APP_ENV: "preview"        # overrides the global value for this job only
  script:
    - echo "Deploying to $APP_ENV"

A variable defined on a job overrides the same name defined globally. Use YAML variables for configuration that is not secret and that belongs with the code: a Python version, a cache directory, a feature toggle.

Project and group variables, set in the web interface under Settings > CI/CD > Variables, are for values that should not live in the repository: API keys, passwords, tokens. Add a variable there, give it a key such as DEPLOY_TOKEN, and every job in the project sees it as $DEPLOY_TOKEN. The Add variable dialog has options that matter:

• Masked hides the value in job logs, replacing it with [MASKED]. To be maskable, the value must be a single line with no spaces and at least eight characters. As of GitLab 18.3, new variables in the interface default to masked visibility.
• Masked and hidden goes further and hides the value in the settings page too, after you save it. Use it for secrets you never need to read back.
• Protected makes the variable available only to pipelines running on protected branches and tags (for example main). A variable holding a production credential should be protected, so a pipeline on someone's experimental branch cannot read it.
• Type: File writes the value to a temporary file and gives you the file's path in the variable. This is right for certificates and key files, which do not work well as environment variable text.
• Expand variable reference controls whether $OTHER_VAR inside the value is substituted. Since GitLab 18.6 it is off by default for variables set in the interface, which means a value containing a literal dollar sign is now kept as written. Turn it on only when you want substitution.

Two behaviours to know. Variables are read by the shell inside the job, so they are referred to as $NAME (or ${NAME} when followed by other characters, like ${NAME}_suffix). And when the same name exists in several places, there is a precedence order. You do not need the full list yet, only the practical version: a value set when you run a pipeline by hand beats a project variable, which beats a variable in the YAML file, which beats a predefined one. When a variable "has the wrong value", search every place it could be defined before you debug anything else.

Rules: deciding when a job runs

So far every job runs on every push to every branch. Real projects need finer control: run the deploy only on the main branch, skip the heavy tests when only the README changed, run the review checks only on merge requests. The keyword for this is rules.

A rules: list is read top to bottom. For each job GitLab finds the first rule that matches, and that rule decides what happens. If no rule matches, the job is not added to the pipeline at all. A rule can have an if: condition, a when: outcome, and a few other keys.

deploy-production:
  stage: deploy
  script:
    - echo "Deploying $CI_COMMIT_SHORT_SHA"
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: manual
    - when: never

Read it aloud: if the branch is the default branch, add this job and require a manual click; otherwise never add it. The final - when: never makes the fallback explicit. The job appears only on main, and even there it waits for a human.

The expressions in if: have a particular syntax that trips beginners constantly, so learn the rules now:

• Variables are written unquoted, with a dollar sign, and compared with a quoted string: $CI_COMMIT_BRANCH == "main".
• The wrong forms are "$CI_COMMIT_BRANCH" == "main", ${CI_COMMIT_BRANCH} == "main" and $CI_COMMIT_BRANCH == main. All three are invalid or do not behave as you expect.
• =~ and !~ match a regular expression written between slashes: $CI_COMMIT_BRANCH =~ /^feature\//.
• &&, || and parentheses combine conditions: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push".
• A bare variable like if: $CI_COMMIT_TAG is true when that variable is set and not empty.

Two more rule keys cover the common cases. changes: runs a job only when certain files changed, which saves compute:

docs-check:
  script:
    - echo "Checking docs"
  rules:
    - changes:
        - docs/**/*
        - README.md

And exists: runs a job only when a file is present. Both can be combined with if: in the same rule, and all keys in one rule must match together.

Why pipelines run twice, and how to stop it. This is the most common puzzle in a real project. When you push commits to a branch that has an open merge request, GitLab can create two pipelines: a branch pipeline (for the push) and a merge request pipeline (for the merge request). If your jobs use rules with no if: on the pipeline source, both pipelines run the same jobs, which doubles your compute and confuses everyone, and GitLab warns Job may allow multiple pipelines to run for a single action.

The standard fix is a workflow:rules block at the top of the file. It decides whether a pipeline is created at all, before any job rules apply:

workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
      when: never
    - if: $CI_COMMIT_BRANCH

Read it as three rules. A merge request event creates a pipeline. A branch push where a merge request is already open creates nothing (because the merge request pipeline already covers it). Any other branch push creates a pipeline. The result: exactly one pipeline per action. Copy this block into every project you write; it is boring and correct.

If workflow:rules matches nothing, the merge request shows Pipeline filtered out by workflow rules. That is not a bug but your own rules excluding the run, and the fix is to adjust the block.

Artifacts: passing files between jobs, and keeping results

Remember that every job starts in a clean environment. If build-package creates a dist/ folder, the deploy job, running later in a different container and possibly on a different machine, will not see it. Artifacts are the mechanism for moving files forward. You list paths in a job's artifacts: key, the runner uploads them to GitLab when the job finishes, and jobs in later stages download them automatically before they start.

build-package:
  stage: build
  script:
    - mkdir -p dist
    - cp app.py dist/
    - echo "$CI_COMMIT_SHA" > dist/VERSION
  artifacts:
    paths:
      - dist/
    expire_in: 1 week

deploy-production:
  stage: deploy
  script:
    - cat dist/VERSION
    - echo "Uploading dist/ to the server"
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: manual

The deploy-production job never built anything, yet dist/VERSION is there, because the artifacts of earlier-stage jobs are downloaded into the working directory first. You can see them in the web interface too: open the build-package job page and use Browse or Download on the right-hand side. That is how a pipeline delivers a compiled binary, a model file, or a report to a human.

The keys to know:

• paths lists files and directories to keep, relative to the project directory. Wildcards work (reports/*.html).
• expire_in says how long GitLab keeps them: 30 minutes, 1 week, 3 months. Without it, the instance default applies (30 days out of the box). Artifacts take up storage, so set a short expiry on anything bulky.
• when decides when to upload. By default artifacts are saved only if the job succeeds. when: always saves them after failures too, which is what you want for test reports, because the failed run is the one you need to read.
• name names the download archive, and exclude removes files matching a pattern from the set.
• reports is a special kind of artifact that GitLab understands and displays instead of just storing.

The last one is worth seeing, because it turns a wall of log text into a proper result. If your tests write a JUnit XML file, GitLab shows the individual test results in the pipeline page and in merge requests:

unit-tests:
  stage: test
  script:
    - pip install -r requirements.txt
    - pytest -v --junitxml=report.xml
  artifacts:
    when: always
    reports:
      junit: report.xml
    expire_in: 1 week

After this job runs, the pipeline page gains a Tests tab listing each test with its status and time, and a failing test in a merge request is named right there without anyone opening the log. The when: always matters: without it, a failing run (the one with the interesting report) would upload nothing.

Artifacts have limits. The default maximum size of one artifact file is 100 MB on most instances, so they are not the place for multi-gigabyte datasets; store large files in object storage (S3, GCS, or a regional equivalent) and pass only a reference. Anything a user can download from an artifact is visible to everyone with access to the job, so do not put secrets in artifacts.

Cache: making repeated work faster

Every job installs its dependencies from scratch in a clean container. For a project with a hundred Python packages or a large node_modules, that can be most of the job's duration, and most of your compute bill. A cache lets jobs reuse the downloaded packages from earlier runs.

variables:
  PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"

unit-tests:
  stage: test
  cache:
    key:
      files:
        - requirements.txt
    paths:
      - .cache/pip
  script:
    - pip install -r requirements.txt
    - pytest -v

Two things are happening. First, the variable PIP_CACHE_DIR tells pip to keep its downloaded packages inside the project directory, because the cache can only save paths that are inside the project. Second, the cache: block says which directory to save (paths) and under what name (key). The runner downloads the cache before the script and uploads it after. The second pipeline's pip install finds the packages already local, and the job gets faster.

The key is how GitLab decides which cache belongs to which job. A fixed string such as key: my-cache shares one cache everywhere, and a stale cache can linger forever. The pattern used here, key:files: with requirements.txt, builds the key from a hash of that file. When your dependencies change, the file changes, the key changes, and a fresh cache is built. When they do not, the same cache is reused. That is almost always the right behaviour, and it keeps the cache honest without anyone clearing it.

Some options you will eventually want:

• policy: pull-push is the default (download at the start, upload at the end). pull only downloads, which suits jobs that only read the cache, and push only uploads.
• fallback_keys: other keys to try if the exact key has no cache yet, so a changed dependency list can start from a nearly-right cache.
• when: on_success by default; always also saves after a failure.

The most important thing to understand about a cache is that it is not guaranteed. The cache may be empty because this is the first run, because it expired, because the job ran on a different runner that does not share the cache storage, or because someone cleared it. Your job must work with an empty cache, only more slowly. If a job passes only when the cache is warm, you have found a bug, not a feature. A job can use up to four caches at once.

Runners and tags: choosing where a job runs

Each job needs a runner, and a project often has several: a hosted fleet, a team runner with special hardware, a cheap one for linting. Tags are how you match jobs to runners. A runner is given tags when it is created (docker, gpu, linux-arm64), and a job lists the tags it needs with tags:. A job runs only on a runner that has all of the job's tags.

train-smoke-test:
  stage: test
  tags:
    - gpu
  script:
    - nvidia-smi
    - python train.py --epochs 1 --dry-run

This job waits for a runner tagged gpu. A job with no tags: can run on any runner that is set to accept untagged jobs. On GitLab.com, untagged jobs go to the small Linux hosted runner, and tags such as saas-linux-medium-amd64 select bigger ones on plans that include them.

Runners also have a scope, which is who may use them. Instance runners serve every project on the GitLab server, group runners serve all projects in a group, and project runners serve only the projects you attach them to. Hosted runners on GitLab.com are instance runners. A runner can also be marked protected, in which case it runs only pipelines for protected branches and tags, and locked, in which case it stays with its current project.

The failure you will see first is this one, and it is shown as a clock-like "pending" job that never starts:

This job is stuck because you don't have any active runners that can run this job.

It means no online runner satisfies the job. The causes, in order of likelihood:

1. A tag mismatch. The job says tags: [gpu] and no runner has gpu (or the tag is spelled differently).
2. The runner refuses untagged jobs and your job has no tags.
3. The runner is offline or paused. Check Settings > CI/CD > Runners for a green Online badge.
4. The runner is protected-only and your branch is not protected, or locked to another project.

Fix it from the Runners page: compare the job's tags with the runner's, and enable Run untagged jobs on the runner if that is what you intend. If nothing matches for an hour, GitLab fails the job with the reason stuck_pending_no_matching_runners.

The executor is the other runner property worth knowing by name. The actively maintained ones are Docker (each job in a container, the common choice), Kubernetes (each job in a pod) and the autoscaling executors that create cloud machines on demand. The older Shell, SSH, VirtualBox, Parallels and Custom executors are in maintenance mode, receiving security fixes only, and the Docker Machine executor is deprecated with removal in GitLab 20.0. As a beginner you only need to know that the executor decides what image: means: on a shell executor, the image: key does nothing, because there is no container.

Environments and deploying safely

A pipeline that only tests is half the job. The other half gets the tested code to where it runs. This guide does not teach a particular hosting platform (that depends on your employer), but the GitLab side of deployment is the same everywhere, and worth learning with a stand-in script.

An environment is a named target such as staging or production. Declaring one on a job tells GitLab that running this job is a deployment, and GitLab records every deployment: which commit, which pipeline, when, and by whom. The project then has an Environments page (under the Operate section of the sidebar) that shows what is currently deployed where, with the history, which turns "what version is in production?" from a question into a click.

deploy-staging:
  stage: deploy
  environment:
    name: staging
    url: https://staging.example.com
  script:
    - echo "Deploying $CI_COMMIT_SHORT_SHA to staging"
    - ./scripts/deploy.sh staging
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

deploy-production:
  stage: deploy
  environment:
    name: production
    url: https://www.example.com
  script:
    - echo "Deploying $CI_COMMIT_SHORT_SHA to production"
    - ./scripts/deploy.sh production
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: manual

Staging deploys automatically when code reaches main. Production waits for a person to press the play button (when: manual). This two-step shape is how most teams work: automatic delivery to a safe place, human approval for the real one. A manual job shows in the graph with a play icon and the pipeline is marked as passed-with-manual-pending rather than blocked, so it never holds up the rest of your work.

Two extra protections are worth knowing about even now. First, a protected environment restricts who is allowed to run deployments to it, which is a paid-tier feature; if someone outside the allowed list clicks play, they see You are not authorized to run this manual job. Second, the resource_group keyword stops two deployments to the same place from running at once, which prevents two quick merges from racing each other:

deploy-production:
  resource_group: production
  environment:
    name: production
  script:
    - ./scripts/deploy.sh production

Jobs that share a resource group run one at a time; the second waits for the first to finish.

If your deploy needs a credential, this is where your earlier variables lesson pays off. Store the token as a masked, protected project variable, reference it as $DEPLOY_TOKEN inside scripts/deploy.sh, and the value never appears in the repository. Because it is protected, only pipelines on protected branches such as main can read it.

Reusing configuration without copy-paste

Once you have a dozen jobs, the same lines appear everywhere. YAML and GitLab give you three tools to stop that. Learn them in this order.

Hidden jobs and extends. A job whose name starts with a dot (.python-job) is never run; it is a template. A real job copies it with extends::

.python-job:
  image: python:3.12
  cache:
    key:
      files:
        - requirements.txt
    paths:
      - .cache/pip
  before_script:
    - pip install -r requirements.txt

unit-tests:
  extends: .python-job
  stage: test
  script:
    - pytest -v

type-check:
  extends: .python-job
  stage: test
  script:
    - pip install mypy
    - mypy app.py

Both jobs inherit the image, cache and install step, and each adds its own script. Keys defined on the job win over the template's. The merge is shallow for most keys: if the job defines its own cache, it replaces the template's whole cache block rather than merging into it, which surprises people, so check the Full configuration tab in the pipeline editor to see the result.

include. As the file grows, or as several projects need the same setup, you can split the configuration into files and pull them in:

include:
  - local: /ci/test-jobs.yml
  - template: Jobs/SAST.gitlab-ci.yml

local pulls a file from the same repository (the path starts with / and is relative to the repository root), and template uses one of GitLab's own ready-made configurations, here a security scan. There are other forms: project (a file from another GitLab project at a chosen ref), remote (a URL) and component. A project can include at most 150 files. The errors to recognise are Local file ... does not exist!, which means a wrong path, and Project ... not found or access denied, which means a wrong project path or missing access.

Components and the CI/CD Catalog. The modern, versioned way to share pipeline pieces is a CI/CD component: a reusable chunk of configuration, published in a catalog, that you include by name and version and can configure with inputs. Teams use them to give every project the same lint, test, and build steps without copying YAML. They belong to Mid-level; for now it is enough to know they exist, so that when a colleague writes include: - component: ..., you know what you are looking at.

Validating and debugging a pipeline

Sooner or later a pipeline will refuse to start, or a job will fail in a way the log does not explain. Debugging pipelines has a method, and using it beats staring.

Step one: is the configuration valid? Before any job runs, GitLab validates the YAML. If it is wrong, the pipeline is not created, and you see a red banner such as This GitLab CI configuration is invalid: jobs:test-job:artifacts config contains unknown keys: path. The message names the job and the exact problem (here path should have been paths). Use the pipeline editor for instant feedback, or check from a terminal with the official CLI, glab:

glab auth login
glab ci lint
glab ci lint --dry-run --ref main

glab ci lint sends your local .gitlab-ci.yml to GitLab for validation and prints any errors. The --dry-run form goes further and simulates pipeline creation, catching rules that would leave you with no jobs. Another error you will see is jobs config should contain at least one visible job, meaning the file has only hidden jobs or your rules excluded everything.

Step two: did the pipeline get created, and with which jobs? If the configuration is valid but the job you expected is missing, the cause is almost always rules or workflow:rules. Open the Full configuration tab, then re-read your conditions with the actual values of CI_PIPELINE_SOURCE and CI_COMMIT_BRANCH for this kind of pipeline in mind. A job restricted to branches will not appear in a tag pipeline or a merge request pipeline, and that surprises nearly everyone once.

Step three: did the job start? A job stuck in pending is a runner-matching problem, covered above. A job that starts and fails instantly is usually the image or the clone step, so read the first twenty lines of the log.

Step four: read the failing command. Scroll to the first red line, not the last. The last line is always ERROR: Job failed: exit code 1, which tells you only that something returned non-zero. The command above it, with its own output, is the evidence. GitLab collapses sections of the log; expand them. Timestamps on each line help you spot a step that took suspiciously long.

Step five: reproduce it. Much of the time you can run the same commands locally in the same image. For the Python example:

docker run --rm -it -v "$PWD":/work -w /work python:3.12 bash
pip install -r requirements.txt
pytest -v

If it fails here too, you have a code problem that the pipeline was right to catch. If it passes locally but fails in CI, compare environment variables, file permissions, and which files exist in a clean checkout (did you forget to commit a file?).

The command-line companions. With glab you can drive pipelines without the browser:

Common errors and how to read them

Most failures repeat. Here are the ones beginners meet, with what the message means and the fix. Keep this table open during your first month.

Two of these deserve a paragraph. The timeout message reports the limit that was hit, so compare it with your timeout: value, the project default of 60 minutes, and any runner maximum before deciding whether the job hung or simply needs longer. And the Docker error appears the moment you try to run docker build inside a job. Building container images in a pipeline needs a Docker daemon the job can reach, usually through a docker:dind service on a privileged runner, and the GitLab.com hosted runners support it. That setup is a Mid-level topic, but knowing the error means you will recognise it when you get there.

Finally, a small rule for reading any error: find the first error in the log, not the last, and ask whether it is about the configuration (before any job), the runner (the job never starts), the environment (the image or the tools), or the script (your commands). Four categories cover nearly everything.

Keeping secrets out of your repository

Every beginner pushes a password to a repository once. In a pipeline the stakes are higher, because the credential is now usable by anything that can run a job. A few habits cover most of the risk.

Never commit secrets. Not in .gitlab-ci.yml, not in a script, not in a .env file, not in a test fixture. Git history is permanent: deleting a file in the next commit leaves the secret in the previous one. If a real secret reaches the repository, treat it as leaked and rotate it, meaning generate a new one and revoke the old, rather than trying to scrub history.

Put them in variables. Project or group CI/CD variables, masked and protected, are the beginner-level solution, as you saw above. Use File-type variables for certificates and key files.

Use the job token for GitLab itself. Inside a job, GitLab provides CI_JOB_TOKEN, a short-lived credential tied to that one job that can do a limited set of things (pull from the project's container registry, call some API endpoints, clone allowed projects). The built-in registry variables use it: CI_REGISTRY_USER and CI_REGISTRY_PASSWORD log you in to the project's registry. Prefer them to creating a personal access token that outlives the job. Since GitLab 19.0 the job token uses the JWT format by default, and one consequence is that wrapping it in base64 with line breaks corrupts it (use base64 -w0).

Do not print secrets. echo $DEPLOY_TOKEN hides the value only when it is masked, and even then a transformed value escapes. Write scripts that pass the variable to the tool that needs it and print nothing.

Mind who can run what. Protected variables reach only pipelines on protected branches. A merge request from a fork runs in the fork's context without your secrets unless a maintainer deliberately runs it in the parent project, and that is a safety feature: read a stranger's changes to .gitlab-ci.yml before approving a pipeline.

The best modern approach, which Mid and Senior levels cover, is to avoid stored secrets altogether: a job asks GitLab for a short-lived OpenID Connect token (id_tokens:) and exchanges it with your cloud provider or vault for temporary credentials. Nothing long-lived is stored, so nothing long-lived can leak.

Putting it all together

Let us assemble everything into one small, complete project: a Python package with tests, a build that produces an artifact, a test report, a cache, a staging deploy on main, and a manual production deploy. Start from the four files you already have (app.py, test_app.py, requirements.txt and, optionally, a scripts/deploy.sh that just echoes). Here is the full pipeline:

default:
  image: python:3.12
  interruptible: true

stages:
  - build
  - test
  - deploy

variables:
  PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"

workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
      when: never
    - if: $CI_COMMIT_BRANCH

.python-cache:
  cache:
    key:
      files:
        - requirements.txt
    paths:
      - .cache/pip

build-package:
  stage: build
  script:
    - mkdir -p dist
    - cp app.py dist/
    - echo "$CI_COMMIT_SHA" > dist/VERSION
  artifacts:
    paths:
      - dist/
    expire_in: 1 week

lint:
  extends: .python-cache
  stage: test
  allow_failure: true
  script:
    - pip install ruff
    - ruff check .

unit-tests:
  extends: .python-cache
  stage: test
  script:
    - pip install -r requirements.txt
    - pytest -v --junitxml=report.xml
  artifacts:
    when: always
    reports:
      junit: report.xml
    expire_in: 1 week

deploy-staging:
  stage: deploy
  environment:
    name: staging
  script:
    - echo "Deploying version $(cat dist/VERSION) to staging"
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

deploy-production:
  stage: deploy
  resource_group: production
  environment:
    name: production
  script:
    - echo "Deploying version $(cat dist/VERSION) to production"
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: manual

Read it once from the top; every line is something you have now learned. The default: block sets the image and marks jobs interruptible, so that a newer push can cancel an older, now-pointless pipeline and save minutes. workflow:rules guarantees one pipeline per action. The hidden .python-cache job carries the cache, and two jobs extend it. build-package produces a dist/ artifact, which the deploy jobs read without building anything. The unit tests publish a JUnit report. Staging deploys on main; production waits for a click and is serialised with a resource group.

Now walk it end to end, as a teammate would:

1. Create a branch, change greet, and push. A branch pipeline runs: build, then lint and tests in parallel. No deploy jobs appear, because the branch is not main.
2. Open a merge request. The workflow rules switch to a merge request pipeline, with the tests shown in the merge request's widget. Push a fix and confirm only one pipeline runs.
3. Merge. A pipeline on main runs everything, and deploy-staging runs automatically.
4. Open the pipeline and click play on deploy-production. Open the Environments page and read the history.
5. Break a test deliberately in a new branch and watch the merge request show the failing test by name.

Commit it, push it, and then change one thing at a time to see how each keyword behaves. Remove the cache and compare times. Delete the workflow block and watch the duplicate pipelines appear. Add allow_failure: false and see how the lint job then blocks. That experimentation is the real lesson, and it costs almost nothing.

If you want to take the image-building route next, the same project becomes a container pipeline once you add a Dockerfile; the Docker guide covers writing one, and the Mid-level part of this guide shows how to build and push it from a job. For deploying onto a cluster, continue with Kubernetes, and for the infrastructure underneath, Terraform.

What you can now do, and what comes next

You can read and write a .gitlab-ci.yml from nothing. You know the four nouns (pipeline, stage, job, runner) and how they relate. You can run tests on every push, choose images and services, pass files between jobs with artifacts, speed up installs with a cache keyed on the lockfile, and keep a secret in a masked, protected variable instead of the repository. You can control when jobs run with rules and workflow:rules, put a manual approval in front of a deployment, record deployments in environments, reuse configuration with hidden jobs and extends, and diagnose a red pipeline by category. You also know which older idioms to avoid: registration tokens, only/except, and top-level image and cache.

The Mid-level guide picks up exactly here. It opens the pipeline and shows how GitLab decides what to run, then teaches the DAG with needs, parallel and matrices, downstream and dynamic pipelines, components and inputs, variable precedence in full, merge request pipelines and merge trains, Docker-in-Docker builds, OpenID Connect secrets, and running your own runners well. The Senior guide treats GitLab CI/CD as a platform you operate for a team: runner fleets and autoscaling, the security and trust model, multi-tenancy, cost, upgrades (with 20.0 in May 2027 as the next breaking-change window), and governance.

Before moving on, keep three habits. Validate before you push, using the pipeline editor or glab ci lint. Read the first error in a log, not the last. And treat a pipeline as code: small commits, reviewed changes, and no secret ever near the repository. Those three habits are worth more than any single keyword.

The Interview and Tips & Tricks companions for this level turn what you just learned into answers you can say out loud and mistakes you can avoid.

Sources

The content of this guide was checked against the official documentation for GitLab 19.4 and GitLab Runner 19.4.1.

• GitLab CI/CD documentation
• CI/CD YAML syntax reference
• Deprecated keywords
• Workflow keyword and rules
• Validate CI/CD configuration
• Job rules
• Job artifacts
• Troubleshooting jobs
• Debugging CI/CD pipelines
• CI/CD variables
• Predefined CI/CD variables
• Caching in GitLab CI/CD
• The pipeline editor
• Hosted runners on GitLab.com
• Linux hosted runners
• Runner scopes
• Creating a runner (new workflow)
• CI/CD job token
• Building Docker images with GitLab CI/CD
• GitLab Runner documentation
• Install GitLab Runner
• Registering runners
• GitLab Runner commands
• Runner executors
• Deprecations and removals
• GitLab 19.4 release notes
• glab CLI, CI commands

Part one of three. A twenty-five-minute review of the Beginner Detailed track, then the questions interviewers actually ask. Mid-level reviews this plus its own material; Senior reviews all three. Everything here was checked against GitLab 19.4 and GitLab Runner 19.4.1.

The thirty-second answer

GitLab CI/CD is the automation built into GitLab. You describe a pipeline in a .gitlab-ci.yml file in the repository: stages, and jobs inside them, each with a script. On every push or merge request GitLab creates a pipeline from that file, and a GitLab Runner, a separate agent that polls GitLab for work, executes each job, usually in a fresh container. Results, logs, test reports and artifacts appear next to the code review.

Then the line that shows you have used it: "the configuration is frozen when the pipeline is created, each job starts from a clean checkout, and only artifacts carry files forward, while a cache is just a best-effort speed-up."

If you only have ten seconds, say it in one sentence: a pipeline is the recipe in .gitlab-ci.yml turned into stages and jobs, and runners are the machines that cook it.

The model

Interviewers check that you hold a correct picture, so be able to draw it. There are four nouns.

A job is a named task with a script. A stage groups jobs that run in parallel, and stages run in order: by default build, then test, then deploy, and a later stage starts only when every job of the previous one succeeded. A pipeline is the whole run for one event, tied to one commit. A runner is the machine that executes a job, and the executor is how it does so (a Docker container, a Kubernetes pod, a plain shell).

The points that separate a person who has read about it from one who has used it:

• GitLab does not run your scripts. Runners do. The runner makes outbound HTTPS requests asking for jobs, so it needs no inbound port and can live behind a firewall.
• The configuration is fixed when the pipeline is created. Retrying a job after editing the YAML reuses the old configuration. You need a new pipeline, which a new commit gives you.
• Each job is isolated. A fresh container, a fresh checkout. Files do not flow between jobs unless you pass them as artifacts.
• Jobs in one stage are parallel. Stages are sequential. The needs keyword can break the stage order, and that is a Mid-level topic.
• Failure means a non-zero exit status. Any command in the script that returns non-zero stops the job and fails it.

The commands and keys to have ready

Interviewers often ask you to write a small pipeline on a whiteboard or explain one they show you. Have this skeleton in your head. Every keyword in it should be explainable.

default:
  image: python:3.12
  interruptible: true

stages: [build, test, deploy]

variables:
  PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"

workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
      when: never
    - if: $CI_COMMIT_BRANCH

test:
  stage: test
  cache:
    key:
      files: [requirements.txt]
    paths: [.cache/pip]
  script:
    - pip install -r requirements.txt
    - pytest --junitxml=report.xml
  artifacts:
    when: always
    reports:
      junit: report.xml
    expire_in: 1 week

deploy:
  stage: deploy
  script: ./deploy.sh
  environment: production
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: manual

And the few command-line tools:

glab auth login                    # once, authenticate the GitLab CLI
glab ci lint                       # validate .gitlab-ci.yml against GitLab
glab ci lint --dry-run --ref main  # simulate pipeline creation
glab ci status --live              # watch the latest pipeline
glab ci trace                      # stream a job's log
glab ci run -b main                # start a pipeline on a branch
gitlab-runner verify               # confirm a runner can reach GitLab
gitlab-runner register --url https://gitlab.com --token glrt-...   # link a host to a runner

The keys to be able to explain in one sentence each: script, before_script, after_script, stage, image, services, variables, rules, workflow:rules, when, allow_failure, artifacts, cache, tags, timeout, retry, environment, extends, include and default. If you can do that, the whiteboard exercise is easy.

Jobs, scripts and exit codes

A job passes when its script finishes with every command returning zero. The runner runs the commands in order and stops at the first non-zero one, so ERROR: Job failed: exit code 1 only tells you a command failed. The cause is the command above it. Expect a follow-up such as "a job fails on grep finding nothing, why?": because grep returns 1 when it matches nothing, and the shell treats that as failure. The fix is to handle the case in the shell (grep pattern file || true) when a miss is acceptable.

before_script runs before the script in the same shell, which makes it the place for shared setup. after_script runs after the script, even when it failed, but in a separate shell, so it cannot see variables exported in script. That difference is a classic question.

allow_failure: true lets a job fail without failing the pipeline, shown as an orange warning. when: chooses when a job runs: on_success (default), on_failure, always, manual, delayed and never. A .post stage always runs last, and .pre first.

Rules, workflow and duplicate pipelines

rules replaced only and except, which are deprecated. A job's rules are checked top to bottom, the first matching rule wins, and if none matches the job is not created. Know the expression syntax, because interviewers enjoy it:

• Unquoted variable, quoted string: $CI_COMMIT_BRANCH == "main".
• Invalid forms: "$CI_COMMIT_BRANCH" == "main", ${CI_COMMIT_BRANCH} == "main" and $CI_COMMIT_BRANCH == main.
• Regular expressions use =~ and !~ with slashes: $CI_COMMIT_BRANCH =~ /^feature\//.
• &&, || and parentheses combine conditions, and a bare $VAR is true when the variable is set and non-empty.

The other famous question is why a push to a branch with an open merge request starts two pipelines. Because the push creates a branch pipeline and the merge request creates a merge request pipeline, and with job-level rules that do not distinguish them, both run the same jobs. GitLab even warns Job may allow multiple pipelines to run for a single action. The fix is a workflow:rules block: create a pipeline for merge request events, skip branch pipelines when a merge request is open ($CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS with when: never), and allow other branch pipelines. workflow:rules decides whether a pipeline exists at all, before job rules apply.

rules:changes runs a job only when certain files changed, which saves compute. It is easy to get a surprise on branch pipelines without a merge request, where "changed" is measured against the previous push; say that you would pair it with a merge request pipeline or an explicit comparison.

Artifacts versus cache

Get this one perfect, because it is asked in nearly every interview.

The model answer: artifacts carry results, caches carry speed. If a later job needs the file to be correct, use an artifact. A cache may be missing on any run, so a job must still work, only slower, with an empty cache. The follow-ups are the cache key (use key:files on the lockfile so the cache changes when dependencies change), the policy (pull-push, pull, push), and that paths must be inside the project directory, which is why PIP_CACHE_DIR is set to a path under $CI_PROJECT_DIR.

Also know artifacts:when: always, which saves the artifacts of a failed job, and artifacts:reports:junit, which makes GitLab display individual test results. A job can use up to four caches, and the default maximum artifact size is 100 MB per file, so large datasets do not belong in artifacts.

Variables and secrets

Four sources to name: predefined variables GitLab provides (CI_COMMIT_SHA, CI_COMMIT_BRANCH, CI_PIPELINE_SOURCE, CI_DEFAULT_BRANCH, CI_REGISTRY_IMAGE and many more), variables in the YAML, project or group variables in Settings > CI/CD > Variables, and values typed when you run a pipeline manually.

The attributes of a settings variable are a reliable question: masked (hidden in logs, needs a single line of at least eight characters with no spaces), masked and hidden (also hidden in the settings page), protected (available only on protected branches and tags), File type (the value is written to a temporary file and the variable holds its path), and expand variable reference (whether $OTHER inside a value is substituted; off by default for interface variables since 18.6).

The sentence to say about masking: it is not a security boundary. It hides an exact match in the log and is bypassed by printing env, encoding the value, or changing the pipeline to exfiltrate it. The protection that matters is scoping: protected variables only on protected refs, and review of any change to .gitlab-ci.yml from people who can reach those variables. The modern answer to "how do you avoid storing long-lived secrets at all" is id_tokens, OpenID Connect tokens exchanged for temporary cloud credentials, which you can mention as something you are aware of even if you have not built it yet.

Runners, executors and tags

A strong answer covers four things. Scope: instance runners serve everyone, group runners a group, project runners specific projects; on GitLab.com the hosted runners are instance runners. Tags: a job runs only on a runner that has all of the job's tags, and a runner may or may not accept untagged jobs. Executor: Docker (a container per job), Kubernetes (a pod per job), the autoscaling executors, and the older ones in maintenance mode (Shell, SSH, VirtualBox, Parallels, Custom). Setup: create the runner in the interface, receive a glrt- authentication token, then gitlab-runner register --url ... --token glrt-.... The old registration-token flow is deprecated and set for removal in 20.0.

A version detail that impresses: keep the runner's major and minor version in step with GitLab's. Another: Docker Machine is deprecated with removal in 20.0, replaced by the Docker Autoscaler and Instance executors.

Environments and deploys

An environment: on a job marks it as a deployment and gives you a history of what was deployed where, by which commit. when: manual adds an approval click. resource_group serialises jobs so two deploys to one environment never overlap. A protected environment restricts who can deploy, and the error for an unauthorised click is You are not authorized to run this manual job. The standard shape is automatic deploy to staging from the default branch and a manual gate for production.

The classic traps, and why they share a cause

Most beginner failures come from forgetting that every job is a fresh, isolated environment and the configuration is resolved before anything runs. Seen that way, the traps stop being a list.

Debugging order

If you are asked "a pipeline is red, what do you do", answer with an order, not a guess.

1. Was the pipeline created? If not, the configuration is invalid or workflow:rules filtered it out. Read the banner, or run glab ci lint and use the pipeline editor.
2. Is the job you expect there? If not, check rules against the real CI_PIPELINE_SOURCE and branch for this pipeline type.
3. Did the job start? A pending job is a runner-matching problem: tags, untagged setting, online status, protected or locked.
4. Where did it fail? Find the first error in the log: the image pull, the clone, or your own command.
5. Can you reproduce it? Run the same image and commands locally, with docker run, and compare.
6. Is it the environment or the code? Variables that exist on protected branches only, files not committed, and a cold cache are the usual differences.

Mention that you would use CI_DEBUG_TRACE only as a last resort because it prints every variable, including secrets, into the log.

Common interview questions

1. What is a pipeline, a stage and a job?A pipeline is one run of the configuration for one commit. It is made of stages, which run in order, and each stage holds jobs, which run in parallel. A job is a named task with a script, and it is the unit that runs on a runner.
2. What does a runner do, and does GitLab run your code?GitLab never runs your scripts. A runner polls GitLab over outbound HTTPS for jobs, executes the script through its executor, streams the log back and uploads artifacts. Because the runner initiates the connection, it works behind a firewall with no inbound ports.
3. Why does my job not see the file the previous job created?Every job starts in a fresh environment with a clean checkout. Files have to be passed as artifacts, and a later stage downloads them automatically. A cache will not do, because it is best effort.
4. Artifacts or cache?Artifacts carry results that later jobs or people need and are guaranteed until they expire. A cache carries repeated downloads for speed, may be missing, and must never be required for correctness. Key the cache on the lockfile with key:files.
5. How do you prevent duplicate branch and merge request pipelines?With a workflow:rules block: run for merge request events, return when: never for a branch with an open merge request, and allow other branch pipelines. It controls whether the pipeline exists, before any job rules.
6. What is the difference between rules and only/except?only and except are deprecated and limited. rules evaluates conditions top to bottom, supports if, changes, exists and when, and cannot be mixed with them in the same job.
7. How do you run a job only on the default branch, with a manual click?rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH, with when: manual, then a final - when: never to be explicit. Add an environment so the deployment is recorded.
8. What makes a variable safe for a production credential?Store it in project or group settings rather than the file, mark it masked and protected, use a File-type variable for certificates, and understand that masking only hides exact text. Protection to protected branches is what limits exposure. Better still, use OpenID Connect id_tokens so no long-lived secret exists.
9. What are tags for?They match jobs to runners. A job runs only on a runner that has every one of its tags, which is how you route work to GPU machines, specific operating systems or larger sizes. A mismatch leaves the job pending with the message that it is stuck.
10. A job is stuck in pending. What do you check?Whether any online runner has all the job's tags, whether the runner accepts untagged jobs, whether it is paused, protected-only or locked to another project, and, on GitLab.com Free, whether identity verification is complete.
11. I fixed the YAML and clicked retry, but it fails the same way. Why?A retry belongs to the old pipeline and reuses its frozen configuration. Push a new commit to get a new pipeline with the fixed file.
12. What is the difference between before_script and after_script?before_script runs in the same shell as the script, so its setup is visible. after_script runs afterwards, even on failure, in a separate shell, so it cannot see exports from the script. Use it for cleanup.
13. What does allow_failure do, and what is the risk?It lets a job fail without failing the pipeline. The risk is a check everyone learns to ignore, so either fix or remove such jobs.
14. How do you use a database in a test job?Add it under services: with an alias, and connect to that hostname from the job. The service container starts before the script and is discarded afterwards.
15. How do you make a pipeline faster?Cache dependencies keyed on the lockfile, run independent jobs in the same stage, use rules:changes to skip irrelevant jobs, choose lean images, set short expire_in values, and mark jobs interruptible so a newer push cancels obsolete pipelines.
16. How do you validate a pipeline before pushing?The pipeline editor, which validates as you type, or glab ci lint, with --dry-run to simulate creating the pipeline.
17. What has changed recently that teachers often get wrong?Runner registration tokens are deprecated in favour of the glrt- authentication token, only and except are replaced by rules, global image and cache moved under default:, and new interface variables are masked by default. The next breaking-change window is GitLab 20.0 in May 2027.
18. What is an environment?A named deployment target. Declaring one on a job records every deployment with its commit and pipeline, so you can see what runs where, and lets you attach protections such as manual approval and a resource group.
19. Why should the lockfile be part of the cache key?So the cache is rebuilt exactly when dependencies change. A fixed key keeps serving a stale cache, and no key means no sharing.
20. How would you run the same pipeline config for a Python and a Node job?Set image per job, or set a default and override it in the one job that differs. Use extends with a hidden job to share everything else.

Sixty-second self-test

• Say what a pipeline, stage, job and runner are, in one sentence each.
• Explain why GitLab does not run your code and what "polls" means for firewalls.
• Explain why a file from one job is missing in the next, and the fix.
• Give the artifacts-versus-cache answer, including the cache key.
• Write the workflow:rules block from memory and say what problem it solves.
• Write a rules entry that runs a manual job only on the default branch, with correct if syntax.
• List what masked, protected and File-type variables do, and why masking is not a boundary.
• Give four reasons a job is stuck pending.
• Say why retrying after a YAML fix does not help.
• Name three outdated idioms and their replacements.
• Describe your debugging order in six steps.

Part one of three. Practical help for your first months with GitLab CI/CD: the mistakes beginners actually make, the habits worth adopting on day one, and a checklist to copy. Everything was checked against GitLab 19.4 and GitLab Runner 19.4.1.

Common errors at this level

Each entry gives the symptom, the cause and the fix. They are ordered roughly by how often they happen.

The job stays pending and never starts. The job page says This job is stuck because you don't have any active runners that can run this job. The cause is that no online runner has every tag the job asks for, or that the only runners refuse untagged jobs, or that the runner is paused, locked to another project, or set to run protected refs only. Fix it from Settings > CI/CD > Runners: compare the job's tags: with the runner's tags, turn on Run untagged jobs if you meant to, and look for a green Online badge. On GitLab.com Free, also check that identity verification is complete, since without it jobs fail with Identity verification is required in order to run CI jobs.

The pipeline was never created. You see This GitLab CI configuration is invalid: jobs:test-job:artifacts config contains unknown keys: path. The message is precise: job test-job, key artifacts, unknown key path. The fix is usually one letter (paths) or a wrong indent. Check before you push with the pipeline editor or glab ci lint.

A file from the previous job is missing. cat: dist/VERSION: No such file or directory in a later job. The cause is that every job starts in a fresh environment and you did not declare the file as an artifact. Add artifacts: paths: [dist/] to the producing job. Do not reach for a cache, because a cache is not guaranteed to exist.

The pipeline only passes when the cache is warm. The cause is using a cache for something the job needs to be correct, or a cache key so broad that stale content is served. The fix is to make the job work from empty (pip install -r requirements.txt always runs, and the cache only makes it faster) and to key the cache on the lockfile.

Two pipelines on every push. One is a branch pipeline and the other a merge request pipeline, because job rules do not separate them. GitLab warns Job may allow multiple pipelines to run for a single action. Add the standard workflow:rules block at the top of the file.

The job you expected is not in the pipeline. The configuration is valid, but rules excluded the job for this pipeline type. A job restricted to $CI_COMMIT_BRANCH does not appear in a tag pipeline, because CI_COMMIT_BRANCH is empty there. Open Full configuration in the pipeline editor and compare your conditions with the real values of CI_PIPELINE_SOURCE and the ref.

"I fixed it and retried, but it still fails." A retry reuses the pipeline's frozen configuration. Push a new commit, or run a new pipeline with Build > Pipelines > New pipeline or glab ci run -b main.

command not found. The image does not contain the tool. python: not found on an alpine image, or npm: command not found on a Python image. Choose an image that has it, or install it in the script.

An if: condition that never matches. The syntax is strict: an unquoted variable compared with a quoted string, $CI_COMMIT_BRANCH == "main". "$CI_COMMIT_BRANCH" == "main", ${CI_COMMIT_BRANCH} == "main" and $CI_COMMIT_BRANCH == main are all wrong. Regular expressions use =~ and slashes.

A variable is empty or has the wrong value. It may be protected and you are on an unprotected branch; it may be scoped to another environment; the same name may be defined in several places with different values; or Expand variable reference is off (the default for interface variables since 18.6) and a $ in the value is kept literally. Search every place the name could be defined before debugging anything else.

A secret appears in the log. Masking hides only an exact match, so a value that was transformed, encoded or split prints in clear. The value also cannot be masked at all unless it is a single line of at least eight characters with no spaces, so a short or multi-line secret shows in full. Revoke and replace the secret, then remove the line that printed it.

ERROR: Job failed: exit code 1 and nothing else useful. That last line is always the same. The evidence is above it. Scroll up to the first error, expand the collapsed sections, and find the command that returned non-zero.

The job timed out. ERROR: Job failed: execution took longer than ... seconds. The project default is 60 minutes, and GitLab.com hosted runners have a hard three-hour cap. Decide whether the job hung (a command waiting for input that never comes) or is honestly long, and only then raise timeout:.

A job cannot pull its image. Failed to pull image ... denied: requested access to the resource is denied means a private image without credentials, or a project whose job token is not allowed to pull from another project's registry. Authenticate to the registry, or add the consuming project to the image project's job token allowlist.

A manual job refuses to run. You are not authorized to run this manual job means a protected environment, or a rule that only lets certain people deploy. Ask to be added to "Allowed to deploy", or deploy from the account that is.

A runner fails to register. Check registration token or 410 Gone - runner registration disallowed is the old flow. Create the runner in the web interface first and register with --token glrt-..., not --registration-token.

x509: certificate signed by unknown authority. A self-managed GitLab or a proxy uses a private certificate authority that the runner does not trust. Put the authority's certificate where the runner looks for it (/etc/gitlab-runner/certs/ named after the host, ending in .crt) and restart the runner.

The habits that pay off most

These are in order of leverage. The first three will save you more time than all the rest together.

1. Validate before you push. Use the pipeline editor, or glab ci lint --dry-run. A typo caught in five seconds beats a failed pipeline in five minutes.
2. Read the first error. In a job log, in a configuration banner, anywhere. The last line of a failed job is boilerplate.
3. Start every project with the workflow block. Copy the three-rule workflow:rules block once and forget about duplicate pipelines.
4. Pin your images. Use python:3.12, not latest. A pipeline should fail because of your change, not because an image moved.
5. Key caches on lockfiles. key:files with requirements.txt or package-lock.json.
6. Set expire_in on every artifact. Storage you forgot about still counts.
7. Keep scripts in files. Long shell in YAML cannot be linted or run locally. A scripts/deploy.sh can.
8. Use default:, rules and runner authentication tokens. Avoid the deprecated top-level image, only/except and registration tokens, so you do not learn habits you must unlearn.
9. Mark jobs interruptible: true. A newer push then cancels the obsolete pipeline and saves compute minutes.
10. Protect the things that matter. Protect main, protect production variables, and protect the production environment where your plan allows it.
11. Make your pipeline run the same commands you run locally. When CI and your laptop differ, bugs hide in the gap. A make test or a scripts/test.sh used by both is worth the small effort.
12. Treat .gitlab-ci.yml as production code. Small commits, review, and a descriptive message when you change it.

Practice cards

Short, self-contained exercises. Each takes a few minutes and ends in a result you can check on screen.

1. Prove jobs are isolatedMake job A run echo hi > note.txt in stage one and job B cat note.txt in stage two. Watch B fail. Add artifacts: paths: [note.txt] to A and watch B pass.
2. Watch a cache workAdd a pip cache keyed on requirements.txt. Run twice and compare the install time. Change the requirements and watch the key change.
3. Cause the stuck message on purposeGive a job tags: [does-not-exist]. Read the message. Then read your runner's real tags and correct the job.
4. Create and cure duplicate pipelinesOpen a merge request on a project with job-level rules and no workflow block, push twice, and count pipelines. Add the workflow block and count again.
5. Make a variable leak, then stop itAdd a masked variable and print it with echo "$VAR" | sed 's/./& /g'. See it spelled out. Now make it protected and run on a feature branch to see it empty.
6. Break the YAML three waysMisspell a key, mis-indent a block, and use "$VAR" == "x" in a rule. Run glab ci lint each time and note how the message reads.
7. Build a manual gateAdd a deploy job that is when: manual on the default branch only, with an environment. Click play, then open the Environments page.
8. Reproduce a CI failure locallyCopy a failing job's image and script into docker run --rm -it -v "$PWD":/work -w /work IMAGE bash and run it by hand.

Reading a job log

A log has the same structure every time, and knowing it makes you fast. The top is setup: the runner and executor, the image being pulled, the repository being cloned at the pipeline's commit, and any cache being restored. Then come your commands. Each command is printed on a line starting with $, followed by its output. After the script, you see the cache being saved and artifacts being uploaded, and finally Job succeeded or an ERROR: line.

Where failures hide, from the top:

• Setup failures happen before your first $ line. A failed image pull, a clone error (fatal: repository ... not found), or a missing service. These are environment or permission problems, not code problems.
• Command failures show the failing command with its own error output right above the final ERROR: Job failed: exit code N. The number is the command's exit code: 1 is a general failure, 2 often means misuse of a command, 126 is found-but-not-executable (a script without the executable bit), and 127 is command not found.
• Upload failures come after your script: WARNING: dist/: no matching files means the artifact path did not exist. The job may still pass, but later jobs will lack the file.

A small trick: in the job page, use the Raw link to get the log as plain text, and the search box on the page to jump to ERROR or WARNING. Warnings often explain a later failure, such as a cache that never saved or an artifact path that matched nothing.

Running pipelines on demand

Pipelines do not only start from pushes, and knowing the other sources saves you from clicking around.

• Run pipeline (Build > Pipelines > New pipeline) starts a pipeline on a branch you choose, and lets you type variable values. From a terminal, glab ci run -b main does the same.
• Schedules (Build > Pipeline schedules) run a pipeline on a cron expression in a time zone you set. A scheduled pipeline has CI_PIPELINE_SOURCE equal to schedule, and runs as the user who owns the schedule, so use a rule such as if: $CI_PIPELINE_SOURCE == "schedule" to give nightly jobs their own behaviour.
• Manual jobs wait for a click. Inside a pipeline, they are the approval gates.
• Merge request pipelines run for every update of an open merge request, when your workflow rules allow it.
• Retrying a single failed job is possible from the job page, and saves the time of re-running the passed ones, but remember that it uses the frozen configuration.

For an MLOps team, schedules are especially useful: a nightly job that re-validates a dataset, a weekly job that retrains a model on a small sample, or a daily check that an external data feed still has the expected columns. Give those jobs an explicit rules entry for schedule so they do not also run on every push.

Naming and organising the file

A pipeline is read far more often than written, so a few conventions pay back quickly.

Name jobs for what they do, using lowercase words joined with hyphens: unit-tests, build-image, deploy-staging. A name like job1 tells the next reader nothing, and the job names are what people see in the graph, in merge request checks and in notification messages.

Order the file the way the pipeline flows. Put default, stages, variables and workflow at the top, then hidden template jobs, then jobs in stage order. Someone scanning the file should be able to follow the pipeline from top to bottom.

Keep one concern per job. A job that lints, tests and builds is hard to read, hard to retry, and gives poor feedback. Three small jobs run in parallel, fail separately, and show you exactly what broke.

Comment the why, not the what. # pinned: 3.13 breaks our numpy build helps. # install dependencies above pip install does not.

Split large files early. When .gitlab-ci.yml passes a couple of hundred lines, move groups of jobs into files under ci/ and pull them in with include: - local: /ci/tests.yml. Remember that paths start with / from the repository root, and that a wrong path gives Local file ... does not exist!.

Saving compute minutes

On GitLab.com, hosted runners spend compute minutes from a monthly allowance, and a busy team can run out. Self-hosted runners cost no GitLab minutes, but they cost machines and time. Either way, waste is worth removing, and most of it is easy.

1. Cancel what nobody needs. Add interruptible: true under default: so a new push cancels the older pipeline on the same branch.
2. Skip irrelevant jobs. Use rules:changes so a documentation edit does not run the whole test suite. Be careful on branch pipelines without a merge request, where "changed" is measured against the previous push.
3. Cache dependencies with a lockfile key.
4. Choose small images. A slim Python image pulls faster than a full one, and the time to pull counts.
5. Run independent jobs in the same stage so they run side by side rather than one after another.
6. Set timeout: on jobs that can hang, so a stuck job does not burn an hour.
7. Expire artifacts you do not need, which saves storage rather than minutes.
8. Retry sparingly. retry: 2 on a flaky test triples the cost of a real failure and hides the flake. Fix flaky tests.

Security habits for beginners

You do not need to be a security engineer to avoid the common mistakes.

• Never commit a secret. If one reaches the repository, rotate it immediately. Deleting the file in a later commit does not remove it from history.
• Store credentials as masked, protected variables and use File type for certificates.
• Prefer the job token (CI_JOB_TOKEN) and the built-in CI_REGISTRY_USER and CI_REGISTRY_PASSWORD over personal access tokens, because they expire with the job.
• Read pipeline changes in merge requests. A .gitlab-ci.yml change can run any command with access to whatever secrets the pipeline can reach. Review it like code that will execute, because it will.
• Do not attach a shell-executor runner to a project that accepts outside contributions, since the job runs as a real user on the machine.
• Avoid CI_DEBUG_TRACE. It prints every variable into the log.
• Know your data rules. Teams working for Gulf and Egyptian employers often have data-residency requirements. If your jobs touch production data, find out where the runners run and where artifacts and caches are stored before you build the pipeline, because GitLab.com hosted runners are not in your country by default and a self-managed runner in the right region may be required.

A starter setup to copy

A reasonable beginning for a Python project. Copy it, change the commands, and grow it:

default:
  image: python:3.12
  interruptible: true

stages: [test, build, deploy]

variables:
  PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"

workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
      when: never
    - if: $CI_COMMIT_BRANCH

.py:
  cache:
    key:
      files: [requirements.txt]
    paths: [.cache/pip]
  before_script:
    - pip install -r requirements.txt

lint:
  extends: .py
  stage: test
  script:
    - pip install ruff
    - ruff check .

unit-tests:
  extends: .py
  stage: test
  timeout: 15 minutes
  script:
    - pytest -v --junitxml=report.xml
  artifacts:
    when: always
    reports:
      junit: report.xml
    expire_in: 1 week

build:
  stage: build
  script:
    - mkdir -p dist
    - cp app.py dist/
    - echo "$CI_COMMIT_SHA" > dist/VERSION
  artifacts:
    paths: [dist/]
    expire_in: 1 week

deploy:
  stage: deploy
  resource_group: production
  environment: production
  script:
    - ./scripts/deploy.sh
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: manual

Notice the choices. The image is pinned, the workflow block prevents duplicates, the cache is keyed on the lockfile, the tests have a timeout and publish a report even when they fail, artifacts expire, and the production deploy is manual and serialised. Nothing in it is clever, and that is the point: a boring pipeline is one you can debug at the end of a long day.

The day-one checklist

Copy this into your project's README or your notes and tick it off for each new pipeline.

• .gitlab-ci.yml is at the repository root and the name is exact.
• A runner is available and online, and I know its tags.
• Every job has an explicit image, directly or through default:, and it is pinned.
• A workflow:rules block prevents duplicate pipelines.
• Jobs use rules, not only and except.
• Anything a later job needs is an artifact; anything only for speed is a cache.
• The cache key is built from the lockfile.
• Every artifact has expire_in, and test reports use when: always.
• Secrets are masked, protected variables, and none are in the repository.
• Production deploys are manual, tied to an environment, and use a resource group.
• I validated the file with the pipeline editor or glab ci lint before pushing.
• Long scripts live in files I can run locally.
• Nothing in the pipeline uses a deprecated feature: registration tokens, only/except, top-level image or cache.
• I know how I would reproduce a failing job locally.

When you can tick every box without looking anything up, you are ready for the Mid-level guide.

This is part two of three. Beginner got you to a green pipeline: a .gitlab-ci.yml with stages, a cached install, a JUnit report and a manual deploy. This level is about the pipeline you inherit at work, the one with forty jobs, three include: lines, a runner nobody remembers registering, and a bill for compute minutes that somebody in finance has started asking about. The gap between "it runs" and "it runs well" is almost entirely a matter of knowing when GitLab makes each decision: which decisions are taken on the server when the pipeline is created, which are taken on the runner while the job runs, and which are taken by you, by hand, at three in the afternoon on a Friday.

This guide is written against GitLab 19.4 (released 17 September 2026) and GitLab Runner 19.4.1. The 19.5 release is due on 15 October 2026, and anything the documentation already lists for 19.5 is marked here as upcoming. CI changed a great deal between GitLab 16 and 19, so a good share of what you find in old blog posts, Stack Overflow answers and inherited pipelines is now deprecated or wrong. Where that is the case, the guide says so and shows the current form.

Where this picks up

Every section opens with the problem it solves and closes with a Try it you can run against a real project in a few minutes.

How a pipeline is born

The first thing to settle is what actually happens between git push and the first job starting, because almost every confusing behaviour at this level is a consequence of it.

When something creates a pipeline (a push, a merge request event, a schedule, the API, a trigger token, an upstream pipeline) GitLab's Rails application does the following on the server, before any runner is involved:

1. It picks the CI configuration file (.gitlab-ci.yml by default, or a custom path) at the exact commit the pipeline is for.
2. It resolves every include, fetching local, project, remote, template and component files, and merges them into one document.
3. It interpolates inputs and component context ($[[ ... ]]), because those are create-time values.
4. It evaluates workflow:rules to decide whether the pipeline exists at all.
5. It evaluates each job's rules, only/except, and needs to decide which jobs are created.
6. It stores the result. From this point the configuration is frozen for that pipeline.

Two practical consequences follow. First, retrying a job re-uses the old configuration. If you fixed a typo in .gitlab-ci.yml and clicked retry on the failed job, you are retrying the pipeline as it was defined when it was created, and you will watch the same failure with the fix sitting unused in the repository. To pick up a change you need a new pipeline. Second, there is a clean split between create time and run time. Anything decided by rules, include, inputs or workflow is settled on the server; anything to do with $VARIABLE expansion inside script, the file system, the network and the container is settled on the runner. When a rule "does not see" a value, it is usually because the value only comes into existence at run time.

The pipeline source is worth memorising as a concept, because it drives most decisions. Every pipeline records why it exists in CI_PIPELINE_SOURCE: push, merge_request_event, schedule, web (the Run pipeline button), api, trigger, pipeline (a multi-project downstream), parent_pipeline (a child), chat, external, security_orchestration_policy, and a few more. The kinds of pipeline you will meet are:

• Branch and tag pipelines, created by pushes.
• Merge request pipelines, created by merge request events and running on the source branch. Their CI_PIPELINE_SOURCE is merge_request_event, and only they have CI_MERGE_REQUEST_* variables such as CI_MERGE_REQUEST_IID.
• Merged results pipelines (Premium), which run on a temporary commit that merges the source branch into the target, so you test what would actually land.
• Merge trains (Premium), queues of merged results pipelines. By default up to 20 run in parallel per train, and since 19.0 that limit is configurable.
• Scheduled pipelines, which run as the schedule's owner.
• Downstream pipelines, either parent-child inside one project or multi-project across projects.

The duplicate pipeline problem

The classic trap at this level is the project where every push to a branch with an open merge request creates two pipelines, one branch pipeline and one merge request pipeline, running the same jobs. It happens whenever jobs have rules that match both sources, or no rules at all while some jobs use rules:if: $CI_PIPELINE_SOURCE == "merge_request_event". GitLab even warns you in the editor: Job may allow multiple pipelines to run for a single action.

The fix is one workflow:rules block at the top of the file, and every project should have one:

workflow:
  name: '$CI_PIPELINE_SOURCE on $CI_COMMIT_REF_NAME'
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
      when: never
    - if: $CI_COMMIT_BRANCH
    - if: $CI_COMMIT_TAG

Read it top to bottom, first match wins. A merge request event creates a pipeline. A branch push where a merge request is already open is refused (it would duplicate the MR pipeline). Any other branch push, and any tag push, creates a pipeline. If nothing matches, the pipeline is not created, and the merge request widget tells you so with Pipeline filtered out by workflow rules. That message is not an error; it is your own workflow:rules doing exactly what you wrote.

workflow:name gives each pipeline a readable title in the pipeline list, which pays for itself the first time you scan twenty runs looking for the scheduled one.

Limits that shape the design

Some limits are worth knowing now, because they shape how you structure a pipeline later. A single git push creates at most four branch or tag pipelines by default (GitLab 18.0 introduced that). A pipeline can include at most 150 files, each up to 2 MB, and a job can have at most 50 needs and 4 caches. These are instance limits, so self-managed admins can change them, but on GitLab.com they are what you get. You will not hit most of them by accident; the ones you do hit are usually a sign that a single pipeline is doing the work of several.

Rules, precisely

Beginner showed rules:if. Underneath, rules is an ordered list evaluated per job at pipeline creation: the first rule that matches decides, and its attributes (when, allow_failure, variables, needs) apply to the job. If no rule matches, the job is not added. This is the most important fact to hold onto: rules answers "does this job exist in this pipeline", once, on the server. It cannot react to anything that happens while the pipeline runs.

The keyword only/except is deprecated in every form. It still works, but it is harder to reason about than rules, and it cannot be mixed with rules in the same job. Convert old pipelines when you touch them, not in a mass rewrite.

The clauses

if is an expression over variables. The syntax is strict. An unquoted variable on the left compares with a quoted string on the right: $ENV == "production". All of "$ENV" == ..., ${ENV} == ... and $ENV == production are invalid. =~ and !~ take a regular expression, written between slashes: $CI_COMMIT_BRANCH =~ /^feature\//. You can combine conditions with &&, || and parentheses. A bare variable (if: $DEPLOY) means "defined and not empty".

changes matches if any file in a list of paths changed, and is how you skip the frontend build when only docs changed:

build-frontend:
  stage: build
  script: npm run build
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
      changes:
        paths:
          - frontend/**/*
          - package-lock.json
        compare_to: refs/heads/main
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

The subtlety is what changes compares against. In a merge request pipeline it diffs the merge request's changes. In a branch pipeline without a merge request it compares with the previous push, which on a new branch, a force push or a tag can mean "everything changed" and therefore "run the job". compare_to makes the comparison explicit, and is the fix for the puzzling case where a docs-only change still runs the expensive jobs. Since 19.2 there is also rules:changes:regexp, which takes a Ruby regular expression of up to 255 characters; it is mutually exclusive with paths: in the same clause.

exists matches if a file exists in the repository, which is how a shared template chooses a build tool: exists: [Dockerfile]. rules:exists:regexp arrived in 19.2 with the same constraints as the changes form.

when sets the outcome of a match: on_success (the default), on_failure, always, manual, delayed (with start_in: 30 minutes) or never. allow_failure marks a matched job as non-blocking, and works with when: manual in a rule to make an optional deploy button that does not hold up the pipeline's status. rules:variables sets variables only for that match, which is the tidy way to express "the same job, different target":

deploy:
  stage: deploy
  script: ./deploy.sh "$TARGET"
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      variables: { TARGET: staging }
    - if: $CI_COMMIT_TAG
      variables: { TARGET: production }
      when: manual

Reading rule failures

When a job you expected is missing, the usual cause is that no rule matched at creation. The pipeline editor's Validate tab lints the file against the server, and the lint API has a dry_run mode that simulates pipeline creation. The CLI route is glab ci lint --dry-run --ref main --include-jobs, which lints the current file with the real server and lists the jobs it would produce. Reach for these before you push a dozen commits to "see if the rule works".

Stages, the DAG, and wall-clock time

A stage-based pipeline has a hidden cost: the next stage waits for the slowest job of the previous one. If test holds a thirty-second lint and a twelve-minute integration suite, then deploy-review cannot start for twelve minutes even though it only needed the image built two stages earlier. Stages are a simple way to express order, and they overconstrain.

needs replaces the stage barrier with an explicit dependency graph (a DAG, a directed acyclic graph). A job with needs starts the moment the jobs it names have finished, regardless of stage. A job with needs: [] starts immediately, at pipeline creation, even in a late stage.

stages: [build, test, package, deploy]

build:
  stage: build
  script: make build
  artifacts:
    paths: [dist/]
    expire_in: 1 day

unit:
  stage: test
  needs: [build]
  script: make unit

lint:
  stage: test
  needs: []              # starts at once, needs nothing
  script: make lint

integration:
  stage: test
  needs: [build]
  script: make integration

package:
  stage: package
  needs:
    - job: build
      artifacts: true
    - job: unit          # wait for the fast gate, not for integration
      artifacts: false
  script: make package

Notice what needs also does to artifacts. By default, a job receives the artifacts of all jobs in earlier stages. With needs, it receives artifacts only from the jobs it names, and you can opt out per need with artifacts: false when you want ordering without the download. That is a real performance lever: a job that only needs to wait for a gate should not pull 400 MB of build output. The stage-based alternative is dependencies:, which limits artifact downloads but does not change ordering.

The forms you will use are needs: [job], the object form with artifacts and optional, needs:project for artifacts from another project's pipeline, needs:pipeline:job for artifacts from a parent pipeline in a child, and needs:parallel:matrix to depend on specific permutations of a matrix job. A job can have at most 50 needs on a default instance. Note that needs:pipeline:job:optional is documented for 19.5 and is not in a stable release yet.

Fan-out with parallel

parallel: N runs N copies of a job (1 to 200). Each copy receives CI_NODE_INDEX (starting at 1) and CI_NODE_TOTAL, and it is your test runner's job to split work accordingly. Most ecosystems have a sharding flag: pytest --splits "$CI_NODE_TOTAL" --group "$CI_NODE_INDEX" with the pytest-split plugin, for instance. The matrix form runs one job per permutation of variables, at most 200:

test:
  stage: test
  image: python:$PY
  parallel:
    matrix:
      - PY: ["3.11", "3.12", "3.13"]
        DB: [postgres, sqlite]
  script:
    - pip install -r requirements.txt
    - pytest --db "$DB"

That produces six jobs, each visible under a name like test: [3.11, postgres]. A downstream job can depend on just some of them with needs:parallel:matrix. Two warnings. First, every permutation costs a full runner allocation, image pull and checkout, so a matrix of six that each take 40 seconds of setup and 20 of work is mostly paying for setup. Second, parallelism only helps if the runner fleet has capacity: if concurrent on your runners is 4, a parallel: 20 job queues sixteen of them and you have reshaped the pipeline without making it faster.

Cancelling work that no longer matters

The cheapest job is the one that never runs. When a developer pushes three commits in a minute, the first two pipelines are obsolete. Two settings let GitLab cancel them safely:

workflow:
  auto_cancel:
    on_new_commit: interruptible
    on_job_failure: all

default:
  interruptible: true

deploy-production:
  interruptible: false
  script: ./deploy.sh

on_new_commit accepts conservative (the default: cancel only if no job has started that is not interruptible), interruptible (cancel only the interruptible jobs), and none. on_job_failure: all stops the rest of a pipeline the moment one job fails, instead of letting ten more jobs run for a result nobody will look at. The discipline is to mark build and test jobs interruptible: true through default, and to mark anything with side effects (deploys, publishing, migrations) interruptible: false. A deploy job that gets cancelled halfway through is worse than one that finishes.

Reuse without chaos: extends, anchors, include

A mature pipeline is mostly repetition. Every service has the same lint job, the same image build, the same deploy skeleton, and the danger is not repetition itself but repetition that drifts. GitLab gives you three mechanisms, from local to organisation-wide, and they do not all merge the same way.

extends and hidden jobs

A job whose name starts with a dot is hidden: it never runs, and exists to be extended.

.python-job:
  image: python:3.12
  before_script:
    - pip install -r requirements.txt
  cache:
    key:
      files: [requirements.txt]
    paths: [.cache/pip]
  variables:
    PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"

unit:
  extends: .python-job
  script: pytest tests/unit

typecheck:
  extends: .python-job
  script: mypy src
  variables:
    MYPY_CACHE_DIR: .mypy

extends does a deep merge of hashes and a replace of everything else. variables is a hash, so typecheck ends up with both PIP_CACHE_DIR and MYPY_CACHE_DIR. script, before_script and rules are arrays, so if the child defines its own, it replaces the parent's entirely; arrays are not concatenated. That single rule explains most "why did my before_script disappear" questions. You can extend several parents (extends: [.base, .with-cache]) and the later ones win conflicts.

!reference for array fragments

When you want to keep the parent's array and add to it, !reference lets you pull in a specific piece of another job:

.setup:
  script:
    - echo "configuring"
    - ./configure.sh

integration:
  script:
    - !reference [.setup, script]
    - pytest tests/integration

This is also how you share a rules block between jobs that need different extra rules. The restriction to remember is that multiple !reference tags inside one array work only in script, rules and stages; anywhere else (for example inside parallel:matrix) you get config should be an array of hashes. YAML anchors (&name and *name) are the third mechanism. They work but only inside a single file, and they cannot be used across include boundaries, which is precisely where you want reuse, so prefer extends and !reference.

default: and the deprecated globals

Top-level image, services, cache, before_script and after_script are deprecated in favour of the default: section. They still work, and you will see them in old files, but new pipelines should write:

default:
  image: python:3.12
  interruptible: true
  retry:
    max: 1
    when: [runner_system_failure]

A job that sets its own value for a keyword overrides the default for that keyword; inherit:default: false turns defaults off for a job entirely. Top-level variables are not part of default; they remain a separate section.

include in five forms

include merges other files into the configuration at creation time. The forms:

include:
  - local: /ci/*.yml                       # from this repo; wildcards allowed
  - project: platform/ci-templates         # from another project
    ref: v4.2.0
    file: /templates/docker.yml
  - remote: https://example.com/ci/shared.yml
    cache: 1 day                           # TTL cache, GA since 19.0
  - template: Jobs/SAST.gitlab-ci.yml      # GitLab-provided templates
  - component: $CI_SERVER_FQDN/platform/ci-components/docker-build@1.4.0

include:local paths are relative to the repository root and start with /; get that wrong and you see Local file <file> does not exist!. include:project needs a ref, and you should always set one. Without it, the include follows the other project's default branch, and someone else's commit changes your pipeline. Pin to a tag or a SHA for anything that matters. A wrong project path gives Project <group/project> not found or access denied, which is also what you get when the triggering user lacks access, so check membership before you check typos. include:remote fetches over HTTP on every pipeline creation unless you set cache:; it is the least trustworthy form, since you have no version and no access control.

The merge order matters: the file containing the include wins over the files it includes, which is how you override one job from a shared template. You override by redefining just the keys you want changed, because the merge is deep for hashes:

include:
  - template: Jobs/SAST.gitlab-ci.yml

semgrep-sast:
  variables:
    SAST_EXCLUDED_PATHS: "tests, docs"

You can also condition includes with include:rules (if, exists, changes), and pass parameters with include:inputs. The limits: at most 150 includes (Maximum of 150 nested includes are allowed!), 2 MB per file. If the error is The parsed YAML is too big, you have an oversized file or one nested too deeply (depth 100), and the answer is child pipelines and moving shell logic into scripts in the repository.

Inputs, components, and job inputs

include and extends share text. They do not give you parameters or versions. That is the gap that CI/CD inputs and components fill, and they are the modern answer to the "shared template with twelve magic variables" problem.

Inputs: typed parameters at creation time

A file can declare a spec:inputs header, then a --- separator, then the jobs. Inputs are interpolated while the pipeline is being created, so they can change anything in the YAML: a job name, a stage, a rules condition, an image tag.

spec:
  inputs:
    stage:
      default: deploy
    env:
      options: [staging, production]
    replicas:
      type: number
      default: 2
    regions:
      type: array
      default: [eu-west-1]
    version:
      regex: '^v\d+\.\d+\.\d+$'
---
deploy-$[[ inputs.env ]]:
  stage: $[[ inputs.stage ]]
  script: ./deploy --env $[[ inputs.env ]] --replicas $[[ inputs.replicas ]]

The types are string (the default), number, boolean and array. An input with no default is required. options restricts to a list, regex validates, and spec:inputs:rules gives conditional options and defaults. Interpolation functions transform a value in place: $[[ inputs.version | truncate(0,8) ]], plus expand_vars, posix_escape and split. Since 19.0 array inputs also support an index, $[[ inputs.regions[0] ]].

The property people miss is that inputs are not environment variables. They do not appear in env inside the job, they take no part in variable precedence, and they cannot be overridden by a stray variable in the UI. That is exactly why GitLab steers you toward them: since 17.7 passing pipeline variables to trigger a pipeline is discouraged, and on GitLab.com new projects in new namespaces default the minimum role for setting pipeline variables to no_one_allowed, after which you see Insufficient permissions to set pipeline variables. Inputs replace that mechanism with something typed and validated.

The same header works for the main file. Pipeline-level inputs (17.11 and later) are declared in .gitlab-ci.yml itself, and appear as fields in the Run pipeline form, in glab ci run --input 'env:staging' and in the API as {"inputs": {...}}. One hard rule: branch and merge request pipelines cannot supply inputs, so every pipeline-level input must have a default, or those pipelines fail. At most 20 inputs are allowed per pipeline.

Components: versioned, published, discoverable

A component is an input-parameterised template stored in a project's templates/ directory (either templates/<name>.yml or templates/<name>/template.yml), with a README, and referenced by an address that includes a version:

include:
  - component: $CI_SERVER_FQDN/platform/ci-components/docker-build@1.4.0
    inputs:
      image_name: $CI_REGISTRY_IMAGE
      context: services/api

Version resolution is worth knowing exactly, because it decides how much you trust a provider. A reference can be a commit SHA, a tag, a branch name, or (for published catalog releases) ~latest or a partial semantic version such as 1 or 1.4. The partial forms resolve only against published catalog releases, and never pick pre-releases. If no release exists you get Component '<...>@~latest' - content not found, which really means "nothing has been published yet". For production pipelines pin a full version or a SHA. A SHA is the most conservative; a full semver tag is the practical balance.

To publish your own, you mark the project as a CI/CD Catalog resource in Settings, give it a description and a README, and run a tag pipeline with a release: job. The Releases API alone does not publish to the catalog. A project can hold at most 100 components (raised from 30 in 18.5). Inside a component, spec:component: [name, sha, version, reference] lets the template read its own coordinates through $[[ component.version ]], which is handy for pulling a matching container image tag. One error worth recognising: an empty spec:inputs: fails with Build component error: Spec must be a valid json schema; use an empty spec: instead.

On a self-managed instance, GitLab.com's public catalog is not automatically available; you need to mirror the component project. Also worth knowing: 19.4 added a flag-gated behaviour (ci_spec_include_own_context, off by default) that changes how spec:include resolves, so do not rely on it yet.

Job inputs, the run-time cousin

GitLab 18.10 added job-level inputs:, which are not interpolated at creation but evaluated by the runner when the job starts, using a different expression syntax, ${{ job.inputs.NAME }}. Their selling point is that a person can override them when manually running or retrying that single job, which makes one-off operations ("re-run this deploy, but with dry_run=true") possible without a new pipeline. Keep the three syntaxes straight:

Treat the run: keyword (GitLab Functions, formerly CI Steps) as an experiment. The docs say so, and this guide does not build on it.

Variables: precedence, scope, and what masking does not do

Variables are where CI configuration and operational reality meet, and so where most incidents start. The mental model is three layers: where a value is defined, which jobs can see it, and who can see it in the output.

Precedence

When the same name is defined in several places, the highest wins. From the top:

1. Security policy variables (pipeline execution and scan execution policies, Ultimate)
2. Variables on a manual job, typed when you click play
3. Pipeline variables: the Run pipeline form, a schedule, the API, a trigger, a push option, an upstream pipeline
4. Project variables
5. Group variables (the nearest subgroup wins)
6. Instance variables
7. Dotenv report variables from earlier jobs
8. YAML variables: rules:variables, then the job's variables, then workflow:rules:variables, then top-level variables
9. Deployment variables
10. Predefined variables

The surprising line is number 4 above 8: a variable set in the project UI beats the same name in the YAML. That is usually what you want (the UI holds the real secret, the YAML holds a harmless default) and it is a nasty surprise when someone added DEPLOY_ENV in settings two years ago and your YAML override never takes effect. When a value is "wrong", print where it came from by checking each layer in order; do not guess.

The four flags

A UI variable can be protected (only exposed to pipelines on protected branches or tags), masked (hidden in job logs), masked and hidden (also hidden in the settings UI after creation), and of file type. Environment scope (production, staging/*) restricts a variable to jobs that deploy to a matching environment.

Protected is the flag that matters most for security, and it has a trap: a protected variable is simply absent in an unprotected pipeline. The symptom, a deploy job that works on main and fails on a feature branch with an empty credential, is a feature, not a bug. Since 18.3 new UI variables are created masked by default, and since 18.6 the "Expand variable reference" option is off by default for them.

Masking has rules the value must satisfy: a single line, at least 8 characters, and (when expansion is on) limited to letters, digits and _ : @ - + . ~ = /. A value that cannot be masked is rejected when you save it, and that is better than finding out in a log.

File variables and dotenv hand-offs

A file-type variable writes the value to a temporary file and sets the variable to its path. Use it for things like kubeconfigs and certificates, where you need a file and where multi-line values do not survive environment variables gracefully. The shape to learn: kubectl --kubeconfig "$KUBECONFIG_FILE" ..., not echo "$KUBECONFIG" > kc.

To pass a computed value from one job to a later one, write a dotenv file and publish it as a report:

build:
  stage: build
  script:
    - echo "IMAGE_TAG=$CI_COMMIT_SHORT_SHA-$(date +%Y%m%d)" >> build.env
  artifacts:
    reports:
      dotenv: build.env

deploy:
  stage: deploy
  needs:
    - job: build
      artifacts: true
  script:
    - ./deploy.sh "$IMAGE_TAG"

The value arrives in the dependent job as an ordinary variable, at precedence 7, which is below UI and pipeline variables. That ordering means a dotenv variable can never override a protected project variable of the same name, which is the sensible safety property.

Expansion and raw values

Variables can reference each other (URL: "https://$HOST/api"), and GitLab expands them when building the job. When a value legitimately contains a dollar sign (a password, a regex) you can mark a YAML variable expand: false so it is passed through unchanged:

variables:
  REGEX:
    value: '^v[0-9]+\$'
    expand: false
  DEPLOY_TARGET:
    description: "Where to deploy on manual runs"
    value: staging
    options: [staging, production]

The description, value and options fields also pre-fill the Run pipeline form, which turns a variable into a documented, constrained parameter for manual runs. For anything typed, prefer inputs; use this for legacy cases where variables are required.

Cache and artifacts: two different contracts

Beginner introduced both with one sentence each. The mistakes at this level come from treating them as interchangeable, so here is the contract of each stated plainly.

An artifact is a guaranteed hand-off. A job uploads files to GitLab; later jobs (and humans) can download them. It is stored by GitLab, it is tied to that pipeline, and it expires on a schedule. If a later job needs a file produced by an earlier one to be correct, it is an artifact.

A cache is a best-effort speed-up. The runner stores a directory (locally, or in S3, GCS or Azure object storage when configured) and tries to restore it next time. It may be missing, stale, or on a different runner host than the last job used. A job must succeed, slowly, with an empty cache. If your build breaks when the cache is cold, you have put correctness in the wrong place.

Cache keys and policies

The key decides who shares a cache. A fixed key means every branch and job shares one directory, which invites corruption. The better key for dependencies is derived from the lockfile, so the cache changes exactly when the dependencies do:

.node-cache:
  cache:
    - key:
        files: [package-lock.json]
      paths: [.npm/]
      policy: pull
      fallback_keys: [npm-default]

install:
  stage: .pre
  cache:
    - key:
        files: [package-lock.json]
      paths: [.npm/]
      policy: pull-push
  script:
    - npm ci --cache .npm --prefer-offline

unit:
  extends: .node-cache
  script: npm test

Three details. policy: pull-push (the default) downloads at the start and uploads at the end; pull only downloads, and is what every job except the one that populates the cache should use, because it saves the upload time and stops a hundred jobs racing to write the same archive. key:files hashes the listed file or files (up to two) so a lockfile change produces a new key. And fallback_keys lists older keys to try when the exact key misses, so a lockfile change restores last week's cache as a starting point rather than starting from nothing. A job may have at most four caches, which is a hard ceiling since 19.2 (the default max caches per job).

cache:when (on_success, on_failure, always) controls whether a cache is saved when the job fails, which is useful when a long dependency install succeeded but a test failed.

Two more realities. On a runner with only local cache, the cache lives on whichever host ran the job, so a fleet of twenty hosts gives you twenty mostly-cold caches. Real sharing needs a distributed cache in object storage, configured in the runner's [runners.cache] section. And cache archives are compressed on every save: GitLab Runner 19.4 uses Fastzip by default for caches and artifacts, which is substantially faster than the old archiver; the FF_USE_FASTZIP flag exists to toggle it and is scheduled for removal in 19.7. CACHE_COMPRESSION_FORMAT (zip, tarzstd or zipzstd) selects the format.

Artifacts that behave

test:
  stage: test
  script:
    - pytest --junitxml=report.xml --cov --cov-report xml:coverage.xml
  coverage: '/TOTAL.*\s+(\d+%)$/'
  artifacts:
    when: always
    expire_in: 1 week
    paths: [htmlcov/]
    exclude: [htmlcov/**/*.map]
    reports:
      junit: report.xml
      coverage_report:
        coverage_format: cobertura
        path: coverage.xml

when: always is the important line: by default artifacts upload only when the job succeeds, so the test report for a failed run, the one you actually want, is thrown away. expire_in is the cost control: set it short for intermediate artifacts (a day) and longer for releases, because the instance default is thirty days and storage is billed on GitLab.com. reports: artifacts feed the UI (the test summary and failed-test list in merge requests, coverage deltas on the diff) and are separate from paths:, so a JUnit report does not need to be downloadable to be useful. A newer API, added in 19.4, fetches one specific report by type: GET /projects/:id/jobs/:job_id/artifacts?file_type=junit.

For access control, artifacts:access (all, developer, maintainer, none) limits who can download; maintainer arrived in 18.4 and the keyword is mutually exclusive with artifacts:public. The runner also prints the artifact size it uploaded, which is where you spot a build that has quietly grown to 300 MB of node_modules.

Building images in CI

Nearly every machine learning platform job ends with "build a container, push it, deploy it", so this is worth doing properly. Start with the Docker guide if multi-stage builds and layer caching are unfamiliar; this section is about what changes when the build runs inside GitLab.

The standard pattern from the docs is Docker-in-Docker (dind): the job runs the docker CLI in one container, and talks to a Docker daemon in a service container.

build-image:
  stage: build
  image: docker:24.0.5-cli
  services: [docker:24.0.5-dind]
  variables:
    DOCKER_TLS_CERTDIR: "/certs"
  script:
    - echo "$CI_REGISTRY_PASSWORD" | docker login "$CI_REGISTRY" -u "$CI_REGISTRY_USER" --password-stdin
    - docker build -t "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA" .
    - docker push "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA"

The version numbers are the docs' examples; pin the CLI and the dind service to the same major version that you test. The mechanics to understand are these. CI_REGISTRY, CI_REGISTRY_IMAGE, CI_REGISTRY_USER and CI_REGISTRY_PASSWORD are predefined, so no credentials are stored; the password is a short-lived job token. The service reachable at host docker is a real Docker daemon that needs privileged = true on the runner, which is the security cost. GitLab.com's hosted runners are privileged; your own runners must opt in, and that is the moment to consider whether building images on a shared runner is wise.

The error that shows up when any piece is misaligned is Cannot connect to the Docker daemon at tcp://docker:2375. Is the docker daemon running?. Read it as: the service did not start (runner not privileged), or TLS settings differ between client and daemon (set DOCKER_TLS_CERTDIR: "/certs" on the job and use a docker:*-dind service of the matching version).

Tagging deserves a policy. Tag with the commit SHA in every pipeline (immutable, traceable), and additionally with a human tag such as the release version in tag pipelines. Do not deploy latest, and do not rebuild per environment; build once, and promote the same digest through staging and production, which is the same discipline the Docker guide calls image promotion. The merge request pipeline can build and push :$CI_COMMIT_SHORT_SHA so reviewers can pull exactly what the pipeline tested. Clean up: the container registry grows without bound unless you configure cleanup policies, and a forgotten registry is one of the quieter line items on a storage bill.

A final trade-off to put on the table. dind works everywhere, but the layer cache lives in an ephemeral daemon and is cold on every job. The fixes are --cache-from with a previously pushed image, BuildKit's registry cache exports (--cache-to type=registry), or a daemonless builder. Whichever you pick, measure a cold versus warm build, because that number is the justification for the complexity.

Runners: how jobs actually get executed

Every job is executed by a runner, and the runner is where most "it works on my machine" and "it is slow" stories end up. You do not need to administer a fleet at this level, but you do need to understand what the runner does, because it is also your debugging surface.

A runner is a long-running process, gitlab-runner, which polls GitLab for work. There is no inbound connection: the runner calls POST /api/v4/jobs/request over outbound HTTPS, every check_interval seconds (default 3), and GitLab answers with a job whose tags, protected status and project scope that runner is allowed to serve. The runner then executes the job through an executor, streams the log back, uploads artifacts and caches, and reports the result.

Tags are the matching mechanism. A job runs only on a runner that carries all of the job's tags, and a runner can be told whether to accept untagged jobs. The error This job is stuck because you don't have any active runners that can run this job. almost always means a tag no runner has, a runner that does not run untagged jobs, a runner that is paused, or a protected-only runner facing an unprotected branch. After an hour with no matching runner the job fails with stuck_pending_no_matching_runners; if matching runners exist but are busy, it is dropped after 24 hours with stuck_pending_with_matching_runners.

The registration workflow changed

Old tutorials tell you to copy a registration token from the project settings and pass it to gitlab-runner register --registration-token. That workflow is deprecated and scheduled for removal in 20.0 (May 2027). The current flow has you create the runner first, in the UI or with POST /user/runners, choosing scope, tags and description there, and receive a runner authentication token beginning with glrt-:

gitlab-runner register --non-interactive \
  --url https://gitlab.com \
  --token glrt-XXXXXXXXXXXXXXXXXXXX \
  --executor docker \
  --docker-image alpine:latest \
  --description "ml-build-runner"
gitlab-runner verify
gitlab-runner lint --config /etc/gitlab-runner/config.toml

Tags, the locked flag, and run-untagged are now set when you create the runner, which is why the matching register flags (--tag-list, --locked, --run-untagged, --access-level, --maximum-timeout, --paused, --maintenance-note) are deprecated. gitlab-runner verify prints Verifying runner... is alive when the token works, and gitlab-runner lint (new) validates the TOML and flags misspelled or unknown keys, which silently do nothing otherwise. If you see Check registration token or 410 Gone - runner registration disallowed, somebody is using the legacy flow on an instance that has turned it off.

Executors and the configuration that matters

The executor determines isolation and cost. The actively developed ones are Docker, Docker Autoscaler, Instance and Kubernetes. Shell, SSH, VirtualBox, Parallels and Custom are in maintenance mode (security fixes only), and the Docker Machine executor is deprecated with removal in 20.0; its replacements are Docker Autoscaler or Instance with a fleeting plugin. If your organisation runs docker+machine autoscaling today, the migration is a 2027 deadline, not a 2029 one. For Kubernetes, see the Kubernetes guide and install through the Helm chart; the Helm guide covers values.yaml mechanics.

The fields in config.toml that an engineer at this level should be able to read:

concurrent = 8
check_interval = 3
log_format = "json"

[[runners]]
  name = "ml-build-runner"
  url = "https://gitlab.com"
  token = "glrt-XXXXXXXXXXXXXXXXXXXX"
  executor = "docker"
  limit = 4
  output_limit = 8192
  [runners.docker]
    image = "alpine:latest"
    privileged = false
    pull_policy = ["if-not-present"]
    volumes = ["/cache"]
  [runners.cache]
    Type = "s3"
    Shared = true
    [runners.cache.s3]
      ServerAddress = "s3.amazonaws.com"
      BucketName = "ci-cache"
      BucketLocation = "me-central-1"

concurrent is the global cap across all runners in the file; limit caps one [[runners]] block. If concurrent is 8 and you queue 30 jobs, 22 wait, whatever the size of the machine. output_limit is the job log cap in KB (4096 by default); hitting it gives Job's log exceeded limit of 4194304 bytes. and the answer is usually less noisy output, not a larger limit. pull_policy defaults to always; if-not-present is faster, but on a shared runner it can expose a private image cached by another job, which is why the security guidance is always on shared hosts. The example places the cache bucket in me-central-1, the AWS UAE region, which is one way to keep cache contents in-region for employers with data-residency requirements; substitute your own region and bucket.

Keep the runner's major.minor aligned with GitLab's (this guide assumes 19.4 for both). Minor versions are backward compatible, but a runner lagging behind a feature you use, such as secrets:gitlab_secrets_manager which needs Runner 19.0 or later, fails in ways that look like YAML problems.

Environments, deploys, and serialising change

A pipeline that only tests is a safety net. A pipeline that deploys is a production system, and the keywords for it are about who, when, and one at a time.

An environment: ties a job to a deploy target and creates a deployment record, so GitLab can show history, the current version, and a rollback button per environment. A dynamic name such as review/$CI_COMMIT_REF_SLUG creates one environment per branch, which is how review apps work:

deploy-review:
  stage: deploy
  script: ./deploy.sh review "$CI_COMMIT_REF_SLUG"
  environment:
    name: review/$CI_COMMIT_REF_SLUG
    url: https://$CI_COMMIT_REF_SLUG.review.example.com
    on_stop: stop-review
    auto_stop_in: 1 week
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"

stop-review:
  stage: deploy
  script: ./teardown.sh "$CI_COMMIT_REF_SLUG"
  environment:
    name: review/$CI_COMMIT_REF_SLUG
    action: stop
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
      when: manual

CI_COMMIT_REF_SLUG is the branch name made safe for URLs and DNS, which is why you use it here instead of CI_COMMIT_REF_NAME. on_stop links the stop job that tears the environment down when the merge request is merged or closed, and auto_stop_in is the safety net for the ones nobody cleans up; forgetting both is how a cluster accumulates hundreds of abandoned namespaces. The action values (start, prepare, stop, verify, access) tell GitLab what a job does to the environment.

One deploy at a time

Two pipelines deploying to the same environment concurrently is a race. resource_group is a mutex:

deploy-production:
  stage: deploy
  script: ./deploy.sh production "$CI_COMMIT_SHA"
  resource_group: production
  interruptible: false
  environment:
    name: production
    deployment_tier: production
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: manual

Jobs sharing a resource group run one at a time. The process_mode setting chooses which waiting job goes next (unordered, oldest_first, newest_first, newest_ready_first); for deploys you usually want newest_first, so an older commit does not run after a newer one. Pair this with the project's "Prevent outdated deployment jobs" setting, which refuses to run a deploy older than what is live. If you see You are not authorized to run this manual job, the cause is typically a protected environment or that setting, not a permissions bug.

Protected environments (Premium) restrict who may deploy and can require approvals, and protected branches and tags restrict which refs can reach protected variables. Design these together: the branch is protected, the secret is protected and scoped to production, and the environment limits who can click deploy. Any one layer alone leaves a gap.

Merge trains

Merged results pipelines test the combination of your branch and the target. Merge trains go one step further: they queue merge requests, and each pipeline runs against the merge of the target plus every merge request ahead of it, so the default branch never contains a combination nobody tested. Up to 20 pipelines run in parallel per train, which is configurable since 19.0, and merge train enforcement (generally available in 19.3) adds a project setting that blocks merges which bypass the train via the UI or API. Both are Premium features. The trade-off is latency: a train is great for a busy repository with a slow test suite and a burden for a small one.

Secrets, OIDC, and the job token

The question to ask of every credential in a pipeline is "what is the damage if this leaks, and how long is it valid?" Stored secrets are valid until someone rotates them, which means forever. GitLab gives you a better ordering:

1. OIDC id_tokens: the job gets a short-lived signed JWT, and your cloud or Vault trusts GitLab's identity provider. No stored secret.
2. secrets: integrations: the runner fetches from Vault, GCP Secret Manager, Azure Key Vault or AWS Secrets Manager at job start.
3. Protected, masked-and-hidden, file-type variables with environment scopes: the fallback.

An ID token is requested with id_tokens: and an audience. The job then exchanges it with the provider:

deploy-prod:
  stage: deploy
  id_tokens:
    VAULT_ID_TOKEN:
      aud: https://vault.example.com
  secrets:
    DB_PASSWORD:
      vault: gitlab/production/db
      token: $VAULT_ID_TOKEN
  script:
    - ./migrate.sh

The secrets: value is exposed as a file-type variable by default; set file: false to get a plain variable. The Vault guide covers the server-side auth method and policy; the important GitLab-side fact is that trust is expressed through claims in the token. The default sub claim is project_path:{group}/{project}:ref_type:{type}:ref:{branch}, and other claims such as ref_protected, environment and pipeline_source let the trust policy say "only protected refs, only for the production environment". A policy that trusts only the project path lets any branch, including an unreviewed feature branch, assume the production role. That is the single most common OIDC misconfiguration.

When OIDC fails, decode the token instead of guessing: echo "$VAULT_ID_TOKEN" | cut -d. -f2 | base64 -d | jq . shows the claims you actually sent. Errors of the form 401: unauthorized are almost always an aud or sub mismatch, or an old pipeline still using CI_JOB_JWT_V2, which was removed in 17.0. GitLab 19.4 also notes that GitLab Secrets Manager (in beta, billed in credits on GitLab.com) no longer supports group-based secret permissions, so grant to users or roles.

The job token and what it may touch

CI_JOB_TOKEN is the credential each job carries. It acts with the permissions of the user who triggered the pipeline, but only against a limited list of API endpoints, and only against other projects that allowlist yours. Since 18.0 that allowlist ("Authorized groups and projects" under Settings > CI/CD > Job token permissions) is enforced by default, and the old "limit access from this project" setting is gone. The consequences you will actually meet:

• Cloning another project with gitlab-ci-token:$CI_JOB_TOKEN fails with fatal: repository '...' not found until the source project adds yours to its allowlist.
• An image pull from another project's registry fails with requested access to the resource is denied for the same reason. (18.11.0 and 18.11.1 had a regression with this symptom for internal and public projects, fixed in 18.11.2.)
• From 19.1, allowlisted projects can push with the job token too, which is useful for release automation and dangerous for the same reason.

Since 19.0 the job token is a JWT by default. Base64-wrapping it with line breaks breaks it (invalid character '\n' in string literal; use base64 -w0), and old Fargate custom-executor versions (0.5.0 and earlier) fail outright. A top-level group can opt back into the legacy format, but only until 20.0.

Downstream pipelines

A pipeline that only grows becomes unreadable. GitLab lets one pipeline start others: a parent-child pipeline runs another YAML file in the same project, and a multi-project pipeline starts a pipeline in a different project.

trigger-service-a:
  stage: test
  trigger:
    include: services/a/.gitlab-ci.yml
    strategy: mirror
  rules:
    - changes: [services/a/**/*]

trigger-platform:
  stage: deploy
  trigger:
    project: platform/deployments
    branch: main
    strategy: mirror
    inputs:
      image_tag: $CI_COMMIT_SHORT_SHA

Two points that are commonly wrong. Use strategy: mirror, not depend; depend is documented as not recommended, and mirror (18.2 onward) makes the trigger job copy the downstream status faithfully. Without either, the trigger job succeeds as soon as the downstream pipeline is created, which reports green while the real work is still running. And pass parameters as inputs, not variables, because of the pipeline variable restrictions covered earlier; trigger:forward controls what is forwarded (yaml_variables: true and pipeline_variables: false are the defaults). A hierarchy can hold at most 1000 downstream pipelines, and the error when you exceed it is downstream pipeline tree is too large.

Dynamic child pipelines generate YAML in one job and run it in the next, which is the way to handle a monorepo where the set of jobs depends on the contents of the change:

generate:
  stage: build
  script: python ci/generate.py > generated.yml
  artifacts:
    paths: [generated.yml]

run-generated:
  stage: test
  trigger:
    include:
      - artifact: generated.yml
        job: generate
    strategy: mirror

The price of parent-child is that the graph is split across several views, and debugging means clicking through. Use it when you have a real reason: separate ownership per service, generated work, or a pipeline that otherwise exceeds the size limits. Do not use it just to make a long file shorter; include does that without the extra pipelines.

Debugging beyond the job log

When a pipeline misbehaves there are four places to look, in order of cheapness.

The merged configuration. Most "my job is not running" and "my variable is wrong" problems are visible in Build > Pipeline editor > Full configuration, or via glab ci lint and the lint API, which resolve includes and report This GitLab CI configuration is invalid: ... with a pointer to the offending key. Since 19.4 the editor also validates keywords that only the backend used to check. Typical messages: jobs:test-job:artifacts config contains unknown keys: path (a typo or bad indentation) and jobs config should contain at least one visible job (only hidden jobs, or rules excluded everything).

The job log. Learn its structure: the runner prints its version and executor, prepares the environment, fetches sources, restores caches, runs before_script and script, then uploads artifacts and saves caches, each in collapsible sections. The exit line tells you the class of failure: ERROR: Job failed: exit code 1 means your script returned non-zero; ERROR: Job failed: execution took longer than 1h0m0s seconds means the timeout (project default 60 minutes, hosted runners cap at 3 hours). If the failure is before your script ran (image pull, clone, cache), the cause is the runner, the registry or the network, not your code.

Known infrastructure errors. A short list worth memorising:

• error: RPC failed; curl 16 HTTP/2 send again with decreased length during clone: set git config --global http.version HTTP/1.1 using hooks:pre_get_sources_script.
• x509: certificate signed by unknown authority: a private CA or TLS-inspecting proxy; put the CA in /etc/gitlab-runner/certs/<host>.crt or set CI_SERVER_TLS_CA_FILE.
• WARNING: Failed to pull image with policy "always" with denied: the job token allowlist, or the 18.11.0/18.11.1 regression.
• incompatible pull policy: the job's image:pull_policy is not in the runner's allowed_pull_policies.

The failure reason. Since 19.0, job failure reasons are much more specific, and this changes retry:when. The values stuck_or_timeout_failure and job_execution_timeout are deprecated as of 19.1, with removal planned for 20.0; old values act as aliases with a lint warning. Use the specific reasons (stuck_pending_no_matching_runners, no_updates_running, server_timeout_running and their siblings). Retrying a script failure is a smell; retry only infrastructure causes:

default:
  retry:
    max: 2
    when:
      - runner_system_failure
      - runner_interrupted

From 19.3 a job can read CI_JOB_RETRY_COUNT (0 on the first run), which is useful for logging and for making a retry behave differently. To debug interactively, add CI_DEBUG_SERVICES: "true" for service container logs, and use CI_DEBUG_TRACE only in a safe environment. Finally, the API gives you scripted access: GET /projects/:id/jobs/:job_id/trace returns a log, and glab ci trace <job-name> streams one from the terminal.

Testing the pipeline itself and keeping it fast

A pipeline is code that nobody tests, which is why it breaks on Friday. Three practices fix most of that.

Lint before you push. glab ci lint against the server is faster than a failed pipeline, and can be a Git pre-commit hook. The CI Lint API (POST /projects/:id/ci/lint with content, dry_run and include_jobs) is what it calls. 19.2 added a rate limit for lint behind a flag that is off by default, so heavy automation should still batch requests.

Run scripts locally. Keep logic in scripts under ci/, and make them runnable with the same environment variables. For the job environment itself, run the job's image: docker run --rm -it -v "$PWD":/work -w /work python:3.12 bash, then execute the script lines by hand. That reproduces ninety percent of "works in CI, not here" cases, and the remaining ten are almost always variables and network reachability.

Treat the pipeline like a service. Track pipeline duration, failure rate and queue time, which the CI/CD analytics page charts. Mind that since 19.1 success and failure rates count only completed pipelines. The levers for speed, in order of typical leverage:

1. Stop work that is not needed: rules:changes, interruptible, and auto_cancel.
2. Stop work from waiting: needs so the critical path is the longest chain, not the sum of stages.
3. Stop re-downloading: lockfile-keyed caches, pull policy, slim images, pinned versions.
4. Stop shipping large artifacts: artifacts:exclude, expire_in, needs: [{job: x, artifacts: false}].
5. Stop waiting for a runner: check queue time before blaming the job. A pipeline that "takes twenty minutes" often spends eight of them pending.
6. Right-size: on GitLab.com a larger runner tag costs more compute minutes per minute, so a bigger machine only saves money if it saves more time than its cost factor.

Cost on GitLab.com is compute minutes times the machine-size cost factor, and arm64 runners and shorter jobs are the direct levers. For self-hosted runners the cost moves to your own infrastructure bill. Either way, git itself can be slow: GIT_DEPTH controls shallow cloning (new projects default to a shallow clone of 20), and GIT_STRATEGY: none skips the clone entirely for jobs that only call an API. The Terraform guide and the Argo CD guide show two common things a deploy job hands off to; often the right answer for a slow deploy job is to make it shorter by letting a GitOps controller do the long part.

Putting it all together

Here is a compact pipeline for a service that builds an image and deploys it to a cluster. It uses everything above, and each choice is one you can now defend.

spec:
  inputs:
    deploy_target:
      default: staging
      options: [staging, production]
---
workflow:
  name: '$CI_PIPELINE_SOURCE $CI_COMMIT_REF_NAME'
  auto_cancel:
    on_new_commit: interruptible
    on_job_failure: all
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
      when: never
    - if: $CI_COMMIT_BRANCH
    - if: $CI_COMMIT_TAG

default:
  image: python:3.12
  interruptible: true
  retry:
    max: 1
    when: [runner_system_failure, runner_interrupted]

stages: [build, test, package, deploy]

.python:
  cache:
    key:
      files: [requirements.txt]
    paths: [.cache/pip]
  variables:
    PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"

lint:
  extends: .python
  stage: test
  needs: []
  script: ruff check .

unit:
  extends: .python
  stage: test
  needs: []
  script:
    - pip install -r requirements.txt
    - pytest --junitxml=report.xml
  artifacts:
    when: always
    expire_in: 1 week
    reports:
      junit: report.xml

image:
  stage: package
  needs: [unit, lint]
  image: docker:24.0.5-cli
  services: [docker:24.0.5-dind]
  variables:
    DOCKER_TLS_CERTDIR: "/certs"
  script:
    - echo "$CI_REGISTRY_PASSWORD" | docker login "$CI_REGISTRY" -u "$CI_REGISTRY_USER" --password-stdin
    - docker build -t "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA" .
    - docker push "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA"
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"

deploy:
  stage: deploy
  needs: [image]
  interruptible: false
  resource_group: $[[ inputs.deploy_target ]]
  id_tokens:
    CLUSTER_TOKEN:
      aud: https://cluster.example.com
  environment:
    name: $[[ inputs.deploy_target ]]
  script: ./ci/scripts/deploy.sh "$CI_COMMIT_SHORT_SHA"
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: manual

Read it the way a reviewer would. One pipeline per push, cancelled when superseded. Lint and unit tests start immediately, in parallel, with a lockfile-keyed cache. The image builds only when the gate is green, and is tagged with the commit. The deploy is manual, serialised per environment, never cancelled midway, authenticates through a short-lived token, and takes its target from a typed input with a default. There is not a stored production credential anywhere in the file.

Where you are now

You can predict how GitLab will treat a pipeline, because you know which decisions happen at creation and which at run time. You write rules that cannot produce duplicate pipelines, shape work as a DAG instead of a staircase, reuse configuration with extends, components and inputs without losing track of what runs, and keep credentials out of the file with OIDC and the job token allowlist. You know the difference between a cache and an artifact, what the runner does between polls, and how to read a failure down to its class. You also know what is changing by 20.0 in May 2027: registration tokens, the Docker Machine executor, legacy job tokens, compliance pipelines and the old retry:when reasons.

The senior level covers what comes next: running runners as a fleet with autoscaling, the trust model behind protected refs and forks, pipeline execution policies, multi-tenant runner design, upgrade planning and the cost of a platform that other teams depend on. The mid-level interview file turns this material into scenario questions, and the tips file collects the errors, practices and recipes you will reach for at work.

Sources

• GitLab CI/CD documentation
• CI/CD YAML syntax reference
• Deprecated keywords
• Use CI/CD configuration from other files
• Workflow keyword
• Job rules
• Job inputs
• Job artifacts
• CI/CD job token
• Troubleshooting jobs
• Debugging CI/CD pipelines
• CI/CD variables
• Predefined variables
• CI/CD inputs
• CI/CD components
• Caching in GitLab CI/CD
• Downstream pipelines
• Merge request pipelines
• Merge trains
• Pipeline editor
• CI/CD secrets
• ID token authentication
• Use Docker to build Docker images
• Runners
• GitLab-hosted runners
• Runner creation workflow
• CI/CD limits
• Pipelines API
• Jobs API
• Lint API
• GitLab deprecations
• GitLab 19.4 release notes
• GitLab 19.0 release notes
• GitLab Runner documentation
• Runner commands
• Runner advanced configuration
• Runner executors
• Runner security

Part two of three. A cumulative review of Beginner and Mid-level material, organised by topic rather than by level. At this level interviewers stop asking "what is a stage" and start asking "the pipeline is slow, what do you check first". The answers they want are reasoned: what you would look at, in what order, and why. Read the fast review first, then work through the scenarios and trade-offs, and finish with the self-test.

Where this picks up

The mental model in one minute

GitLab CI/CD builds a pipeline on the server when an event happens, from the configuration at that commit, and freezes it. Runners poll for jobs, execute them through an executor, and stream results back. Decisions about which jobs exist are made at creation; decisions about what a script sees are made on the runner.

Two sentences interviewers love: "Retrying a job re-uses the old configuration", and "a cache is best effort, an artifact is a guarantee." If you can say both and explain them, you have already separated yourself from half the candidates.

Commands and keywords to have ready

glab auth login
glab ci lint --dry-run --ref main --include-jobs   # validate against the real server
glab ci run -b main --input 'env:staging'          # pipeline-level inputs
glab ci status --live                              # watch the current pipeline
glab ci trace build-image                          # stream one job's log
gitlab-runner verify                               # is the token accepted
gitlab-runner lint --config /etc/gitlab-runner/config.toml

Scenario questions

These are the questions that separate people who have run a pipeline from people who have owned one. Each has a model answer and a note on what a strong answer includes.

The pipeline takes thirty minutes. What do you check first?

Model answer. I would first decide whether the time is spent running or waiting. In the pipeline view and the duration charts, I look at queue time: a job that sat pending for eight minutes is a runner-capacity problem, not a job problem, and no amount of tuning the script fixes it. Then I find the critical path, the longest chain of dependent jobs, because that is what sets wall-clock time. Stage-based pipelines often make the critical path the sum of each stage's slowest job, so I check whether needs would let jobs start earlier. After that I look inside the longest job: how much is setup (image pull, clone, cache restore, dependency install) versus real work, using the timestamps in the log. Only then do I reach for fixes: lockfile-keyed caches, slimmer images, rules:changes to skip irrelevant jobs, interruptible and auto-cancel so obsolete pipelines do not clog the queue, and parallel for genuinely splittable test suites.

What a strong answer adds. It says "measure first" and names queue time separately. It notices that parallel only helps if runner concurrency allows it. A weak answer jumps to "add more runners" or "use a bigger machine".

Two pipelines run for every push to a merge request branch. Why, and how do you fix it?

Model answer. Jobs are matching both the branch pipeline (CI_PIPELINE_SOURCE == "push") and the merge request pipeline (merge_request_event). GitLab even warns Job may allow multiple pipelines to run for a single action. The fix is a workflow:rules block: create a pipeline for merge request events, refuse a branch pipeline when $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS, and allow other branches and tags. Jobs with when but no if are a common cause, because that rule matches everything.

What a strong answer adds. It explains that workflow rules decide whether the pipeline exists while job rules decide which jobs do, and mentions that Pipeline filtered out by workflow rules. is the expected message, not an error.

A job works on main and fails on a feature branch with an empty credential. What happened?

Model answer. Almost certainly a protected variable: it is exposed only to pipelines on protected branches and tags, so on an unprotected branch it is simply absent. That is deliberate, it keeps production secrets away from unreviewed code. I confirm by checking the variable's flags and the branch's protection, and I would not "fix" it by unprotecting the variable. If the feature-branch job genuinely needs a credential, the right answer is a separate lower-privilege credential with an environment scope, or an OIDC role that trusts non-protected refs only for a non-production resource.

What a strong answer adds. It connects protection to the trust model: protected branch, protected variable, and protected environment form one boundary, and any one alone leaves a gap.

A deploy job was retried and deployed an old version over a new one. How do you prevent it?

Model answer. Two controls. A resource_group so deploys to one environment run one at a time, with process_mode: newest_first so when several are waiting the newest goes next. And the project's "Prevent outdated deployment jobs" setting so a job older than the current deployment cannot run. I would also make the deploy job interruptible: false so a superseding pipeline does not cancel it halfway, and deploy an immutable image reference such as the commit SHA so "retry" always means "the same thing".

What a strong answer adds. It recognises the difference between serialising (mutex) and ordering (outdated prevention), which are separate problems.

Someone runs terraform apply from a manual job with a long-lived cloud key in a CI variable. What do you change?

Model answer. I would remove the stored key. The job requests an OIDC token with id_tokens: and an audience, and the cloud provider trusts GitLab as an identity provider, issuing short-lived credentials. The trust policy matters more than the plumbing: it must check claims such as ref_protected, the environment, and the project path, so that only a protected branch deploying to production can assume the production role. If OIDC is not available for the target, I would fall back to secrets: from Vault or a cloud secret manager, and as a last resort a protected, masked-and-hidden, environment-scoped file variable.

What a strong answer adds. It warns that masking is not a security boundary, and that a policy trusting only the project path lets any branch assume the role. Note also that the official Terraform CI templates were removed in 18.0; the OpenTofu component is the replacement, and the Terraform guide covers the tool itself.

A job is stuck in pending forever. Walk me through it.

Model answer. GitLab says This job is stuck because you don't have any active runners that can run this job. I check four things: does any online runner carry all of the job's tags; does the runner accept untagged jobs if the job has none; is the runner paused, locked to other projects, or protected-only while this is an unprotected branch; and does the runner's scope include this project. A job with no matching runner fails after an hour with stuck_pending_no_matching_runners; one that has matching runners but no capacity is dropped after 24 hours as stuck_pending_with_matching_runners. If runners match and are busy, I look at concurrent and limit in config.toml.

What a strong answer adds. It distinguishes "no match" from "no capacity", which are different failures with different fixes.

Builds pass locally but fail in CI with "Cannot connect to the Docker daemon at tcp://docker:2375". Why?

Model answer. The job uses Docker-in-Docker and the daemon service is not reachable. Usual causes: the runner is not privileged, so the docker:dind service never started; a mismatch between client and daemon TLS settings (the docs pattern sets DOCKER_TLS_CERTDIR: "/certs" with a docker:*-dind service of the matching version); or the job's image lacks the Docker CLI. I would read the service's logs with CI_DEBUG_SERVICES, and ask whether building images on a privileged shared runner is the right design at all.

What a strong answer adds. The security trade-off, and the alternatives: daemonless builders or a dedicated protected runner pool. See the Docker guide for the image side.

The pipeline is green, but the downstream deploy failed. How?

Model answer. The trigger job has no strategy, so it succeeds as soon as the downstream pipeline is created. The fix is trigger:strategy: mirror (the older depend is documented as not recommended), which copies the downstream result onto the trigger job. I would also pass parameters as inputs rather than variables, given the pipeline-variable restrictions.

What a strong answer adds. It knows the 18.2 introduction of mirror, and the hierarchy cap of 1000 downstream pipelines.

Trade-off questions

Stages versus needs. Stages are simple and read top to bottom; they overconstrain and waste time on the slowest job of each stage. needs gives a faster pipeline and finer artifact control, at the cost of a graph that is harder to read at a glance, and of jobs that can start before something you implicitly relied on. A balanced answer: keep stages as the coarse lanes, use needs where the time saved is measurable.

include versus components versus copy-paste. Copy-paste is fast and drifts. include:project shares text but has no parameters and, without a ref, follows someone else's branch. Components add typed inputs, versions and discoverability, at the cost of maintaining a component project with releases. A strong answer: use components once three or more projects need the same thing, pin a full version or SHA, and accept that partial versions and ~latest resolve only against published catalog releases.

Parent-child pipelines versus one big file. Child pipelines isolate ownership and enable dynamic generation, but split the view and add pipelines to count against limits. Use them for monorepo services or generated work, not for file length; include fixes length.

GitLab-hosted versus self-hosted runners. Hosted runners are zero-maintenance, ephemeral and clean, billed in compute minutes by machine size. Self-hosted runners cost infrastructure, patching and security work, but give control of networking, hardware (GPUs), data residency, and cost at volume. For employers in the Gulf or Egypt with in-region data requirements, self-hosted runners in a regional cloud are frequently the practical answer. A strong answer mentions ephemeral executors, since long-lived runners accumulate state.

Cache in the runner versus distributed cache. Local caches are fast and free but per-host. Object-storage caches are shared and consistent, cost egress and latency, and need configuration. The question to ask is how many runner hosts you have.

Retrying jobs. Retry masks flakiness; it also wastes minutes. Retry only infrastructure reasons such as runner_system_failure and runner_interrupted, and treat a repeatedly retried test as a bug to fix.

Merge trains. They guarantee the default branch is never broken by an untested combination, at the cost of latency and runner load. They fit busy repositories with expensive suites and not quiet ones.

Questions about change and deprecation

These show that you keep up, which matters in a tool that has a major release every May.

"What should be migrated before GitLab 20.0?" Registration-token runner registration (create runners in the UI or API and use glrt- tokens), the Docker Machine executor (move to Docker Autoscaler or Instance), compliance pipelines (to pipeline execution policies), the legacy job token format, and the retry:when reasons stuck_or_timeout_failure and job_execution_timeout. 20.0 is expected in May 2027.

"What changed about the job token recently?" 18.0 enforced the allowlist by default and removed the "limit access from this project" setting; 19.0 made the JWT format the default, with an opt-out for top-level groups until 20.0; 19.1 made cross-project push with the job token generally available for allowlisted projects.

"Why is only/except bad?" It is deprecated, cannot be combined with rules in a job, and expresses less. Convert when you touch the file.

"What are the top-level image and cache keys?" Deprecated; put them under default:.

"How do I pass a variable to another pipeline?" Prefer inputs. Pipeline variables are discouraged since 17.7, and on GitLab.com new projects in new namespaces default to no_one_allowed for setting them.

Common interview questions

What is the difference between a cache and an artifact?

An artifact is a guaranteed, pipeline-scoped hand-off stored by GitLab, used for build results and reports. A cache is a best-effort, cross-pipeline speed-up stored by the runner or its object storage, used for dependencies. A job must succeed with an empty cache; if it does not, correctness depends on something that is not guaranteed.

How does a runner get a job?

It polls GitLab over outbound HTTPS every check_interval (3 seconds by default), and GitLab hands it a job whose tags, protection level and scope it satisfies. It executes the job through its executor, streams the log, uploads artifacts and cache, and reports the status. No inbound port is needed.

What is CI_JOB_TOKEN?

A short-lived credential per job, carrying the triggering user's permissions but limited to a set of API endpoints, and to other projects only if those projects allowlist yours. Since 19.0 it is a JWT.

What are CI/CD inputs, and how do they differ from variables?

Inputs are typed parameters (string, number, boolean, array) interpolated when the pipeline is created with $[[ inputs.name ]]. They are not environment variables, do not participate in variable precedence, and are validated by options or regex. Pipeline-level inputs need defaults because branch and merge request pipelines cannot supply them.

How would you reduce a pipeline's cost?

Skip unnecessary work with rules:changes and auto-cancel, shorten the critical path with needs, key caches to lockfiles, set artifacts:expire_in, choose runner sizes by measured benefit, use arm64 runners where they work, and read the analytics before changing anything.

How do you roll out a change to a shared pipeline template safely?

Version it. Publish the change as a new release, have consumers pin a full version or SHA, and move them forward on their own schedule with a bot or a scheduled job. Never change a template that consumers follow by branch name.

How would you debug an include that does not do what you expect?

Open the merged configuration in the pipeline editor, check which file each job came from, and check merge rules: hashes merge deeply, arrays are replaced. Then check the ref of the include and the job's rules.

What a strong answer covers that a weak one misses

• Timing. Strong answers separate creation time from run time without being prompted. Weak ones describe rules as if they could react to results.
• Measurement. Strong answers start with data: queue time, critical path, log timestamps. Weak ones start with a fix.
• Trust boundaries. Strong answers link protected branches, protected variables, protected environments and OIDC claims. Weak ones say "use secret variables".
• Failure classes. Strong answers distinguish script failure, runner failure, and configuration failure, and say what each implies.
• Deprecation awareness. Strong answers know which keywords and workflows are going away and what replaces them.
• Trade-offs with a recommendation. Strong answers give both sides and then pick, with the condition under which they would pick the other.

Final self-test

Answer aloud, then check against the sections above.

1. At what point is the pipeline's configuration frozen, and what does that imply for retrying a job?
2. Write the workflow:rules that prevents duplicate branch and merge request pipelines.
3. Why can rules:changes run a job unexpectedly on a branch without a merge request, and what fixes it?
4. What does needs change about artifacts, besides ordering?
5. Does extends merge or replace before_script? And variables?
6. What is the precedence of a project UI variable versus a YAML variable of the same name?
7. Why is masking not a security boundary?
8. A job must succeed with an empty cache. Why?
9. What do you set so a failed test run still uploads its report?
10. What is the difference between concurrent and limit?
11. Which token starts with glrt-, and how do you get one?
12. What does resource_group prevent, and what does "Prevent outdated deployment jobs" prevent?
13. Name the claims you would put in an OIDC trust policy for a production deploy.
14. Why does a downstream pipeline need strategy: mirror?
15. List three things scheduled for removal in 20.0.

If you can answer twelve of fifteen without looking, you are ready for a mid-level conversation. Senior takes every one of these further: fleet design and autoscaling, the trust model behind forks and protected refs, pipeline execution policies, upgrade planning across required stops, multi-tenant runner pools, and the conversation about cost and ownership with leadership.

Part two of three. At this level your pipelines run and go green, so the problems change character. They stop being "why is my YAML invalid" and become "why do two pipelines run for one push", "why is the deploy job missing on this branch" and "why did a pipeline that has not changed in a month suddenly fail". Start with the error table, then the practices in order of leverage, the practice cards, and the operational recipes at the end.

Common errors at this level

Cumulative: Beginner's errors still apply, and these are the ones that appear once things mostly work.

The practices that pay off most

Ordered by leverage, meaning the ones that prevent the most pain per minute of effort come first.

1. One workflow:rules block in every project. It removes the most common source of doubled cost and confusing results. Five lines, forever.
2. Lint before you push. glab ci lint --dry-run --ref main --include-jobs catches typos, bad includes and missing jobs without creating a pipeline. Make it a pre-commit hook.
3. Pin everything you include. A tag or SHA for include:project, a full version for components, a digest or exact tag for images. Unpinned references turn someone else's commit into your outage.
4. Mark work interruptible, and deploys not. default: interruptible: true, auto_cancel, and interruptible: false on anything with side effects.
5. Use needs on the critical path. Not everywhere; on the longest chain, after measuring.
6. Key caches on lockfiles and use pull policy in consumers. One job populates, the rest read.
7. Put scripts in files. Shellcheck, run locally, review in diffs.
8. Replace stored cloud credentials with OIDC. The trust policy, not the plumbing, is where the security lives: pin it to protected refs and the environment.
9. Serialise deploys with resource_group and prevent outdated deployments.
10. Set artifacts:expire_in on everything intermediate. Storage is a real cost line and the default is thirty days.
11. Pass parameters as inputs, not variables, and give them defaults.
12. Track your deprecations. Keep a short list of what must change before 20.0, and do one item each sprint.

Practice cards

Each card is a five-minute drill. Do them on a scratch project.

Reading a failing job in order

A fixed order saves more time than intuition. Follow it, top to bottom, and stop when you find the cause.

1. Did a pipeline and the job get created?If the job is missing, it is a rules or workflow question, not a runtime one. Use the merged config and lint.
2. Did a runner pick it up?Pending with a stuck message is a tags, scope or capacity question. Check the runner's status and the job's tags.
3. Did it get past preparation?Image pull, clone, cache restore and services happen before your script. Failures there belong to the runner, registry, network or job token.
4. Did your script fail?exit code 1 is yours. Read above the last line, not the last line, and run the script in the job's image locally.
5. Did something after the script fail?Artifact upload and cache save have their own sections, sizes and limits.

Making a pipeline debuggable

Some habits cost nothing now and save an hour later.

default:
  before_script:
    - echo "runner=$CI_RUNNER_DESCRIPTION job=$CI_JOB_NAME retry=${CI_JOB_RETRY_COUNT:-0}"
    - set -euo pipefail

workflow:
  name: '$CI_PIPELINE_SOURCE: $CI_COMMIT_REF_NAME'

Printing the runner description at the start of each job answers "is it always the same host that fails" immediately. set -euo pipefail makes the shell stop on the first error, on an unset variable, and on a failure anywhere in a pipe, which turns silent mis-steps into loud failures. A readable workflow:name makes the pipeline list scannable. Use CI_JOB_RETRY_COUNT (available since 19.3) when logging, so you can tell a first run from a retry.

Operational recipes

Recipe: a safe review-app lifecycle

review:
  stage: deploy
  script: ./ci/scripts/review.sh start "$CI_COMMIT_REF_SLUG"
  environment:
    name: review/$CI_COMMIT_REF_SLUG
    url: https://$CI_COMMIT_REF_SLUG.review.example.com
    on_stop: review-stop
    auto_stop_in: 3 days
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"

review-stop:
  stage: deploy
  script: ./ci/scripts/review.sh stop "$CI_COMMIT_REF_SLUG"
  environment:
    name: review/$CI_COMMIT_REF_SLUG
    action: stop
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
      when: manual

Always pair a dynamic environment with on_stop and auto_stop_in. Without both, clusters fill with abandoned review apps, each holding CPU and a load balancer.

Recipe: a lockfile-keyed cache with one populating job

deps:
  stage: .pre
  script: pip install -r requirements.txt
  cache:
    key:
      files: [requirements.txt]
    paths: [.cache/pip]
    policy: pull-push

unit:
  script: pytest
  cache:
    key:
      files: [requirements.txt]
    paths: [.cache/pip]
    policy: pull

One writer avoids archive races and wasted uploads. If unit works only when the cache is warm, it is using the cache for correctness; fix that first.

Recipe: split a test suite with parallel

test:
  stage: test
  parallel: 4
  script:
    - pytest --splits "$CI_NODE_TOTAL" --group "$CI_NODE_INDEX"
  artifacts:
    when: always
    reports:
      junit: report.xml

This assumes a sharding plugin such as pytest-split; the pipeline only supplies the two variables. Check runner concurrent before raising the number, and keep the shards balanced by their measured durations.

Recipe: pass a build result to a deploy job

build:
  stage: build
  script:
    - echo "IMAGE_TAG=$CI_COMMIT_SHORT_SHA" >> build.env
  artifacts:
    reports:
      dotenv: build.env

deploy:
  stage: deploy
  needs:
    - job: build
      artifacts: true
  script: ./ci/scripts/deploy.sh "$IMAGE_TAG"

Recipe: call the API from a pipeline with the job token

curl --fail --header "JOB-TOKEN: $CI_JOB_TOKEN" \
  "$CI_API_V4_URL/projects/$CI_PROJECT_ID/pipelines/$CI_PIPELINE_ID/jobs"

If you get {"message":"404 Not Found"}, the endpoint is probably not on the job token's allowed list, or the target is outside its allowlist. Check the job token doc's access table before reaching for a personal token, which is a far bigger credential.

Recipe: trigger a pipeline with inputs from the terminal

glab ci run -b main \
  --input 'env:staging' \
  --input 'replicas:int(3)' \
  --input 'regions:array(eu-west-1,me-south-1)'

The int(...) and array(...) wrappers give the type, and the same values can be posted to the Pipelines API as {"inputs": {...}}. Every input must match its declared type and options, or the call is rejected.

Recipe: find out why a runner is not picking jobs

gitlab-runner --version
gitlab-runner status
gitlab-runner list
gitlab-runner verify
sudo journalctl -u gitlab-runner -n 100 --no-pager
gitlab-runner lint --config /etc/gitlab-runner/config.toml

Then in the UI: is the runner online, unpaused, scoped to your project, and does its tag list cover the job's? If the lint reports an unknown key, a setting you believed was active never was. On a container runner, restart the container after changing config, because gitlab-runner restart inside it only starts a new process.

Recipe: self-signed GitLab or a TLS-inspecting proxy

sudo mkdir -p /etc/gitlab-runner/certs
sudo cp corp-ca.crt /etc/gitlab-runner/certs/gitlab.example.com.crt
sudo gitlab-runner restart

The file must be named after the host and sit in that directory. This is the answer to x509: certificate signed by unknown authority in a Gulf or Egyptian enterprise network where outbound TLS is inspected, and it is worth doing once, properly, instead of disabling verification in each job.

Recipe: migrate off a registration token

1. In the UI, open Settings > CI/CD > Runners and choose Create project runner (or the group or admin equivalent). Set tags, description and untagged behaviour there.
2. Copy the glrt- token shown once.
3. Run gitlab-runner register --non-interactive --url https://gitlab.com --token glrt-... --executor docker --docker-image alpine:latest.
4. Run gitlab-runner verify, then run a test job and check it lands on the new runner.
5. Retire the old runner entry and remove registration steps from your automation before 20.0.

Why pipelines rot, and how to slow it down

A pipeline that was tidy in January is rarely tidy in October, and the reasons are predictable. Each one has a cheap countermeasure, and knowing them turns maintenance from a rescue into a habit.

Unpinned inputs drift. An image tagged python:3 changes under you, an include:project without a ref follows another team's branch, and a component referenced as @~latest moves whenever a release is published. The pipeline did not change, yet it fails on a Tuesday. The countermeasure is pinning, and the discipline of moving pins forward deliberately, in a commit you can revert, rather than having them move by themselves.

Copies diverge. Five services started from the same template and now have five slightly different lint jobs, one of which quietly stopped failing the build. The countermeasure is to share through include or components from the start, and to make the shared piece small enough that people do not fork it. A component with two inputs gets used; one with twenty gets copied and edited.

Workarounds outlive their reasons. A retry: 2 added for a flaky network three years ago now hides a real race condition. A GIT_STRATEGY: clone added to fix a corrupted workspace costs ninety seconds on every job. A comment with the date and the reason beside each workaround makes it possible to remove it later, and a quarterly pass through the pipeline with "is this still needed" as the only question is worth an afternoon.

Tokens and scopes age. A trigger token belonging to a person who left, a runner registered under a personal token, a job token allowlist with projects that no longer exist. None of these fail loudly until the day they do. Review the allowlists and runner list at the same time as the workaround pass.

The platform moves. GitLab releases a minor version every month and a major one every May, and pipelines that depended on deprecated behaviour break at the major boundary. The next one, 20.0 in May 2027, removes registration tokens, the Docker Machine executor and the legacy job token format, among other things. The countermeasure is the deprecation list: keep it in the repository, link each item to a ticket, and read the upgrade notes for each release when it ships rather than when a pipeline turns red.

Reviewing a pipeline change

Pipeline changes get reviewed less carefully than application code, and they are at least as dangerous, because they run with credentials and can reach production. When you review one, read it in the following order.

First, who can trigger this and on which refs. Look at rules and workflow: did the change widen the set of pipelines that can run a privileged job? A deploy job that used to run only on the default branch and now runs on merge request pipelines is a serious change in one line.

Second, what credentials does it touch. Any new variable reference, id_tokens block or secrets: entry deserves a question about its scope and protection. Ask whether the trust policy on the other end is pinned to protected refs or merely to a project path.

Third, what does it pull in from outside. A new include:remote, an unpinned image, a curl | sh line in a script, or a component from a project the team does not own is a supply-chain decision, and should be made consciously.

Fourth, what does it cost. A new parallel: 20, a matrix that doubled, a cache with no policy, artifacts with no expire_in, or a job that now runs on every push rather than on changes. None of these is wrong, but each has a price, and the review is the cheapest place to name it.

Fifth, how do you roll it back. A change to a shared template should be a new version, not an edit in place. A change to a deploy job should leave the previous way of deploying recoverable.

The most useful single habit is to ask the author to run glab ci lint --dry-run --ref main --include-jobs and paste the job list into the merge request. It takes seconds, and it makes the effect of the change visible to a reviewer who would otherwise have to execute the YAML in their head.

Habits worth adopting now

• Read the merged configuration, not just your file, when something surprises you.
• Treat the pipeline's YAML as code: review it, lint it, version shared pieces.
• Start every performance conversation with a measurement: queue time, critical path, log timestamps.
• Keep a runner's version in step with GitLab's major.minor.
• Assume masked means "hidden in the log", nothing more.
• Give every pipeline-level input a default.
• Prefer the specific retry:when reasons, and retry infrastructure failures only.
• Delete what you cannot explain. A job nobody understands is a job nobody dares remove, and it costs minutes on every push.

Checklist before you merge a pipeline change

• workflow:rules prevents duplicate pipelines
• glab ci lint --dry-run --ref main --include-jobs passes
• Every include:project has a ref, and every component has a pinned version
• Build and test jobs are interruptible; deploy jobs are not
• Caches are keyed on lockfiles, and jobs succeed with a cold cache
• artifacts: when: always on test jobs, expire_in on intermediates
• Deploys use resource_group and an immutable image reference
• No long-lived cloud credential in a variable; OIDC claims pinned to protected refs
• Trigger jobs use strategy: mirror
• Nothing new relies on a deprecated keyword from the table in the detailed guide

Senior takes these further: runner fleet design, pipeline execution policies, secrets strategy at organisation scale, upgrade paths across required stops, and the cost and governance of a shared CI platform.

This is part three of three. It closes the series by taking every topic from Beginner and Mid one level further, and adds the work you own when GitLab CI/CD is a platform that other teams depend on rather than a tool you use yourself.

At the earlier levels the question was "how do I make my pipeline pass?" From here the questions change. What happens to two hundred engineers when the runner fleet falls over at nine on a Monday morning? Who can push a change that reaches production without review? Which component saturates first when the company doubles in size? How do you upgrade the thing that builds everything else, and how do you retire a feature you told everyone to use last year? A pipeline that works is a personal achievement. A pipeline system that keeps working while forty teams change it every day is an engineering discipline, and it has its own failure modes.

Everything here is verified against GitLab 19.4 (released 17 September 2026) and GitLab Runner 19.4.1. GitLab 19.5 is due on 15 October 2026, so anything I describe as arriving in 19.5 is upcoming and not in a stable release. Two dates matter for your planning: the next major release, 20.0, is expected in May 2027, and it removes several things that still work today. Those removals are called out wherever they apply, and gathered in one place in the migrations section.

Where this picks up

I am starting with the architecture, because every scaling and security decision in the rest of this guide follows from one fact: runners pull work, and GitLab never reaches into your infrastructure.

The architecture and where it breaks

GitLab CI/CD is two systems joined by an HTTPS API. The first is GitLab itself: Rails, Sidekiq background workers, PostgreSQL, Gitaly (the Git storage service), Redis and object storage. GitLab decides what should run. When a push arrives, it reads the CI configuration at that commit, resolves every include, interpolates every input, evaluates rules, and writes a pipeline and its jobs into the database. The second system is the runner fleet: one or more gitlab-runner processes that decide where and how a job runs.

The link between them is a polling loop. Each runner manager asks GitLab POST /api/v4/jobs/request every check_interval seconds (default 3). If a job matching the runner's scope and tags is pending, GitLab hands back a payload containing the script, the variables, the image, the job token and the list of artifacts to download. The runner executes it, streams the log back in chunks, and uploads artifacts and caches when it finishes. There is no inbound connection to the runner at all, which is the property that makes this model easy to secure: a runner in a private subnet needs only outbound HTTPS to GitLab and to whatever your jobs themselves reach.

This split predicts a good deal of behaviour. Because configuration is frozen at pipeline creation, retrying a job never picks up a fix to .gitlab-ci.yml; you need a new pipeline. Because includes and inputs are resolved on the server, a component you changed five minutes ago affects every pipeline created after that moment and none created before it. And because runners are pull-based, pending time (the gap between a job being created and a runner accepting it) is the clearest single measure of whether your fleet is keeping up.

Which component breaks first

Platform owners tend to assume the runners will be the bottleneck. On GitLab.com you never see that because the fleet is GitLab's problem; on Self-Managed, experience points to a consistent ordering. Knowing it tells you where to spend attention.

1. Runner capacity and autoscaler latency. The first thing users notice is jobs sitting in pending. Usually the cause is too few workers at peak, or an autoscaler that provisions capacity too slowly, not a GitLab fault.
2. The container registry and package traffic. Two hundred jobs starting simultaneously each pull a base image, and Docker Hub rate limits or an overloaded internal registry show up as failures that look like flaky jobs.
3. Artifact and cache transfer. A monorepo with a 400 MB dependency cache uploaded by every job saturates the network and object storage long before CPU matters.
4. Sidekiq queues. Pipeline creation and job state transitions run as background jobs. A very large number of pipelines created at once (a mass schedule firing at 02:00, or a bot pushing to a hundred projects) delays pipeline creation, which users perceive as "CI is broken" even though no runner is involved.
5. PostgreSQL. CI tables are the largest and busiest in the database. GitLab partitions them and, in 18.7, added new metadata tables (p_ci_job_definitions) with a large background migration on upgrade. On a big instance that migration can run for a long time, and you tune it with the environment variables GITLAB_DB_CI_JOBS_PROCESSING_DATA_CUTOFF and GITLAB_DB_CI_JOBS_MIGRATION_CUTOFF, together with pipeline archival.

The practical lesson is to measure before you scale. Section "Observability" below shows the three metrics that separate these cases.

The limits you will meet

GitLab enforces instance limits that protect the control plane. Self-Managed administrators can change most of them; on GitLab.com the SaaS values apply. Learn the defaults, because pipelines that fail with an opaque message are very often hitting one of them.

Two of these deserve a warning because they are silent. The four-pipelines-per-push limit means a tag-heavy release script that pushes ten tags at once creates four pipelines and ignores the rest without an error. And from 18.11 through 19.2.1 the compiled per-job configuration had a 1 MiB cap that surfaced as Config is too large. Maximum size allowed is 1 MiB; the fix made it configurable through ci_max_total_yaml_size_bytes. If you run a version in that range with heavy templating, that is the first thing to check.

Choosing and operating executors

The executor is the part of the runner that decides where a job's commands run. The choice is the most consequential runner decision you make, because it sets your isolation model, your scaling model and your upgrade path. The current landscape is simpler than it was a few years ago, because GitLab has moved most of the older options into maintenance.

The actively developed executors are Docker, Docker Autoscaler, Instance and Kubernetes. The Shell, SSH, VirtualBox, Parallels and Custom executors are in maintenance mode, meaning security fixes only. And the Docker Machine executor (docker+machine) was deprecated in 17.5 and is removed in 20.0, so it is a dated dependency if you still run it. Its replacement is the pair of Docker Autoscaler and Instance executors, built on fleeting, a plugin system for managing cloud instance groups.

The Shell row is a warning in table form. On a shell executor a job is a process on your machine, with the runner user's permissions and whatever the previous job left behind. Use it for code you fully control, on hardware you can wipe, and never on a runner that serves merge requests from contributors you do not trust.

The Kubernetes executor

If your organisation already runs Kubernetes, this is where to start, because you inherit scheduling, autoscaling and quotas instead of building them. The runner manager runs as a small Deployment, and every job becomes a pod with a build container (your image), a helper container (the GitLab helper image that clones the repository and handles artifacts and caches) and one container per services: entry. You install it with the official chart:

helm repo add gitlab https://charts.gitlab.io
helm search repo -l gitlab/gitlab-runner
helm install --namespace gitlab-runner gitlab-runner -f values.yaml gitlab/gitlab-runner

gitlabUrl: https://gitlab.example.com
runnerToken: ""            # supplied from a Secret, never committed
rbac:
  create: true
concurrent: 40
runners:
  config: |
    [[runners]]
      [runners.kubernetes]
        namespace = "ci-jobs"
        image = "alpine:3.20"
        cpu_request = "500m"
        memory_request = "1Gi"
        memory_limit = "4Gi"
        service_account = "ci-job"
        [runners.kubernetes.node_selector]
          "workload" = "ci"

Notice what the configuration controls. The runner manager lives in one namespace (gitlab-runner) with RBAC to create pods, while the jobs run in another (ci-jobs) under a service account with almost no permissions. That separation matters more than anything else in the file: a job is arbitrary code, so the identity it runs as should be able to do nothing inside your cluster. The node_selector keeps CI pods on a dedicated node pool so a runaway build cannot starve production workloads, and requests and limits make each job a known quantity to the scheduler.

Two traps are specific to this executor. First, resource requests are what the scheduler reserves, so a job that requests 500m CPU but uses four cores throttles its neighbours on the node; set requests near real usage and limits above it. Second, the *_overwrite_allowed settings decide whether a job can raise its own CPU, memory or service account by setting a variable in .gitlab-ci.yml. Left unset, they are off, which is what you want on a shared runner. If you permit them, give each a regular expression that bounds the range, for example allowing memory overrides only up to a known ceiling, so one team's "just need more memory" does not become the whole node.

GitLab 19.4 added suspend and resume support to the Kubernetes executor, and the kubectl-level details of namespaces, quotas and node pools are covered in the Kubernetes guide; the chart itself is ordinary Helm, so the Helm guide applies to upgrading it.

Autoscaling VMs with fleeting

When you cannot or do not want to run Kubernetes, the Docker Autoscaler executor gives you ephemeral VMs with container-level jobs on top. The runner manager talks to a fleeting plugin for your cloud, which scales an instance group: an Auto Scaling group on AWS, a managed instance group on Google Cloud, or a scale set on Azure. The configuration lives in an [runners.autoscaler] block:

concurrent = 50

[[runners]]
  name = "autoscale-docker"
  url = "https://gitlab.example.com"
  token = "glrt-REDACTED"
  executor = "docker-autoscaler"

  [runners.docker]
    image = "alpine:3.20"
    privileged = false

  [runners.autoscaler]
    plugin = "aws:latest"
    capacity_per_instance = 1
    max_use_count = 1
    max_instances = 50

    [runners.autoscaler.plugin_config]
      name = "ci-workers-asg"

    [[runners.autoscaler.policy]]
      idle_count = 5
      idle_time = "20m0s"

Read the four capacity settings as a cost-versus-latency dial. max_instances is the hard ceiling and therefore your worst-case bill. capacity_per_instance = 1 means one job per VM, which is the strongest isolation because no job ever shares a kernel with another. max_use_count = 1 destroys the VM after one job, so nothing survives to be poisoned. And the policy block sets how many idle machines to keep warm: idle_count = 5 means the fifth developer to push at 09:00 does not wait for a cold boot. Every idle machine costs money around the clock, which is why the policy supports periods and timezone, letting you keep a large warm pool during working hours and a small one overnight.

The trade-off is unavoidable and worth saying plainly to your stakeholders: isolation and speed are purchased with money. One-job-per-VM with no warm pool is cheap and safe but slow to start. A big warm pool with shared VMs is fast and cheap per job but shares kernels and caches between jobs. You pick a point on that line per trust tier, not once for the company, and that idea returns in the multi-tenancy section.

Concurrency: three numbers that get confused

Runner throughput is governed by three settings that people constantly mix up, and getting them wrong produces either idle capacity or overload.

• concurrent in the global section caps the total number of jobs this runner process runs at once across every [[runners]] block. It is the top-level budget.
• limit inside a [[runners]] block caps that one runner configuration. 0 means unlimited. On an autoscaler, limit is the total number of workers.
• request_concurrency (default 1) sets how many jobs a runner may request from GitLab in parallel. Adaptive request concurrency is on by default in modern runners (FF_USE_ADAPTIVE_REQUEST_CONCURRENCY), so the runner raises its polling parallelism when there is a queue.

A runner with concurrent = 10 and two [[runners]] blocks each set to limit = 10 will still never run more than ten jobs in total. A common incident is an operator raising limit and seeing no change, because concurrent was the binding constraint. Another is setting concurrent far above what the host can actually run, so jobs start and then fail with out-of-memory kills. The number to size against is the host's real capacity at peak, not the number of jobs you would like to run.

Several runner managers may share one runner configuration and one glrt- token, distinguished by a system_id. That is how a Kubernetes deployment with three replicas, or an autoscaling group of runner managers, presents as a single runner in the UI.

Runners, registration and the new token workflow

If you operate runners that predate 2024, this section is about a change that is easy to postpone and expensive to rush. Runner registration tokens are deprecated and are removed in 20.0. The old flow was: copy a shared registration token from a group or project, run gitlab-runner register --registration-token ..., and pass scope settings such as tags and --locked as flags. The token was long-lived and shared, so anyone who had it could register runners into your pool.

The current flow inverts this. You create the runner in GitLab first (in the UI, or with the API call POST /user/runners), setting its scope, tags and description there. GitLab returns a runner authentication token beginning glrt-, and you register the runner host with it:

gitlab-runner register --non-interactive \
  --url https://gitlab.example.com \
  --token "$RUNNER_AUTH_TOKEN" \
  --executor docker \
  --docker-image alpine:3.20 \
  --description "shared-linux-docker"

Together with the registration token, the register flags --locked, --access-level, --run-untagged, --maximum-timeout, --paused, --tag-list and --maintenance-note are deprecated with removal in 20.0, because those properties now belong to the runner object in GitLab rather than to the command line. This has two consequences for a platform owner. It is now scriptable through the API, so a Terraform or Ansible run can create the runner object, capture its token and feed it to the host; and it is rotatable, because POST /runners/:id/reset_authentication_token or gitlab-runner reset-token --name x replaces a compromised token without touching anything else.

For a fleet, treat the glrt- token as a credential with a rotation policy, store it in your secret manager, and keep the runner hosts themselves disposable. An error such as Check registration token or 410 Gone - runner registration disallowed on an old automation script is the signal that it still uses the legacy flow and will break outright in 20.0.

The scope of a runner is the other decision made at creation time. Instance runners serve every project; group runners serve one subtree; project runners are locked to selected projects. Combine that with the protected option, which restricts a runner to jobs on protected branches and tags, and you already have the skeleton of a trust model: a production-deploying runner should be protected, group-scoped and tagged, so untrusted branches can never reach it. The next sections build on that idea.

Caches and artifacts at fleet scale

At small scale a cache is a convenience. At fleet scale it is a piece of shared infrastructure with its own capacity, its own failure modes and, because it carries files between jobs, its own security implications. You already know the rule from Mid: a cache is a best-effort speed-up, and an artifact is a guarantee. Nothing correctness-critical may travel through the cache, because a cache miss, an evicted object or a different runner must never change whether the build is right.

Where the cache lives

With the Docker executor on a single host, caches are stored in local volumes, which is fast but invisible to every other host. As soon as you have more than one runner, a job lands on a machine that has never seen the cache, and you get a miss. A distributed cache fixes this: the runner uploads to and downloads from object storage configured in [runners.cache].

[[runners]]
  name = "shared-linux"
  executor = "kubernetes"
  [runners.cache]
    Type = "s3"
    Shared = true
    Path = "ci-cache"
    MaxUploadedArchiveSize = 2147483648
    [runners.cache.s3]
      ServerAddress = "s3.amazonaws.com"
      BucketName = "example-ci-cache"
      BucketLocation = "eu-central-1"
      AuthenticationType = "iam"

Three settings deserve a sentence each. AuthenticationType = "iam" uses the runner's cloud identity instead of a stored access key, which removes one secret you would otherwise have to rotate. Shared = true lets different runners read each other's caches, which raises the hit rate but widens the trust boundary, a point I return to below. And MaxUploadedArchiveSize puts a ceiling on what a single job can write, so one bad cache key cannot fill the bucket. Put the bucket in the same region as the runners: for employers in the Gulf or Egypt that also means choosing a region such as a Middle East cloud region deliberately, both for latency (cache traffic is large and chatty) and for any data-residency commitment that covers build inputs.

Since Runner 19.4, Fastzip is the default archiver for caches and artifacts (FF_USE_FASTZIP defaults to true, and the flag is slated for removal in 19.7). It is considerably faster than the old implementation, so a runner upgrade can shorten cache steps with no pipeline change. You can also choose the compression with CACHE_COMPRESSION_FORMAT: zip, tarzstd or zipzstd. Zstd trades a little CPU for smaller archives and is usually the right choice when the bottleneck is the network. GitLab 19.4 also added an option to suppress cache upload and download URLs in job logs, which matters because those URLs are presigned and sometimes sensitive.

Keys, size and the four-cache limit

A cache is only as good as its key. The pattern you want is a key derived from the lockfile, so the cache is invalidated exactly when dependencies change, and fallback_keys so a miss falls back to a recent nearby cache instead of starting from nothing:

default:
  cache:
    key:
      files: [package-lock.json]
      prefix: $CI_JOB_NAME
    paths: [.npm/]
    fallback_keys: [$CI_DEFAULT_BRANCH-npm]
    policy: pull

Notice policy: pull as the default. In a fleet, every job downloading and then re-uploading the cache is a large hidden cost: a hundred jobs each upload 300 MB after a run in which nothing changed. The discipline is that one dedicated job uses policy: pull-push (or push) and refreshes the cache, and everything else is pull. The default limit of four caches per job (generally available in 19.2) is a useful constraint, not an annoyance; a job that wants ten caches is carrying state it should bake into its image.

Size is where caches quietly hurt. Compression and upload happen on the critical path of every job that writes, so a 2 GB cache that saves ninety seconds of install but costs two minutes to move is a net loss. Measure it: look at the "Saving cache" and "Restoring cache" timings in a job log, and compare them with the install step they replace.

Cache poisoning and trust tiers

Because a shared cache carries files from one job into another, it is a path for one job to influence a different one. If an untrusted merge request pipeline can write to a cache key that a protected-branch release job later reads, then code from the merge request has shaped the release build. The mitigation is to keep the trust tiers apart. GitLab has a project setting to use separate caches for protected branches, and you should keep it on and preserve that separation in any custom key: never build a key that deliberately collapses them, and never let a release or deploy job restore a cache written by a job that ran untrusted code.

Artifacts: retention, access and the cost of forgetting

Artifacts are stored in GitLab's object storage and count against it. The default maximum is 100 MB per artifact file, and the default expiry is the instance setting (30 days out of the box), which a job overrides with artifacts:expire_in. A platform with no retention discipline discovers, eventually, that most of its storage is build outputs nobody has opened since they were created. Three habits keep that in check. Set expire_in explicitly on anything large and short-lived; pass artifacts between jobs with needs: and artifacts: paths narrow enough that you do not ship the entire workspace; and enable automatic pipeline cleanup so old pipelines and their artifacts are removed on a schedule.

Access matters too. The artifacts:access keyword takes all, developer, maintainer or none, and is mutually exclusive with the older artifacts:public. maintainer (added in 18.4) is the right setting for artifacts that contain anything you would not show a contributor: coverage reports with file paths, internal logs, unsigned binaries. The default, all, means anyone who can see the pipeline can download the artifact, and on a public project that is the whole internet. Set artifacts:access in a shared template so the safe choice is the default that teams inherit.

GitLab 19.4 added one small API convenience worth knowing: GET /projects/:id/jobs/:job_id/artifacts?file_type=junit fetches a specific report artifact, which lets a dashboard or a flaky-test tracker read test results without downloading the whole archive.

The trust model

Security on a CI platform is a single question asked from different angles: when a job runs, what can it reach, and who could have influenced what it runs? GitLab gives you a set of controls that, used together, answer it. Used separately, each one leaves a gap the next would have closed.

Start from the fact that a job is arbitrary code running with credentials. It runs with the permissions of the user who triggered it, expressed through a short-lived CI_JOB_TOKEN. It may also hold variables, cloud tokens and cached files. So the chain of custody is: who can change the code or the configuration (branch protection, code owners, merge approvals), which runner executes it (protected, scoped and isolated), and what secrets are exposed to it (protected variables, OIDC claims, external secret managers). Break any link and the rest stop meaning much.

Protected branches, protected runners, protected variables

Three features are named "protected" and form one mechanism. Protected branches and tags decide who may push and merge. Protected variables are exposed only to pipelines running on protected refs. Protected runners accept jobs only from protected refs. Together they mean: a person who can open a merge request from a feature branch can cause a pipeline to run, but that pipeline cannot read the production credential and cannot land on the runner that holds network access to production.

The failure mode is the gap between two of the three. A protected variable with an unprotected runner is fine, because the variable never reaches the unprotected pipeline. But a runner with production network access that is not protected can be used by any branch, and the variables are then irrelevant because the job can reach the database directly. Audit the set rather than each item: for every runner that can reach production, check that it is protected; for every variable that unlocks production, check that it is protected and environment-scoped.

Merge requests from forks

A merge request from a fork is a stranger's code. By default the pipeline runs in the fork's context, not in yours, so it does not receive your protected variables. The danger is the maintainer who helpfully runs it in the parent project. That pipeline runs with parent-project runners and variables, and the contributor controls .gitlab-ci.yml. The rule is to review the pipeline configuration changes before running a fork's pipeline in the parent project, and to have a written policy that says when that is acceptable. A reviewer who skims a diff for application bugs and approves a pipeline run has just given the contributor your runner.

The job token and its allowlist

CI_JOB_TOKEN is how a job authenticates to GitLab itself: pulling an image from the project registry, downloading another project's artifacts, calling a small set of API endpoints. Since GitLab 18.0, access into a project is governed by that project's allowlist ("Authorized groups and projects" under Settings > CI/CD > Job token permissions), and it is enforced by default. The old "Limit access from this project" setting was removed. The mental model is that the target decides who may come in, which is the correct direction: the owner of a sensitive project controls its own exposure instead of depending on every consumer to behave.

The token itself changed format in GitLab 19.0: it is now a JWT by default. A top-level group can switch back to the legacy format, but only until 20.0. Two practical breakages are worth knowing. Base64-encoding the token with line wrapping corrupts it, producing invalid character '\n' in string literal, so use base64 -w0. And older custom-executor Fargate driver versions (0.5.0 and earlier) fail with JWT tokens. Separately, a bug in versions up to 18.8 could return 403 Forbidden on PUT /api/v4/jobs/<id> for jobs using needs, child pipelines or long silent steps, fixed from 18.9.

Later releases widened what the token may do, carefully. Cross-project Git push with CI_JOB_TOKEN to allowlisted projects became generally available in 19.1 (it was behind the flag allow_push_to_allowlisted_projects in 19.0), and 19.3 let the token fetch repository archives. Each of these is a capability you should grant per project, not globally. Fine-grained job token permissions and the job token authentication log exist to let you see which projects are actually using the token against yours; read that log before tightening an allowlist so you do not cut off a consumer you did not know about.

One version trap belongs here: GitLab 18.11.0 and 18.11.1 had a regression that stopped CI_JOB_TOKEN pulling images from internal and public projects, with the same denied message. It is fixed in 18.11.2. If you diagnosed it as an allowlist problem on those versions, you would have been chasing the wrong thing, which is a good argument for always checking the version first.

Hardening the runner host

The runner is where untrusted code actually executes, so most of the runner guidance is about containing it. The rules are concrete:

• Never run untrusted code on the shell executor. The job has the runner user's rights on the host.
• Avoid privileged = true and --pid=host. A privileged container is close to root on the host. If you genuinely need Docker-in-Docker, use dedicated, ephemeral, protected-only runners, and accept that a privileged job can escape to its VM.
• Use pull_policy = "always" on shared runners. With if-not-present, a private image pulled by one project's job is cached on the host and can be used by another project that never had pull access.
• Avoid GIT_STRATEGY: fetch on shared hosts. It reuses a working directory that a previous tenant's job has touched.
• Restrict what jobs may request with allowed_images, allowed_services and allowed_privileged_images in [runners.docker], and the *_overwrite_allowed regexes for Kubernetes.
• Segment the network and block cloud metadata endpoints (the instance metadata address) from job containers, so a job cannot read the runner VM's cloud credentials.

A fragment of a hardened Docker runner looks like this:

[[runners]]
  name = "shared-hardened"
  executor = "docker"
  [runners.docker]
    image = "alpine:3.20"
    privileged = false
    pull_policy = ["always"]
    allowed_images = ["registry.example.com/ci/*", "alpine:*"]
    allowed_services = ["registry.example.com/ci/services/*"]
    cap_drop = ["NET_RAW", "SYS_ADMIN"]
    disable_cache = false

The 19.4 keys services_cap_add and services_cap_drop extend the same idea to service containers, which previously inherited capabilities you could not tune independently. For background on what capabilities and privileged mode actually expose, the Docker guide covers the isolation model in depth. Note allowed_pull_policies too: since 18.0 a job uses only pull policies that appear in the runner's allowlist, and a mismatch produces the error incompatible pull policy.

Secrets, OIDC and the supply chain

The Mid level taught you to use id_tokens so a job proves its identity to Vault or a cloud provider without a stored credential. The senior question is how you design the trust those tokens carry, because an OIDC integration is only as tight as the claim it checks.

Ranking secrets by how much they can leak

There is a clear order of preference, and it reflects how much an attacker gains from each:

1. OIDC id_tokens into the cloud provider or Vault. No secret is stored in GitLab at all. The job receives a short-lived signed token and exchanges it for short-lived credentials.
2. The secrets: integrations (HashiCorp Vault, Google Cloud Secret Manager, Azure Key Vault, AWS Secrets Manager). The value lives outside GitLab and is fetched at job start. GitLab's own Secrets Manager (the secrets:gitlab_secrets_manager keyword, needing Runner 19.0 or later) is in public beta and, on GitLab.com, in limited availability and billed in GitLab Credits. In 19.4 the Maintainer role got read/write by default and group-based secret permissions were removed, so grant access to users or roles. Treat it as promising but not yet the foundation of a platform.
3. Masked, hidden, protected, file-type CI/CD variables with environment scopes. A stored secret, but as contained as a stored secret gets.

Masking deserves a blunt sentence because teams over-rely on it: masking is not a security boundary. It replaces exact matches of the value in the job log. A job that runs env | base64, or prints the value with a character between each letter, walks straight past it. Anyone who can change the pipeline script can read every variable the job can see. Masking protects against accidents; protected scopes, file-type variables and external secret managers protect against people. Since 18.3, new UI variables default to the Masked visibility, and the Masked and hidden option (reworked in 19.0) hides the value in the settings UI as well. Masking also has format rules: the value must be a single line of at least eight characters, and with variable expansion on, only the characters _ : @ - + . ~ = / beyond letters and digits. A secret with spaces or a newline cannot be masked, which is one more reason to prefer file-type variables for certificates and keys.

Also remember CI_DEBUG_TRACE: "true", which prints every variable in the job log, including ones you believed were masked. On a platform, set debug_trace_disabled = true in the runner's configuration for shared pools, so one engineer debugging at midnight cannot publish a production credential to everyone who can read the log.

Designing the OIDC claim

When a job requests an ID token, GitLab signs a JWT whose claims describe the job. The default sub (subject) claim has the shape project_path:{group}/{project}:ref_type:{type}:ref:{branch}. Your cloud or Vault trust policy then says which subjects may assume which role. The security of the integration is the precision of that policy:

deploy-production:
  stage: deploy
  image: registry.example.com/ci/aws-cli:2
  id_tokens:
    AWS_ID_TOKEN:
      aud: https://sts.amazonaws.com
  environment: production
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  script:
    - >
      aws sts assume-role-with-web-identity
      --role-arn "$AWS_ROLE_ARN"
      --role-session-name "gitlab-$CI_JOB_ID"
      --web-identity-token "$AWS_ID_TOKEN"

The mistake that turns this into a vulnerability is a trust policy that checks only the audience, or that matches project_path:myorg/*. Then any project or branch in the group can assume the production role. The fix is to bind the role to the most specific claims you can: the exact project_path, the ref, and ideally ref_protected (true only for protected refs) or environment together with environment_protected. Extra claims available include pipeline_source, job_project_path (added in 18.4) and job_source (18.9). Decode a token while designing the policy, rather than guessing what it contains:

echo "$AWS_ID_TOKEN" | cut -d. -f2 | base64 -d 2>/dev/null | jq .

Two gotchas. A 401: unauthorized from Vault or AWS often means the job still uses the removed CI_JOB_JWT_V2 (removed in 17.0), or that aud or sub does not match the policy. And the error ID token issuance is disabled in CI because this project's path was previously used by a different project. protects you from a subtle attack: if a project is deleted and another is created at the same path, a policy keyed on project_path would trust the newcomer. GitLab refuses to issue tokens in that case until you adjust the claim or policy.

For teams on Kubernetes or GitOps, pair this with the Vault guide for the server side: a JWT auth role bound to the claims above, issuing short-lived dynamic credentials.

The supply chain, from source to artifact

Your platform is a supply chain: source goes in, artifacts come out, and other systems trust them. The controls that matter are ones you can enforce centrally.

• Pin what you include. A component referenced as @1.2.3 or a commit SHA is reproducible. @~latest and a moving branch mean a change by the component's author reaches every consumer on their next pipeline. Partial versions (1, 1.2) and ~latest resolve only against published catalog releases and never pick pre-releases. For anything that touches production, pin a full version or a SHA and bump it deliberately through a merge request.
• Pin images by digest in jobs that build releases, and pair it with a scheduled rebuild so you still receive patches.
• Sign keylessly. With id_tokens and aud: sigstore, a job can sign images or files with cosign and no signing key to store. The signature records the identity of the pipeline that produced it, which a verifier can then check.
• Generate provenance. The runner can produce SLSA provenance metadata for artifacts when RUNNER_GENERATE_ARTIFACTS_METADATA is set, recording which job and which inputs produced the file.
• Scan in the pipeline. SAST, secret detection, dependency scanning (Dependency-Scanning.v2) and container scanning ship as templates and components, and GitLab 19.1 added ingestion of third-party SARIF 2.1.0 results so you can bring your own scanner into the same report.

A signing job then looks like this (the image and cosign usage are illustrative; check your registry's current cosign flags):

sign-image:
  stage: publish
  image: registry.example.com/ci/cosign:2
  id_tokens:
    SIGSTORE_ID_TOKEN:
      aud: sigstore
  rules:
    - if: $CI_COMMIT_TAG
  script:
    - cosign sign --yes "$CI_REGISTRY_IMAGE@$IMAGE_DIGEST"

The value of these controls is realised at the point of consumption: a cluster admission policy that refuses unsigned images, or a release process that verifies provenance before promotion. A signature nobody checks is decoration. The Docker guide covers the image side of digests, SBOMs and admission, and the Kubernetes guide the enforcement side.

Config as an API: components, inputs and what teams may override

When forty teams each copy a hundred lines of pipeline YAML, you have forty slightly different pipelines and no way to change them all. The platform answer is to treat shared pipeline configuration as a versioned interface, the way you would treat a library, and to decide deliberately what consumers may change.

Components and the catalog

A CI/CD component is a reusable unit of configuration that lives in a component project's templates/ directory and is referenced by version:

include:
  - component: $CI_SERVER_FQDN/platform/ci-components/build-image@2.3.0
    inputs:
      image_name: $CI_REGISTRY_IMAGE
      platforms: linux/amd64,linux/arm64

spec:
  inputs:
    image_name:
      description: Image repository to push to
    platforms:
      default: linux/amd64
    stage:
      default: build
---
build-image:
  stage: $[[ inputs.stage ]]
  script:
    - ./build.sh "$[[ inputs.image_name ]]" "$[[ inputs.platforms ]]"

Two details are easy to get wrong. The header (spec:) must be separated from the jobs by ---. And an empty spec:inputs: fails with Build component error: Spec must be a valid json schema; use an empty spec: instead. To publish a component to the CI/CD Catalog, mark the project as a catalog resource, give it a description and README, and run a tag pipeline that uses the release: keyword. Creating a release through the Releases API does not publish to the catalog. A project can hold up to 100 components (raised from 30 in 18.5). If you use GitLab.com components on Self-Managed, you must mirror the component project, since the catalog is per instance.

Versioning is the contract

A component is an API, and the failure that matters is a breaking change delivered silently. Decide your versioning rules and enforce them:

• Semantic versions with real meaning. A major bump is any change that can fail a consumer who changed nothing: a renamed input, a removed job, a changed default stage.
• Consumers pin exact versions for critical pipelines and use partial versions (@2) only where they accept automatic minor updates. ~latest is for experimentation.
• Release from a protected tag pipeline so only maintainers can publish a version that other teams will consume.
• Test the component in its own pipeline with a matrix of the input combinations consumers use, before tagging.

Component authors should also use the context values available inside templates, such as $[[ component.version ]] (generally available in 18.7), so a component can reference its own version, for example to pull a matching helper image. And use spec:inputs validation: options for enumerations, regex for formats like ^v\d+\.\d+\.\d+$, type: number or boolean, and since 19.0 array inputs with an index operator ($[[ inputs.list[0] ]]). Validation at pipeline creation turns a mistake into a clear error instead of a failed deploy.

Inputs instead of pipeline variables

For years the way to parameterise a pipeline was to set variables, through the Run pipeline form, the API or a trigger. The problem is that variables are untyped, unvalidated environment-wide strings with a long precedence chain, and anyone who can set a pipeline variable can change the behaviour of every job. GitLab has discouraged this since 17.7, and new projects in new GitLab.com namespaces default ci_pipeline_variables_minimum_override_role to no_one_allowed. The consequence you will see is Insufficient permissions to set pipeline variables, including from trigger jobs that pass variables to a downstream project.

The platform move is therefore to set that minimum role deliberately at the group level, and migrate consumers to inputs, which are typed, validated and interpolated at creation with $[[ inputs.name ]]. They take no part in variable precedence, so they cannot be used to smuggle an override into a job. One pitfall: pipeline-level inputs must be declared in the main .gitlab-ci.yml and must have defaults, because merge request and branch pipelines cannot supply inputs. 18.10 added job-level inputs: with runtime evaluation (${{ job.inputs.NAME }}), overridable on a manual run or retry, which suits manual deploy jobs that need a parameter. The run-time Moa expression syntax and the run: keyword (GitLab Functions, formerly CI Steps) are still experimental, so teach them as experiments and do not build your platform on them.

What a consumer may override

Once shared configuration exists, the next question is how much a team can change. A component can be as open as a blank template or as closed as a fixed pipeline, and the right answer depends on the risk of the job. Low-risk jobs (linting, unit tests) should be easy to extend: expose extends and let teams add variables and rules. High-risk jobs (build, sign, deploy) should expose a small set of inputs and nothing else, because every extra knob is something a mistake or an attacker can turn.

The strongest mechanism for the high-risk case is pipeline execution policy (Ultimate), which injects jobs into project pipelines from a separate, centrally owned policy project. Those jobs run whether the project wants them or not, and their variables take the highest precedence in the variable order, above anything a project sets. Policies replace the older compliance pipelines, which are deprecated and removed in 20.0. Policies offer variables_override controls for what a project may change, and since 18.10 dotenv variables honour that setting. Scheduled pipeline execution policies are generally available as of 19.2. The design principle is that enforcement belongs in a place the development teams cannot edit, which is exactly what a policy project is.

Deployment safety: environments, merge trains and ordering

A platform owner's nightmare is not a failed build but a successful wrong deploy: two pipelines deploying over each other, an old pipeline finishing after a newer one, or a merge that passed alone and broke the default branch in combination with another. GitLab has a specific control for each.

Ordering. resource_group is a mutex: jobs sharing a group run one at a time. Its process_mode decides what happens to the queue: unordered (any order), oldest_first (the default sensible choice for serialised deploys), newest_first, or newest_ready_first. For production deploys, oldest_first preserves commit order, while newest_first skips a stale deploy in favour of the latest commit, which is often what you actually want when each deploy ships the whole tree. Pair it with the project setting "Prevent outdated deployment jobs" so a slow, old pipeline cannot overwrite a newer release. The error You are not authorized to run this manual job can come from that setting as well as from a protected environment.

Who may deploy. Protected environments (Premium) restrict which users and roles may deploy, with approval rules for production requiring sign-off from named groups. The platform habit is to make "Allowed to deploy" a small, audited group, and to use approvals so the person who wrote the change is never the only one who approved it reaching production.

Merge order. A branch that passes CI alone can still break main once it merges next to another change. Merged results pipelines test the source branch combined with the current target; merge trains (Premium) queue merge requests and test each one on top of those ahead of it. By default up to 20 pipelines run in parallel per train, configurable since 19.0. GitLab 19.3 made merge train enforcement generally available: a project setting that blocks merges that bypass the train through the UI or the API. Without enforcement a train is a suggestion; with it, it is a rule.

workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
      when: never
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

deploy-production:
  stage: deploy
  resource_group: production
  environment:
    name: production
    deployment_tier: production
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: manual
  script: ./deploy.sh

The workflow:rules block matters on a platform because it prevents the duplicate branch-plus-merge-request pipelines that double your runner bill; the warning Job may allow multiple pipelines to run for a single action is the lint telling you a template lacks it. Put this block in the shared base template every team includes, and the duplicate-pipeline problem disappears fleet-wide. For the Kubernetes delivery side, the GitLab agent for Kubernetes works with Flux, and the pull-based alternative is covered in the Argo CD guide and the Flux guide; GitLab's role then shrinks to building and promoting the artifact, and the cluster pulls it.

Multi-tenancy, cost and pipeline economics

A shared CI platform is an economy. Teams consume a scarce resource (runner time, cache, artifact storage, registry bandwidth) and, without feedback, each optimises for its own convenience. Multi-tenancy is therefore partly security (keep tenants apart) and partly cost (make the price of a pipeline visible and reasonable).

Partitioning the fleet by trust and by shape

Separate runner pools rather than one big pool with one configuration. A sensible starting set has four: a general pool (ephemeral, unprivileged, unprotected) for merge request pipelines; a protected pool for default-branch builds and releases, with access to signing and deploy networks; a privileged pool for the few jobs that need Docker-in-Docker, small and ephemeral; and a specialised pool for GPU, arm64 or macOS work. Route jobs with tags, and lock the dangerous pools with the protected flag and group scoping. Each tenant gets its own Kubernetes namespace or service account where you use that executor, so a quota or a compromise stays contained. On GitLab.com you get a version of this for free: every hosted job runs on an ephemeral VM deleted after the job, with no VM-to-VM traffic.

What a pipeline costs

On GitLab.com, hosted runners bill compute minutes multiplied by a cost factor by machine size, and a namespace has a monthly quota per plan. On self-hosted runners you pay your cloud provider instead, and GitLab minutes do not apply. Either way, the levers are the same, and the order of leverage is worth memorising because the first three usually outweigh the rest together:

1. Do not run what is not needed. rules:changes runs only the jobs a change affects. One trap: on a branch pipeline with no merge request, changes compares against the previous push and can run far more than expected, so set compare_to to anchor the comparison. Since 19.2, rules:changes:regexp and rules:exists:regexp exist, and they are mutually exclusive with paths:.
2. Cancel what is obsolete. Mark most jobs interruptible: true and leave workflow:auto_cancel:on_new_commit at its default conservative (or interruptible), so a new push cancels the now-pointless pipeline. Do not mark deploy jobs interruptible.
3. Use the DAG. needs: starts jobs as soon as their inputs finish instead of waiting for the whole stage, which cuts wall-clock time without adding cost.
4. Cache by lockfile, pull by default, as discussed above.
5. Expire artifacts and clean pipelines.
6. Right-size runners, use arm64 where it works, and set autoscaler idle policies to match working hours.

Wall-clock time and cost are different things and sometimes opposed: parallelising a test suite four ways finishes sooner but uses a little more total compute (each shard pays setup). Decide which you are buying, and give teams the number: a pipeline-duration chart and a rough cost per pipeline make the conversation concrete. Platform owners who can say "your pipeline costs forty times your neighbour's, and here is why" get changes made; those who send a generic "please optimise" do not.

Observability: knowing the platform is healthy

If you cannot answer "is CI slow right now, and why?" in two minutes, you run it by rumour. The runner exposes Prometheus metrics; you turn them on with listen_address = ":9252" in the global section of config.toml, and scrape /metrics. The metrics that separate the failure modes from the architecture section are:

• gitlab_runner_jobs and gitlab_runner_jobs_running_total: how busy the runner is against gitlab_runner_concurrent and gitlab_runner_limit. A gauge pinned at the limit with jobs pending means you are capacity-bound.
• gitlab_runner_errors_total: runner-side errors by type. A rise after a change tells you the change is the suspect.
• Autoscaling histograms: how long provisioning takes. If pending time is high but the runner is below its limit, the autoscaler is the slow part.

High-cardinality metrics (per-job labels) are behind FF_EXPORT_HIGH_CARDINALITY_METRICS, off by default because they can overwhelm your metrics system. The documentation provides Grafana dashboards for a runner fleet, and Runner 19.0 started emitting OpenTelemetry job spans, so you can see where a job spent its time (waiting for the pod, cloning, restoring cache) in your tracing system. The Prometheus guide covers alert design; for CI, alert on symptoms: pending time above a threshold, runner error rate, and autoscaler provisioning time, rather than on CPU of the runner host.

On the GitLab side you have CI/CD analytics (from 19.1, success and failure rates count only completed pipelines, so a still-running pipeline no longer skews them), pipeline and job duration charts, the Runner fleet dashboard (Ultimate), the job token authentication log and audit events. Use the pending-time measure above all, because it is the one users feel. Publish it, and publish your target, so the platform team and its customers argue about a shared number.

groups:
  - name: gitlab-runner
    rules:
      - alert: RunnerAtLimit
        expr: gitlab_runner_jobs / gitlab_runner_limit > 0.9
        for: 10m
        labels: { severity: warning }
        annotations:
          summary: "Runner {{ $labels.runner }} has been above 90% of its limit for 10 minutes"

Treat the expression above as a starting point: the exact label names depend on your scrape configuration, so check them against a live /metrics response before you rely on the alert.

Upgrades, deprecations and the road to 20.0

Because GitLab releases monthly and CI is the thing everything else depends on, upgrades are a standing operational duty rather than an event. The rules are few but strict, and each has a failure that teaches it.

Follow the required upgrade stops. On Self-Managed you cannot jump across certain minor versions, because database migrations are staged. For 18.x the stops were 18.2, 18.5, 18.8 and 18.11; for 19.x they are 19.2, 19.5, 19.8 and 19.11. Use the official upgrade path tool, and read the version-specific notes (gitlab_19_changes and its predecessors) before the window, not during it. The platform minimums also moved in 19.0: PostgreSQL 17 is the minimum, Redis 6 support was removed, and the Helm chart no longer bundles PostgreSQL, Redis or MinIO, which means a chart-based installation must bring its own.

Keep the runner's major.minor aligned with GitLab's. Minor versions are backward compatible, but features arrive in matching pairs. For example, secrets:gitlab_secrets_manager needs Runner 19.0 or later. When you pin a runner package, pin the helper images with it (since 17.7.1, gitlab-runner=19.4.1-1 gitlab-runner-helper-images=19.4.1-1). Upgrading is otherwise sudo apt update && sudo apt install gitlab-runner, and note that the package prunes unused Docker images unless NO_DOCKER_PRUNE=1 is set, which can surprise a host that depends on a pre-pulled image. Drain before you restart: SIGQUIT lets running jobs finish and then stops the process, where SIGTERM aborts them.

Watch for long migrations. The 18.7 CI builds metadata tables produced a large background migration. On a large instance plan for it, and tune it with the cutoff variables mentioned earlier.

Roll runners in waves. Upgrade a canary pool first, run a synthetic pipeline on it, then the general pool, then protected pools. A runner regression stops every job on that pool, which is why the protected pool, which builds releases, goes last.

The 20.0 list

GitLab deprecates on a schedule and removes in major releases, and the next one is 20.0, expected May 2027. GitLab.com applies breaking changes in announced windows (for 19.0 that was 4 to 6 May 2026). The things that still work today and will stop are the platform owner's backlog:

Older items already removed, which still appear in copied pipelines, are worth a repository-wide search: CI_JOB_JWT and its variants (removed in 17.0, replaced by id_tokens), and the Terraform CI/CD templates (removed in 18.0; use the OpenTofu component, which the Terraform guide places in context). Deprecated but still functioning: only and except (use rules), top-level image, services, cache, before_script and after_script (use default:), the magic pages job name and job-level publish (use the pages: keyword and pages.publish), trigger:strategy: depend (use mirror, from 18.2), and environment:kubernetes:namespace (use environment:kubernetes:dashboard:namespace).

The 19.1 reclassification of failure reasons has a subtle effect: some failures that used to be script_failure are now runner_configuration_error, and some runner_system_failure are now runner_external_dependency_failure or runner_interrupted. If any of your automation retries on a specific reason, re-test it, because the set of failures that match changed under it. Old retry:when values still work as aliases with a lint warning, which is precisely the state in which they get ignored until they break.

The efficient way to run this is a deprecation burndown: a script that searches every project's configuration and every runner host for the patterns in this section, opens an issue per hit, and reports a count each month. A number trending to zero is a plan; a wiki page of "things to fix someday" is not.

# Find projects still using deprecated keywords (example using the GitLab API and jq)
for id in $(curl -s --header "PRIVATE-TOKEN: $TOKEN" \
  "https://gitlab.example.com/api/v4/groups/$GROUP_ID/projects?per_page=100&include_subgroups=true" | jq '.[].id'); do
  curl -s --header "PRIVATE-TOKEN: $TOKEN" \
    "https://gitlab.example.com/api/v4/projects/$id/repository/files/.gitlab-ci.yml/raw?ref=HEAD" \
    | grep -lE '^(only|except):|registration-token' >/dev/null && echo "project $id"
done

Incident playbooks

Platform incidents have a recognisable shape: many people report different symptoms of one cause. The skill is to establish the blast radius first, then narrow it. The timeline below is the order I use for the most common report, "CI is down."

Jobs are stuck in pending

The message This job is stuck because you don't have any active runners that can run this job. means no online runner satisfies the job. Check four things in order. Is any runner online? (gitlab-runner verify on the host; "Online" in the UI.) Does a runner carry all of the job's tags? Does the runner accept untagged jobs if the job has no tags? Is the runner protected while the job is on an unprotected ref? After an hour the failure reason becomes stuck_pending_no_matching_runners; a job with matching runners that simply never got capacity is dropped after 24 hours as stuck_pending_with_matching_runners. If runners are online and matching but the queue is long, you are capacity-bound, not misconfigured: look at the running gauge against the limit, and at autoscaler provisioning time.

A bad component version went out

Symptom: many unrelated projects fail on the same job within minutes. Mitigation is to pin the last good version, not to hurry a fix into the component. If consumers reference @2 (a partial version), the fastest safe action is to publish a corrected patch release, or move consumers back to the last good exact version. Then add the missing test to the component's own pipeline. This is why the versioning rules matter more than the code.

A secret was exposed

If a credential appeared in a job log, treat it as public: rotate first, then clean up. Deleting a job log or the job removes access to the text but not the fact that anyone with pipeline access may have already read or copied it. Then find the cause, which is usually one of three things: CI_DEBUG_TRACE was on, a script printed the environment, or the value was not masked. Then fix it structurally: disable debug trace on shared runners, move the credential to OIDC or an external secret manager, and scope what remains by protected and environment.

The runner host or a job was compromised

Because a job is arbitrary code, assume a compromised job is always possible and design so the blast radius is one ephemeral environment. Revoke by resetting the runner's authentication token, terminate the VM or pod, and rotate anything the job could reach: its variables, its OIDC roles' sessions, and cached data. Check the job token authentication log for which projects the token touched. Then ask the question that determines whether it could happen again: was the runner one-job-per-VM and protected-only? If not, that is the fix.

The upgrade went wrong

Symptoms are data integrity failure after an upgrade (a database sequence or Sidekiq version skew: check PostgreSQL and Sidekiq logs and restart any stale Sidekiq processes), or mass pipeline-creation delay from a background migration. Keep the last backup and the runbook to hand before the window, and do not start the next upgrade stop until the background migrations from the last one have completed.

Where GitLab CI/CD stops

An honest platform owner knows what the tool is not for. GitLab CI/CD is a pipeline engine tightly bound to a Git forge, and it excels when the source, the review, the pipeline, the registry and the security scanning live in one product. It becomes awkward in four situations.

Continuous delivery to Kubernetes at scale. GitLab can push deployments from a job, but for many clusters and many services a pull-based GitOps controller is safer and gives continuous reconciliation and drift correction. A job that runs kubectl apply has no idea whether the cluster still matches an hour later. Use GitLab to build and promote artifacts and let Argo CD or Flux deploy them.

Infrastructure provisioning with state and plans. Pipelines can run tofu apply, but locking, plan review and policy checks are what purpose-built Terraform automation adds. The CI/CD templates for Terraform were removed in 18.0; if you run infrastructure as code from GitLab, use the OpenTofu component and read the Terraform guide for the state and review model.

Workflows that are not CI. Long-running data and ML pipelines with backfills, sensors, retries over days and rich dependency graphs belong in an orchestrator such as Airflow. A .gitlab-ci.yml with 200 jobs that "orchestrates" a nightly data job is a signal to move it.

Teams that live elsewhere. If an organisation's code is on GitHub and the teams are productive with GitHub Actions, forcing GitLab CI on them is a migration with a cost. GitLab supports CI/CD for external repositories through mirroring, and migration guides exist for GitHub Actions, Jenkins, CircleCI and Bamboo, but run that decision on evidence: where does the code live, and who has to change their habits? Consolidation is a benefit only if it is cheaper than the friction it causes.

Finally, the experimental features (the run: keyword, Moa expressions, job-router dispatch behind FF_USE_JOB_ROUTER) are not a basis for production standards. Follow them, try them, and tell teams plainly that they are experiments.

Running GitLab CI/CD as a platform

Everything above is mechanism. Running it as a platform is mostly about the relationship with the teams that use it, and a few habits make the difference.

Publish a paved road. Provide a small set of components and a base template (with workflow:rules, interruptible, the default: section, the safe artifact defaults, and mirrored images) so the secure, cheap, correct pipeline is also the easiest one. Teams that can get a working pipeline in ten lines do not copy a stranger's two hundred.

Version and communicate like a product. Keep a changelog for your components, announce deprecations with dates, and run the burndown. Every breaking change should have a migration note and, if you can manage it, an automated merge request that performs it across projects.

Set service levels you can measure. Pending-time targets, pipeline-creation latency, runner availability. Publish the numbers. A platform without a stated objective is judged by its worst day.

Offer a fast lane for exceptions. There will be a team with a legitimate need for a privileged runner or a special image. A documented, time-limited, reviewed exception process keeps those requests from becoming uncontrolled workarounds.

Keep the break-glass path, and rehearse it. Restore a runner from scratch. Roll back a component. Upgrade a staging GitLab through a required stop. Procedures you have rehearsed are the ones that work at three in the morning.

Give every decision an owner and a review date. A runner pool, a policy project, a component: each has a named owner, and each appears on a review calendar. Orphaned platform pieces are how privileged runners outlive the team that needed them.

The review checklist

Use this when reviewing a pull request to shared templates, a design document for a new runner pool, or an audit of a team's pipeline. Each line is something this guide taught you to check.

1. Is the runner isolated for the trust it servesEphemeral, unprivileged, and protected-only where it touches production. Shell executor never on untrusted code. Privileged only on a dedicated small pool.
2. Does the job token follow least privilegeAllowlists set on sensitive projects; fine-grained permissions; no cross-project push unless needed; the authentication log reviewed.
3. Where do the secrets come fromOIDC first, then an external manager, then protected, hidden, file-type variables. No masked-only reliance, no debug trace on shared runners.
4. Is the OIDC trust specificExact project path, protected ref or environment, one role per environment, and a refusal tested.
5. Are shared includes pinnedA full version or SHA for anything that reaches production; no ~latest and no moving branch.
6. Is enforcement in a place teams cannot editPolicy, protected branch, approval. Not a wiki page.
7. Will a duplicate or obsolete pipeline runworkflow:rules, interruptible, auto_cancel, and no interruptible deploys.
8. Is the cost boundedCaches keyed by lockfile and pull by default, artifacts expiring, autoscaler ceiling and idle policy set, images mirrored.
9. Is the deploy ordered and gatedresource_group, protected environment, approvals, outdated-deploy prevention, and merge train enforcement where it fits.
10. Can you see it failingPending time, runner saturation and autoscaler latency measured, with an alert that has fired in a test.
11. Is anything on the 20.0 listRegistration tokens, Docker Machine, legacy job tokens, compliance pipelines, old retry:when values.
12. Can you recover without the pipelineA break-glass path exists and has been exercised.

The complete picture

Read the picture left to right as a chain of custody. At each step the question is who could have changed this, and would we know? The commit is gated by protected refs and review. The pipeline is assembled from pinned, versioned components and cannot drop the controls a policy injects. The runner that executes it is isolated for the trust it serves. The artifact it produces is signed and carries provenance, so the consumer can verify it. The deploy is serialised, approved and ordered. A weakness in any one link is the one an attacker, or a bad day, will find.

Where the series leaves you

You started by making a job pass. You now have the vocabulary to describe a whole delivery system: how work reaches a runner, what that runner is allowed to touch, who can change the configuration everyone depends on, what a pipeline costs, how to see it failing and how to keep it current through a release cadence that never stops. The most important habit in this guide is the review one: nearly every platform incident is visible in a diff, a settings page or a dashboard before it becomes an outage, if someone is looking with the right questions.

Next, follow the delivery chain outward. The Kubernetes guide is where your runners and your deploys meet a cluster, the Helm guide and the Argo CD guide cover packaging and pull-based delivery, and the Terraform guide covers the infrastructure those runners sit on. Keep the release notes in your calendar, because the next breaking window is May 2027, and the teams that treat that date as a project rather than a surprise are the ones whose platform stays boring.

Sources

• GitLab CI/CD documentation
• CI/CD YAML syntax reference
• Deprecated keywords
• Includes
• Workflow keyword
• CI/CD inputs
• CI/CD components
• CI/CD variables
• Caching
• Job artifacts
• Job token
• Job inputs
• Secrets
• ID token authentication
• GitLab Secrets Manager
• Merge trains
• Merge request pipelines
• Downstream pipelines
• Runners
• Runner scope
• Creating runners (new workflow)
• CI/CD instance limits
• Instance limits
• Deprecations and removals
• Breaking change windows
• GitLab 19 upgrade notes
• GitLab 19.4 release notes
• GitLab 19.0 release notes
• GitLab Runner
• Runner installation on Linux
• Runner on Kubernetes (Helm)
• Runner commands
• Runner advanced configuration
• Runner feature flags
• Runner autoscaling
• Kubernetes executor
• Docker Autoscaler executor
• Instance executor
• Runner fleet scaling
• Runner monitoring
• Runner security
• GitLab Runner changelog

Part three of three. At senior level the interview stops asking what a keyword does and starts asking you to design, diagnose and defend. Expect a whiteboard prompt about a CI platform for a growing company, an incident you must triage out loud, and a question about how you would persuade a sceptical director. This file is organised the way those conversations run: the ideas to have ready, system-design prompts, incident questions, trade-off arguments, and a self-test. Verified against GitLab 19.4 and Runner 19.4.1. About fifty minutes.

The ideas to have ready

Before any prompt, hold five ideas in your head, because nearly every good senior answer is a combination of them.

Runners pull; GitLab never pushes. A runner polls POST /api/v4/jobs/request and executes what it is handed. This single fact explains why runners need only outbound HTTPS, why pending time is the best health measure, and why a compromised job cannot reach back into the control plane except through the job token.

Configuration is frozen at pipeline creation. Includes, inputs and rules are resolved on the server when the pipeline is created. Retrying a job never picks up a YAML fix. This distinguishes creation-time failures (invalid config, missing job) from execution-time failures (a job that started and failed).

A job is arbitrary code holding credentials. Every security question reduces to: what can it reach, and who influenced what it runs? The chain of custody is protected refs, then protected and isolated runners, then scoped secrets.

A cache is best-effort; an artifact is a guarantee. Nothing correctness-critical travels through a cache, and a release build should not trust a cache that untrusted code could have written.

Enforce, do not recommend. A control that a project can delete from its own YAML is a suggestion. Real enforcement lives in a place the team cannot edit: pipeline execution policy, protected branches, required approvals.

Interviewers at this level listen for whether you walk that chain unprompted. A candidate who designs the runner fleet and forgets to ask who can edit .gitlab-ci.yml has answered half a question.

The facts that get checked

These are the specifics interviewers use to see whether you have run the platform or only read about it.

System-design prompts

The format is the same each time: state your assumptions, draw the shape, name the failure modes, then say what you would measure. Practise answering each in about ten minutes.

Prompt one: a CI platform for 300 engineers

"We are moving 300 engineers and about 600 repositories onto GitLab CI. Design the runner platform."

A strong answer begins with questions, not infrastructure. What is the workload mix: short unit tests, long builds, Docker image builds, GPU or arm64? What trust levels exist: employees only, or contractors and external contributors opening merge requests? Do they already run Kubernetes? Is there a data-residency requirement, which for employers in the Gulf or Egypt often means running the fleet and the cache bucket in a specific region?

Then the shape. I would propose four pools split by trust and by shape: a general pool for merge request pipelines (ephemeral, unprivileged); a protected pool for default-branch builds and releases; a small privileged pool for Docker-in-Docker; and a specialised pool for GPU or macOS. On Kubernetes, I would use the Kubernetes executor with jobs in a dedicated namespace under a near-powerless service account, on a dedicated node pool, with explicit resource requests. Without Kubernetes, the Docker Autoscaler executor with one job per VM, scaling on the cloud's instance group.

Then the failure modes, in the order I expect them to appear. Pending time at peak, handled with an idle policy by working hours and a max-instances ceiling sized to budget. Registry rate limits from every job pulling base images, handled with an internal mirror. Cache and artifact transfer, handled with a same-region object store, lockfile-keyed caches that default to pull, and expiring artifacts. Sidekiq and database load from mass scheduled pipelines, handled by staggering schedules.

Finally, what I would measure: pending time (median and 95th percentile), runner saturation against the limit, autoscaler provisioning time, and cost per pipeline by team. What a weak answer misses: the trust tiers, the registry, and any measurement. A weak answer says "we autoscale" and stops.

Prompt two: standardising pipelines across 600 repositories

"Every team copied a different pipeline. How do you standardise without blocking them?"

The strong answer separates what must be enforced from what should be easy. I would build a small paved road: a base template containing the standard workflow:rules, default: settings, interruptible, safe artifact access and mirrored images, plus versioned CI/CD components for build, test, scan and deploy, published to the catalog from protected tag pipelines. Consumers pin exact versions; a bot opens merge requests to bump them. Inputs are typed and validated, with options and regex, so a wrong value fails at pipeline creation.

For what must be enforced (the security scan, the signing step), I would use pipeline execution policies, because a project cannot remove a job that a policy injects, and policy variables take the highest precedence. I would retire compliance pipelines in favour of this, since they are deprecated and removed in 20.0. I would also set ci_pipeline_variables_minimum_override_role so contributors cannot override behaviour with pipeline variables, and steer them to inputs.

The part that shows seniority is the rollout: migrate the highest-traffic and highest-risk repositories first, measure the change in pipeline duration and failure rate, and keep an exception process for teams with a real need. What a weak answer misses: versioning, the difference between enforcing and recommending, and the migration path. "Write a shared template" is the answer of someone who has not had to change it.

Prompt three: secrets and cloud access without stored credentials

"Pipelines deploy to AWS and read from Vault. No long-lived keys. How?"

The design is OIDC federation. Each job declares id_tokens with an audience, GitLab signs a short-lived JWT, and AWS or Vault exchanges it for short-lived credentials. The trust policy is the security boundary. I would bind each role to exact claims: the project_path, ref_protected being true or a protected environment, and never a wildcard over a group. I would use one role per environment with least privilege, and I would test a refusal, running the job from a feature branch and confirming the assumption fails.

I would mention the operational details that show experience: the default sub claim has the shape project_path:{group}/{project}:ref_type:{type}:ref:{branch}; a 401 often means the pipeline still uses the removed CI_JOB_JWT_V2 or that aud or sub mismatch; and GitLab refuses to issue tokens when a project path was previously used by a different project, which protects against path-reuse attacks. For secrets that must stay in GitLab, use masked-and-hidden, protected, file-type variables scoped by environment, and be candid that masking is not a security boundary.

Prompt four: a supply chain you can defend

"How do you prove the image in production came from your pipeline and your source?"

Walk the chain. Source is protected and reviewed. The pipeline is assembled from pinned components. The build runs on an ephemeral protected runner. The image is tagged with the commit SHA and referenced by digest. The job signs it keylessly with cosign using an ID token with aud: sigstore, and the runner emits SLSA provenance metadata. Scanners (SAST, secret detection, dependency and container scanning) run as templates or components, with third-party SARIF ingestion available. Then the vital final step: an admission policy in the cluster refuses anything unsigned or unattested. A signature nobody verifies has no effect.

Incident-response questions

Answer these aloud in sequence: blast radius, what changed, mitigate, then diagnose. Interviewers are watching whether you restore service before you seek the root cause.

"Everyone's pipelines are pending."

First, is anything matching? Check that runners are online, that the tags on the stuck jobs match a runner, and that the runner is not protected while the job is on an unprotected ref. If runners are online and matching, you are capacity-bound: compare the running-jobs gauge to gitlab_runner_limit and concurrent. If the runner is below its limit but jobs are still pending, the autoscaler is slow or failing to provision: check the cloud quota, the instance group and the provisioning histogram. Mitigate by raising max_instances or idle count, or by routing to a fallback pool. Only then investigate why demand jumped, perhaps a scheduled pipeline storm or a matrix someone widened.

"A component release broke forty projects."

Mitigate by moving consumers to the last good exact version, or by publishing a corrected patch. Do not debug the component while forty teams are blocked. Then root-cause: why did the component's own pipeline not catch it? Add a test with the input combinations consumers use, and strengthen the rule that breaking changes need a major version. I would also ask whether consumers used a floating reference like ~latest, because that is how a change reaches everyone at once.

"A production secret appeared in a job log."

Rotate first. Deleting the log does not recall a value that people with pipeline access may already have seen. Then find the cause (debug trace enabled, env printed, value unmasked), and fix it structurally: disable CI_DEBUG_TRACE on shared runners with debug_trace_disabled, move the secret to OIDC or an external secret manager, scope what remains by protection and environment. Check the audit events for who could have seen it. Report the detection time.

"We suspect a runner was compromised."

Assume arbitrary code ran with the job's permissions. Contain: reset the runner token, terminate the VM or pod. Rotate everything the job could reach (variables, cloud roles, cache contents). Review the job token authentication log for which projects it touched. Then ask what made the blast radius what it was: was the runner one-job-per-VM, protected-only, unprivileged, with metadata endpoints blocked? The fix is whichever of those was missing.

"An upgrade stalled the instance."

Identify whether it is a background migration (the 18.7 CI metadata migration is a known long one, tunable with cutoff variables), a Sidekiq version skew (data integrity failure), or a database sizing issue. Do not start the next upgrade stop until background migrations complete. Afterwards, rehearse the next stop on a staging copy, and keep a tested backup.

"Deploys are overwriting each other."

Check resource_group on the deploy job, its process_mode, and whether "Prevent outdated deployment jobs" is enabled. For merge-time breakage, check whether merged results pipelines or merge trains are in use and whether enforcement is on, since without enforcement the UI or API can bypass the train.

Arguing trade-offs to leadership

At this level the hardest question is often not technical: "why should we spend money or time on this?" The pattern that works is to convert each technical control into a cost or risk the listener already understands, offer two options, and recommend one with a stated cost.

"Why are we paying for one-job-per-VM autoscaling when shared runners are cheaper?" Frame it as risk: shared long-lived runners let one job's leftovers influence the next, including cached files and credentials. One job per VM removes a class of incident (cross-job contamination) at a known price, roughly the cost of idle warm machines plus startup latency. Offer the middle path: isolate the protected and privileged pools fully, and let the low-risk pool share. Recommend that, and quote the number.

"Why not let teams write their own pipelines? It is faster for them." Agree that it is faster today. Then show the cost curve: forty divergent pipelines mean a security fix takes forty merge requests, an audit takes forty reviews, and a deprecation such as the 20.0 removals is forty migrations. Shared components with versioning make it one. The honest trade-off is that a paved road slows down the teams with unusual needs, which is why you offer an exception process.

"Why should we spend engineer time on deprecations that still work?" Because the removals have a date. Registration tokens, the Docker Machine executor and the legacy job token all go in 20.0, expected May 2027. A migration done in a planned quarter costs less than one done in the week the upgrade breaks production builds. Present it as a burndown with a count and a target.

"Can we just use the hosted runners and avoid running anything?" It is a legitimate option and a good one for many organisations: no fleet to operate, ephemeral VMs per job, and you pay compute minutes with a cost factor by machine size. The trade-offs are cost at high volume, the three-hour hard job cap, less control over network access and data location, and fewer options for specialised hardware. Give the crossover: when monthly compute-minute spend exceeds the cost of running and staffing your own fleet, or when you need private networking or specific regions, self-hosting wins.

"Should we migrate from GitHub Actions or Jenkins to GitLab?" Answer with the evidence you would gather: where the code lives, how much pipeline logic exists, who must change habits, and what consolidation buys (one place for review, pipeline, registry and scanning). GitLab has migration guides for both, and can run CI for external repositories. The right answer may be "not yet" if the cost of change exceeds the benefit.

Common interview questions

What is the difference between concurrent, limit and request_concurrency? concurrent is the global cap on jobs one runner process runs across every [[runners]] block. limit caps a single runner configuration (total workers on an autoscaler). request_concurrency is how many jobs it may request from GitLab in parallel, with adaptive behaviour on by default. A common mistake is raising limit and seeing no change because concurrent is the binding cap.

How do you register a runner today? Create the runner in GitLab (UI or POST /user/runners) so scope, tags and description live on the runner object, receive the glrt- authentication token, then gitlab-runner register --url ... --token glrt-.... Registration tokens and the flags that carried scope settings are deprecated and removed in 20.0.

Which executor would you choose and why? Kubernetes when the company runs it, because you inherit scheduling and autoscaling. Docker Autoscaler when you need VM-level isolation without Kubernetes. Instance when a job needs a whole machine. Docker for a single host. Shell only for trusted code such as macOS builds. Docker Machine is deprecated and goes in 20.0.

How does the job token work, and what changed recently? It is a short-lived per-job token with the triggering user's permissions but a limited API surface. Since 18.0 the target project's allowlist controls incoming access, enforced by default. Since 19.0 it is a JWT by default, with a legacy opt-out until 20.0. Cross-project push to allowlisted projects became generally available in 19.1.

Why is masking not enough? It matches the literal value in the log. Printing the environment, encoding the value or inserting characters bypasses it, and anyone who can edit the script can read every variable the job sees. Use protected scopes, file-type variables and external secrets, and treat masking as protection from accidents.

How do you keep a cache from becoming an attack path? Keep trust tiers apart, keep the setting that separates protected-branch caches on, and do not let release jobs restore a cache written by untrusted pipelines. Accept a slower release build over an influenced one.

Merge request pipelines, merged results and merge trains? A merge request pipeline tests the source branch. Merged results tests the source combined with the target. A merge train queues merge requests and tests each on top of those ahead. Enforcement (19.3) stops merges bypassing the train.

What are inputs and why prefer them to variables? Typed, validated parameters interpolated at pipeline creation with $[[ inputs.name ]]. They take no part in variable precedence, so they cannot be used to override a job. Pipeline-level inputs need defaults because merge request pipelines cannot supply them.

How do you version a component? Semantic versions, exact pins for critical consumers, release from a protected tag pipeline with the release: keyword, tests that cover consumer input combinations. Partial versions and ~latest resolve only to published catalog releases.

What would you watch on a runner fleet? Pending time, running jobs against the limit, runner errors, autoscaler provisioning time, and cost per pipeline.

Where does GitLab CI stop? GitOps-style delivery to many clusters, state-heavy infrastructure provisioning, long-lived data orchestration and teams whose code is elsewhere.

What a strong answer covers that a weak one misses

The difference is rarely knowledge of a keyword. It is four habits.

Sixty-second self-test

Answer each aloud without looking back. If you hesitate on more than two, reread the matching section of the detailed guide.

1. Why do runners need only outbound HTTPS, and what does that imply for compromise?
2. Which three numbers govern runner throughput, and which one is the top-level cap?
3. Name the four actively developed executors and the one removed in 20.0.
4. What is the current way to register a runner, and what is deprecated?
5. What does the job token allowlist control, and in which direction?
6. Why is masking not a boundary, and what do you use instead?
7. What makes an OIDC trust policy tight rather than loose?
8. Why pin a component to an exact version?
9. What is the difference between enforcing and recommending a pipeline control?
10. What are the next required upgrade stops for 19.x?
11. How do resource_group and outdated-deployment prevention differ from a merge train?
12. Name three things on the 20.0 list.

Your goal is not recall of trivia. It is that, given any prompt, you can walk commit, pipeline, runner, secrets and deploy and say who could have changed each, and how you would know.

Part three of three. At this level the problems are operational: a template that forty teams include, a runner pool that saturates on Monday morning, a token that can reach more than it should, a deprecation nobody tracked. Start with the error table, then the practices, the verification recipes, the review checklists and the platform habits. Verified against GitLab 19.4 and Runner 19.4.1.

Production traps that only appear at scale

Cumulative: everything from Beginner and Mid still applies. These rarely fail a single pipeline; they degrade a platform, often without an error message.

The practices that pay off most

Practice cards

Card: the shared base template. Every team includes one base template. It contains the workflow:rules that prevent duplicate pipelines, a default: section (image from your mirror, interruptible: true), safe artifact access, and a short retry policy for runner failures. One merge request to this file changes the whole company, so it needs code owners and a test project that includes it.

workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
      when: never
    - if: $CI_COMMIT_BRANCH

default:
  image: registry.example.com/mirror/alpine:3.20
  interruptible: true
  retry:
    max: 1
    when:
      - runner_system_failure
      - runner_unsupported

Card: the hardened runner pool. Ephemeral, one job per VM or pod, unprivileged, pull_policy = ["always"], allowed_images restricted, metadata endpoints blocked, debug_trace_disabled = true. For protected pools add the protected flag and group scope. Treat any deviation as a documented exception with an owner and an expiry.

Card: the trust policy. A role per environment, bound to exact project_path and ref_protected or a protected environment. Decode a token to write it and test a refusal from a feature branch.

Card: the component release. Semantic version, protected tag pipeline with release:, a test matrix in the component's own pipeline, a changelog entry, and a note on whether it is breaking. Consumers pin the exact version.

Card: the deploy job. resource_group, a protected environment with approvals, rules restricting it to the default branch, "Prevent outdated deployment jobs" on, and never interruptible.

Recipes: verifying instead of assuming

A platform accumulates settings you believe are in force. The recipes below check them. Each takes a few minutes and finds real problems.

Check the runner configuration before you restart

A misspelled key is silently ignored, and you find out from behaviour or the bill. Runner 19 includes a linter:

gitlab-runner lint --config /etc/gitlab-runner/config.toml
echo "exit code: $?"

It validates the TOML, the schema and unknown keys, and exits non-zero on a problem. Run it in the pipeline that deploys the runner configuration, and refuse to deploy on failure. Then confirm the runner is alive:

gitlab-runner verify
gitlab-runner list

verify should print Verifying runner... is alive. A runner that GitLab has deleted shows as no longer valid, and gitlab-runner verify --delete removes stale entries from the local file.

Check what a runner host exposes

From inside a job on each pool, test what a compromised job could reach. The cloud metadata address is the one to check first.

probe-metadata:
  stage: test
  script:
    - curl -s -m 3 http://169.254.169.254/ && echo "REACHABLE: fix this" || echo "blocked"
    - id
    - cat /proc/self/status | grep -E 'CapEff|NoNewPrivs'
  rules:
    - if: $CI_PIPELINE_SOURCE == "web"

If the metadata address answers, a job can read the runner VM's cloud credentials. If CapEff shows a wide capability set on a pool that should be unprivileged, the pool is not what you think it is.

Check what the lint says a pipeline will become

glab ci lint --dry-run --include-jobs --ref main

The dry run creates a simulated pipeline and lists the jobs, so you see the effect of rules and workflow:rules before anything runs. Use it in a merge request check for changes to shared templates.

Check what a token can do

Decode an ID token to see the claims your trust policy will see:

echo "$MY_ID_TOKEN" | cut -d. -f2 | base64 -d 2>/dev/null | jq '{sub, ref, ref_protected, environment, project_path}'

Run the same job from a protected branch and a feature branch and compare. A policy you wrote from documentation rather than from a real token is a guess.

Check the API for a runner's status and pipelines

curl -s --header "PRIVATE-TOKEN: $TOKEN" \
  "https://gitlab.example.com/api/v4/runners/all?status=online&per_page=100" | jq 'length'

curl -s --header "PRIVATE-TOKEN: $TOKEN" \
  "https://gitlab.example.com/api/v4/projects/$PROJECT_ID/pipelines?status=pending&per_page=50" | jq 'length'

A rising count of pending pipelines beside a steady count of online runners is the shape of a capacity problem; a count of online runners that drops is a runner problem. Put both on a dashboard.

Rotate a runner token

gitlab-runner reset-token --name "shared-linux"

Do this on a schedule in a quiet window, and after any suspected leak. The point of the new workflow is that rotation is a one-command operation; if yours is not, fix that before an incident forces it.

Drain a runner without killing jobs

sudo kill -SIGQUIT "$(pgrep -f 'gitlab-runner run' | head -1)"

SIGQUIT stops the runner accepting new jobs and exits after running jobs finish; SIGTERM or SIGINT aborts them. Use the graceful form for every rolling change, and inside a container remember that gitlab-runner restart starts a new process, so restart the container itself to apply configuration.

Review checklists

A platform is protected mostly at review time. Two checklists follow, for the two kinds of review a platform owner does most: a merge request that changes configuration, and a design document that proposes something new.

Reviewing a merge request to shared pipeline configuration

1. Who is affectedEvery project that includes this file. If you cannot say how many, find out before approving. A change to a base template reaches everyone on the next pipeline.
2. Is it breakingA renamed input, a removed job, a changed default stage or a new required input is breaking and needs a major version, not a patch.
3. Is anything unpinnedNew include lines must reference an exact version or SHA. No ~latest, no moving branch.
4. Does it widen trustNew variables exposed to merge request pipelines, new runner tags, privileged, new allowed_* entries, a broader OIDC audience.
5. Does it increase costNew matrix or parallel fan-out, a new cache, larger artifacts without expire_in, a job missing interruptible.
6. Did it use a deprecated formonly and except, top-level image, trigger:strategy: depend, old retry:when reasons. Do not add to the burndown.
7. Was it testedA project that includes the change and ran it, with a link. "Should work" is not a test.
8. Can it be rolled backBy pinning consumers back to the previous version. If not, say why.

Reviewing a design document for a runner pool or a pipeline platform

1. What trust does it serveEmployees only, contractors, external contributors? The pool's isolation must match its least trusted user.
2. What is the executor and whyKubernetes, Docker Autoscaler, Instance or Docker. Anything on Shell, SSH or Docker Machine needs a migration plan for 20.0 and a justification.
3. What are the limits and ceilingsconcurrent, limit, max_instances, idle policy. A maximum is a cost statement: what is the worst-case bill?
4. How does it failName the first component to saturate, and the signal you will see. "It scales automatically" is not an answer.
5. How is it observedMetrics scraped, dashboards, an alert that has fired in a test, and a pending-time target.
6. How are secrets handledOIDC first; where the runner token lives and how it rotates; how debug trace is disabled.
7. How is it upgradedCanary pool first, version alignment with GitLab, drain procedure, and a rollback.
8. Where is the dataRegion for runners, cache bucket, artifacts and registry, particularly where an employer or client has residency requirements in the Gulf or Egypt.
9. What is the exitWhat happens when a cloud or a tool is retired. Fleeting plugins and executors are replaceable; hand-tuned hosts are not.

Platform-ownership habits

The difference between a platform and a collection of runners is a set of habits that are dull and cumulative.

Write down the contract. What a team can expect: pending-time target, supported executors and images, how long artifacts live, how long a deprecated feature stays supported, how to ask for an exception. A contract that is written can be met or missed; one that is assumed is merely argued about.

Change in small, reversible steps. Canary the new runner version, the new component, the new policy. A change that affects everyone at once should be rare and scheduled.

Announce with a date and a migration note. "We are removing registration tokens" is noise. "Registration tokens stop working in GitLab 20.0, expected May 2027; here are the eleven repositories that still use them and the three-line change" is a work item.

Own the boring middle of the lifecycle. Stale runners (not seen for seven days count as stale for registration-limit purposes), orphaned projects with scheduled pipelines still firing, artifacts nobody has downloaded, variables in groups nobody owns. Run a quarterly sweep.

Treat the platform's own pipeline as production. The pipeline that deploys runner configuration, templates and policies needs review, tests and a rollback. It is the most privileged code in the company.

Keep the people side healthy. Office hours, a channel for questions, a visible backlog, a short blameless write-up after each incident. Teams that feel heard raise problems early; teams that do not route around you.

Cost and performance levers worth knowing

These are ordered by typical leverage, most first.

1. Do not run unaffected jobs. rules:changes, with compare_to on branch pipelines, and the regex forms added in 19.2. Test the rule: a changes rule that never matches saves money and skips tests.
2. Cancel what is obsolete. interruptible plus workflow:auto_cancel:on_new_commit, and on_job_failure where fail-fast is acceptable.
3. Use the DAG. needs: reduces wall-clock time without adding compute.
4. Cache carefully. Lockfile key, pull by default, size measured against the install it replaces, CACHE_COMPRESSION_FORMAT chosen for the network. Fastzip is the default archiver from 19.4.
5. Shape the autoscaler. Idle policy by periods and timezone; max_use_count and capacity_per_instance for the isolation you need.
6. Use arm64 where your workload allows. Often cheaper per minute; check the images you depend on publish arm64.
7. Clean up. artifacts:expire_in, automatic pipeline cleanup, stale-runner removal.
8. Mirror images. Fewer failures and faster starts, and one place to patch.

A warning about over-optimising: shaving thirty seconds from a pipeline that runs twice a day is not worth a week of your time. Optimise by impact: the pipelines that run hundreds of times a day, the jobs on the critical path, and the pools that are the bottleneck for everyone.

Version notes worth keeping on the wall

Four changes from the last year account for a large share of surprises, so keep them where the team can see them.

• 19.0 (May 2026): job tokens are JWT by default; array inputs; include:cache GA; PostgreSQL 17 minimum and Redis 6 removed; runner OpenTelemetry job spans.
• 19.1: cross-project push with the job token GA; retry:when legacy reasons deprecated; SARIF 2.1.0 ingestion; get_sources_timeout.
• 19.2: rules:changes:regexp and rules:exists:regexp; four-caches-per-job limit GA; scheduled pipeline execution policies GA.
• 19.3 and 19.4: CI_JOB_RETRY_COUNT and CI_JOB_TAGS; merge train enforcement GA; Fastzip default; services_cap_add and services_cap_drop; group-based Secrets Manager permissions removed.

GitLab 19.5 is due on 15 October 2026. Features documented as 19.5 (for example needs:pipeline:job:optional) are not yet in a stable release, so do not put them in a template teams rely on today. The next major release, 20.0, is expected in May 2027.

The checklist to run before you call a platform change done

Tick these, honestly, before approving a rollout.

• I can say how many projects the change affects, and I tested it in at least one.
• Nothing new is unpinned, privileged or wider than it needs to be.
• Every trust boundary I changed has a refusal I have seen, not only a success.
• The change is reversible, and I know the rollback command.
• Pending time, runner saturation and autoscaler latency are measured, and I know the baseline.
• Cost impact is estimated, including the worst case of the ceiling I set.
• No new use of a deprecated form, and nothing added to the 20.0 burndown.
• The runner configuration passed gitlab-runner lint and the runner version matches GitLab's major.minor.
• Consumers know what is changing, by when, and what to do about it.
• A break-glass path still works if this change breaks the pipeline that deploys it.

A platform is a set of promises to other teams. Every item above is a way of checking a promise before it is tested by an outage.