The Complete Ansible Guide

This is part one of three. It covers everything you need to do real work with Ansible, not a teaser. By the end you can describe a server's desired state in a YAML file, run it against one machine or fifty, see exactly what changed, run it again and watch nothing change, keep a password out of your repository, and read the error messages when something goes wrong. Mid-level and Senior take the same topics further; nothing here is thrown away.

This guide targets ansible-core 2.21, which is the engine inside the Ansible community package 14. Where an older tutorial you find online disagrees with what you read here, trust this page, and the "Common errors" section explains the most frequent differences.

Each section ends with a Try it task. Do them as you go. They take a few minutes each, and Ansible only makes sense once you have watched your own playbook change something, run again, and report that there was nothing left to do.

What Ansible is, and the problem it solves

Ansible is a tool that configures machines from a description you write in text files. You list the machines you care about, you write down what state each one should be in (this package installed, this file present with this content, this service running), and Ansible connects to each machine and makes it so.

The diagram is the whole idea. Compare it to what came before.

Before tools like this, a new server was set up by a person following a wiki page, or by a shell script someone wrote three years ago and nobody dared touch. Both approaches share a flaw: they describe steps, not state. A script that says "append this line to the config file" is correct the first time and wrong the second, because now the line is there twice. So scripts grow if statements to check whether each step was already done, the checks get subtly wrong, and eventually every server in the fleet is slightly different from every other one. Engineers call this configuration drift, and it is the reason the phrase "nobody knows how this server was built" exists.

Ansible's answer is that you write the desired end state and each building block (called a module) is responsible for checking whether the machine is already in that state. If it is, the module does nothing and reports ok. If it is not, the module fixes it and reports changed. That property is called idempotency, and it is the single most important idea in this guide. It means you can run the same playbook every day, against a machine that is brand new or a machine that has run for five years, and the result is the same.

Three consequences of the design explain most of what follows. Notice them now rather than discovering them later.

The description is a file you commit. Your playbooks are plain YAML that sit next to your application code, so they are versioned, reviewed and diffed like any other code. When a server behaves strangely, git log tells you who changed its configuration and why.

There is nothing to install on the machines you manage. Ansible connects over SSH, the same way you log in by hand, and uses the Python that is already on the target. There is no agent to deploy, patch, secure or monitor. This is why people call Ansible agentless, and it is why you can start using it on a server that already exists today without any preparation beyond SSH access.

It runs from wherever you are. The machine you type ansible-playbook on is called the control node, and it can be your laptop. Nothing runs permanently anywhere. When the run finishes, Ansible has left no daemon behind.

What people use it for:

You need little to follow along: a terminal on Linux or macOS (Windows users can use WSL, covered in the install section) and Python. You do not need a server. The first half of this guide runs entirely against your own machine, and the later sections show what changes when the target is a real remote host.

The agentless model: control node and managed nodes

Two words describe every Ansible setup, and getting them clear early saves confusion later.

The control node is the machine where Ansible is installed and where you run commands. It must be Linux, macOS, or a BSD. Native Windows cannot be a control node; Windows users run Ansible inside WSL (the Windows Subsystem for Linux), a virtual machine, or a container. The managed nodes are the machines Ansible configures. You will also see them called hosts or targets. They need no Ansible software at all.

What a managed node does need is a way in and a Python interpreter. For Linux and other POSIX machines that means SSH access and Python 3.9 or newer (for ansible-core 2.21, Python 3.9 through 3.14 on the target; the control node needs 3.12 through 3.14). Windows managed nodes need PowerShell and are reached over WinRM, PSRP or SSH instead. A handful of modules (raw and script) work even without Python, which matters when you need to install Python itself on a bare machine.

Here is what actually happens when a task runs, because it explains both the speed characteristics and several error messages.

1. Ansible reads your playbook and inventory on the control node.
2. For each task, it takes the module (a small piece of Python code) and packages it with its arguments.
3. It copies that package to the target over the connection, which is SSH by default, into a temporary directory.
4. It runs the package on the target using the target's Python interpreter.
5. The module checks the current state, changes what needs changing, and prints a JSON result.
6. Ansible reads the JSON, reports ok, changed or failed, and deletes the temporary files.

That is called a push model: the control node pushes work out. The opposite, where each machine pulls its instructions from a central place on a schedule, also exists (ansible-pull), but beginners should start with push.

Because the target's Python runs the module, Ansible has to find that Python. By default it does this automatically (interpreter_python = auto) and sometimes prints a warning that it found one and a different one could appear later. That warning is harmless and we will silence it in the configuration section. The older discovery modes auto_legacy and auto_legacy_silent were removed in ansible-core 2.21, so if you see them in an old blog post, skip that advice.

The four nouns: inventory, module, task, playbook

Ansible has a lot of vocabulary, but a beginner needs four words, and every other term hangs off them.

Two more terms appear constantly. A play connects a group of hosts to a list of tasks: "on these machines, do these things". A playbook contains one or more plays, which is why a playbook can first prepare the database servers and then configure the web servers in a single file. Handlers, which you will meet later, are tasks that run only when something changed.

Modules deserve a closer look, because the quality of your Ansible code is mostly the quality of your choice of modules. A module is a small program with one job: ansible.builtin.user manages user accounts, ansible.builtin.file manages files and directories, ansible.builtin.dnf installs packages on Fedora and Red Hat systems. Each module takes arguments (the user's name, the file's permissions), knows how to check the current state, and knows how to change it. Hundreds of modules exist. You almost never write your own as a beginner; you pick the right one from the documentation.

The long name matters. ansible.builtin.copy is a fully qualified collection name, abbreviated FQCN. It has three parts: a namespace (ansible), a collection (builtin) and the module (copy). Collections are how Ansible distributes modules: ansible.builtin ships with the engine, and others such as community.general or amazon.aws are installed separately. Older tutorials write just copy, which still works, but writing the full name is now the standard. It removes any doubt about which copy you mean, and linters expect it. This guide uses FQCNs everywhere.

Installing Ansible and checking the setup

The community package, called simply ansible, contains ansible-core plus a large set of curated collections. The smaller package, ansible-core, contains the engine and the ansible.builtin collection only. As a beginner, install the full ansible package, because it saves you from installing collections one at a time later. Both need Python 3.12 or newer on the control node.

The officially recommended installer on Linux is pipx, which gives each Python tool its own isolated environment so it cannot break your system Python.

pipx install --include-deps ansible       # the full community package
pipx install ansible-core==2.21.4         # or: only the engine, pinned
pipx upgrade --include-injected ansible   # later, to upgrade

If you prefer a plain virtual environment, which is easy to delete and recreate, this works on any system with Python 3.12 or newer:

python3 -m venv ~/.venvs/ansible
~/.venvs/ansible/bin/pip install ansible
export PATH="$HOME/.venvs/ansible/bin:$PATH"

Modern Linux distributions refuse pip install into the system Python and print externally-managed-environment. That is a deliberate protection (PEP 668), and the virtual environment or pipx is the correct answer, not a workaround flag. Distribution packages also exist and are fine for a first look, though they may lag behind:

sudo dnf install ansible-core        # Fedora, RHEL family (full ansible on Fedora)
sudo apt install ansible             # Ubuntu, after adding the Ansible PPA
sudo pacman -S ansible               # Arch
brew install ansible                 # macOS with Homebrew

On macOS, Homebrew's ansible formula tracks the community package and works well. On Windows, open an Ubuntu shell in WSL and follow the Linux instructions. A version of this problem you may meet later is managing Windows machines from Linux; that uses the winrm, psrp or ssh connection types and modules named win_*, which a later guide covers.

Now verify. Three commands tell you everything that matters.

ansible --version
ansible localhost -m ansible.builtin.ping
ansible-galaxy collection list

The first prints something like this (paths and Python version will differ):

ansible [core 2.21.4]
  config file = None
  configured module search path = ['/Users/you/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/you/.venvs/ansible/lib/python3.13/site-packages/ansible
  ansible collection location = /Users/you/.ansible/collections:/usr/share/ansible/collections
  executable location = /Users/you/.venvs/ansible/bin/ansible
  python version = 3.13.9 (main, ...)
  jinja version = 3.1.6
  pyyaml version = 6.0.3 (with libyaml v0.2.5)

Read it line by line, because it answers the questions you will ask when something breaks. config file = None means no ansible.cfg was found in the directory you are in; we will create one shortly. The collection location is where Ansible looks for installed collections. The python version is the control node's interpreter, not the target's. If you installed the full package, ansible-community --version prints the community version (for example 14.4.0), which is a different number from the core version; both are real and they answer different questions.

The second command is the smallest possible end-to-end test. It asks Ansible to run the ping module against the implicit localhost:

localhost | SUCCESS => {
    "changed": false,
    "ping": "pong"
}

The ping module is not a network ping. It checks that Ansible can reach the host and run Python there, which is exactly what every other module needs. "changed": false tells you it did not modify anything. You may see a warning before it about the discovered Python interpreter; that is the harmless warning mentioned earlier.

A practice lab you can run anywhere

Learning configuration management usually demands a spare server, and that blocks many beginners. You do not need one. Ansible can treat your own machine as a managed node using the local connection, which skips SSH and runs modules directly. Everything in the first half of this guide uses that, working only inside a folder you create, so nothing here can damage your system.

mkdir -p ~/ansible-lab && cd ~/ansible-lab

Every command from here on assumes you are inside this folder. The first file to create is a small configuration file. Ansible reads ansible.cfg from the current directory, and keeping one per project means a teammate who clones your repository gets the same behaviour you had.

[defaults]
inventory = inventory.ini
interpreter_python = auto_silent

The first line says where the inventory is, so you can leave off -i from every command. The second silences the interpreter discovery warning by using the auto_silent mode (the other valid values are auto, which warns, or an explicit path). Now the inventory:

[lab]
localhost ansible_connection=local

That is one group called lab containing one host called localhost, and the variable ansible_connection=local tells Ansible to run modules directly instead of through SSH. Check that Ansible agrees with you:

ansible-inventory --graph

@all:
  |--@ungrouped:
  |--@lab:
  |  |--localhost

@all is a group that always exists and contains every host. @ungrouped holds hosts that belong to no group, and yours is in lab. Finally, check the lab works:

ansible lab -m ansible.builtin.ping

If you see SUCCESS, the lab is ready. You now have a fully working Ansible project in three small files.

If you do have a spare machine, such as a cloud virtual machine, a Raspberry Pi or a virtual machine on your laptop, the later sections on real servers show the two changes needed: an address in the inventory and SSH access. Everything else in this guide is identical.

Inventory: telling Ansible about your machines

The inventory answers "which machines?" and it carries the facts that are specific to each machine: its address, the user to log in as, the port. The simplest format is INI, which is what you just used, and the other common format is YAML. Both describe the same thing.

[web]
web01 ansible_host=10.0.0.11
web02 ansible_host=10.0.0.12

[db]
db01 ansible_host=10.0.0.21 ansible_user=admin ansible_port=2222

[prod:children]
web
db

[web:vars]
http_port=80

Every line deserves a reading. A name on its own (web01) is the inventory name, the label you use in commands and playbooks. ansible_host is the real address to connect to, which lets you use friendly names while connecting by IP. ansible_user and ansible_port override the login user and the SSH port for that machine. A section in square brackets starts a group. A section named [prod:children] makes prod a group whose members are other groups, so prod contains everything in web and db. A section named [web:vars] sets variables for every host in web. Ranges save typing: web[01:10].example.com expands to ten hosts.

The same inventory in YAML:

all:
  children:
    web:
      hosts:
        web01:
          ansible_host: 10.0.0.11
        web02:
          ansible_host: 10.0.0.12
      vars:
        http_port: 80
    db:
      hosts:
        db01:
          ansible_host: 10.0.0.21
          ansible_user: admin
          ansible_port: 2222

Pick one style and stay with it; INI is shorter for tiny inventories and YAML is easier once variables get structured. As a project grows, per-group and per-host variables move out of the inventory file into folders named group_vars/ and host_vars/ next to it, a pattern the variables section explains.

Once you have an inventory, you select hosts from it with patterns. A pattern appears in a playbook's hosts: line and in the --limit option.

Because the pattern language is small and powerful, always check what a pattern selects before running anything that changes state.

ansible-inventory --graph                 # the tree of groups and hosts
ansible-inventory --host web01            # the variables Ansible will use for one host
ansible web:!db --list-hosts              # what a pattern matches, without running anything

Two other warnings tell you the inventory itself was not found: No inventory was parsed, only implicit localhost is available means you forgot -i and there is no inventory = line in ansible.cfg. The fix is one of those two.

Ad hoc commands: one task, right now

Before playbooks, there is the quickest way to use Ansible: a single module run from the command line. These are called ad hoc commands, and they are what you use to check something across many machines or to do a one-off job you will never repeat.

The shape is always the same:

ansible <host-pattern> -m <module> -a "<arguments>"

ansible lab -m ansible.builtin.ping
ansible lab -m ansible.builtin.command -a "uptime"
ansible lab -m ansible.builtin.setup -a "filter=ansible_distribution*"
ansible lab -m ansible.builtin.file -a "path=/tmp/ansible-lab-demo state=directory mode=0755"

The command module runs a command and shows you its output. The setup module gathers facts, which are data Ansible discovers about a host (the operating system, the IP addresses, the memory), and the filter argument narrows the output to names that match. Facts get their own section later. The file module with state=directory creates a directory.

Run that last command twice and read both outputs. This is the most important thirty seconds of your first day.

localhost | CHANGED => {
    "changed": true,
    "gid": 20,
    "group": "staff",
    "mode": "0755",
    "owner": "you",
    "path": "/tmp/ansible-lab-demo",
    "size": 64,
    "state": "directory",
    "uid": 501
}

The first time you see CHANGED, in yellow if your terminal shows colours, because the directory did not exist and the module created it. The second time:

localhost | SUCCESS => {
    "changed": false,
    "gid": 20,
    "group": "staff",
    "mode": "0755",
    "owner": "you",
    "path": "/tmp/ansible-lab-demo",
    "size": 64,
    "state": "directory",
    "uid": 501
}

SUCCESS with "changed": false. Same command, but now the module looked, found the directory already in the desired state, and did nothing. You have just watched idempotency happen. Contrast this with the command module: run ansible lab -m ansible.builtin.command -a "uptime" twice and it reports CHANGED both times, because Ansible cannot know whether an arbitrary command changed anything. The warning sign to remember: command, shell and raw are not idempotent by themselves, so prefer a purpose-built module whenever one exists.

Privilege escalation, running as root, is requested with -b (for "become"). Installing a package needs it on a real server:

ansible web -b -m ansible.builtin.dnf -a "name=httpd state=present"

That line would install the httpd package on the web group of a Fedora or RHEL-family system. Do not run it on your laptop. On Debian and Ubuntu you would use the apt module, and the generic ansible.builtin.package module picks the right one for the platform, though package names still differ between distributions.

Your first playbook

An ad hoc command is one task with no memory. A playbook is a file that holds many tasks in order, so the work is written down and repeatable. Playbooks are YAML, and YAML cares about indentation, so use spaces (never tabs) and two spaces per level.

Create welcome.yml in your lab folder:

- name: Create a small site folder
  hosts: lab
  tasks:
    - name: Make the output directory
      ansible.builtin.file:
        path: "{{ playbook_dir }}/out"
        state: directory
        mode: "0755"

    - name: Write a welcome page
      ansible.builtin.copy:
        dest: "{{ playbook_dir }}/out/index.html"
        content: |
          <h1>Hello from Ansible</h1>
          <p>This file is managed by a playbook.</p>
        mode: "0644"

Read it from the outside in. The file is a list (every top-level line starts with - ), and each item in the list is a play. This play has a name, which is a human label printed while it runs; hosts: lab, which is a pattern selecting which machines; and tasks:, the ordered list of things to do.

Each task has a name and then a module with its arguments as an indented map. ansible.builtin.file with state: directory ensures a directory exists. ansible.builtin.copy with content: writes a file with exactly that text. The | after content: is YAML's way to start a block of literal text that keeps its line breaks. The value {{ playbook_dir }} is a variable, written inside double braces, which Ansible replaces with the folder holding the playbook. Because the value begins with {{, it must be wrapped in quotes; an unquoted value that starts with a brace is read by YAML as the start of a dictionary, and that mistake is so common it has its own error message later in this guide.

Two habits to adopt now. Give every task a name, because the name is what you read in the output when something fails at two in the morning. And quote file modes: mode: "0644" with quotes, since an unquoted 0644 is read by YAML as a number and can be silently misread.

Before running anything, ask Ansible to check your work:

ansible-playbook welcome.yml --syntax-check
ansible-playbook welcome.yml --list-tasks

--syntax-check parses the file without running it and catches broken YAML. --list-tasks prints the tasks it found. Now run it:

ansible-playbook welcome.yml

PLAY [Create a small site folder] **********************************************

TASK [Gathering Facts] *********************************************************
ok: [localhost]

TASK [Make the output directory] ***********************************************
changed: [localhost]

TASK [Write a welcome page] ****************************************************
changed: [localhost]

PLAY RECAP *********************************************************************
localhost                  : ok=3    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Check the result with cat out/index.html. Then run the same playbook a second time, and compare the recap: ok=3 changed=0. Nothing changed, because the directory and the file were already exactly as described. The playbook is now a statement of fact about the machine rather than a list of steps you performed once.

Reading the output

Ansible's output looks noisy until you know its structure, and then it tells you everything. It has four layers.

A PLAY line announces which play is starting. A TASK line announces which task is starting. Under each task is one line per host giving that host's result. At the end the PLAY RECAP summarises every host on one line.

The per-host result is one of a small set of words, and learning them is worth five minutes:

The recap columns count those outcomes per host. The numbers you check first are failed and unreachable: both should be zero. A healthy first run shows changed greater than zero, and a healthy second run, with nothing in the world altered, shows changed=0. Developing the reflex of reading the recap before anything else is a real productivity gain.

When something fails, the task prints fatal: and a JSON blob. Since ansible-core 2.19 it also prints an [ERROR] line naming the file, line and column, with an excerpt of your source and a <<< caused by >>> chain that walks from the symptom to the root cause. Read that chain from the bottom up. The wording of these messages is not guaranteed to stay the same between versions, so search for the meaning, not the exact text.

Increase detail with the verbosity flag. One -v shows each task's returned data; -vvv shows the connection details, including the exact ssh command line; -vvvv adds more connection debugging. You will use -v often and -vvv when you cannot connect.

ansible-playbook welcome.yml -v
ansible-playbook welcome.yml -vvv

By default the result data is printed as JSON. If you find that hard to read, ask for YAML instead in ansible.cfg:

[defaults]
inventory = inventory.ini
interpreter_python = auto_silent
callback_result_format = yaml

Idempotency, check mode, and diff mode

You have seen idempotency; now use it deliberately. Two flags turn it into a safety tool, and using them before every real run is the habit that separates careful Ansible users from the rest.

Check mode (--check, or -C) runs the playbook as a dry run. Each module reports what it would change without changing it. Diff mode (--diff, or -D) shows, for modules that support it such as copy, template and lineinfile, the before-and-after text of files. Together:

ansible-playbook welcome.yml --check --diff

Edit out/index.html to add a stray word, then run that command. You will see changed for the copy task and a diff showing the stray word being removed, yet the file on disk is untouched until you run the playbook for real. This lets you review a change the way you review a pull request.

Check mode has limits you should know. A task that depends on an earlier task's real effect can misreport: a task that reads a file created by a previous task sees no file in check mode. Modules that cannot predict their effect, such as command and shell, are skipped in check mode rather than guessed at. A task can be marked check_mode: false to force it to run even during a dry run (useful for read-only queries whose results later tasks need), and ansible_check_mode is a variable you can test. Treat check mode as a strong hint, not a guarantee.

The deeper lesson is how to make your own tasks idempotent when no purpose-built module exists. command and shell always report changed, which makes every run look like it changed something and ruins the recap as a health signal. Two arguments and two keywords fix that.

- name: Generate a report once
  ansible.builtin.command:
    cmd: /usr/local/bin/make-report --out /var/lib/report.txt
    creates: /var/lib/report.txt

- name: Read the kernel version (changes nothing)
  ansible.builtin.command: uname -r
  register: kernel
  changed_when: false

creates: says "if this file exists, skip the task", which makes a one-time command idempotent. removes: is the mirror image. changed_when: false tells Ansible that this task never changes anything, so it reports ok instead of changed. register: kernel saves the task's result into a variable called kernel, which a later task can read as kernel.stdout. The general rule: if you must run a command, teach Ansible how to tell whether it changed something.

The modules you will use every week

There are hundreds of modules, but a beginner's daily work uses a dozen. Learn these well and you can automate most of a typical Linux server. All are in ansible.builtin, so they need no extra installation.

Three of these deserve an example, because each shows a distinct style of idempotent editing.

- name: Ensure the deploy group exists
  ansible.builtin.group:
    name: deploy
    state: present

- name: Ensure one setting in a config file
  ansible.builtin.lineinfile:
    path: "{{ playbook_dir }}/out/app.conf"
    regexp: '^log_level='
    line: 'log_level=info'
    create: true
    mode: "0644"

- name: Make sure a scratch file is gone
  ansible.builtin.file:
    path: "{{ playbook_dir }}/out/scratch.txt"
    state: absent

lineinfile is the workhorse. The regexp finds the line that should be replaced, line is what it should become, and if no line matches, the new line is added at the end. Running it again finds the line already correct and does nothing. Without a regexp, lineinfile appends a new copy each time the text differs from any existing line, so always give a regexp when you are managing a setting rather than adding a fixed line.

When you choose between shell and command, choose command. It runs the program directly, with no shell involved, so there is no pipe, redirect, glob or &&, which is also why it is safer with untrusted values. Use shell only when you truly need shell features, and then quote any variable you interpolate.

Variables: one playbook, many situations

A playbook with hard-coded values works for one machine. Variables let the same playbook serve many, by separating what is the same (the tasks) from what differs (the values). A variable is a name whose value you use inside {{ double braces }}.

The most direct place to define variables is the play's vars: section:

- name: Variables in action
  hosts: lab
  vars:
    app_name: demo
    app_port: 8080
    packages:
      - git
      - curl
    settings:
      log_level: info
      workers: 4
  tasks:
    - name: Show some values
      ansible.builtin.debug:
        msg: "{{ app_name }} listens on {{ app_port }}, first package is {{ packages[0] }}, workers are {{ settings.workers }}"

Variables come in the types YAML gives you: strings, numbers, booleans, lists (items introduced with - ) and dictionaries (key: value pairs). You read a list item by position, packages[0], and a dictionary value by key, settings.workers or settings['workers']. The debug module with msg: prints the result, which makes it your print statement.

Where a variable is defined decides who can see it, and there are many places. You do not need all of them on day one. The ones you will actually use are these, and the typical beginner layout is:

• Play vars: as above, for values tied to one play.
• group_vars/: a folder beside your inventory with a file per group, for example group_vars/web.yml holding values for every host in web. A file named group_vars/all.yml applies to every host.
• host_vars/: the same, for one host, for example host_vars/web01.yml.
• Inventory variables, like ansible_host=10.0.0.11 you already used.
• Extra vars on the command line, -e "app_port=9090", or -e @vars.yml to load a whole file.

When the same name is defined in two places, the higher-priority one wins. Ansible has 22 precedence levels, but a practical summary is enough. Role defaults are the weakest (they exist to be overridden). Then inventory and group_vars, then host_vars, then play vars, then facts and registered results, and extra vars (-e) always win. That last one is a useful emergency override and a dangerous surprise, because a forgotten -e in a script silently defeats everything in your files.

ansible-playbook vars-demo.yml
ansible-playbook vars-demo.yml -e "app_port=9090"

The second command prints 9090 instead of 8080 without touching the file. To see what value a host actually gets after all that merging, ask the inventory:

ansible-inventory --host localhost

You can also define variables while the playbook runs. register: stores a task's result (you used it earlier) and ansible.builtin.set_fact creates a variable from an expression:

- name: Compute a release name
  ansible.builtin.set_fact:
    release_name: "{{ app_name }}-{{ app_port }}"

Name your variables with lowercase letters, digits and underscores, starting with a letter, and prefix them by purpose (nginx_port, not port) so they do not collide with a variable from somewhere else. Avoid names that are Ansible keywords or magic variables.

Facts: what Ansible knows about each host

Before the first task of a play, Ansible quietly runs the setup module. That is the Gathering Facts task in your output. It collects dozens of facts about each host: the operating system and version, the CPU count, memory, network interfaces, mounted disks, and more. They are stored in a variable called ansible_facts, and they are how one playbook adapts to different machines.

- name: Use facts
  hosts: lab
  tasks:
    - name: Show a few facts
      ansible.builtin.debug:
        msg: >-
          This host runs {{ ansible_facts['distribution'] }}
          {{ ansible_facts['distribution_version'] }}
          ({{ ansible_facts['os_family'] }} family)
          with {{ ansible_facts['processor_vcpus'] }} vCPUs

To explore what is available, print everything for one host, which is a large block of JSON, and narrow with a filter:

ansible lab -m ansible.builtin.setup | less
ansible lab -m ansible.builtin.setup -a "filter=ansible_os_family"

Use the dictionary form, ansible_facts['os_family'], and not the older short form ansible_os_family. Both currently work, because Ansible also injects facts as top-level variables. But that injection (the setting INJECT_FACTS_AS_VARS) is deprecated and will be switched off by default in ansible-core 2.24. Playbooks that use the short form will break then, and every tutorial from before 2.20 uses the short form, so you will copy it by accident. Start with the right habit.

Facts cost time, since gathering runs on every host at the start of every play. When a play needs none of it, turn it off:

- name: A play that needs no facts
  hosts: lab
  gather_facts: false
  tasks:
    - name: Do something simple
      ansible.builtin.debug:
        msg: no facts needed here

On a fleet of hundreds of machines that saves a noticeable amount of time. There is also a gather_subset option to collect only some categories. Custom facts, which are files on a host under /etc/ansible/facts.d/ ending in .fact, are exposed under ansible_local, and they let a machine describe itself to Ansible, for instance "I am the primary database".

The facts you will use most are the ones that drive decisions: os_family (for example Debian or RedHat), distribution, distribution_version, hostname, default_ipv4.address, processor_vcpus and memtotal_mb. Facts describe the machine at the moment of gathering, so if a task changes something (installs a package, for instance), the fact is stale until you gather again with the ansible.builtin.setup module.

Conditions and loops

Real playbooks make decisions and repeat things. Two keywords do that work: when and loop.

A conditional runs a task only if an expression is true:

- name: Only on Debian-family systems
  ansible.builtin.debug:
    msg: "this is a Debian-family host"
  when: ansible_facts['os_family'] == "Debian"

- name: Only when a list is non-empty
  ansible.builtin.debug:
    msg: "we have packages"
  when: packages | length > 0

The when value is a Jinja2 expression without {{ }}. Writing braces inside when is an error in current Ansible: when: "{{ a }} == 'b'" fails with Template delimiters are not supported in expressions, and the correct form is when: a == 'b'. The same applies to failed_when, changed_when and until.

Since ansible-core 2.19 there is a second strictness rule that catches people migrating old playbooks: a condition must evaluate to a true boolean. A variable that holds a non-empty string, used directly as when: myvar, now fails with Conditional result (True) was derived from value of type 'str'. Conditionals must have a boolean result. Make the intent explicit instead:

when: myvar | length > 0     # is it non-empty?
when: enable_feature | bool  # convert "yes"/"true"/"on" strings to a real boolean
when: myvar is defined       # is it set at all?

The bool filter converts recognised truthy strings such as "yes", "true" and "on", and recognised false ones such as "no", "false" and "off". You can combine conditions with and, or and not, or by listing several lines, which are joined with and:

when:
  - ansible_facts['os_family'] == "RedHat"
  - ansible_facts['distribution_major_version'] | int >= 9

There are also tests, which read naturally with is: result is changed, result is failed, path_stat.stat.exists. A skipped task prints skipping: and counts in the recap.

A loop repeats a task once per item, and the current item is the variable item:

- name: Create several directories
  ansible.builtin.file:
    path: "{{ playbook_dir }}/out/{{ item }}"
    state: directory
    mode: "0755"
  loop:
    - logs
    - data
    - backups

- name: Create users from a list of dictionaries
  ansible.builtin.debug:
    msg: "would create {{ item.name }} in group {{ item.group }}"
  loop:
    - { name: alice, group: dev }
    - { name: bob, group: ops }
  loop_control:
    label: "{{ item.name }}"

loop_control with label makes the output show just alice instead of the whole dictionary for each pass, which keeps logs readable and, importantly, keeps secrets out of logs when items contain passwords. Older playbooks use with_items; it still works, but loop is the current style. Many modules accept a whole list directly and do it in one go, which is faster than looping:

- name: Install several packages in one transaction
  ansible.builtin.package:
    name:
      - git
      - curl
    state: present

Conditions and loops combine. A when on a looped task is evaluated once per item, so when: item.enabled decides per pass.

Templates: files that adapt

Configuration files usually differ slightly per machine: a hostname here, a port there. Copying a static file cannot express that. A template can. Ansible uses the Jinja2 templating language: you write a file with placeholders, and the template module fills them with variables and writes the result to the target.

Create a folder templates/ and a file in it:

# Managed by Ansible. Changes made by hand will be overwritten.
[server]
name = {{ app_name }}
port = {{ app_port }}
host = {{ ansible_facts['hostname'] }}

[features]
{% for feature in features %}
{{ feature }} = on
{% endfor %}
{% if debug_enabled | default(false) %}
debug = true
{% endif %}

Three Jinja2 constructs cover nearly everything. {{ expression }} prints a value. {% for ... %} ... {% endfor %} repeats a block. {% if ... %} ... {% endif %} includes a block conditionally. The .j2 suffix is a convention that tells humans this is a template; Ansible does not require it. Now the task that uses it:

- name: Render configuration
  hosts: lab
  vars:
    app_name: demo
    app_port: 8080
    features:
      - search
      - metrics
  tasks:
    - name: Create the output directory
      ansible.builtin.file:
        path: "{{ playbook_dir }}/out"
        state: directory
        mode: "0755"

    - name: Render app.conf
      ansible.builtin.template:
        src: app.conf.j2
        dest: "{{ playbook_dir }}/out/app.conf"
        mode: "0644"

Ansible finds app.conf.j2 in the templates/ folder beside the playbook automatically. Run it, read out/app.conf, then run it again and confirm changed=0. Change app_port and run --check --diff to watch the diff show exactly the changed line.

template is idempotent: it renders in memory, compares with the file on disk, and only writes if the content differs. That is what makes it safe to run constantly. It also has a superpower for risky files: validate. For a file where a typo would take a service down, have a checker approve the rendered result before it replaces the real one:

- name: Install the web server config only if it is valid
  ansible.builtin.template:
    src: nginx.conf.j2
    dest: /etc/nginx/nginx.conf
    mode: "0644"
    validate: nginx -t -c %s

The %s becomes the path of the temporary rendered file, and if the command exits non-zero the task fails and the live file is left alone. Use this wherever the service offers a config check.

Whitespace in Jinja2 blocks can surprise you: each {% %} line leaves an empty line unless you trim it. Ansible's template module already removes the newline after a block tag by default, which keeps the example above tidy; if output has odd blank lines, look at the block tags first.

Handlers: act only when something changed

Suppose a task changes a web server's config file. The server must be reloaded to use it, but reloading every run, even when nothing changed, would interrupt traffic for no reason. Handlers solve this. A handler is a task that runs only if another task reported changed and notified it.

- name: Config change triggers a reload
  hosts: lab
  vars:
    app_name: demo
    app_port: 8080
    features: [search]
  tasks:
    - name: Create the output directory
      ansible.builtin.file:
        path: "{{ playbook_dir }}/out"
        state: directory
        mode: "0755"

    - name: Render app.conf
      ansible.builtin.template:
        src: app.conf.j2
        dest: "{{ playbook_dir }}/out/app.conf"
        mode: "0644"
      notify: Reload the app

  handlers:
    - name: Reload the app
      ansible.builtin.debug:
        msg: "pretend we reloaded the app now"

The notify line names the handler. The handler lives in the handlers: section and has a name that must match. On a real server the handler would be ansible.builtin.service with state: reloaded; here debug stands in so the lab stays safe. Run the playbook and the handler fires, because the template created the file (changed). Run it again and nothing prints: no change, no notification, no reload. Change a value and the handler fires again.

Three rules govern handlers, and each has caught everyone at least once.

1. Handlers run once, at the end of the play's task section, no matter how many tasks notified them. If five tasks change five config files and all notify "Restart nginx", nginx restarts one time.
2. They run in the order they are defined in the handlers: section, not the order they were notified.
3. If a task fails, the play stops on that host before the handlers run, and a change that already happened will not trigger its handler on the next run (because the next run sees no change). If a failed run leaves the service out of step with its config, use --force-handlers or set force_handlers: true on the play.

If a handler must run earlier, say a service restart has to happen before a later task uses the service, you can force it with ansible.builtin.meta: flush_handlers:

- name: Run any pending handlers now
  ansible.builtin.meta: flush_handlers

Handling failure: register, failed_when, blocks

By default, when a task fails on a host, Ansible stops running tasks on that host and carries on with the others. That default is right most of the time, but real automation needs finer control, and Ansible gives you a few keywords for it.

register captures a task's result so later tasks can inspect it. failed_when and changed_when redefine what "failed" and "changed" mean for that task, which matters for commands whose exit codes are not a simple success or failure.

- name: Check whether the app answers
  ansible.builtin.command: /usr/local/bin/app-check
  register: check
  changed_when: false
  failed_when: check.rc not in [0, 2]

- name: Report the result
  ansible.builtin.debug:
    msg: "check returned {{ check.rc }}: {{ check.stdout }}"

Here exit codes 0 and 2 both count as acceptable, and anything else fails the task. check.rc is the exit code, and check.stdout is the printed output; every registered result of a command-like module has these and more, which -v shows.

ignore_errors: true on a task lets the play continue even if it fails. Use it sparingly: it hides problems, and a failed task that is ignored still looks like a failure in the output, which confuses readers. Prefer failed_when to define precisely what is acceptable. If a host cannot be reached, that is a separate situation (unreachable), handled with ignore_unreachable: true, and since ansible-core 2.19 a timeout while becoming another user counts as unreachable rather than failed.

When several tasks belong together and you want try-and-recover behaviour, group them in a block:

- name: Upgrade with a safety net
  block:
    - name: Run the migration
      ansible.builtin.command: /opt/app/migrate
  rescue:
    - name: Report the failure and roll back
      ansible.builtin.debug:
        msg: "migration failed on {{ inventory_hostname }}, rolling back"
  always:
    - name: Remove the maintenance flag either way
      ansible.builtin.file:
        path: /tmp/maintenance
        state: absent

The block runs; if any of its tasks fails, the rescue tasks run; the always tasks run regardless. That is exactly the try, except and finally of a programming language. A block also lets you apply a keyword, such as become: true or a when, to a group of tasks once.

One more guard deserves a place in every serious playbook: stating your assumptions so they fail early and clearly.

- name: Stop early if this is not a supported system
  ansible.builtin.assert:
    that:
      - ansible_facts['system'] == "Linux"
      - app_port | int > 1024
    fail_msg: "This playbook needs Linux and a non-privileged port"
    success_msg: "Preconditions hold"

assert is to a playbook what an input check is to a function. It turns a confusing failure halfway through into a clear one at the start.

Connecting to real servers: SSH, users, and sudo

Everything so far ran on your own machine. The jump to real remote servers needs three things to be true: Ansible can log in over SSH, it knows which user to log in as, and, when a task needs root, it can become root.

Step one: SSH access. Test it by hand first, because if ssh user@host does not work, Ansible will not either.

ssh-keygen -t ed25519 -C "ansible control node"
ssh-copy-id deploy@10.0.0.11
ssh deploy@10.0.0.11 'python3 --version'

A key-based login with no password prompt is what you want, since automation cannot type passwords. The last command also confirms Python exists on the target.

Step two: describe the machines and tell Ansible who to be, in the inventory or ansible.cfg:

[web]
web01 ansible_host=10.0.0.11
web02 ansible_host=10.0.0.12

[web:vars]
ansible_user=deploy

If the key is not in the default location, name it with ansible_ssh_private_key_file=~/.ssh/ansible_ed25519 in the inventory or --private-key on the command line. Check the connection for the whole group with ansible web -m ansible.builtin.ping.

Step three: privilege escalation, called become. Installing packages and editing files in /etc require root. You ask for it per play or per task with become: true. By default Ansible uses sudo to become root.

- name: Install and start nginx
  hosts: web
  become: true
  tasks:
    - name: Install nginx
      ansible.builtin.package:
        name: nginx
        state: present

    - name: Start nginx now and at boot
      ansible.builtin.service:
        name: nginx
        state: started
        enabled: true

If the deploy user needs a sudo password, add --ask-become-pass (or -K) and Ansible prompts once. A dedicated automation account with passwordless sudo for the commands it needs is the usual solution for unattended runs, configured once by your operations or security team. Never put the sudo password into a plain file in the repository; the vault section below shows the right place.

Now the first connection problems you will meet, and what they mean.

The second row tempts everybody to set host_key_checking = False. Do that only in a throwaway lab. Host key checking is what protects you from connecting to an impostor, and it is on by default for good reason. When you are stuck, add -vvv to see the exact ssh command Ansible ran; you can copy it and run it yourself, which takes most of the mystery out of connection problems.

If you have a real machine available, swap your lab inventory for one like the above and re-run welcome.yml: the hosts: line and the tasks do not change at all. Only the inventory changed, which is the point of separating them.

The everyday commands, grouped by what you are trying to do

You now know enough pieces to see the command line as a toolbox. Here it is, grouped by intention rather than alphabetically.

Checking before you run. These commands change nothing and should precede any real run.

ansible-playbook site.yml --syntax-check       # is the YAML valid?
ansible-playbook site.yml --list-hosts         # which hosts will be touched?
ansible-playbook site.yml --list-tasks         # which tasks will run, in which order?
ansible-playbook site.yml --list-tags          # which tags exist?
ansible-playbook site.yml --check --diff       # dry run with file diffs

Narrowing a run. You rarely want to run everything everywhere.

ansible-playbook site.yml --limit web01        # only this host (or pattern): -l
ansible-playbook site.yml --tags config        # only tasks tagged "config": -t
ansible-playbook site.yml --skip-tags slow     # everything except the "slow" tag
ansible-playbook site.yml --start-at-task "Render app.conf"
ansible-playbook site.yml --step               # confirm each task interactively

Tags are labels you attach to tasks, so you can run a subset:

- name: Render app.conf
  ansible.builtin.template:
    src: app.conf.j2
    dest: /etc/app/app.conf
    mode: "0644"
  tags: [config]

Supplying values and credentials.

ansible-playbook site.yml -e "app_port=9090"               # one extra variable
ansible-playbook site.yml -e @extra-vars.yml               # a file of variables
ansible-playbook site.yml -u deploy --private-key ~/.ssh/key
ansible-playbook site.yml -K                               # ask for the sudo password
ansible-playbook site.yml --ask-vault-password             # ask for the vault password

Seeing more or less.

ansible-playbook site.yml -v        # task results
ansible-playbook site.yml -vvv      # connection details
ansible-playbook site.yml -f 20     # 20 hosts in parallel (forks, default 5)

Learning and inspecting.

ansible-doc ansible.builtin.copy             # manual for a module
ansible-doc -l                               # list every module
ansible-doc -t keyword when                  # what a keyword means
ansible-inventory --graph                    # groups and hosts
ansible-config dump --only-changed           # settings that differ from defaults
ansible-galaxy collection list               # installed collections

Two details about the options. Tag selection has special tags: always (runs even when you select other tags, unless skipped), never (runs only when asked for explicitly), tagged and untagged. And when you limit a run with --limit, the hosts: pattern in the play still applies on top, so --limit can only narrow the selection, never widen it.

Secrets: a first look at Ansible Vault

Sooner or later a playbook needs a password, an API token or a private key. You cannot leave it in plain text in a repository that other people can read, and you cannot leave it out. Ansible Vault encrypts files, or single values, with a password (AES256) so the encrypted text can be committed safely and decrypted when a playbook runs.

Create an encrypted file, which opens your editor and saves the contents encrypted:

ansible-vault create group_vars/lab/vault.yml

Put a variable inside, prefixed clearly so it is obvious where it comes from:

vault_db_password: "correct-horse-battery-staple"

On disk the file now starts with a header such as $ANSIBLE_VAULT;1.1;AES256 followed by hexadecimal ciphertext. You can still use the variable in a playbook as usual, and supply the password when you run:

ansible-playbook site.yml --ask-vault-password
ansible-playbook site.yml --vault-password-file ~/.vault_pass

The other vault commands are what you would guess: ansible-vault view, edit, encrypt (an existing file), decrypt and rekey (change the password). To encrypt just one value inside an otherwise readable file:

ansible-vault encrypt_string 'supersecret' --name db_password

It prints a block beginning db_password: !vault | that you paste into a normal vars file. A popular pattern keeps two files per group: a plain vars.yml that refers to {{ vault_db_password }}, so anyone reading the repository sees which variables exist, and a vault.yml that holds the encrypted values.

Keep these rules from day one:

1. Never commit the vault password. If you keep it in a file such as .vault_pass for convenience, add that file to .gitignore immediately. You can point Ansible at it with vault_password_file = .vault_pass in ansible.cfg.
2. Use no_log: true on any task that handles a secret, so the value does not appear in the terminal or logs: for example a user task that sets a password, or a uri task sending a token.
3. Remember the error messages. Forgetting the password shows Attempting to decrypt but no vault secrets found. A wrong password shows Decryption failed (no vault secrets were found that could decrypt). Both mean the password did not reach Ansible, not that the file is damaged.

Vault protects secrets at rest, in the repository. It is not a full secrets platform: anyone with the password can read everything, and there is no audit trail or rotation. Larger teams move to an external secret manager, which the Mid-level and Senior guides cover.

Roles and collections: where this grows

A single playbook file works for a small job. A real project reaches a point where the same tasks, such as "install and configure nginx", are needed in several playbooks. Copying them around recreates the drift problem. Roles and collections are Ansible's answer, and a beginner should know what they are, even if you build none yet.

A role is a folder with a fixed layout that packages a related set of tasks, handlers, templates, default variables and files:

roles/
  nginx/
    tasks/main.yml        # the tasks
    handlers/main.yml     # the handlers
    templates/            # Jinja2 templates
    files/                # static files
    defaults/main.yml     # default variable values (the weakest, meant to be overridden)
    vars/main.yml         # role-internal variables (stronger)
    meta/main.yml         # metadata and dependencies

Ansible looks inside those folders by name, so a template task in tasks/main.yml finds its template in templates/ without a path. You apply a role to hosts like this:

- name: Configure web servers
  hosts: web
  become: true
  roles:
    - nginx

ansible-galaxy role init nginx creates the skeleton folder for you. When a role needs a value to differ between hosts, it reads a variable that has a default in defaults/main.yml, and the caller overrides it from group_vars or the play. That is the design: behaviour is fixed, configuration is passed in.

A collection is the larger distribution format: a bundle of modules, plugins and roles under a namespace.collection name. You already use one, ansible.builtin. The community package includes many more; with only ansible-core you install collections yourself from Ansible Galaxy, the public hub at galaxy.ansible.com.

ansible-galaxy collection install community.general
ansible-galaxy collection list

To keep a project reproducible, list what it needs in a requirements.yml file and install from it, so a teammate or a CI job gets the same content:

collections:
  - name: community.general
    version: ">=11.0.0,<12.0.0"

ansible-galaxy collection install -r requirements.yml

Roles downloaded from Galaxy are other people's code that will run as root on your machines. Read them before you use them, pin a version, and prefer maintained ones with clear documentation. Since ansible-core 2.21, ansible-galaxy collection install skips collections that declare themselves incompatible with your core version, and if you see couldn't resolve module/action 'community.general.something', the collection is usually missing or the name is misspelled.

A sensible ladder for a learner: one playbook first, then group_vars and templates, then extract repeated tasks into a role, then pull in collections when you need a module that ships outside ansible.builtin. Do not start with roles; they are a refactoring you earn once there is something to refactor.

Configuration: ansible.cfg and the settings worth knowing

You have used ansible.cfg already. Here is how Ansible decides which configuration to read, and which settings a beginner should know.

Ansible reads one configuration file, searching in this order and stopping at the first it finds. Files are not merged:

1. The file named by the ANSIBLE_CONFIG environment variable.
2. ansible.cfg in the current directory (ignored if that directory is world-writable, a safety measure).
3. ~/.ansible.cfg in your home directory.
4. /etc/ansible/ansible.cfg.

Because the current directory wins over the home directory, a project-level ansible.cfg beside your playbooks is the right place for project settings, and it travels with the repository. You can always check which one is active and what differs from the defaults:

ansible --version | head -3                  # shows "config file = ..."
ansible-config dump --only-changed           # only what you have customised
ansible-config init --disabled > ansible.cfg # a fully commented template to read

Every setting can also be given as an environment variable, which is handy for one-off overrides and for CI. The pattern is the key name in capitals with an ANSIBLE_ prefix, although a few have different names, so check the docs rather than guessing.

A sensible starting ansible.cfg for a project:

[defaults]
inventory = inventory.ini
interpreter_python = auto_silent
callback_result_format = yaml
forks = 10

A few settings have surprises. interpreter_python is overridden per host by the variable ansible_python_interpreter, which is how you point a machine at a specific Python, such as /usr/bin/python3.12. The key collections_path is singular, not plural, in both the file and the environment variable. Retry files, the .retry leftovers that old Ansible versions wrote, are off by default now, despite what older pages say.

The precedence for settings, from weakest to strongest, is the built-in default, the config file, environment variables, and command-line options. An environment variable left exported in your shell can therefore override ansible.cfg without any sign in the file. When behaviour puzzles you, ansible-config dump --only-changed is the first place to look.

The errors you will actually meet

Every beginner meets the same ten errors. Since ansible-core 2.19 the format is [ERROR]: Task failed: ... followed by Origin: with the file, line and column, an excerpt of your source with a marker, and sometimes a <<< caused by >>> chain. Read the top line for the symptom, the origin for where, and the bottom of the chain for why.

1. The YAML did not parse: missing quotes around a template.

[ERROR]: YAML parsing failed: This may be an issue with missing quotes around a template block.

The hint prints raw: {{ some_var }} and Should be: raw: "{{ some_var }}". A value that starts with {{ must be quoted. Fix: add double quotes.

2. A variable is undefined.

[ERROR]: Task failed: Finalization of task args for 'ansible.builtin.debug' failed: Error while resolving value for 'msg': 'not_defined_var' is undefined

Older versions said The task includes an option with an undefined variable, and you will see that wording in old posts. The meaning is the same: a typo, a variable defined in a scope the task cannot see, or facts disabled. Fix the name, define it, or use | default(...) if it is optional.

3. A condition is not a boolean.

A 'when' expression failed: Conditional result (True) was derived from value of type 'str' ... Conditionals must have a boolean result.

You wrote when: some_string. Write a comparison: when: some_string | length > 0, or when: some_value | bool.

4. Template delimiters inside a condition.

A 'when' expression failed: Syntax error in expression. Template delimiters are not supported in expressions

You wrote when: "{{ x }} == 'y'". Remove the braces: when: x == 'y'.

5. The module cannot be found.

[ERROR]: couldn't resolve module/action 'community.general.not_a_module'. This often indicates a misspelling, missing collection, or incorrect module path.

Check the spelling and the FQCN, then run ansible-galaxy collection list. If the collection is absent, install it with ansible-galaxy collection install. A module that used to exist may also have moved to a different collection.

6. No hosts matched.

[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'
[WARNING]: Could not match supplied host pattern, ignoring: web

The inventory was not loaded or the group name is wrong. Check -i, the inventory = line, and ansible-inventory --graph.

7. The host is unreachable.

fatal: [web01]: UNREACHABLE! => {"changed": false, "msg": "Task failed: Failed to connect to the host via ssh: ... Permission denied (publickey,password).", "unreachable": true}

This is about the connection, not your tasks. Test with plain ssh, check ansible_user, the key and the address, and add -vvv.

8. Sudo wants a password.

fatal: [web01]: FAILED! => {"changed": false, "msg": "Task failed: Premature end of stream waiting for become success.\n>>> Standard Error\nsudo: a password is required"}

Use -K to be prompted, or configure passwordless sudo for the automation user.

9. Python is missing on the target.

[ERROR]: Task failed: Action failed: The module interpreter '/usr/bin/python9' was not found.

On a bare host you may instead see MODULE FAILURE ... /usr/bin/python3: not found. Install Python using the raw module, which needs none: ansible host -m ansible.builtin.raw -a "dnf -y install python3". Or point ansible_python_interpreter at the correct path.

10. The vault password did not arrive.

Attempt to use undecryptable variable: Attempting to decrypt but no vault secrets found.

Add --ask-vault-password or --vault-password-file. A wrong password gives Decryption failed (no vault secrets were found that could decrypt).

Two more environmental ones come up in CI and minimal containers. ERROR: Ansible requires blocking IO on stdin/stdout/stderr means something wrapping the process set its pipes non-blocking; piping the output through cat or running under a real terminal fixes it. ERROR: Ansible could not initialize the preferred locale: unsupported locale setting means the environment lacks a UTF-8 locale; export LC_ALL=C.UTF-8.

A last habit that makes all ten easier: read the first error, not the last. Ansible stops a host at its first failure, so everything after it is either skipped or fallout. Fix the first, re-run, and expect the list to shrink.

Putting it all together

Time to combine everything into one small, real project that you could hand to a colleague. It builds a tiny "service configuration" folder on your own machine, using variables, group variables, a vaulted secret, a template, a handler, tags, an assertion, and a block, then proves it is idempotent.

The layout:

ansible-lab/
  ansible.cfg
  inventory.ini
  site.yml
  group_vars/
    lab/
      vars.yml
      vault.yml
  templates/
    service.conf.j2
  .gitignore
  .vault_pass

Start with the settings, pointing at the vault password file:

[defaults]
inventory = inventory.ini
interpreter_python = auto_silent
callback_result_format = yaml
vault_password_file = .vault_pass

[lab]
localhost ansible_connection=local

Create the password file and keep it out of git, then create the encrypted secret file. Because ansible.cfg names the password file, there is no prompt:

echo "a-long-random-lab-password" > .vault_pass
chmod 600 .vault_pass
printf '.vault_pass\nout/\n' > .gitignore
ansible-vault create group_vars/lab/vault.yml

In the editor, enter one line and save:

vault_api_token: "lab-token-12345"

The plain variables file refers to the secret by name, so anyone reading it knows what exists:

service_name: demo-api
service_port: 8080
service_workers: 2
features:
  - search
  - metrics
api_token: "{{ vault_api_token }}"

The template:

# Managed by Ansible. Do not edit by hand.
[service]
name = {{ service_name }}
port = {{ service_port }}
workers = {{ service_workers }}
host = {{ ansible_facts['hostname'] }}
token = {{ api_token }}

[features]
{% for feature in features %}
{{ feature }} = on
{% endfor %}

And the playbook, which ties it together:

- name: Configure the demo service
  hosts: lab
  tasks:
    - name: Check our assumptions
      ansible.builtin.assert:
        that:
          - service_port | int > 1024
          - service_workers | int >= 1
          - features | length > 0
        fail_msg: "service_port, service_workers and features need sensible values"
      tags: [always]

    - name: Prepare the service directories
      ansible.builtin.file:
        path: "{{ playbook_dir }}/out/{{ item }}"
        state: directory
        mode: "0755"
      loop:
        - conf
        - logs
      tags: [dirs]

    - name: Render and install the configuration
      block:
        - name: Write service.conf
          ansible.builtin.template:
            src: service.conf.j2
            dest: "{{ playbook_dir }}/out/conf/service.conf"
            mode: "0600"
          notify: Reload the service
          no_log: true

        - name: Verify the file landed
          ansible.builtin.stat:
            path: "{{ playbook_dir }}/out/conf/service.conf"
          register: conf_stat
          failed_when: not conf_stat.stat.exists
      rescue:
        - name: Explain the failure
          ansible.builtin.debug:
            msg: "Could not install the configuration on {{ inventory_hostname }}"
      tags: [config]

  handlers:
    - name: Reload the service
      ansible.builtin.debug:
        msg: "service would be reloaded now"

Read it the way a reviewer would. The assert runs first and carries the always tag, so it checks the inputs even when you run only one tag. The directory task uses a loop. The template task is wrapped in a block with a rescue; it has no_log: true because the rendered content contains the token; it sets the mode to 0600 so only the owner reads the secret; and it notifies the handler. The stat task verifies the file exists and fails the block if not.

Run the sequence you learned, in order:

ansible-playbook site.yml --syntax-check
ansible-playbook site.yml --list-tasks
ansible-playbook site.yml --check --diff
ansible-playbook site.yml
ansible-playbook site.yml

The first run creates two directories, writes the file, and runs the handler, so the recap shows changed above zero. The second run shows changed=0 and no handler message. Now edit service_workers to 4, run --check --diff (the diff is hidden because no_log: true covers that task, which is the price of hiding a secret), run for real, and watch the handler fire once. Finally run only a slice:

ansible-playbook site.yml --tags config
ansible-playbook site.yml --skip-tags dirs

If you have a remote machine, the only change needed is the inventory (an address and a user) and adding become: true where the tasks touch system locations. The playbook and templates stay the same.

What you can now do, and what comes next

You started with no Ansible and you can now do the whole loop that makes it useful. You can install the tool and verify it. You can describe machines in an inventory and select them with patterns. You can run one-off ad hoc commands and, more importantly, write playbooks that declare a desired state. You can read the output, distinguish ok from changed from failed, and use --check --diff to preview changes. You can use variables, facts, conditions, loops and templates, act on change through handlers, handle failures with failed_when and blocks, keep secrets in Vault, connect to real servers over SSH with become, and read the errors that stop most beginners. That is enough to automate a real small server, and the habits (name every task, use FQCNs, check before you run, prefer modules to commands) are the ones that keep larger projects healthy.

What comes next depends on what you want to do.

1. Go deeper on Ansible itself. The Mid-level guide covers roles in full, collections and requirements.yml, dynamic inventory from cloud providers, variable precedence in detail, testing with Molecule, linting with ansible-lint, and debugging beyond -vvv. The Senior guide covers running Ansible as a platform, scaling, security and upgrades.
2. Connect it to neighbouring tools. Ansible configures machines, and something has to create them. Terraform provisions the servers and networks, and Ansible then configures them. If your workloads run in containers, start with Docker, and for clusters continue with Kubernetes. Ansible can drive all of them, but it is a different tool with a different job, and knowing where each one stops is half of using them well.
3. Practise on something real. Take a server you maintain by hand today and write the playbook that rebuilds its setup from nothing. Run it in --check mode against the live server first; every difference the diff reports is a place where your documentation and your reality disagreed. Fixing those is the most valuable automation work there is.

A final point about the region. Teams in the Gulf and Egypt often manage servers in a mix of cloud regions and local data centres, sometimes under data-residency requirements. Ansible is well suited to that, because it needs only SSH and runs from wherever your control node sits, so you can keep the control node, the inventory and the secrets inside the jurisdiction your employer requires. Check that your vault passwords, SSH keys and any fact caches live where your policy says they may.

Sources

• Ansible documentation home
• Installing Ansible
• Installing Ansible on specific operating systems
• Ansible configuration settings
• Interpreter discovery
• Using variables (and precedence)
• ansible.builtin collection index
• Releases and maintenance
• Porting guide for ansible-core 2.19
• Porting guide for ansible-core 2.20
• Porting guide for ansible-core 2.21
• ansible-core 2.21 changelog
• ansible-core on PyPI
• Ansible community package on PyPI

Part one of three. A twenty-five-minute review of the Beginner Detailed track, then the questions interviewers ask. Mid-level reviews this plus its own material; Senior reviews all three. Everything here targets ansible-core 2.21, the engine of the Ansible community package 14.

The thirty-second answer

Ansible configures machines from a description of their desired state, written as YAML playbooks. It is agentless: the control node connects over SSH, copies a small module to each managed host, runs it with the host's Python, and reads back a JSON result. Because each module checks the current state before acting, a playbook is idempotent: run it twice and the second run changes nothing.

Then the line that shows you have used it: "I read the play recap first, failed and unreachable must be zero, and on a second run changed should be zero, and I always run check mode with diff before a real change."

The model

Keep the push-model sequence ready, because interviewers love it. Ansible reads the playbook, packages the module, copies it over the connection to a temporary directory on the target, runs it with the target's Python, receives a JSON result and deletes the files. Nothing stays resident on the host.

Agentless, and what that costs

"Agentless" is the headline claim, so be ready to qualify it. The benefit is that there is nothing to install, patch or secure on the managed nodes: SSH access and Python are enough. The costs are honest ones. Each task opens work over a network connection, so very large fleets need tuning (forks, SSH multiplexing, pipelining) that a pull-based agent system would not. And the control node becomes a sensitive place, because it holds the keys and secrets that reach every machine.

A strong follow-up: Python must exist on the target for most modules, which is a real requirement even in an "agentless" tool. The exceptions are the raw and script modules and network devices, and the usual way to bootstrap a bare host is to use raw to install Python.

The commands to have ready

ansible --version                               # core version, config file in use, Python
ansible all -m ansible.builtin.ping             # can I reach and run Python on every host?
ansible web -m ansible.builtin.command -a uptime
ansible web -b -m ansible.builtin.dnf -a "name=httpd state=present"

ansible-inventory --graph                       # groups and hosts
ansible-inventory --host web01                  # the variables one host gets

ansible-playbook site.yml --syntax-check
ansible-playbook site.yml --list-hosts --list-tasks
ansible-playbook site.yml --check --diff        # dry run with diffs
ansible-playbook site.yml -l web01 -t config -v # limit, tag, verbose
ansible-playbook site.yml -K                    # ask for the sudo password

ansible-doc ansible.builtin.copy                # the manual, offline
ansible-config dump --only-changed              # which settings are not default
ansible-vault create|edit|view secrets.yml
ansible-galaxy collection install community.general

If asked to explain a line, be able to decode -b (become), -K (ask become password), -k (ask SSH password), -l (limit), -t (tags), -e (extra vars), -C (check), -D (diff) and -v through -vvvv (verbosity).

Idempotency, stated precisely

Interviewers use idempotency to separate people who have used Ansible from people who have read about it. The precise statement: a module compares the current state with the desired state and acts only if they differ, then reports changed or ok accordingly. Applying the same playbook repeatedly converges on the same state, and a second run against an already-configured host shows changed=0.

The follow-up is always where idempotency breaks. The answer is command, shell and raw: Ansible cannot know what an arbitrary command did, so they report changed every time. You repair that with creates: or removes: arguments (skip the command if a file exists or is absent), with changed_when (declare your own rule, including changed_when: false for read-only commands), and above all by choosing a purpose-built module when one exists.

A second trap in the same area is lineinfile without a regexp. With no pattern to match, it cannot find the line it should replace, so a changed line: value adds another line instead of editing the existing one.

Check mode and diff mode

--check makes each module report what it would change without changing it. --diff shows the before and after text for file-oriented modules. Together they are the review step before a change.

Know the limits, because a good answer includes them. Modules that cannot predict their effect are skipped in check mode (command and shell are the usual examples). A task that depends on the real effect of an earlier task can misreport, since the earlier task did nothing. A task can opt out with check_mode: false, and the variable ansible_check_mode lets a playbook know it is in a dry run. Check mode is strong evidence, not proof.

Variables and precedence

The short version that satisfies most interviewers: variables come from the play, from group_vars and host_vars beside the inventory, from the inventory itself, from facts, from registered task results and from set_fact, and from the command line. Ansible has 22 precedence levels, but you can give the skeleton. Role defaults are the weakest, meant to be overridden. Inventory and group variables are stronger than role defaults, host variables stronger than group variables, play vars stronger than both, and extra vars (-e) always win.

The practical consequences are what to say out loud. Put defaults in role defaults or group variables, put per-host exceptions in host_vars, and treat -e as an emergency override, because a forgotten -e in a script silently beats every file. Check what a host really receives with ansible-inventory --host <name>.

Also be ready for the facts question. Use ansible_facts['os_family'], the dictionary form, and not the short injected form ansible_os_family. The injection setting INJECT_FACTS_AS_VARS is deprecated and will be off by default in ansible-core 2.24, so the short form has a scheduled end.

Handlers

Handlers are tasks that run only when notified by a task that reported changed. The classic use is reloading a service after its configuration file changed.

The three rules an interviewer expects you to know: a handler runs once, at the end of the tasks section of the play, no matter how many tasks notified it; handlers run in the order they are defined, not notified; and if the play fails before the handlers run, the change has been made but the handler has not, so the next run (which sees no change) will not fire it. The cures are meta: flush_handlers to run pending handlers at a chosen point, and --force-handlers or force_handlers: true to run them even after a failure.

Become, SSH and connection problems

State the order of diagnosis, because it shows method. First, can I ssh user@host by hand? If not, Ansible cannot either. Second, ansible <target> -m ansible.builtin.ping isolates connection from task logic. Third, -vvv prints the exact ssh command Ansible ran, which you can copy and try yourself.

Match the message to the cause:

Two modern details score points. Since ansible-core 2.19 a timeout while becoming a user counts as unreachable, not failed, so ignore_unreachable (not ignore_errors) is the right keyword there. And the old requirement to install sshpass for password logins is gone: the ssh connection now uses an askpass mechanism by default. If asked about the paramiko transport, say it was removed in ansible-core 2.21 and the default OpenSSH-based ssh connection is what you use.

Conditionals that broke in recent versions

Two rules changed in ansible-core 2.19 and they catch everyone upgrading old playbooks. Expect a question on them.

# Wrong in 2.19+: the string is not a boolean
when: my_string

# Right: say what you mean
when: my_string | length > 0
when: enable_feature | bool
when: my_var is defined and my_var

# Wrong: template braces inside an expression
when: "{{ env }} == 'prod'"

# Right
when: env == 'prod'

The first fails with Conditional result (True) was derived from value of type 'str' ... Conditionals must have a boolean result. The second fails with Template delimiters are not supported in expressions. The temporary escape hatch for the first is ALLOW_BROKEN_CONDITIONALS, and the honest answer is that it is a migration aid to be removed, not a setting to leave on.

Vault in two lines

Ansible Vault encrypts a file, or a single value, with AES256 so ciphertext can live in git. The password is supplied at run time with --ask-vault-password, --vault-password-file or vault_password_file in ansible.cfg, and the password itself is never committed.

What to add: use no_log: true on tasks that handle secrets; keep plain variable files that reference vault_-prefixed variables so readers can see what exists; and know that Vault protects secrets at rest in the repository but offers no rotation or audit trail, so larger teams use an external secret manager.

The two errors: Attempting to decrypt but no vault secrets found means no password arrived, and Decryption failed (no vault secrets were found that could decrypt) means the wrong one arrived.

Configuration

Ansible reads exactly one config file, the first found of: the ANSIBLE_CONFIG variable, ./ansible.cfg, ~/.ansible.cfg, /etc/ansible/ansible.cfg. They are not merged. A current-directory ansible.cfg in a world-writable directory is ignored, as a safety measure. ansible-config dump --only-changed shows what differs from defaults, and ansible --version names the file in use.

Environment variables override the file, and command-line options override both. The interpreter setting has the surprising environment name ANSIBLE_PYTHON_INTERPRETER, and collections_path is singular. Defaults worth reciting: five forks, ten seconds connection timeout, host key checking on, retry files off.

Debugging order

When a run fails, work in this order and say it aloud in an interview.

1. Read the first error, not the last. A host stops at its first failure, so later lines are fallout.
2. Read the origin. Current versions print the file, line and column, a source excerpt and a caused by chain. Read the chain from the bottom.
3. --syntax-check for YAML problems.
4. -v, then -vvv for task data and connection detail.
5. Reproduce the task alone with an ad hoc command of the same module.
6. ansible-inventory --host when a variable has the wrong value, and ansible-config dump --only-changed when behaviour is strange.
7. debug to print the variable in question.

The traps, and why they share one cause

Most beginner failures come from treating Ansible like a script. A script runs steps; Ansible declares state. The traps are the places where the two models collide.

• Appending instead of declaring. A shell: echo line >> file reports changed forever and duplicates the line. Declare it with lineinfile and a regexp.
• Commands everywhere. shell: apt-get install is not idempotent and skips the module's lock handling and cache options.
• Unquoted templates. A value beginning {{ without quotes is read by YAML as a dictionary, which raises the missing-quotes parsing error.
• Facts disabled but used. gather_facts: false saves time and then the first ansible_facts[...] reference fails.
• Forgetting become. A task that edits /etc works in your root shell and fails as the automation user.
• A typo in a handler name. Names are case-sensitive, and the notification fails to find its handler.
• Stale facts. Facts are gathered once per play, so a package installed midway is not reflected until you gather again.
• ignore_errors as a fix. It silences the symptom and keeps the cause.

Common interview questions

1. What is Ansible, and why use it?A tool that configures machines from declarative YAML descriptions over SSH, with no agent. It removes configuration drift, makes setup repeatable and reviewable in git, and lets one engineer manage many machines consistently.
2. What does agentless mean, and what is the catch?Nothing is installed on managed hosts; Ansible pushes modules over SSH and uses the target's Python. The catch is that Python is still required on the target for most modules, and that the control node holds the credentials to everything.
3. What is idempotency, and which modules are not?Running the same play repeatedly gives the same result, and an already-correct host reports ok. command, shell and raw are not idempotent by themselves; fix with creates, removes or changed_when.
4. Play versus playbook versus task?A task is one module call. A play maps hosts to tasks. A playbook is a file containing one or more plays.
5. What is an inventory, and how do you select hosts?The list of hosts and groups, in INI or YAML or dynamic. Select with patterns such as web, web:db, web:&prod and web:!staging in hosts: or --limit.
6. What is an ad hoc command, and when do you use it?A single module run from the command line, like ansible web -m ansible.builtin.ping. Use it for quick questions or one-off actions. Anything repeatable belongs in a playbook.
7. What is a fully qualified collection name, and why use it?The namespace.collection.module form, such as ansible.builtin.copy. It removes ambiguity about which module runs and is what linters and modern documentation expect.
8. How does a playbook preview its changes?--check for a dry run and --diff for file diffs. Limits: tasks that cannot predict their effect are skipped, and dependent tasks can misreport. It is strong evidence, not proof.
9. What is a handler, and when does it run?A task that runs only when notified by a changed task. It runs once at the end of the play's tasks, in definition order, and will not run if the play fails first unless you force it.
10. What are facts, and how do you use them?Data gathered by the setup module, stored in ansible_facts. Use the dictionary form, ansible_facts['os_family'], to adapt tasks per host. gather_facts: false skips gathering.
11. Which variable definition wins?Extra vars (-e) always win. Roughly: role defaults weakest, then inventory and group variables, host variables, play variables, registered variables, and extra vars on top.
12. How do you make a task conditional or repeat it?when takes an expression without braces; loop repeats with item. Since 2.19 a condition must evaluate to a true boolean, so use | length > 0 or | bool on strings.
13. How does Ansible connect, and how do you escalate privilege?SSH by default, using the OpenSSH program. Escalate with become: true, which uses sudo by default, and supply a password with -K if needed.
14. A play says the host is unreachable. What do you check?Plain ssh by hand, the ansible_host, ansible_user and key, then -vvv to see the exact command. It is a connection problem, not a task problem.
15. How do you keep a password out of a repository?Ansible Vault: ansible-vault create or encrypt_string, with the vault password supplied at run time and never committed. Add no_log: true to tasks that handle the value.
16. What is the difference between command and shell?command runs a program directly with no shell features; shell runs through a shell, allowing pipes and redirects. Prefer command, and prefer a real module over either.
17. What is a role, and what is a collection?A role is a folder layout packaging tasks, handlers, templates and defaults for reuse. A collection is a larger distribution of modules, plugins and roles under a namespace.collection name, installed with ansible-galaxy.
18. Why does a second run matter?It proves idempotency. A healthy playbook shows changed=0 on the second run, which also makes the recap a real health signal.
19. How do you handle a task that might fail?Define the acceptable outcomes with failed_when, group recoverable steps in block/rescue/always, and reserve ignore_errors for the rare case where failure truly does not matter.
20. Where does Ansible read configuration from?One file only, the first found of ANSIBLE_CONFIG, ./ansible.cfg, ~/.ansible.cfg, /etc/ansible/ansible.cfg. They are not merged. Check with ansible-config dump --only-changed.
21. Can Windows be a control node?No, not natively. Use WSL, a Linux VM or a container. Windows can be a managed node, over WinRM, PSRP or SSH.

Sixty-second self-test

• Say what Ansible is in two sentences, including the word idempotent.
• Describe the six steps of what happens when a task runs on a remote host.
• Name the four things you must be able to do before Ansible can manage a Linux host.
• Explain why shell: echo x >> file is a bad task and what replaces it.
• State what --check --diff does and two of its limits.
• Give the rule for handler timing: how many times, in what order, and what a failure does.
• Explain the two conditional rules that changed in ansible-core 2.19, with a wrong and a right example of each.
• Explain why ansible_facts['os_family'] is preferred over ansible_os_family.
• Name the diagnosis order for an unreachable host, and what -vvv shows.
• Give the order in which Ansible looks for a config file, and say whether the files are merged.
• Say what no_log: true is for, and what Vault does and does not protect.

Part one of three. Three mistakes produce most of the errors below: you write steps when Ansible wants a state, you forget that YAML and Jinja2 each have their own rules, or you forget that a playbook runs on a different machine with a different user than you. Start with the table, then the habits and practice cards. Everything targets ansible-core 2.21, the engine of the Ansible community package 14.

Common errors at this level

The habits that pay off most

Practice cards

Short, self-contained exercises. Each one takes a few minutes and ends in a result you can check on screen. Use the ~/ansible-lab folder from the detailed guide.

1. Watch idempotency happenRun ansible lab -m ansible.builtin.file -a "path=/tmp/tip-demo state=directory" twice. Then run ansible lab -m ansible.builtin.command -a "uptime" twice. One flips from CHANGED to SUCCESS, the other never does.
2. Make a command idempotentWrite a task that runs touch /tmp/tip-made through ansible.builtin.command. Run the playbook twice and see changed both times. Add creates: /tmp/tip-made and run it twice again.
3. Double a line on purposeUse lineinfile without a regexp to set mode=a, then change it to mode=b and rerun. Count the lines. Add regexp: '^mode=' and repeat.
4. Break a conditionWrite when: "{{ ansible_facts['system'] }} == 'Linux'", read the error, and fix it. Then write when: some_string with a string variable, read that error, and fix it with | length > 0.
5. Climb the precedence ladderDefine a variable in group_vars/lab.yml, then in the play's vars, then on the command line with -e. Print it each time with debug and note who wins.
6. Fire a handler exactly onceGive three tasks the same notify. Make all three change something and confirm the handler runs once at the end. Run again and confirm it does not run at all.
7. Lose a secret, then recover itCreate a vaulted file, run a playbook using it without the password, read the error, then supply the password. Then try a wrong password and compare the two messages.
8. Diagnose a fake outageAdd a host with an address that does not exist and run ping with -vvv. Find the exact ssh command in the output and read the failure it reports.

Reading a failure without panicking

A red failure looks like a wall of text. It is actually a short message wrapped in structure, and you can learn to pull it apart in seconds.

Start at the top of the failure, not the bottom. Current versions print [ERROR]: Task failed: followed by the reason in a sentence. Below it is Origin: with a file name, a line and a column, then a small excerpt of your own source with a marker under the offending spot. Below that, sometimes, a <<< caused by >>> chain, which is the sequence of deeper reasons; the last link is usually the real root cause. The JSON blob after fatal: [host]: FAILED! => holds the module's own detail, such as rc, stdout and stderr, and is where a failed command tells you what the program said.

Then ask which of three families the problem belongs to, because each has a different fix.

1. It never ran. YAML would not parse, a module name could not be resolved, or a variable was undefined while arguments were being prepared. The task itself never touched the host. Fix the text of the playbook.
2. It could not reach the host. UNREACHABLE means a connection or authentication problem, often SSH keys, users, addresses, firewalls or a become timeout. Your tasks are probably fine.
3. It ran and the host said no. A command returned a non-zero code, a package does not exist, a service would not start. Read stderr, then reproduce the same thing by hand on that host.

The two distinguishing words are FAILED (it ran and failed) and UNREACHABLE (it never got there). If you handle them with the wrong tool, such as ignore_errors for an unreachable host, you will chase the wrong problem. Since ansible-core 2.19, a become timeout is classed as unreachable, so use ignore_unreachable: true there.

Keeping a playbook readable

Playbooks are read far more often than they are written, often by someone who is stressed and in a hurry. A few conventions make that person's life easier, and some of them stop errors too.

Give plays and tasks descriptive names that state the outcome. "Install nginx" and "Ensure the deploy user exists" read as state. "Run the thing" does not.

Keep each task to one job. A task that installs a package and also edits a file and also restarts a service hides failures and cannot be tagged or skipped sensibly.

Use two-space indentation throughout, and never tabs. Put the module name on its own line and its arguments indented under it, as in the detailed guide, rather than in the compact key=value style. The YAML map form is easier to diff, supports lists, and avoids quoting surprises.

Write booleans as true and false, and quote file modes ("0644"), version numbers ("3.10", which unquoted would become the number 3.1) and anything that starts with {{, *, &, !, or a colon-space. YAML is forgiving in ways that silently change values: no, off and yes have historically been read as booleans, and a country code NO for Norway famously became false. Quoting removes the doubt.

Prefer loop to with_items. Both work, but loop is the current style and what documentation and linters expect. Prefer passing a list to modules that accept one, such as ansible.builtin.package with name: as a list, over looping, since it is faster and lets the package manager resolve everything in one transaction.

Keep values that differ out of tasks. A port number, a package list or a path belongs in a variable, ideally in group_vars, so that the tasks never change between environments and only the values do.

Finally, comment why, not what. The task name already says what happens. A short comment explaining why a setting has an unusual value saves the next reader an afternoon.

Variables, facts and templates: the traps

The variable system is the most common source of "it works here but not there". A few specifics are worth learning early.

Precedence surprises come from -e. Extra vars beat everything, including values in host_vars. If a value will not change no matter what you edit, search your shell history and your CI configuration for -e. The command ansible-inventory --host <name> shows the value from inventory sources, which helps you tell a file problem from a command-line override.

Variable names in defaults versus vars matter once you use roles. Role defaults are the weakest variables, so callers override them easily. Role vars are much stronger and are hard to override. Put settings a user should change in defaults.

Facts are a snapshot. They are collected once at the start of a play. If a task installs a package or changes a network setting, a fact that describes it is stale until you re-gather with ansible.builtin.setup. If you need the current state mid-play, use a purpose-built query such as ansible.builtin.stat, or ansible.builtin.package_facts and service_facts.

Do not define your own variables with names that could clash. Prefix them by purpose (nginx_port, app_user). Avoid reserved and magic names such as hostvars, groups, environment, playbook_dir, and _task, which ansible-core 2.21 uses for a new implicit object.

Jinja2 whitespace and types. Since ansible-core 2.19, templates always return native types: "{{ 5 }}" is the integer 5, not the string "5". That is usually what you want, but a version number like "{{ '3.10' }}" stays a string while an unquoted 3.10 in YAML is a float. Be explicit with the string and int filters when a module cares. A range() result cannot be stored in a variable directly; add | list.

Undefined and optional values. An undefined variable is an error, which is good. For a genuinely optional argument, pass omit so the module argument is left out entirely:

- name: Create a user with an optional comment
  ansible.builtin.user:
    name: alice
    comment: "{{ user_comment | default(omit) }}"

Templates and validate. Whenever a service has a config-check command, use the validate argument of template or copy so a broken file never replaces a working one. validate: nginx -t -c %s is the standard example. It turns a potential outage into a failed task.

Safer connections and privilege

Connection and privilege choices are easy to get wrong in ways that work until they are dangerous.

Do not turn off host key checking outside a lab. host_key_checking = False removes the protection against connecting to an impostor. The right fix for Host key verification failed is to add the key to known_hosts, with ssh-keyscan or by connecting once, or to use the ansible.builtin.known_hosts module in a bootstrap play.

Use a dedicated automation user. A separate account for Ansible, with a key, and with sudo rights limited to what it needs, keeps an audit trail and keeps your personal account out of automation. Passwordless sudo is acceptable for that account because it cannot be used interactively.

Do not put sudo passwords in plain text. Use -K while learning, and Vault or an external secret store later.

Do not use become everywhere by default. Add it only to the plays or tasks that need root. Tasks that run as the normal user leave files owned by the right person, and a mistake does less damage.

Mind shell and untrusted values. A variable interpolated into shell runs as shell code. If the value could come from outside, such as a branch name or a user input, quote it with the quote filter: {{ branch | quote }}. Better, avoid shell and use a module with proper arguments.

Keep the control node safe. Anyone who can change your playbooks, inventory or group variables can run commands as root on managed hosts. Protect the repository like production credentials. Do not keep private keys or vault passwords in the repository. A .gitignore entry for .vault_pass and keys is the minimum.

Know where output goes. With -vvv, secrets passed as variables can appear in the terminal or in a log file. Use no_log: true on sensitive tasks, and never debug a secret.

Making runs faster

A playbook that takes ten minutes on ten servers is tedious, and a few beginner-level settings cut that time sharply without any risky change.

Skip facts you do not use. gather_facts: false on a play that never reads facts saves a round trip to every host. For plays that need only a little, gather_subset narrows what is collected.

Raise forks. Ansible works on five hosts at a time by default. With twenty hosts, forks = 20 in ansible.cfg (or -f 20) lets them all run at once. The limit is your control node's CPU and memory, since each fork is a process, and the network.

Pass lists to modules. One package task with ten names is far quicker than a loop of ten tasks, because the package manager runs once.

Limit what you run while developing. Use --limit to try a change on one host, and --tags to re-run only the slice you edited. Use --start-at-task to resume partway through a long playbook after a fix.

Avoid work that does nothing. A task that runs apt update every time costs seconds per host per run. Use the module's own cache options (update_cache: true with cache_valid_time) so the refresh happens only when the cache is old.

Larger-scale tuning, including SSH pipelining, fact caching and execution strategies, belongs to the Mid-level and Senior guides, because it matters only once the basics are in place.

Stale tutorials: what changed

A lot of Ansible content online was written for versions many releases old. Knowing the main differences lets you use it safely, because the ideas are usually fine and the syntax is stale.

A starter setup to copy

A small project skeleton you can copy whenever you start. It has everything the detailed guide used, in the shape most teams adopt.

my-project/
  ansible.cfg
  inventory.ini
  site.yml
  requirements.yml
  group_vars/
    all.yml
    web/
      vars.yml
      vault.yml
  host_vars/
  templates/
  files/
  roles/
  .gitignore

[defaults]
inventory = inventory.ini
interpreter_python = auto_silent
callback_result_format = yaml
forks = 10
# vault_password_file = .vault_pass   # uncomment once you have a password file (and gitignore it)

.vault_pass
*.retry
out/

And a checklist to run through before you point a playbook at real machines.

• Every task has a name and uses a fully qualified module name.
• No command or shell task where a module exists; any that remain are idempotent.
• Values that differ per environment are in group_vars or host_vars, not in tasks.
• Secrets are in Vault or an external store, and sensitive tasks have no_log: true.
• The vault password file and private keys are in .gitignore.
• --syntax-check passes and --list-hosts shows exactly the hosts I expect.
• --check --diff has been read, and every change in it is one I intended.
• A second real run reports changed=0.
• Host key checking is on, and privilege escalation is used only where needed.
• Nothing in the playbook depends on a deprecated name I copied from an old tutorial.

This is part two of three. It picks up where Beginner ended and assumes you already run playbooks at work. The aim is to make you the person on the team who can predict what Ansible will do before it does it: why a task reports changed every run, why a variable has the value it has, why a rolling restart took the whole pool down, and why the same playbook that passed last month now fails after an upgrade. This guide targets ansible-core 2.21 (Ansible community package 14), and it flags every place where older tutorials on the internet will now mislead you.

Where this picks up

Each section starts with the problem it solves and ends with a Try it that you can run against a scratch host or a container in a few minutes. Anywhere a section needs a neighbouring tool, it links to that guide in this catalogue: Terraform for provisioning, Kubernetes for the platform, Docker for test targets, and GitHub Actions for the pipeline.

What happens when a task runs

Ansible has no agent. That sentence is easy to recite and hard to reason with, so here is what it means mechanically. For every task on every host, the controller does roughly the following. It resolves the task's arguments by rendering templates against that host's variables. It picks the module (or, for things like template, copy, debug and set_fact, an action plugin that runs on the controller first and may call a module afterwards). For a Python module it builds a self-contained payload with the module code, its module_utils dependencies and the arguments baked in, a wrapper usually called AnsiballZ. It opens a connection through the connection plugin (ssh by default), makes a temporary directory on the target, copies the payload there, runs it with the target's Python interpreter, reads back the JSON it prints, and deletes the directory. The result JSON is what you see as ok, changed, failed or skipped.

Three practical consequences follow from that model, and each one explains a class of incidents.

Every task costs at least one round trip, usually several. Without pipelining, a task with become creates a temp directory, uploads the module, makes it executable, runs it through sudo, and removes the directory, as separate SSH operations. That is why a playbook with two hundred tasks feels slow against a host with a high-latency link even though each module takes milliseconds. The performance section later covers pipelining and connection reuse; for now, just hold the model.

The target must run the module with a Python it can find. For core 2.21, targets need Python 3.9 through 3.14, and the control node needs 3.12 through 3.14. Ansible discovers the interpreter automatically (interpreter_python = auto, the default). The older modes auto_legacy and auto_legacy_silent were removed in 2.21, and some pages on docs.ansible.com still mention them, so if you see them in a colleague's ansible.cfg they will no longer work. The valid values now are auto, auto_silent, or an explicit path such as /usr/bin/python3.12. You will meet the discovery warning in practice:

[WARNING]: Host 'web01' is using the discovered Python interpreter at '/usr/bin/python3.12', but future installation of another Python interpreter could cause a different interpreter to be discovered.

It is a warning about stability, not a fault. Fix it by pinning ansible_python_interpreter in group_vars, or silence it with interpreter_python = auto_silent once you are happy. The raw and script modules are the exceptions to the Python requirement, which is why raw is the standard way to bootstrap Python onto a bare host.

Modules report; the controller decides. A module returns a dictionary. The controller interprets changed, failed, rc and friends, applies your changed_when and failed_when overrides, stores the dictionary if you used register, and only then moves on. Debugging is therefore mostly a matter of looking at the dictionary, which is what -v and debug: var= are for.

Two debugging switches belong in your toolkit from now on:

ANSIBLE_KEEP_REMOTE_FILES=1 ansible-playbook -vvv site.yml   # leave the module payload on the target
ansible-playbook site.yml -vvvv                               # show the exact ssh command line

With ANSIBLE_KEEP_REMOTE_FILES=1 the temp directory under the remote user's ~/.ansible/tmp is not deleted, so you can log in, read the payload and rerun it by hand. -vvvv prints the SSH invocation and the connection details, which resolves most "works from my shell but not from Ansible" mysteries within a minute.

Inventory beyond a flat file

Beginner gave you a host list with a few groups. At work the inventory is the most important file in the repository, because it decides what a run touches. Mistakes here are not cosmetic: a typo in a group name means a play targets nothing and exits green, or targets everything.

The first design decision is separating structure from data. The inventory file says which hosts exist and which groups they belong to. Variables belong next to it in group_vars/ and host_vars/ directories, one file or directory per group or host, named exactly like the group or host:

inventories/
  production/
    hosts.yml
    group_vars/
      all.yml
      web.yml
      db/
        main.yml
        vault.yml
    host_vars/
      db01.yml
  staging/
    hosts.yml
    group_vars/
      all.yml
      web.yml

Keeping one directory per environment, each a complete inventory, is the simplest reliable multi-environment design. You run ansible-playbook -i inventories/staging site.yml and nothing from production leaks in. A single inventory with [staging] and [production] groups works too, but then one wrong --limit or one forgotten hosts: pattern applies production variables to the wrong machines. Separate directories make the blast radius a property of the command line you typed.

Notice db/main.yml and db/vault.yml. A group_vars/<group> entry can be a directory, and every file in it is loaded. This is the conventional way to keep encrypted secrets beside plain variables without encrypting the whole file, which matters because encrypted files produce unreadable diffs. The vault section returns to this.

all:
  children:
    web:
      hosts:
        web01: { ansible_host: 10.20.0.11 }
        web02: { ansible_host: 10.20.0.12 }
    db:
      hosts:
        db01: { ansible_host: 10.20.0.21 }
    production:
      children:
        web:
        db:

Host patterns and --limit

Patterns appear in hosts: and in --limit/-l. The ones you need at this level:

--limit narrows whatever the play's own hosts: already selected; it never widens it. That is a safety property you can rely on. If a play says hosts: web and you pass -l db01, nothing runs and Ansible tells you no hosts matched. Two always-present groups, all and ungrouped, exist without being declared. The implicit localhost is deliberately not a member of all, which surprises people who write hosts: all and wonder why their localhost play skipped.

Dynamic inventory

Static files go stale the moment autoscaling exists. Inventory plugins build the host list at run time from a source of truth: amazon.aws.aws_ec2, azure.azcollection.azure_rm, google.cloud.gcp_compute, kubernetes.core.k8s, and a VMware plugin among others. You configure a plugin with a small YAML file whose name the plugin recognises, conventionally ending in the plugin's suffix, for example aws_ec2.yml. The plugin-specific keys go inside. The part that matters for operations is grouping and variables derived from metadata, such as cloud tags:

plugin: amazon.aws.aws_ec2
regions:
  - me-central-1
filters:
  tag:managed-by: ansible
keyed_groups:
  - key: tags.role
    prefix: role
hostnames:
  - private-ip-address
cache: true
cache_timeout: 300

With that file, every instance tagged role=web lands in a group called role_web. You write hosts: role_web and never maintain a host list. Two warnings. First, turn caching on for any plugin that calls a cloud API, because every playbook run, every ansible-inventory call and every --list-hosts hits the API otherwise, and large accounts hit rate limits. Second, the tag convention is part of your interface: if someone retags an instance, the next run targets a different set of hosts. That is powerful, and it means tag changes deserve review like code changes. (The region shown, me-central-1, is AWS's UAE region; use whichever region your employer's data-residency rules allow.)

You can point -i at a directory mixing static files and plugin configs, and Ansible merges them. That is how you keep a few hand-maintained hosts, such as a bastion, next to a dynamic fleet.

Inventory as a debugging tool

When a variable is wrong, the first question is whether the inventory gave the host what you think. ansible-inventory answers without running anything:

ansible-inventory -i inventories/production --graph            # group tree
ansible-inventory -i inventories/production --graph --vars     # with variables
ansible-inventory -i inventories/production --host web01       # merged vars for one host
ansible-inventory -i inventories/production --list --yaml      # everything, as YAML

--host shows the host's variables as the inventory layer sees them, after merging group_vars and host_vars. It does not show play-level variables, role defaults or facts, so a value missing here may still exist at runtime, but a value present here with the wrong number is an inventory problem for certain.

Variables, precedence and facts

Ansible has twenty-two places a variable can be defined, and the official docs list them in a fixed order. You do not need to memorise all twenty-two. You need the shape of the ladder, and a handful of rules that cover nearly every real conflict.

The shape, lowest precedence to highest: role defaults sit at the bottom, deliberately, so that anything can override them. Then inventory variables, with group_vars/all below more specific groups, and groups below host_vars. Then play variables (vars, vars_prompt, vars_files), then role vars/, then block and task variables, then include_vars, then registered variables and set_fact, then role and include parameters, and finally extra vars passed with -e, which beat everything.

The rules that follow from the ladder:

1. Put a role's tunable settings in defaults/, never in vars/. Defaults are the public interface. vars/ sits above play vars and inventory, so a value there cannot be overridden from group_vars, which surprises the next person who tries. Use vars/ only for internal constants the role's users should not change.
2. More specific beats less specific inside the same layer. host_vars/web01 beats a group file, and a child group beats its parent. When two groups at the same level disagree, Ansible merges groups in alphabetical order, and the later one wins; you can set ansible_group_priority on a group to control this. Relying on alphabetical order is a smell. Restructure instead.
3. -e is for emergencies and pipelines, not habits. Because it beats everything, an extra var also beats the fix you commit. If -e version=1.2.3 is in a CI command, no amount of editing group_vars will change what deploys.
4. set_fact has a high precedence and persists for the rest of the play on that host. It is convenient and it is also how variable values become untraceable. Prefer a computed variable in vars: when the value does not depend on a task result.

Dictionaries are replaced, not merged, by default. The setting DEFAULT_HASH_BEHAVIOUR defaults to replace, so if group_vars/all defines app: {port: 80, log_level: info} and a host defines app: {port: 8080}, the host's dictionary wins entirely and log_level disappears. The merge setting exists and is discouraged, because it changes behaviour globally for every role you use. Merge explicitly where you mean to:

- name: Build the effective app settings
  ansible.builtin.set_fact:
    app_effective: "{{ app_defaults | combine(app_overrides | default({}), recursive=true) }}"

Facts

Facts are variables Ansible discovers about a host. They are collected at the start of each play by an implicit ansible.builtin.gather_facts task (gathering = implicit), and stored under ansible_facts. The hygiene rule that matters in 2.21: always read facts through the ansible_facts dictionary, as ansible_facts['os_family'] or ansible_facts['default_ipv4']['address']. The old injected names such as ansible_os_family still work today because INJECT_FACTS_AS_VARS defaults to true, but that default is deprecated and flips to false in ansible-core 2.24. Every playbook that uses the bare names will break then, so migrating is cheap now and expensive later. A quick audit: search your repository for ansible_distribution, ansible_os_family, ansible_hostname and ansible_default_ipv4, and to catch what remains, run a canary job with ANSIBLE_INJECT_FACT_VARS=false set.

Gathering facts is the slowest thing many plays do. If a play does not use facts, set gather_facts: false. If it needs a few, limit the subset:

- name: Only what this play needs
  hosts: web
  gather_facts: true
  gather_subset:
    - "!all"
    - "!min"
    - network
  tasks:
    - ansible.builtin.debug:
        msg: "{{ ansible_facts['default_ipv4']['address'] }}"

You can also define custom facts by dropping an INI or JSON file ending in .fact into /etc/ansible/facts.d/ on the target. They appear under ansible_local, which is a neat way for a host to report what it is, for example which application release is installed, without any central database.

Finally, magic variables are the ones Ansible maintains for you: hostvars (every host's variables, which is how host A reads host B's facts), groups, group_names, inventory_hostname, ansible_play_hosts, ansible_play_batch, ansible_check_mode, ansible_version, role_path and playbook_dir. The old play_hosts is deprecated, so use ansible_play_batch. Reading another host's facts through hostvars['db01']['ansible_facts'] only works if db01's facts were gathered earlier in the same run, or cached.

Templating after the 2.19 rewrite

If you learned Ansible from a blog post or an older book, this is the section most likely to correct you. Ansible-core 2.19 (July 2025) overhauled templating under the banner of "data tagging", and the consequences reach everyday playbooks. Core 2.21 inherits all of them. The changes are deliberate and mostly make errors earlier and clearer, but they also broke a lot of playbooks that had worked by accident.

Conditionals must produce a boolean. when: used to accept any value and coerce it, so when: my_string was true for any non-empty string and when: "false" was also true, which caused quiet bugs. Now a non-boolean result is an error:

[ERROR]: Task failed: A 'when' expression failed: Conditional result (True) was derived from value of type 'str' at 'site.yml:4:12'. Conditionals must have a boolean result.

Say what you mean instead:

when: my_list | length > 0          # not: when: my_list
when: enable_feature | bool         # for strings like "yes"/"true" from -e or inventory
when: my_value is truthy            # the test form
when: result is changed             # tests beat inspecting result.changed

There is an escape hatch, ALLOW_BROKEN_CONDITIONALS=True, which downgrades the error to a warning. Treat it as a bridge while you migrate, not a setting to leave on; it is precisely the setting that hides the bug class the change was meant to remove.

Templates inside expressions are rejected. The fields when, failed_when, changed_when and until are already expressions, so writing {{ }} inside them is wrong, and is now an error rather than a tolerated habit:

# Bad: Syntax error in expression. Template delimiters are not supported in expressions
when: "{{ env }} == 'prod'"
# Good
when: env == 'prod'

Templates return native types. Jinja native mode is now always on, and the old jinja2_native setting is deprecated with no effect. A template that is just "{{ 5 }}" yields the integer 5, and "{{ some_list }}" yields a list, not its string rendering. You will notice this when you used to need | int or | from_json round trips; many of those are now redundant. One related trap: a range() result cannot be stored in a container, so write range(3) | list.

The trust model inverted. Only strings from trusted sources, which means your playbooks, roles, vars files and the built-in inventory plugins, are rendered as templates. Strings that arrive as module output, lookup results, facts or other external data are not re-templated. This closes a class of template-injection problems: a hostname or tag containing {{ ... }} from a cloud API cannot execute. If you ever need to process such data, use filters on it explicitly rather than hoping it will be templated later.

Lazy templating means that nested data structures are rendered when accessed, which makes large templated structures far faster. You do not need to do anything, but it explains why some errors now surface later, at the point a value is used, instead of when a variables file loads. Vault behaves the same way, as we will see.

Error messages got better and are not a stable interface. An error now shows the file, line and column, a source excerpt, and a chain of causes separated by <<< caused by >>>. Read the innermost cause first. Do not write scripts that parse error text; Ansible explicitly does not promise its wording. Tracebacks are controlled by display_traceback (ANSIBLE_DISPLAY_TRACEBACK) which takes error, warning, deprecated, deprecated_value, always or never, with never the default.

Two more details that bite during migration. Variable omit placeholders resolved inside loop items no longer leak through to the task, so use | default(omit) on the argument itself. And the bool filter coercing unrecognised strings is deprecated and becomes an error in 2.23. The filter recognises True, 1 and the strings "yes", "on", "true" and "1" in any case as true, and False, 0 and "no", "off", "false", "0" as false. Anything else, such as "enabled", should be compared explicitly.

And a hint for 2.21 specifically. The new _task object and register projections reduce boilerplate in conditionals and registration. A register: can now be a mapping of names to Jinja expressions (written without {{ }}), and _task.result gives the current task's result inside changed_when, failed_when, when and until without registering a variable first. _task is now a reserved name; do not use it as a variable.

- name: Query build info
  ansible.builtin.command: /opt/app/bin/version
  register:
    version_raw: _task.result.stdout
    version_ok: _task.result.rc == 0
  changed_when: false

Idempotency, check mode and diff mode

An idempotent task can be run again and again and converges to the same state. The second run reports ok, not changed, and touches nothing. Idempotency is the property that lets you run a playbook on a schedule, rerun after a partial failure, and trust a "0 changed" report as evidence that production matches the repository. It is also the property people lose without noticing.

Most modules are idempotent by construction, because they describe state: package with state: present, file with a mode, user, service. The trouble comes from three places.

command, shell and raw know nothing about state. They run, and the module reports changed every time, because it cannot tell whether your command altered anything. You have three tools to teach it:

- name: Initialise the database schema once
  ansible.builtin.command: /opt/app/bin/init-db
  args:
    creates: /var/lib/app/.db-initialised

- name: Read the current version (never a change)
  ansible.builtin.command: /opt/app/bin/version
  register: app_version
  changed_when: false

- name: Apply migrations, changed only if any ran
  ansible.builtin.command: /opt/app/bin/migrate --apply
  register: migrate
  changed_when: "'Applied' in migrate.stdout"

creates and removes skip the task when a path exists or does not exist. changed_when: false is right for read-only queries. A computed changed_when is right when the command's output says whether it acted. Before you reach for command, search ansible-doc -l for a real module. There almost always is one, and it will be idempotent and will support check mode.

Templates that render differently each time. A template containing a timestamp, a random value or unordered dictionary output changes the file on every run, so the template task is perpetually changed and its notify restarts the service every night. Keep volatile content out of templates. If you want a header saying the file is managed, use the ansible_managed variable, which is constant; do not interpolate ansible_date_time. (Note also that the ansible_managed ini key is deprecated in favour of setting an ansible_managed variable.)

Order-dependent or time-dependent state. A task that appends a line, downloads "latest", or writes something derived from the wall clock will churn. Constrain lineinfile with a regexp so it replaces instead of appending; pin versions; use checksum with get_url.

Check and diff mode

--check (-C) asks every module that supports it to report what it would change without changing anything. --diff (-D) shows a before and after for files and templates. Together they are the standard pre-flight review:

ansible-playbook -i inventories/production site.yml --check --diff --limit web01

Treat check mode as strong evidence, not proof, and know where it is blind:

• A command or shell task is skipped in check mode unless it has check_mode: false, because Ansible cannot know whether it is safe. Anything that depends on that skipped task's result, such as a register used in a later when, sees nothing.
• A later task can depend on an earlier one having really run. Package installed in task 3, service configured in task 4: in check mode task 3 does not install, so task 4 may report a false failure.
• Modules that do not support check mode are skipped, silently in some cases.

The task keyword check_mode: false forces a task to run for real even under --check, which is the right setting for a read-only query whose result later tasks need. check_mode: true does the opposite and forces a dry run for that task even on a normal run. The magic variable ansible_check_mode lets a task behave differently, for example skipping a health wait that cannot succeed in a dry run.

- name: Discover the installed version (safe to run in check mode)
  ansible.builtin.command: /opt/app/bin/version
  register: installed
  changed_when: false
  check_mode: false

- name: Wait for health (pointless in check mode)
  ansible.builtin.uri:
    url: http://localhost:8080/health
  register: h
  until: h.status == 200
  retries: 10
  delay: 3
  when: not ansible_check_mode

--diff leaks data. A diff of a templated file that contains a password prints that password in the job log. Put no_log: true on tasks that render secrets, or diff: false on the task, and read the output of a diff run before you paste it into a chat.

Handlers, blocks and failure handling

Real playbooks need three things that a flat list of tasks lacks: actions that run only when something changed, grouped tasks with cleanup, and a deliberate policy for when something fails. Ansible provides handlers, blocks, and a set of failure keywords. Each has a precise behaviour that is worth knowing exactly.

Handlers

A handler is a task that runs only if a task notified it and that task reported changed. Handlers run once per play at a defined point, no matter how many tasks notified them, which is what lets you change five config files and restart the service once.

- name: Validate nginx config
  ansible.builtin.command: nginx -t
  changed_when: false
  listen: Reload nginx

- name: Reload nginx
  ansible.builtin.service:
    name: nginx
    state: reloaded

The details that matter:

• Handlers run at the end of each section: after pre_tasks, after tasks (including roles), and after post_tasks. They run in the order they are defined, not the order they were notified.
• meta: flush_handlers runs pending handlers right now. Use it when a later task in the same play needs the restarted service, for example a health check.
• listen: subscribes a handler to a topic string, so notify: Reload nginx can trigger several handlers. This decouples tasks from handler names, which helps across roles.
• If a task fails on a host, pending handlers for that host do not run, by default, and this is a real trap: the config file was changed, the restart handler was queued, the play failed a task later, and the service keeps running with the old config while the file on disk is new. The next run reports the file as unchanged (it already matches), so the handler is never notified and the drift persists. Set force_handlers: true on the play or pass --force-handlers if you want handlers to run even after failures.
• Handlers are not run when a notifying task reports ok or skipped. A handler can be made to run unconditionally only by calling the module as a normal task.

Blocks: group, recover, clean up

A block groups tasks so that keywords such as when, become, vars or tags apply to all of them, and so that you can attach error handling. block, rescue and always behave like try, except and finally:

- name: Deploy with rollback
  block:
    - name: Install the new release
      ansible.builtin.unarchive:
        src: "https://artifacts.example.com/app-{{ app_version }}.tar.gz"
        dest: /opt/app/releases
        remote_src: true
    - name: Switch the current symlink
      ansible.builtin.file:
        src: "/opt/app/releases/app-{{ app_version }}"
        dest: /opt/app/current
        state: link
    - name: Restart and verify
      ansible.builtin.uri:
        url: http://localhost:8080/health
      register: health
      until: health.status == 200
      retries: 10
      delay: 3
  rescue:
    - name: Roll back to the previous release
      ansible.builtin.file:
        src: "{{ previous_release }}"
        dest: /opt/app/current
        state: link
    - name: Fail loudly after rolling back
      ansible.builtin.fail:
        msg: "Deploy of {{ app_version }} failed on {{ inventory_hostname }}; rolled back."
  always:
    - name: Remove the lock file
      ansible.builtin.file:
        path: /tmp/deploy.lock
        state: absent

Notes on behaviour: rescue runs when any task in the block fails, and if the rescue section succeeds, the host counts as recovered, not failed, so the play continues. That is why the explicit fail at the end of the rescue is often wanted: you roll back, then still report failure so the pipeline goes red. Inside rescue, the variables ansible_failed_task and ansible_failed_result describe what went wrong. always runs regardless of success, which is the place for cleanup. A when on a block is evaluated for each task inside it, so a condition that changes mid-block can produce confusing partial execution; keep block conditions stable.

Failure keywords

Two precise points. ignore_errors hides task failures but not unreachable hosts, and since 2.19 a become timeout is classed as unreachable, so the flag you need there is ignore_unreachable, not ignore_errors. And failed_when replaces the default failure logic, so a failed_when: false will mask everything including a missing binary, while a precise one like failed_when: chk.rc not in [0, 2] documents intent. Also note that a result exception suppressed by failed_when is now stored under failed_when_suppressed_exception if you need to inspect it.

A related signal in 2.21: relying on a non-zero rc alone to mark failure, in modules you write yourself, is deprecated; modules must set failed: true or call fail_json(). If you maintain custom modules, fix them before 2.22 starts warning at runtime.

Roles that survive a team

A role is Ansible's unit of reuse: a directory with a fixed layout that Ansible knows how to load. At beginner level a role is a folder of tasks. At mid level it is an interface: a set of documented inputs, a known effect, and a promise about behaviour on rerun.

roles/nginx/
  defaults/main.yml        # public inputs, lowest precedence
  vars/main.yml            # internal constants, high precedence
  tasks/main.yml
  handlers/main.yml
  templates/nginx.conf.j2
  files/
  meta/main.yml            # dependencies, platforms
  meta/argument_specs.yml  # the validated interface

Defaults versus vars, again

Everything a caller may change goes in defaults/main.yml, and every variable in a role is prefixed with the role name, such as nginx_worker_processes. The prefix is not style. Variables from every role share one namespace for a host, so port in one role collides with port in another. The prefix makes collisions unlikely and makes greps useful. Internal scratch variables can be prefixed with a double underscore, __nginx_conf_path, by convention.

Validate the interface with argument specs

Since meta/argument_specs.yml exists, use it. It declares each input's type, default, choices and whether it is required, and Ansible validates it automatically when the role runs, failing fast with a clear message rather than halfway through with an obscure template error.

argument_specs:
  main:
    short_description: Install and configure nginx
    options:
      nginx_worker_processes:
        type: str
        default: auto
        description: Value for worker_processes.
      nginx_listen_port:
        type: int
        default: 80
      nginx_tls_enabled:
        type: bool
        default: false
      nginx_upstreams:
        type: list
        elements: dict
        required: true
        description: Backends to proxy to.

With this in place, nginx_listen_port: "eighty" fails at role entry with a type error instead of rendering a broken config. Core 2.20 added a tech-preview play keyword, validate_argspec, which does the same for play variables using a <playbook>.meta.yml beside the playbook, so site.yml pairs with site.meta.yml. The ansible.builtin.validate_argument_spec module lets you validate explicitly anywhere.

Import versus include

Both pull tasks or roles in from elsewhere, and the difference is when. import_* (import_tasks, import_role, import_playbook) is static: processed at parse time, so the imported tasks exist before the play starts, --list-tasks and --tags see them, and any keyword on the import, such as when or tags, is copied onto each imported task. include_* (include_tasks, include_role) is dynamic: processed when reached at runtime, so it can loop, can use variables computed earlier, and keywords on the include apply to the include itself.

The old bare include: was removed in 2.16, so replace it with one of the above. The classic surprise: putting when: on an import_tasks does not skip the import, it copies the condition onto every child task and evaluates it per task, which matters when a child task changes the variable the condition reads.

roles: in a play is treated like a static import, and runs before tasks: but after pre_tasks. For control over ordering between plain tasks and role runs, use include_role or import_role inside tasks:.

Keep roles honest

• A role should do one thing and be safe to run on a host twice. If it needs unrelated knowledge (firewall rules, monitoring agents, DNS), it is a playbook in disguise.
• Do not let a role reach into hostvars of other hosts unless that is its entire purpose, and document it if so.
• Avoid meta/main.yml dependencies on roles that take variables; a long dependency chain is hard to reason about and runs dependency roles even when you do not want them. Prefer explicit composition in the playbook.
• Put platform differences in vars/ files loaded per OS: include_vars: "{{ ansible_facts['os_family'] }}.yml" with a first_found lookup for fallbacks.

- name: Load platform variables
  ansible.builtin.include_vars: "{{ lookup('ansible.builtin.first_found', params) }}"
  vars:
    params:
      files:
        - "{{ ansible_facts['distribution'] }}-{{ ansible_facts['distribution_major_version'] }}.yml"
        - "{{ ansible_facts['os_family'] }}.yml"
        - default.yml
      paths:
        - vars

Collections, requirements and reproducible environments

Since ansible-core split from the batteries, almost every module outside ansible.builtin lives in a collection: a namespaced bundle of modules, plugins, roles and docs, named like community.general or amazon.aws. The community ansible package bundles about eighty-five collections, and plain ansible-core ships only the built-ins. Knowing which you installed explains why "it works on my laptop" but not in CI.

Always call modules by fully qualified collection name: ansible.builtin.copy, not copy. Short names still resolve in most cases through a search list, but they are ambiguous, collide with collection-provided modules of the same name, and are flagged by ansible-lint. An unresolved module produces:

[ERROR]: couldn't resolve module/action 'community.general.not_a_module'. This often indicates a misspelling, missing collection, or incorrect module path.

The three usual causes are a typo, a collection that is not installed where Ansible looks, or a module that was removed or redirected. Check with ansible-galaxy collection list and ansible-doc <fqcn>.

Pin everything in requirements.yml

A playbook that depends on whatever version of a collection happens to be installed is not reproducible. Put roles and collections in one file, with version constraints, and install from it in every environment:

collections:
  - name: amazon.aws
    version: ">=10.0.0,<11.0.0"
  - name: community.general
    version: ">=12.0.0,<13.0.0"
  - name: kubernetes.core
    version: ">=6.0.0,<7.0.0"
roles:
  - name: geerlingguy.docker
    version: "7.4.1"

ansible-galaxy collection install -r requirements.yml -p ./collections
ansible-galaxy role install -r requirements.yml -p ./roles
ansible-galaxy collection list
ansible-galaxy collection verify community.general

Installing into ./collections inside the repository, together with collections_path = ./collections in ansible.cfg, makes a checkout self-contained. The version ranges above are illustrative, so replace them with what your team has tested. Note that the collection path setting is singular, collections_path, with the environment variable ANSIBLE_COLLECTIONS_PATH; the plural name appears in old posts.

Two behaviours worth knowing. Since core 2.21, ansible-galaxy collection install skips collections whose requires_ansible is incompatible with your core version, instead of installing something that then fails. And since 2.20, Galaxy v2 API servers are no longer supported, which matters if you run an old private Galaxy mirror. If you must work offline, ansible-galaxy collection download -r requirements.yml -p ./offline makes a local set you can install from.

Match the core version to the collections

Each collection declares the core versions it supports in meta/runtime.yml as requires_ansible. When a collection does not support your core, you get a warning of the form Collection X does not support Ansible version 2.21.x, governed by collections_on_ansible_version_mismatch. A warning is easy to ignore in a long log. The safest posture is a lock of the whole toolchain: one ansible-core version plus the collection versions you tested.

Execution environments

An execution environment (EE) is a container image holding ansible-core, ansible-runner, collections, and their Python and system dependencies. It is built with ansible-builder and run with ansible-navigator or by Red Hat's automation platform. The appeal is that the exact toolchain travels as one image, so a laptop, a CI runner and a controller all behave alike. That is Docker applied to the control node.

version: 3
images:
  base_image:
    name: registry.fedoraproject.org/fedora:42
dependencies:
  python_interpreter:
    package_system: python3
  ansible_core:
    package_pip: ansible-core==2.21.4
  ansible_runner:
    package_pip: ansible-runner
  galaxy: requirements.yml
  system: bindep.txt

ansible-builder build --tag myorg/ansible-ee:1.0 --container-runtime docker
ansible-navigator run site.yml -i inventories/staging --eei myorg/ansible-ee:1.0 -m stdout

The definition schema is version: 3 with ansible-builder 3.x. ansible-builder uses Podman by default, and --container-runtime docker switches it. Do not forget the lag at the enterprise end: Red Hat's automation platform 2.7 defaults to ansible-core 2.16 in its supported EEs, with 2.18 and 2.20 also supported. If your company runs it, your playbooks may need to work on several core versions at once, so test against the EE the platform actually uses, not the newest core on your laptop.

Secrets and Vault in practice

Your automation holds the keys to your servers, so the secrets it handles deserve a real design. Ansible Vault encrypts files or individual values with AES256, and it is good enough for many teams. It is also easy to use in ways that leak, so this section covers layout, identity and failure modes.

Layout: encrypt values, not whole files

The most useful convention is the split-file pattern you saw earlier: a plain main.yml that references variables, and a vault.yml beside it that holds only the secrets, encrypted. The plain file names every variable and where its value comes from; the encrypted file stays small.

db_user: app
db_password: "{{ vault_db_password }}"
db_host: "{{ hostvars['db01']['ansible_host'] }}"

ansible-vault create inventories/production/group_vars/db/vault.yml
ansible-vault edit inventories/production/group_vars/db/vault.yml
ansible-vault rekey inventories/production/group_vars/db/vault.yml
ansible-vault encrypt_string 'hunter2' --name vault_db_password

Searching the repository for db_password finds the plain file, and only the secret itself is opaque. In code review, a diff on main.yml is readable, and a diff on vault.yml is, correctly, a blob. The encrypt_string form embeds a single encrypted value in a YAML file with a !vault | tag, useful for one-off values, but many of them scattered through files make rotation painful.

An encrypted file starts with a header: $ANSIBLE_VAULT;1.1;AES256 when it has no label, or $ANSIBLE_VAULT;1.2;AES256;prod when it carries the vault ID prod. You can recognise encrypted content at a glance, and you can grep -rl 'ANSIBLE_VAULT' . to check that nothing sensitive is sitting in cleartext beside it.

Vault IDs: separate passwords per environment

A single vault password for everything means everyone who can run staging can decrypt production. Vault IDs attach a label to each password so you can use several:

ansible-vault encrypt --vault-id prod@prompt inventories/production/group_vars/db/vault.yml
ansible-playbook site.yml --vault-id dev@dev.txt --vault-id prod@prompt

The format is label@source, where the source is prompt, a file path, or an executable. If a password file is executable, Ansible runs it and reads the password from its output. That is the standard hook for pulling the password from a secret manager at run time, so that no password lives on disk in CI. Configure defaults in ansible.cfg with vault_password_file or vault_identity_list. The setting vault_id_match (default false) makes Ansible try only the password whose label matches the file's, which is stricter but makes label mistakes into failures instead of silent fallbacks.

Lazy decryption changes when errors appear

Since 2.19, vaulted variables are decrypted lazily, at the point of use. A missing or wrong password therefore does not fail when the file loads; it fails when a task first touches the variable:

Attempt to use undecryptable variable: Attempting to decrypt but no vault secrets found.
Attempt to use undecryptable variable: Decryption failed (no vault secrets were found that could decrypt).

The first means no password was supplied at all, and the second means a password was supplied but none of them worked. A production playbook can therefore run half of its tasks before the first encrypted variable is reached. Add an early ansible.builtin.assert or a cheap debug that touches a known vaulted variable in pre_tasks, so a wrong password fails in the first second instead of in the middle of a deploy.

Keeping secrets out of logs

• no_log: true hides a task's arguments and results from output and callbacks. Put it on every task that handles a password, token or key. Note that it also hides the failure detail, which makes debugging those tasks harder, so temporarily disable it only in a safe environment.
• no_log on a loop: use loop_control: label: to print a safe label, and no_log: true to hide the values.
• Verbose output and --diff can reveal secrets that no_log would have hidden, and callback plugins that ship results elsewhere see whatever the task returns. Review your callbacks.
• The global DEFAULT_NO_LOG switch defaults to false. Do not rely on it as a substitute for marking tasks.
• ansible_become_password and similar connection secrets belong in vault or an external store, not in -e on a command line that shows up in the process list and shell history.

Since core 2.19, the SSH connection plugin accepts a private_key option, holding the key contents, and private_key_passphrase. That lets a vaulted variable supply the key without a file on the controller's disk. Teams that run from ephemeral CI runners find this useful.

When Vault is not enough

Vault solves "secrets in git". It does not solve rotation, per-request audit, or short-lived credentials, and every person who can decrypt the repository has the keys. When you need those properties, fetch secrets at run time with lookup plugins from an external store: community.hashi_vault.hashi_vault, amazon.aws.secretsmanager_secret, azure.azcollection.azure_keyvault_secret, google.cloud.gcp_secret_manager and others. The price is another dependency and credentials to reach it, so you move the trust problem rather than deleting it. A common shape is Vault for the one bootstrap credential and an external manager for everything else.

Rolling updates and orchestration

The default behaviour of a play, running every task on every host in the batch, is fine for configuration and dangerous for deployments. If an update is bad, you want it to be bad on one host, notice, and stop. Ansible gives you the controls, but none of them is on by default.

serial: batch size

serial splits a play's hosts into batches, and runs the whole play on one batch before starting the next. Give it a number, a percentage, or a list that grows:

- name: Rolling update of the web tier
  hosts: web
  serial:
    - 1
    - "25%"
    - "100%"
  max_fail_percentage: 10
  tasks:
    - name: Take host out of the load balancer
      ansible.builtin.include_tasks: lb_disable.yml
    - name: Update the application
      ansible.builtin.include_role:
        name: app
    - name: Put host back into the load balancer
      ansible.builtin.include_tasks: lb_enable.yml

That plan updates one host first as a canary, then a quarter of the fleet, then everything else. max_fail_percentage aborts the play if too large a share of a batch fails. Be careful of the arithmetic: with serial: 1, one failure is 100% of that batch, so any max_fail_percentage under 100 aborts on a single failure, which is often what you want for a canary. With larger batches, a percentage above zero can let a play continue past failures. For a strict "stop on the first failure anywhere" policy, any_errors_fatal: true fails the whole play on every host when one host fails.

Two more details. serial interacts with run_once: a run_once task in a serial play runs once per batch, not once per play, which surprises people who use it for a notification. And ansible_play_batch lists the hosts in the current batch, while ansible_play_hosts lists all remaining hosts in the play.

delegate_to, run_once, throttle

• delegate_to: host runs a task on a different machine while still using the current inventory host's variables. The load-balancer calls above are typically delegated to the balancer or to localhost (local_action is the old shorthand). Note that facts gathered in a delegated task attach to the delegated host only if you set delegate_facts: true; otherwise they attach to the original host.
• run_once: true runs a task on one host of the current batch and applies the registered result to all the hosts in the batch. Good for a database migration that must run once per deployment.
• throttle: N limits how many hosts run a task concurrently, regardless of forks, which is the right control for a task that calls a rate-limited API or a shared service.
• order (inventory, reverse_inventory, sorted, reverse_sorted, shuffle) sets the host order for the play, for example shuffle to avoid always updating the same host first.

- name: Run the migration once, from one host, carefully
  ansible.builtin.command: /opt/app/bin/migrate --apply
  run_once: true
  delegate_to: "{{ groups['db'][0] }}"
  throttle: 1

Strategies

The strategy decides how tasks are scheduled across hosts. linear (the default) runs each task on all hosts in the batch before moving to the next, so the play moves at the speed of the slowest host per task. free lets each host run through the play independently, which is faster when tasks are independent and hosts differ in speed, but makes the output harder to read and breaks any assumption that all hosts finished task N before task N+1. host_pinned is like free but keeps a worker on a host until it finishes. debug drops into an interactive debugger when a task fails.

Use free for independent bulk work such as package updates. Do not use it when tasks coordinate across hosts, such as a cluster that needs all members upgraded at each step.

Long-running tasks: async

A task that takes longer than the SSH timeout, or that you want to run in parallel with others, can be started asynchronously and polled later:

- name: Start a long backup
  ansible.builtin.command: /opt/app/bin/backup --full
  async: 3600
  poll: 0
  register: backup_job

- name: Do other work here
  ansible.builtin.debug:
    msg: "Backup running in the background"

- name: Wait for the backup
  ansible.builtin.async_status:
    jid: "{{ backup_job.ansible_job_id }}"
  register: backup_result
  until: backup_result is finished
  retries: 120
  delay: 30

Use the test is finished rather than comparing finished == 1: since 2.19 async_status returns started and finished as integers that are no longer usable as booleans in conditionals. poll: 0 means "fire and forget", and the job keeps running if the connection drops, which is the point. But nothing kills a forgotten job, so always follow with an async_status or an explicit cleanup.

Performance: measure first, then tune

Ansible runs are often slow for boring reasons. Tuning blindly wastes time; every knob below has a measurable signal.

Measure. Enable the profiling callback from the ansible.posix collection (callbacks_enabled = ansible.posix.profile_tasks, ansible.posix.timer) to get a per-task timing table and a total. The top five lines of that table are nearly always where the time goes. Use -vvvv and ANSIBLE_LOG_PATH for ssh-level detail.

Then apply the levers in this order, re-measuring each:

1. Skip or narrow fact gathering. gather_facts: false where unused, or gather_subset where partly used. On a large inventory, full fact collection can dominate the run. With gathering = smart plus a fact cache (fact_caching = jsonfile and fact_caching_connection, or Redis), facts gathered in one run are reused in the next until fact_caching_timeout expires (86,400 seconds by default).
2. Enable SSH pipelining. pipelining = True in [ssh_connection] (or [connection]) sends the module to the target's interpreter over the existing SSH channel instead of copying it to a temp file first. It removes several round trips per task. It defaults to false because it requires that requiretty is not set in sudoers for the become user.
3. Keep SSH connections alive. The default ssh_args already include -C -o ControlMaster=auto -o ControlPersist=60s, so each host reuses one connection across tasks. If you overwrite ssh_args in your config, you will silently lose that. Prefer ssh_extra_args, or copy the defaults when you override.
4. Raise forks. The default is five hosts in parallel. Raising forks to 25, 50 or 100 is common, limited by the controller's CPU and memory because each fork is an operating-system process. Watch memory before you raise it again.
5. Use strategy: free where tasks are independent, and async where a task is long.
6. Cache the inventory for dynamic plugins (cache: true in the plugin file), as discussed earlier.
7. Reduce the work itself. Replace a loop with a module call that accepts a list (package with name: "{{ packages }}" installs all in one transaction instead of N). Replace command with stat and conditions. Use --limit and tags to run only what you need.

[defaults]
forks = 25
gathering = smart
fact_caching = jsonfile
fact_caching_connection = .ansible_facts_cache
fact_caching_timeout = 3600
callbacks_enabled = ansible.posix.profile_tasks, ansible.posix.timer
interpreter_python = auto_silent

[ssh_connection]
pipelining = True

Some teams use a third-party strategy plugin called Mitogen for further speed, but it is unofficial, not supported by Red Hat, and historically lags new ansible-core releases, so check compatibility with your core version before depending on it.

Linting and testing

Ansible code is code, so it gets linted, tested and reviewed. At this level you do not need an elaborate framework, only three layers that each catch a different kind of bug.

Layer one: static checks (seconds)

ansible-playbook site.yml --syntax-check
ansible-lint
ansible-lint --profile production

--syntax-check catches YAML and structural errors but does not resolve variables. ansible-lint goes much further: it flags short module names instead of FQCNs, missing name: on tasks, command where a module exists, risky file permissions, handler naming problems, tasks that are never idempotent, deprecated syntax, and more. Lint has profiles, ordered from least to most strict: min, basic, moderate, safety, shared and production. Start at the level your repository can pass today, commit a config that pins it, and raise it over time. A lint run that nobody can pass gets ignored.

ansible-lint follows a CalVer version (26.9.0 at the time of writing). Because rules change between releases, pin its version in your CI image or pre-commit config, and upgrade it deliberately, or your pipeline will fail on a Monday for reasons unrelated to your commit.

Layer two: behaviour tests (minutes)

Molecule runs a role against a real throwaway target and checks the outcome. The scenario typically creates an instance (a container or VM, depending on the driver), converges the role, runs it a second time to check idempotence (the second run must report zero changes), and runs a verification step. The command is short:

molecule init scenario
molecule test
molecule converge     # just apply, leave the instance for poking at
molecule login        # shell into it

Molecule's real value is the automated idempotence check. A role that passes molecule test has proved it converges and converges to the same state. Use a container driver for quick feedback, with the caveat that containers have no real init system or kernel features such as firewall rules, so anything touching systemd, sysctl or the firewall needs a VM target or a systemd-enabled container image. Driver packaging and scenario file names have changed across Molecule releases, so follow the Molecule version you install rather than an old tutorial. For verification, asserting with ansible.builtin.assert, ansible.builtin.stat, ansible.builtin.uri and service_facts inside a verify playbook is enough for most roles, and keeps tests in the same language as the code.

Layer three: staged runs (your real environment)

Nothing replaces running the real playbook against a staging copy of the real inventory. The sequence most teams converge on is: lint, syntax check, Molecule for changed roles, --check --diff on staging, real run on staging, then --check --diff on production for review and the real run there, limited by serial.

For collections you write, ansible-test (shipped with ansible-core) provides sanity, units and integration runs, and must be run from inside the collection's ansible_collections/<namespace>/<name> tree.

Ansible in the delivery pipeline

A playbook run by one person from a laptop is a script. A playbook run by a pipeline, from a reviewed commit, against a known inventory, with a recorded result, is delivery. The mechanics differ less than the discipline.

What a pipeline job needs

The job needs the toolchain (a pinned ansible-core and collections, ideally an execution environment image), the repository, credentials for the hosts and for vault, and an inventory selected by the pipeline and not by the human. Everything else is a thin wrapper. A minimal GitHub Actions workflow shows the shape:

name: ansible
on:
  pull_request:
  push:
    branches: [main]

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.13"
      - run: pip install -r requirements-dev.txt
      - run: ansible-galaxy collection install -r requirements.yml -p ./collections
      - run: ansible-lint
      - run: ansible-playbook site.yml -i inventories/staging --syntax-check

  plan-staging:
    needs: lint
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.13"
      - run: pip install -r requirements-dev.txt
      - run: ansible-galaxy collection install -r requirements.yml -p ./collections
      - name: Check mode against staging
        env:
          ANSIBLE_VAULT_PASSWORD_FILE: ./scripts/vault-pass.sh
          VAULT_PASSWORD: ${{ secrets.VAULT_PASSWORD }}
        run: ansible-playbook site.yml -i inventories/staging --check --diff

The workflow is a sketch. The action versions and the vault wiring are yours to adapt, and scripts/vault-pass.sh is the executable password script pattern from the vault section, printing $VAULT_PASSWORD. The lesson is the shape: lint on every push, a check-mode run on the pull request so reviewers see the diff, and the real run only from main, in a protected environment that needs approval for production.

If CI reports ERROR: Ansible requires blocking IO on stdin/stdout/stderr. Non-blocking file handles detected, the runner has put its pipes in non-blocking mode. Pipe the command through | cat, redirect stdin from /dev/null, or run under a TTY. And a locale error, Ansible could not initialize the preferred locale, is fixed with export LC_ALL=C.UTF-8.

Hand-offs from and to Terraform

Terraform creates infrastructure and Ansible configures it. The interface between them is the only fragile part, and there are three workable ways to cross it.

1. Tag-driven dynamic inventory. Terraform tags instances with role and env; Ansible's cloud inventory plugin groups by tag. There is no file to hand over, and no drift between what exists and what Ansible sees.
2. Terraform outputs as inventory. The cloud.terraform collection can read Terraform state, through an inventory plugin and a terraform_output lookup, so the addresses Terraform created feed straight into a play.
3. A generated inventory file. Terraform writes a file that the pipeline passes to -i. Simple, and fine for small setups, but the file is one more artifact to keep in step.

Whichever you pick, decide who owns what. A common rule: Terraform owns everything that exists in the cloud API (networks, instances, disks, IAM), Ansible owns what is inside the instance (packages, files, services). Crossing that line, for example Ansible creating cloud resources in one place and Terraform elsewhere, produces two sources of truth and state fights.

Ansible and Kubernetes

For a cluster, prefer declarative tooling (Helm, Kustomize, Argo CD or Flux) over imperative Ansible runs. Ansible still belongs to the parts around Kubernetes: preparing the nodes the cluster runs on, installing container runtimes, tuning the operating system, rotating host credentials, and bootstrapping the first manifests. The kubernetes.core collection provides k8s, k8s_info and helm modules and a kubectl connection plugin when you do need to reach into a cluster from a play. If you find yourself writing hundreds of lines of Ansible to template manifests, step back: that is what the declarative tools are for.

Ansible and containers

Ansible is a reasonable way to configure VMs that run Docker hosts, and community.docker and containers.podman collections have modules for containers, images and networks. Containers themselves are better described by their Dockerfile or Compose file than by a play that mutates a running one. A sensible use of containers with Ansible is as test targets for roles, through Molecule.

Observability of runs

Keep the output. The junit callback gives CI a test-result format, log_path writes a file, and third-party tools such as ARA record runs in a database. Set callback_result_format = yaml in the default callback when you want YAML-formatted results; the older external YAML stdout callback from community.general is deprecated in favour of this built-in option. In regulated environments in the Gulf region or Egypt where change records must name what ran and when, the combination of a pipeline log, a git commit and a stored run artifact is often enough to satisfy an auditor, and is much better than a shell history.

Debugging beyond -v

Beginner taught -v and debug. When those run out, reach for these, roughly in the order they pay off.

See what the playbook will do before it runs. --list-hosts, --list-tasks and --list-tags print the plan. --start-at-task "Name" jumps to a task and --step asks before each one. --syntax-check and lint catch the cheap errors.

Read the merged state. ansible-inventory --host X for inventory variables. ansible-config dump --only-changed for which settings differ from defaults and where each came from. ansible-config validate (with -t all for plugins) checks your config file. ansible --version shows which config file is being read. Nothing is more common than editing a file Ansible is not reading: the search order takes the first file found and never merges them, in the order ANSIBLE_CONFIG, ./ansible.cfg, ~/.ansible.cfg, /etc/ansible/ansible.cfg. And ./ansible.cfg is ignored if its directory is world-writable, which is a security measure that can look like a bug in a shared folder or a CI workspace mounted from a container.

Use the debugger. strategy: debug, or the debugger: keyword on a task (on_failed, on_unreachable, on_skipped, always, never), drops you into an interactive prompt on failure where you can print task_vars, change an argument and retry with redo. It saves rerunning a twenty-minute play for one typo.

- name: Something that might fail
  ansible.builtin.command: /opt/app/bin/start
  debugger: on_failed

Inspect the module. ANSIBLE_KEEP_REMOTE_FILES=1 keeps the payload on the target. ANSIBLE_DEBUG=1 prints internal engine messages, which is noisy and helps with plugin issues. display_traceback shows Python tracebacks when you need them.

Read errors bottom-up. With the 2.19+ format, the lines after <<< caused by >>> are closest to the root cause, and the Origin: line names the file, line and column. Here is a set of messages you will actually meet, with their usual causes:

A note on how the SSH password mechanism changed in 2.19: the option password_mechanism now defaults to ssh_askpass, so sshpass is no longer needed for password authentication. To go back to the old behaviour, set ansible_ssh_password_mechanism: sshpass. Older Windows and SSH pages still say sshpass is required; they are out of date on this point.

Upgrades and staying current

Ansible moves twice a year, and each release removes things that were deprecated a year earlier. The teams who suffer are those who upgrade only when forced. A steady process costs an hour each cycle.

Know the support window. Each ansible-core release is maintained for about eighteen months. Core 2.21 reached general availability in May 2026 and receives fixes until November 2027. Core 2.20 gets security-only fixes from November 2026 and reaches end of life in May 2027. Core 2.19 reaches end of life in November 2026. The community package follows: a new major about every six months, with only the latest major maintained. Red Hat's automation platform is slower, so if you work with it, your playbooks may have to run on an older core than you use locally.

Read the porting guides for every version you skip. The 2.19 guide is the big one because of templating. Skim the core and package porting guides, then search your repository for the specific removals that hurt most:

The paramiko connection plugin was removed in 2.21. Setting connection: smart or persistent no longer falls back to it. If you manage network devices through network_cli, that collection has its own SSH implementation and is not affected in the same way. A PSRP default changed: negotiate_service is now host instead of WSMAN, which can affect Windows hosts using Kerberos.

Make deprecation warnings a worklist. Leave DEPRECATION_WARNINGS on, collect the warnings from one full staging run, and fix them before the next upgrade, rather than after. The config settings marked for removal in 2.23 include DEFAULT_JINJA2_NATIVE, DEFAULT_ALLOW_UNSAFE_LOOKUPS, DEFAULT_JINJA2_EXTENSIONS, DEFAULT_MANAGED_STR, DEFAULT_NULL_REPRESENTATION, DEFAULT_UNDEFINED_VAR_BEHAVIOR and STRING_TYPE_FILTERS; if your ansible.cfg sets any of those, plan now.

Upgrade in a fixed order: read the guides, bump the pinned core in a branch, run lint and syntax check, run Molecule on roles, run --check --diff against staging, run for real on staging, then production. If you have an execution environment, bump the EE and promote the image through your registry the same way you promote application images.

Putting it all together

To tie the section topics into one piece, here is the skeleton of a small but production-shaped repository for a web tier. It is not big, and each part matches a section above.

infra-ansible/
  ansible.cfg
  requirements.yml
  requirements-dev.txt
  .ansible-lint
  site.yml
  inventories/
    staging/
      hosts.yml
      group_vars/
        all.yml
        web/
          main.yml
          vault.yml
    production/
      hosts.yml
      group_vars/ ...
  roles/
    nginx/
      defaults/main.yml
      meta/argument_specs.yml
      tasks/main.yml
      handlers/main.yml
      templates/nginx.conf.j2
      molecule/default/ ...
    app/ ...
  .github/workflows/ansible.yml

- name: Configure and update the web tier
  hosts: web
  become: true
  gather_facts: true
  gather_subset: ["!all", "!min", "network", "distribution"]
  serial:
    - 1
    - "25%"
    - "100%"
  max_fail_percentage: 10
  force_handlers: true
  pre_tasks:
    - name: Fail fast if the vault password is wrong
      ansible.builtin.assert:
        that: vault_db_password | length > 0
        quiet: true
      no_log: true
  roles:
    - role: nginx
      nginx_upstreams:
        - { host: 127.0.0.1, port: 8080 }
  tasks:
    - name: Deploy the application release
      block:
        - ansible.builtin.include_role:
            name: app
        - name: Verify health
          ansible.builtin.uri:
            url: http://localhost:8080/health
          register: health
          until: health.status == 200
          retries: 10
          delay: 3
          when: not ansible_check_mode
      rescue:
        - name: Report the failure with context
          ansible.builtin.fail:
            msg: "Deploy failed on {{ inventory_hostname }} at task {{ ansible_failed_task.name }}"

Read what this one file encodes. Inventory directories bound the blast radius. The facts are limited to what the play uses. A serial list runs a canary and then widens, with max_fail_percentage and force_handlers covering failure. A pre-task touches a vaulted variable so that a wrong password fails in the first second. The nginx role validates its inputs through argument_specs. The deploy is a block with a health gate and a rescue that reports. Check mode skips the health wait. Everything is called by FQCN, pinned in requirements.yml, and gated by lint and Molecule in CI.

And the day-to-day flow, as steps:

1. Branch and changeEdit a role or variable on a branch. Run ansible-lint and --syntax-check locally.
2. Test the roleRun molecule test for the roles you touched, so idempotence is proved automatically.
3. Open a pull requestCI runs lint and a --check --diff against staging. The reviewer reads the diff as well as the code.
4. Apply to stagingOn merge, the pipeline runs the real play against staging and then a second time to confirm changed=0.
5. Promote to productionA protected step requires approval, runs --check --diff once more for the record, and then applies with serial and health gates.
6. Keep the recordThe commit, the log and the diff are the change record. Deprecation warnings go onto the upgrade worklist.

Where you are now

You can now reason about Ansible rather than just invoke it. You know what a task does on the wire, so slow runs and strange failures have a place to start. You can read the variable ladder, avoid the dictionary-replacement trap, use facts safely in a world where the injected names are going away, and write conditionals that survive the 2.19 templating changes. You treat idempotency as a property you test, not hope for, and you know where check mode lies. You can build roles with validated interfaces, pin the toolchain, and ship it as an execution environment. You handle secrets with vault IDs and executable password scripts and know when to move to an external store. You can roll a change out in batches, stop it when it goes wrong, and recover with a rescue instead of a page.

What comes next is the senior level: running Ansible as a platform for other teams. That means the architecture of the automation platform (controller, execution environments, automation mesh and private hub), scaling limits, the trust model behind who may run what against which hosts, multi-team governance, extending Ansible with your own modules and plugins, event-driven automation, and knowing where Ansible stops and other tools should take over. Read the neighbouring guides as well: Terraform for what creates the hosts, Kubernetes and Helm for what replaces imperative runs on a cluster, and GitHub Actions for the pipeline around all of it.

Sources

• Ansible documentation home
• Releases and maintenance
• Porting guide for ansible-core 2.21
• Porting guide for ansible-core 2.20
• Porting guide for ansible-core 2.19
• Porting guide for Ansible 14
• ansible-core 2.21 changelog
• Installing Ansible
• Ansible configuration settings
• Interpreter discovery
• Using variables and precedence
• Building an execution environment
• ansible.builtin collection index
• Red Hat Ansible Automation Platform life cycle
• Register projections and _task (Ansible forum)

Part two of three. At mid level an interviewer stops asking what a playbook is and starts asking what you would do when one misbehaves. Expect scenarios ("a run that used to take five minutes now takes forty"), trade-offs ("Vault or an external secret store?") and follow-ups that test whether you understand the mechanism or have only memorised commands. This file is organised the way those conversations go: a fast refresh of the model, then scenarios, then trade-offs, then what separates a strong answer from a weak one. It targets ansible-core 2.21 (Ansible community package 14).

Where this picks up

The model in one screen

Ansible is an agentless automation engine. The controller renders each task's arguments, packages the module, copies it to the target over SSH (or WinRM or PSRP for Windows), runs it there, reads the JSON it prints, and deletes it. Modules describe desired state, so a well-written one reports ok when nothing needed to change. That property, idempotency, is what makes reruns, check mode and scheduled runs safe.

Have these facts ready, because they get used as tie-breakers in follow-up questions:

• The control node needs Python 3.12 to 3.14 for ansible-core 2.21, and targets need Python 3.9 to 3.14. Native Windows is not a supported control node; use WSL, a VM or a container.
• The default connection is ssh, using the OpenSSH binary. The paramiko connection plugin was removed in 2.21.
• Default forks is 5, default strategy is linear, and fact gathering is implicit.
• ansible.cfg is found in a fixed order (ANSIBLE_CONFIG, ./ansible.cfg, ~/.ansible.cfg, /etc/ansible/ansible.cfg), the first found wins, and they are not merged.
• Call modules by fully qualified name, such as ansible.builtin.copy.

Commands to have ready

ansible-inventory -i inventories/staging --graph --vars       # what the inventory says
ansible-inventory -i inventories/staging --host web01         # merged inventory vars for one host
ansible-config dump --only-changed                            # non-default settings and their origin
ansible-playbook site.yml --syntax-check
ansible-playbook site.yml --check --diff --limit web01
ansible-playbook site.yml --list-hosts --list-tasks --list-tags
ansible-playbook site.yml --start-at-task "Deploy config" --step
ANSIBLE_KEEP_REMOTE_FILES=1 ansible-playbook site.yml -vvv
ansible-galaxy collection install -r requirements.yml -p ./collections
ansible-lint --profile production
molecule test

If you can say what each of these tells you that the others do not, you are ready for the scenarios.

Scenario: the run is slow

"A playbook that took five minutes now takes forty. What do you check first?"

A strong answer starts with measurement, not tuning. Turn on the ansible.posix.profile_tasks and timer callbacks and read the per-task table, because the slowest three tasks usually account for most of the total. Then look at which category the slow tasks fall into.

• Fact gathering. Did an inventory grow, or did someone add gather_facts to many plays? Fact collection per host per play is slow on large inventories. Fix: gather_facts: false where unused, gather_subset where partly used, or gathering = smart with a fact cache.
• Round trips. Is pipelining off? Without it each task with become makes several SSH operations. Enabling pipelining = True needs requiretty off in sudoers.
• Connections. Did someone override ssh_args and drop ControlMaster=auto and ControlPersist=60s, losing connection reuse?
• Parallelism. forks is 5 by default. With 200 hosts, that is 40 sequential batches for every task. Raise it, bounded by controller CPU and memory.
• Dynamic inventory. Is the plugin calling a cloud API on every invocation, without cache: true?
• A single slow host. Under linear, each task waits for the slowest host in the batch. One host with a failing disk can make every task take minutes.
• A loop of single-item calls. package accepts a list; a loop of forty installs is forty transactions.

The follow-up is usually "what changes between before and after?", so mention checking git history on ansible.cfg, the inventory size, and the collection versions. A weak answer says "raise forks" without measuring.

Scenario: the variable has the wrong value

"A service is deployed with port: 8080 but the group variable says 9090. What do you do?"

Do not guess. The value comes from somewhere on a ladder of twenty-two levels, and the job is to find which. Start by asking the inventory what it believes (ansible-inventory --host), because that covers group_vars and host_vars. If it says 9090, the override is elsewhere: play vars, vars_files, role vars/ (which beats inventory, which is why tunables belong in defaults/), a set_fact or register earlier in the play, include_vars, role or include parameters, or -e on the command line, which beats everything. Search the repository for the variable name, then check the pipeline's command line for -e. A strong answer also mentions that dictionaries are replaced rather than merged by default, so a host-level app: dictionary can silently erase keys defined at group level.

A follow-up worth preparing: "how would you make that less likely?" Prefix role variables with the role name, keep tunables in defaults, forbid -e for anything but one-off overrides, and validate inputs with argument_specs.

Scenario: the service was not restarted

"We changed a config file with Ansible but the running service has the old config. The playbook had a handler."

The classic cause is that the play failed after the change task and before handlers ran. By default, handlers are skipped for a host that has failed, and because the file already matches on the next run, it reports ok and never notifies the handler. The state is stuck: file new, process old. Answers that show you know the trap: use force_handlers: true (or --force-handlers), meta: flush_handlers right after the change, or put the restart in a rescue or always block. Other reasons to mention briefly: the notifying task was ok and not changed; the handler name did not match the notify string exactly; or the handler is defined in a role that the play did not include in that order.

Scenario: "it passed in check mode, but production broke"

"Why can --check pass and the real run fail?"

Because check mode is a prediction made by modules that support it, and several things make predictions wrong. command and shell tasks are skipped unless marked check_mode: false, so later tasks depending on their results see nothing. A task that depends on an earlier real change, like a service configuration after a package install, cannot succeed because the install did not happen. Modules without check-mode support are skipped. And check mode cannot detect runtime problems such as a full disk, a port in use, or a failing health check. A strong answer adds how you compensate: a real run against staging, the ansible_check_mode variable to skip pointless waits, check_mode: false on read-only queries, and canary batches with health gates in production.

Scenario: a rolling update took the whole tier down

"We updated twenty web servers and all of them were down at once. What went wrong and how do you prevent it?"

The play had no serial, so every host in the batch (up to the fork count, five by default, but often everything with a larger forks) restarted in the same task. The fix is structural. Set serial to a list such as [1, "25%", "100%"] so the first batch is a canary. Take hosts out of the load balancer before updating them and put them back afterwards, typically with delegate_to. Gate each batch with a health check using until and retries. Bound the damage with max_fail_percentage or any_errors_fatal. Mention the interaction details: with serial: 1, a single failure is 100% of the batch, and run_once under serial runs once per batch.

Strong answers also ask about the things outside Ansible: the load balancer's own health checks, connection draining time, and whether the application supports two versions running side by side during the roll.

Scenario: upgrading to a new ansible-core

"We are moving from 2.18 to 2.21. What breaks?"

Split the answer by version. Moving through 2.19 is the big one: conditionals must be boolean, so when: my_string now errors; {{ }} inside when and other expression fields is an error; templating is always native; only trusted sources are re-templated; play_hosts is deprecated; and the SSH password mechanism changed from sshpass to ssh_askpass. 2.20 raises the controller to Python 3.12, targets to Python 3.9, drops the smart transport value and Galaxy v2 support, and deprecates INJECT_FACTS_AS_VARS (it defaults to false in 2.24, so use ansible_facts['os_family']). 2.21 removes the paramiko connection plugin and the auto_legacy interpreter discovery modes, deprecates apt_key and apt_repository, and changes the PSRP default. The method matters more than the list: read the porting guides, run lint and syntax check, run staging with deprecation warnings on, use ALLOW_BROKEN_CONDITIONALS only temporarily, and pin versions so the upgrade is one deliberate commit.

Scenario: secrets in the repository

"Someone committed a database password in group_vars. What do you do?"

Two actions, in order. First rotate the password, because it is in git history and in every clone; removing it from the current tree does not make it secret again. Second, fix the process so it cannot recur: move the value into an encrypted vault.yml next to a plain main.yml that references it, add a pre-commit or CI check that flags unencrypted secret-looking values, and use separate vault IDs per environment. Mention no_log: true on the tasks that handle it and the --diff leak. A senior-flavoured follow-up you can pre-empt: Vault gives you encryption at rest in git, not rotation, audit, or short-lived credentials, so for those you fetch from an external secret manager with a lookup plugin at run time.

Scenario: Terraform creates the servers. Now what?

"How do Terraform and Ansible fit together?"

Terraform provisions what exists in the cloud API; Ansible configures what is inside the machine. The hand-off options are tag-driven dynamic inventory (Terraform tags, the cloud inventory plugin groups by tag), reading Terraform outputs or state with the cloud.terraform collection, or a generated inventory file. A strong answer names the ownership boundary and explains why crossing it, such as Ansible creating cloud resources that Terraform also manages, yields two sources of truth. It also notes the timing problem: a host reachable on SSH is not necessarily finished with cloud-init, so the first play should wait for connection with ansible.builtin.wait_for_connection or wait_for. Link the context: the Terraform guide covers the provisioning side.

Trade-offs

Ansible versus Terraform. Terraform holds a state file and is declarative about infrastructure objects; it is superb at create, change and destroy with a plan. Ansible is procedural with declarative modules, holds no state of its own, and is superb at configuring a machine and orchestrating actions across machines. They overlap at the edges, and the choice is about which tool's model fits the job. "Terraform for what exists in the cloud, Ansible for what runs on it" is the usual line.

Ansible versus Kubernetes-native tooling. On a cluster, Helm, Kustomize, Argo CD and Flux reconcile desired state continuously and are the better fit for workloads. Ansible is better for node preparation, host configuration and one-off orchestration. Running imperative Ansible against a cluster you also reconcile with GitOps leads to fights over ownership.

Push versus pull. Normal Ansible pushes from a controller. ansible-pull inverts this: each host clones a repository and applies it to itself, often from cron or a timer. Push gives you control, ordering and central visibility; pull scales without a central controller and handles hosts that are not reachable from outside, at the cost of weaker coordination and harder reporting.

Vault versus an external secret store. Vault is self-contained, versioned with the code, and free; it has no rotation, no per-access audit, and everyone who can decrypt sees everything for that vault ID. A store such as HashiCorp Vault or a cloud secret manager gives rotation, audit and short-lived credentials but adds a dependency and a bootstrap credential. Mixed is common.

import_* versus include_*. Static import for predictability: tags and --list-tasks work, and keywords are copied to children. Dynamic include for flexibility: loops, runtime-computed names, and conditions applied once.

linear versus free strategy. linear is predictable and right whenever tasks coordinate across hosts. free is faster for independent work and harder to read. Do not use free for a cluster rolling upgrade.

Fact caching versus fresh facts. Caching saves minutes and makes when conditions operate on stale data. Short timeouts and explicit refresh for decisions that depend on current state is the usual compromise.

Execution environments versus a virtualenv. A virtual environment is simple and fast to start. An execution environment packages the control node's whole toolchain as a container image, so laptop, CI and platform behave the same, at the cost of an image build pipeline and a container runtime. The larger the team, the more EEs pay off, and they are required for Red Hat's automation platform.

Molecule versus staged runs. Molecule tests a role in isolation with an idempotence check and is cheap and fast; it cannot reproduce your real inventory. A staged run against a staging copy is realistic and slower. You want both, for different bugs.

Questions with model answers

1. How does Ansible execute a task on a host?The controller renders the arguments, selects the module or action plugin, packages a Python module with its dependencies and arguments, copies it over the connection, runs it with the target's Python, reads JSON back, and removes the temporary files. Raw and script modules skip the Python requirement.
2. What makes a task idempotent and how do you test it?Modules describe desired state and report ok if it already holds. Test by running twice and requiring changed=0 on the second run, which Molecule automates. Guard command with creates, removes or changed_when.
3. Explain variable precedence without listing all twenty-two.Role defaults are lowest on purpose. Inventory comes next, specific over general. Then play variables, role vars, block and task vars, then registered variables and set_fact, then role parameters, and extra vars last, which always win.
4. Defaults or vars in a role?Defaults for anything a caller may change, because they are overridable from inventory. Vars only for internal constants, since they beat inventory and surprise users.
5. When does a handler run?After a task that reported changed notified it, once per play at the end of the current section, in definition order. meta: flush_handlers runs them early. They do not run for a failed host unless force_handlers is set.
6. What is the difference between ignore_errors and ignore_unreachable?The first continues after a task failure; the second after an unreachable host. Since 2.19 a become timeout counts as unreachable, so ignore_errors will not cover it.
7. What is block/rescue/always?Try, except, finally. A successful rescue marks the host recovered, so add a fail at the end of it if you want the pipeline to go red after a rollback.
8. Why must conditionals be boolean now?Since 2.19 a non-boolean result is an error, because implicit truthiness caused silent bugs, for example a non-empty string "false" evaluating true. Use | length > 0, | bool, is truthy or explicit comparisons.
9. What changed about templating trust?Only strings from trusted sources such as playbooks, roles and vars files are templated. Strings from module results, facts or lookups are not re-templated, closing template-injection paths.
10. How do you keep a role's interface safe?Prefix variables with the role name, keep tunables in defaults, and validate with meta/argument_specs.yml so bad input fails on entry with a clear message.
11. How do you pin dependencies?A requirements.yml with version ranges for collections and roles, installed into a repository-local path, or a built execution environment. Pin ansible-core as well.
12. What are vault IDs for?Labelling passwords so different environments use different ones. The format is label@source, and an executable password file lets the password come from a secret manager.
13. A vaulted variable fails only halfway through the run. Why?Since 2.19 decryption is lazy, so an undecryptable variable errors when first used, not when the file loads. Touch it in a pre-task to fail fast.
14. Difference between serial, forks and throttle?Forks is how many hosts run in parallel in total. Serial splits the play into batches and runs the whole play per batch. Throttle caps concurrency for one task.
15. What does run_once do with serial?It runs once per batch, not once per play.
16. How do you run a long task without SSH timing out?async with poll: 0, then async_status with until: job is finished. Use the finished test, not integer comparison.
17. How would you debug an unreachable host?Run with -vvvv to see the SSH command. Check the user and key, host key verification, connectivity and the banner timeout. Then reproduce the exact ssh command by hand.
18. What is an execution environment?A container image with ansible-core, ansible-runner, collections and dependencies, built with ansible-builder and run with ansible-navigator or a controller, so every environment uses the same toolchain.
19. When is command acceptable?When no module exists, with creates or removes for idempotency, changed_when for accuracy, and failed_when for acceptable exit codes.
20. Why prefer ansible_facts['os_family'] over ansible_os_family?The injected top-level names are controlled by INJECT_FACTS_AS_VARS, which is deprecated and becomes false in 2.24. The dictionary form works now and later.

What a strong answer covers that a weak one misses

Three habits separate the answers more than any fact. Say what you would measure before you change anything. Name the failure mode of your own proposal, such as fact caches serving stale data. Tie the answer to a consequence, such as a wrong serial taking down a tier. Interviewers at this level are listening for the sound of someone who has been paged.

Final self-test

• Describe, from memory, what happens from the moment a task starts until its result is recorded.
• State where -e sits on the precedence ladder, where role defaults sit, and why.
• Explain why a failed play can leave a service running old config, and three ways to prevent it.
• Name three places check mode is blind and what you do about each.
• Write the serial and failure-budget settings for a canary-then-widen rollout.
• Say what changed in 2.19, 2.20 and 2.21 that a playbook written in 2024 might trip over.
• Explain lazy vault decryption and how you fail fast because of it.
• Give two reasons to choose an execution environment over a virtual environment, and one reason not to.
• Explain where Ansible stops and Terraform or Helm should take over.

Part two of three. At this level your playbooks run and do useful work, so the problems change character. They stop being "why will it not connect" and become "why does this report changed every night", "why did the restart never happen", and "why did the upgrade break a playbook that has not changed in a year". Start with the error table, then the practices ranked by leverage, then practice cards and recipes you can copy. Everything targets ansible-core 2.21 (Ansible community package 14).

Common errors at this level

Cumulative: Beginner's errors still apply, and these are the ones that appear once things mostly work.

The practices that pay off most

In order of leverage. If you can only adopt three, adopt the first three.

Why that order

Idempotency first, because everything else assumes it. Check mode, scheduled runs, reruns after partial failure and "0 changed means production matches the repository" all depend on tasks telling the truth.

Pinning second, because it makes failures attributable. When a run breaks, the first question is "what changed", and if nothing is pinned the answer is "anything, anywhere". With a pinned toolchain the answer is a git diff.

Rollout discipline third, because it converts a bad change from an outage into a blip. It costs a few lines in a play, and is the difference between one broken host and twenty.

The rest (lint, secrets layout, fact hygiene, argument specs, profiling) each remove a class of recurring problem, but none of them is as valuable as the first three.

Practice cards

1. Audit idempotencyRun your most-used playbook twice against a scratch host. List every task that reports changed on the second run and fix each with creates, changed_when, or a real module.
2. Fail a handler on purposeChange a config file, notify a restart handler, then fail the next task. Confirm the restart did not happen, then fix it with force_handlers: true and compare.
3. Climb the ladderDefine one variable in role defaults, group_vars/all, host_vars, play vars, and -e. Delete them from the top down and watch each lower level take over.
4. Lose a dictionary keyDefine a dictionary in two layers, observe the lost key, then fix it with combine.
5. Break a conditional, then mend itWrite when: some_string and read the 2.19-style error, then fix it three ways.
6. Migrate a fact nameRun with ANSIBLE_INJECT_FACT_VARS=false and fix every bare ansible_* fact that breaks.
7. Canary a changeOn four hosts, add serial: [1, "50%", "100%"] and a health gate. Break the health endpoint on one and confirm the rollout stops.
8. Fail fast on a vault typoAdd a pre-task that asserts a vaulted variable is non-empty. Run with a wrong password and see the failure arrive in the first second.
9. Profile one runEnable profile_tasks, find the slowest three tasks, and make one of them faster. Record before and after.
10. Add a role contractWrite argument_specs.yml for a role you own, pass a wrong type, and read the message.

Operational recipes

Copy these, then adapt. Each is small and does one job.

A sane ansible.cfg for a repository

[defaults]
inventory = inventories/staging
collections_path = ./collections
roles_path = ./roles
forks = 20
gathering = smart
fact_caching = jsonfile
fact_caching_connection = .ansible_facts_cache
fact_caching_timeout = 3600
callbacks_enabled = ansible.posix.profile_tasks, ansible.posix.timer
callback_result_format = yaml
interpreter_python = auto_silent
log_path = ./ansible.log

[ssh_connection]
pipelining = True

Defaulting the inventory to staging is deliberate: a forgotten -i should hit the safe environment, and production should need an explicit flag. Add ansible.log and .ansible_facts_cache to .gitignore. Check that the file is the one in use with ansible --version.

The pre-flight command

ansible-lint \
  && ansible-playbook site.yml -i inventories/production --syntax-check \
  && ansible-playbook site.yml -i inventories/production --list-hosts \
  && ansible-playbook site.yml -i inventories/production --check --diff --limit web01

Four quick gates, each cheap, in order of cost. --list-hosts shows what you are about to touch, which stops the most embarrassing mistakes. Limit check mode to one host to keep the diff readable.

Fail fast on a wrong vault password

- name: Assert the vault is readable
  ansible.builtin.assert:
    that: vault_db_password | length > 0
    quiet: true
  no_log: true
  run_once: true

Put it in pre_tasks. Because vaulted variables are decrypted lazily in 2.19 and later, this converts a mid-deploy failure into a first-second failure.

Read-only query that still runs in check mode

- name: Read current version
  ansible.builtin.command: /opt/app/bin/version
  register: current_version
  changed_when: false
  check_mode: false

Without check_mode: false, the task is skipped during --check, and any later when: current_version.stdout ... sees an empty result.

Wait for a freshly created host

- name: Wait for SSH on new hosts
  hosts: new_hosts
  gather_facts: false
  tasks:
    - name: Wait until the host accepts connections
      ansible.builtin.wait_for_connection:
        delay: 5
        timeout: 300
    - name: Now gather facts
      ansible.builtin.setup:

Useful straight after Terraform creates instances: the address exists before SSH does. Setting gather_facts: false on the play prevents the implicit fact gather from failing before the wait.

Bootstrap Python on a bare host

ansible new_host -i inventories/staging -m ansible.builtin.raw -a "dnf -y install python3"

The raw module needs no Python on the target, which is why it is the tool for this chicken-and-egg. On Debian-family hosts use the apt-get equivalent.

Show exactly which config and collections are active

ansible --version
ansible-config dump --only-changed
ansible-galaxy collection list

Three commands, and they answer most "works on my machine" reports. Paste their output into the ticket.

Rolling deploy skeleton

- name: Rolling deploy
  hosts: web
  serial: [1, "25%", "100%"]
  max_fail_percentage: 10
  force_handlers: true
  tasks:
    - name: Drain host from the balancer
      ansible.builtin.include_tasks: lb_disable.yml
    - name: Deploy
      ansible.builtin.include_role:
        name: app
    - name: Flush handlers before checking health
      ansible.builtin.meta: flush_handlers
    - name: Wait for healthy
      ansible.builtin.uri:
        url: http://localhost:8080/health
      register: health
      until: health.status == 200
      retries: 20
      delay: 3
    - name: Return host to the balancer
      ansible.builtin.include_tasks: lb_enable.yml

The flush_handlers line is the one people forget: without it, the restart handler runs at the end of the play, after the host has already been put back into rotation with the old process.

A safe Molecule loop

molecule converge         # apply and keep the instance
molecule verify           # run the checks
molecule login            # poke around interactively
molecule test             # full: create, converge, idempotence, verify, destroy

Iterate with converge and verify while developing and finish with molecule test, which includes the second run that asserts nothing changes. Driver and scenario layout differ between Molecule releases, so follow the documentation for the version you installed.

Find unencrypted secrets before a reviewer does

grep -rEn '(password|secret|token|api_key)\s*:' inventories roles --include='*.yml' \
  | grep -v 'vault_' | grep -v 'lookup('
grep -rL 'ANSIBLE_VAULT' inventories/*/group_vars/*/vault.yml

The first lists lines that look like inline secrets, filtering out references to vaulted variables and lookups. The second lists any vault.yml that is not actually encrypted. Both are crude, so treat them as a tripwire, not a scanner.

Compare what two environments get

diff <(ansible-inventory -i inventories/staging --host web01 --yaml) \
     <(ansible-inventory -i inventories/production --host web01 --yaml)

Everything in the diff is an environment-specific assumption. If there is more in there than you can explain, the environments have drifted or a variable is sneaking in from somewhere unintended.

Search for deprecated patterns before an upgrade

grep -rnE 'ansible_(os_family|distribution|hostname|default_ipv4)\b' roles playbooks
grep -rnE '^\s*- include:' roles playbooks
grep -rnE 'play_hosts' roles playbooks
grep -rnE 'connection:\s*(paramiko|smart|persistent)' roles playbooks inventories
grep -rnE 'apt_key|apt_repository' roles playbooks

Each line maps to a removal or deprecation: injected fact names (off by default in 2.24), the removed bare include:, play_hosts, the removed paramiko and smart transports, and the deprecated apt modules (use deb822_repository).

Collections and requirements in practice

Most module problems at this level are dependency problems wearing a disguise. A few habits make them rare.

Keep one requirements.yml and use it everywhere. The same file installs collections on your laptop, in CI and in the execution environment build. Two files that drift are how "it works locally" starts. Use version ranges that your team has tested, and raise them deliberately in a commit of their own.

Install into the repository. ansible-galaxy collection install -r requirements.yml -p ./collections with collections_path = ./collections keeps dependencies with the code, so a fresh checkout and CI see the same thing. Add collections/ to .gitignore unless you must vendor them for an air-gapped environment, in which case ansible-galaxy collection download prepares an offline set.

Verify what you installed. ansible-galaxy collection list shows names, versions and paths, which is the fastest way to see that an old copy in ~/.ansible/collections is shadowing the one you pinned.

Use FQCNs everywhere. ansible.builtin.copy, not copy. The short form works through a search list and is ambiguous when a collection ships a module of the same name, and lint flags it.

Mind the core lag in enterprise. Red Hat's automation platform 2.7 supports ansible-core 2.16, 2.18 and 2.20 in its execution environments, with 2.16 the default. If your company runs it, test with the core version the platform will use, and avoid features that only exist in the newest core, such as register projections and _task from 2.21, in content that must run there.

Secrets habits

The mistakes here are expensive, so the habits are worth stating plainly.

• Rotate after any leak. A secret that has been in git history is compromised even after the file is fixed.
• One vault ID per environment. A single password for everything gives everyone who can run staging the ability to read production.
• Keep the password out of the repository. Use an executable password script that fetches it from your secret manager or CI variable, and .gitignore any local password file.
• no_log: true on every secret-handling task, and review --diff output before sharing.
• Encrypt values, not structure. Keep the list of variable names in a plain file so reviewers can see what exists.
• Prefer runtime lookups for things that rotate. A database password that changes weekly is a poor fit for a static encrypted file.

In regulated environments across the Gulf and Egypt, where an auditor may ask who could read a credential and when, the ability to say "only the pipeline, through this secret manager, and here is its access log" is a stronger story than "anyone with the vault password".

Testing habits

• Lint first, always. It is the fastest feedback and the cheapest to enforce. Pin ansible-lint's version so its rule changes do not break your pipeline unannounced.
• Test roles in isolation and playbooks in staging. Molecule for the former, a real staging inventory for the latter.
• Assert, do not just run. A verify step that checks the port is open, the file exists with the right mode, and the health endpoint answers, is what turns a converge into a test.
• Test the upgrade path. Run the suite against the next ansible-core in a branch before you need to move.
• Keep test targets honest. A container that lacks systemd cannot test a role that manages services. Choose a systemd-enabled image or a VM for those roles, and say in the role's README which driver it was tested with.

Habits worth adopting now

• Read the diff of a check run like a code review. Ask of each change whether you expected it.
• Use --limit on every production run, even if only to name the tier. It documents intent and shortens the blast radius.
• Name every task, in the imperative, and mention the object: "Install nginx", "Template nginx.conf". Names are what appear in logs and in --start-at-task.
• Prefer modules to commands, and look at ansible-doc before writing a shell line.
• Never disable host key checking in production. Put keys in known_hosts with the known_hosts module or ssh-keyscan, and keep host_key_checking on.
• Treat deprecation warnings as tickets, not noise. They are a dated list of things that will break.
• Keep playbooks small and roles focused. A role that does one thing can be tested in a container in seconds.
• Write down the order of operations for production in the playbook itself, using serial, health checks and comments, not in a wiki page that will drift.
• Keep a scratch environment. A few containers or VMs that you can destroy and recreate in minutes are the cheapest safety net you will ever own.
• Link your neighbours. Know what Terraform creates for you, what Kubernetes and Helm should own instead of you, and how GitHub Actions runs you.

A one-page checklist before you run against production

If the answer to any of those is "I do not know", you have found the next thing to fix, and it is cheaper to fix it now than during an incident.

This is part three of three. It closes the series by taking every topic from Beginner and Mid one level further, because each one now has a blast-radius, scale or ownership dimension, and it adds the work you own when Ansible is a platform that other teams depend on rather than a tool you run yourself.

Up to this point the questions were about making a playbook work. From here they change. Who can make this control node run commands as root on four thousand machines? What happens to a fleet when a single task is wrong and the play is not stopped after the first batch? Which component of the setup slows down first when the inventory doubles? How do you upgrade the engine under forty teams' playbooks without a flag day? This guide targets ansible-core 2.21 (the Ansible community package 14), and every command and setting in it was checked against that release.

Where this picks up

I am starting with the architecture, because every other decision in this guide follows from one fact: Ansible has no resident component on the machines it manages.

The architecture and what it fails on

You know from Beginner that Ansible is agentless. The consequence you may not have followed to its end is that the control node is the system. There is no daemon on a managed host to keep state, no central database recording what the fleet looks like, and no scheduler that notices a host has drifted. A run is a process on the control node. It reads inventory and variables, renders templates, packages each module with the AnsiballZ wrapper, copies that payload to the target over the connection plugin, runs it, reads the JSON that comes back, and deletes the payload. When the run ends, everything Ansible knew is gone except what you chose to keep: the git repository, a fact cache, a log file.

That shape gives Ansible its best properties and most of its failure modes at once.

The best property is that nothing needs installing on targets, so you can manage a machine the moment it accepts an SSH login. The failure modes come from the same absence. There is no state store, so "what did the last run change?" is answered only by whatever logging you built. There is no continuous enforcement: a configuration change made by hand at two in the morning stays there until the next run, and the next run may be next Thursday. There is no queue or scheduler, so two runs that touch the same hosts at the same moment simply collide, for example both restarting a service or both holding the package manager lock. And there is no built-in coordination of who may run what: that is whoever can reach the control node and read its credentials.

For the platform owner this sorts into three practical questions that the rest of the guide answers in turn.

1. Where does the control node live, and who can reach it? A laptop, a bastion, a CI runner and an automation controller are four very different answers, with very different audit trails.
2. Where does state live? Git for intent, a fact cache for facts, a run database or log pipeline for history. If you cannot say where each one is, you are relying on memory.
3. What happens when a run fails halfway? With no transaction and no rollback, the fleet is left partly converted. The design of your plays decides whether that is recoverable.

Ansible also has a pull-shaped variant that changes the picture. ansible-pull inverts the model: each host clones a git repository and runs a playbook against itself, usually from cron or a systemd timer.

ansible-pull -U https://git.example.com/infra.git -C main -i localhost, local.yml

The trade is real. Pull removes the single control node as a bottleneck and as a credential vault, because each host needs only read access to the repository, and it gives you continuous convergence. In exchange you lose central visibility (there is no single run to read), you lose coordinated rolling updates (hosts converge whenever their timer fires), and you need a way to know that a host has stopped pulling. Pull suits large fleets of similar machines, edge devices and autoscaled instances that must configure themselves at boot. Push suits changes that need ordering across hosts, such as a database failover or a rolling deploy behind a load balancer.

The one-line mental model of failure

Because each task is applied host by host without a transaction, the useful question for any play is: if this stops after task N, on host M, what state are the fleet and that host in? A play that is safe to interrupt at any point is written with idempotent tasks, with handlers that are harmless to run late, and with changes ordered so that a half-finished run leaves the system working. A play that is not safe to interrupt should say so in its name and run behind serial and health checks. Sections below build on this.

The trust model: who can run code as root

The security model of Ansible can be stated in one sentence: anyone who can change what the control node runs can run code as root on every host it manages. Everything else is detail.

That sentence makes the trust boundary wider than the control node itself. It includes the git repository that holds playbooks (a merged pull request is a code deployment to your fleet), the CI system or controller that checks it out, the inventory and group_vars files (a variable can change a shell command or a dest path), the collections and roles the playbooks import, and the execution environment image. Each is an input that a playbook executes with the privileges of the automation user. Protecting the fleet means protecting all of them, and the protection is usually code-review rules, not network rules.

The credentials are the second half. A control node holds the SSH keys, the become passwords and the cloud tokens that let it act. That makes it a concentrated target: compromising it is worth more than compromising any single managed host. Four design decisions reduce the damage.

A dedicated automation user. Use one account on the managed nodes, reachable by key-based SSH only, rather than a named administrator's login or a shared root password. It gives you one identity to audit and one to revoke.

Least-privilege sudoers. become: true with an unrestricted NOPASSWD rule is the easy default and the widest one. If a class of automation only ever restarts a service and edits one directory, it can be given a sudoers rule for exactly that. You will not achieve this for a general configuration management role that installs packages, but you can separate roles of automation: one user that can deploy an application and another, used rarely, that can change the operating system. Note that the become plugin in core is sudo by default, with su and runas also built in.

Host key checking left on. host_key_checking defaults to True and should stay True in production. Setting it to False in ansible.cfg "to stop the prompt" removes the protection against a host being impersonated when the connection is made, which matters more when the same key can log in to many machines. Populate known hosts deliberately instead, with the ansible.builtin.known_hosts module or ssh-keyscan at provisioning time.

Different credentials for different environments. The key that reaches production should not be the key that reaches staging, and a pipeline that deploys staging should not be able to read the production key. This sounds obvious and is routinely violated by one ~/.ssh/id_ed25519 and one shared vault password.

What the 2.19 templating change bought you

ansible-core 2.19 changed the trust model of templating, and that is a security improvement you should be able to explain. Before 2.19, a string that came back from a module, a lookup or a fact could be rendered as a template if it happened to contain Jinja delimiters. That made a class of template-injection bugs possible: a hostname, an HTTP response body or a cloud tag containing {{ ... }} could be evaluated by the engine on the control node. From 2.19, only strings from trusted sources are rendered: playbooks, roles, variable files and the built-in inventory plugins. Strings from module results, lookups, facts and other external data are not re-templated. Plugin authors can mark a string as trusted with ansible.template.trust_as_template().

The operational consequence is a rule for your platform: never call trust_as_template on external data in a plugin you write or review, because doing so reopens the hole the change closed. A second consequence is that playbooks that accidentally relied on re-templating of data they fetched will break with an error, and the right fix is to restructure the data flow, not to paper over it.

Credentials and secrets

Secrets are where the trust model meets daily work. The question is not "how do I encrypt the file" (you answered that at Mid level with Ansible Vault) but "how does a secret reach a task, who can read it on the way, and what leaks it afterwards".

Vault at team scale

Ansible Vault encrypts files or single values with AES256. At scale the unit that matters is the vault ID: a label attached to an encrypted item, so that different environments can use different passwords in the same run. The header tells you which is which, $ANSIBLE_VAULT;1.1;AES256 without a label and $ANSIBLE_VAULT;1.2;AES256;<label> with one.

ansible-vault encrypt_string --vault-id prod@prompt 's3cr3t' --name db_password
ansible-playbook site.yml --vault-id dev@dev.txt --vault-id prod@prompt

Use a label per environment, and set vault_id_match if you want a wrong-environment password to be refused instead of tried. By default vault_id_match is False, so Ansible tries every supplied password against an item regardless of its label, which is convenient and means a label is documentation rather than enforcement. Decide which you want and write it down.

A vault password file that is executable is run as a script, and its standard output is the password. That is the integration point with a real secret manager: the script fetches the vault password from your secret store at run time, using whatever identity the job has, so the password never sits on disk in the repository or on a laptop.

#!/usr/bin/env bash
# vault-pass-prod.sh: print the vault password for the prod environment
set -euo pipefail
vault kv get -field=password secret/ansible/prod-vault

Since 2.19, vaulted variables are decrypted lazily, at the point they are used. That has an operational edge: a missing or wrong password no longer fails when the file loads. It fails when a task first needs the value, which may be halfway through a play. Expect Attempt to use undecryptable variable: Attempting to decrypt but no vault secrets found. for no password and Decryption failed (no vault secrets were found that could decrypt) for a wrong one, and have a cheap first task in every play that touches a vaulted value so that the failure is early.

Vault is a file format, not a secrets platform

Ansible Vault encrypts data at rest in git. It does not rotate secrets, does not audit who read them, cannot revoke one person's access (everyone with the vault password has everything in that vault ID), and leaves the decrypted value in memory on the control node for the life of the run. For anything you must rotate or audit, prefer fetching at run time from a store that does those things. The lookup plugins exist for that: community.hashi_vault.hashi_vault or vault_kv2_get, community.general.onepassword, the AWS, Azure and Google secret manager lookups, and the credential plugins in the automation platform. The neighbouring guide on HashiCorp Vault covers the server side; from Ansible's side the pattern is always the same, a lookup that runs on the control node with the job's identity and returns a value that is then used and never written to a file.

- name: Create the application database user
  community.postgresql.postgresql_user:
    name: app
    password: "{{ lookup('community.hashi_vault.vault_kv2_get', 'apps/db', engine_mount_point='secret').secret.password }}"
  no_log: true

The decision table is short. Secrets that never change and are low-impact can live in Vault-encrypted files. Anything rotated, shared across teams, or covered by a compliance requirement should come from an external store at run time. Note that the ssh connection plugin gained private_key and private_key_passphrase options in 2.19, so a vaulted or externally fetched key can be used directly, with no key file written to disk on the control node.

What no_log does and does not do

no_log: true on a task hides that task's arguments and results from the console output and from callbacks. It is necessary and not sufficient. It does not protect the value if you register it and print it elsewhere, it does not stop a verbose run (-vvv) from being a place to inspect the connection, and it does not cover a callback plugin you install that logs differently. DEFAULT_NO_LOG is a global switch that defaults to False; turning it on globally makes debugging production failures painful, so most platforms apply no_log on the tasks that handle secrets, and review for it.

Supply chain: collections, roles and execution environments

A playbook that says ansible.builtin.copy runs code that ships with ansible-core. A playbook that says community.general.something runs code that somebody published, that your control node downloaded, and that executes with your credentials. Since the community package bundles around eighty-five collections and every real project adds more, the supply chain is most of the code you run.

Pin everything, and pin it in one place

The first control is reproducibility. A requirements.yml with ranges like >=11.0.0,<12.0.0 is a policy (accept patch and minor updates); an exact version is a lock. Decide deliberately which you use for each collection, and never run an unpinned ansible-galaxy collection install community.general in a production pipeline, because that gives you a different fleet-changing codebase on every run day.

collections:
  - name: community.general
    version: "==11.4.0"
  - name: ansible.posix
    version: ">=2.0.0,<3.0.0"
  - name: https://git.example.com/platform/ansible-collection-corp.git
    type: git
    version: v3.2.0
roles:
  - name: geerlingguy.docker
    version: "7.4.1"

The versions above are examples of the shape, not recommendations; look up current ones for your own pins. The git form is how a platform team ships its own collection without a private server. Since 2.21 ansible-galaxy collection install skips, by default, a collection whose requires_ansible is incompatible with your core, which turns a class of "works on my laptop" surprises into a visible skip. The related setting collections_on_ansible_version_mismatch defaults to warning.

Verify, mirror, and keep an offline path

ansible-galaxy collection verify compares an installed collection with the copy on the server, and Automation Hub signs collections so that verification can use GPG keyrings (--keyring, or the GALAXY_GPG_KEYRING setting). Verification only helps if a failure blocks the pipeline, so decide what a failure does before you need it.

For a platform, the real control is not fetching from the public internet at run time. Stand up one controlled source: a private automation hub, or an internal mirror or artefact store that you populate after review. ansible-galaxy collection download -r requirements.yml -p ./offline produces tarballs you can carry into an air-gapped network, which matters for regulated estates in the Gulf and in Egypt where production networks have no outbound access. The Galaxy server list lives in ansible.cfg under [galaxy] with one [galaxy_server.<name>] section per server, and a platform usually sets it on the execution environment so that teams do not configure it themselves. Only the v3 Galaxy API is supported from 2.20, because v2 was removed, which matters if you run an old internal server.

The execution environment is the lock file

An execution environment (EE) is an OCI image that contains ansible-core, ansible-runner, the collections, and the Python and system packages they need. It is the unit that makes "this playbook ran with exactly this code" true. Build it with ansible-builder, run it with ansible-navigator or an automation platform, and version it like any other artefact.

version: 3
images:
  base_image:
    name: registry.fedoraproject.org/fedora:42
dependencies:
  python_interpreter:
    package_system: python3
  ansible_core:
    package_pip: ansible-core==2.21.4
  ansible_runner:
    package_pip: ansible-runner
  system:
    - openssh-clients
  galaxy: requirements.yml

ansible-builder build --tag corp/ansible-ee:2026.10 --container-runtime docker
ansible-navigator run site.yml -i inventory --eei corp/ansible-ee:2026.10 -m stdout

Three habits make the EE a real control. Tag it with a version you can trace to a commit (and record the image digest in the run record), so a rerun of last month's deploy uses last month's code. Rebuild it on a schedule, because its base image and its Python dependencies age like any container; the Docker guide covers pinning, scanning and signing, and all of it applies here. And keep a small number of EEs: a platform that lets every team build a private one ends up patching sixty images.

Scaling: which component breaks first

Ansible does not scale like a service, because there is no service. It scales like a batch job fanned out from one machine, and the things that break first are usually in this order.

The control node's CPU and memory, through forks. Each fork is an operating system process. The default forks is 5, which is conservative enough that most teams raise it (to 25, 50 or 100), and then discover that the real limit is the control node's memory, because each fork is a full Python process that holds the playbook and variables. Raise forks in steps and watch memory, not just speed.

Round trips. Each task, by default, costs several SSH operations: create a temporary directory, copy the module, run it, remove it. Two settings cut this substantially. SSH multiplexing is on by default (ssh_args = -C -o ControlMaster=auto -o ControlPersist=60s), which reuses one connection per host. Pipelining (pipelining = True, environment variable ANSIBLE_PIPELINING, default False) sends the module over the existing connection's standard input instead of copying a file first, removing round trips per task. Pipelining requires that requiretty is not enabled in sudoers, which is why it is not the default.

[defaults]
forks = 50
gathering = smart
fact_caching = jsonfile
fact_caching_connection = ~/.ansible/fact_cache
fact_caching_timeout = 86400

[ssh_connection]
pipelining = True
ssh_args = -C -o ControlMaster=auto -o ControlPersist=120s

Fact gathering. The setup module runs on every host at the start of every play by default, and on a large fleet that is the largest single block of time in a short playbook. Options, from gentlest to strongest: limit what is collected with gather_subset (for example ['!all', '!min', 'network']), turn gathering off with gather_facts: false for plays that do not use facts, or use gathering = smart with a fact cache so that a host's facts are reused for the cache timeout. The cache plugin defaults to memory, which means nothing is reused between runs, so a persistent cache (jsonfile, or Redis when several controllers must share) is a deliberate setting. The price is staleness: a cached ansible_facts['distribution_version'] may be a week old after a patch run. Use a cache for facts that rarely change, and gather fresh for the ones you branch on after a change.

The slowest host. Under the default linear strategy, every task waits for every host in the current batch, so one slow or hung host holds up the play on each task. strategy: free lets each host run to the end independently, which improves throughput when hosts do not depend on each other, at the cost of output that interleaves and of no longer having a clean "everyone finished task 3" point. host_pinned is like free but keeps a fork on a host until that host finishes. Two defences against a hung host are DEFAULT_TIMEOUT (connection timeout, default 10 seconds) and task_timeout (default 0, meaning no limit): set a task timeout on platforms so that a hung command cannot hold a fork forever.

Inventory size and dynamic sources. With thousands of hosts, parsing inventory, merging variables and building hostvars has a cost before the first task runs. Dynamic inventory plugins query a cloud API, which is slow and rate limited, so enable the plugin's cache (cache: true, with a cache_plugin and cache_timeout) and pick a timeout that matches how fast your estate changes.

Long-running tasks. A task that runs for an hour holds a fork and a connection, and loses it if the network blips. Use async with poll: 0 to start it, and async_status with until: job is finished to collect it. Note that since 2.19 the started and finished values are integers and must not be used as booleans in conditionals; use the finished and started tests.

- name: Start a long database reindex
  ansible.builtin.command: /opt/db/reindex --all
  async: 3600
  poll: 0
  register: reindex

- name: Wait for it
  ansible.builtin.async_status:
    jid: "{{ reindex.ansible_job_id }}"
  register: job
  until: job is finished
  retries: 120
  delay: 30

Shared infrastructure behind the play. When a hundred hosts download a package at once, the thing that breaks is the package mirror, not Ansible. throttle: 5 on a task caps concurrency for that task regardless of forks, and is the right control for a rate-limited API, a licence server or a mirror.

When one controller is not enough

At some size a single control node is the limit regardless of tuning. The options are to split the estate by inventory, running independent controllers per region or per environment (which also limits blast radius and keeps data inside a region, a requirement that Gulf and Egyptian employers often have); to slice a job across workers, which the automation platform calls job slicing and which you can emulate in CI with --limit on inventory subsets; or to move to pull with ansible-pull so that each host runs its own play. Mitogen, a third-party strategy plugin, is often mentioned for speed, but it is not supported by Red Hat and tends to lag new core releases; treat it as an experiment rather than a platform dependency.

Rolling updates and the blast radius

Scaling answers "how fast can I change the fleet". The senior question is "how do I stop a wrong change from reaching all of it". Ansible gives you several play-level controls; a platform team's job is to make them the default rather than an afterthought.

serial splits a play into batches. The whole play, every task and handler, runs against a batch before the next batch starts. It accepts a number, a percentage or a list, and the list form is the standard pattern for canary deployments.

- name: Roll the web tier
  hosts: web
  serial: [1, "10%", "50%", "100%"]
  max_fail_percentage: 5
  any_errors_fatal: false
  tasks:
    - name: Take the host out of the load balancer
      ansible.builtin.include_role:
        name: lb_drain
    - name: Deploy the release
      ansible.builtin.include_role:
        name: app_release
    - name: Wait until healthy
      ansible.builtin.uri:
        url: "http://{{ inventory_hostname }}:8080/healthz"
        status_code: 200
      register: health
      until: health.status == 200
      retries: 20
      delay: 5
    - name: Put the host back
      ansible.builtin.include_role:
        name: lb_enable

The first batch is one host, which is the canary; the next is a tenth of the fleet, then half, then everything. max_fail_percentage decides how many hosts in a batch may fail before the whole play aborts, and the subtlety to know is that with serial, the percentage is evaluated against the batch, not the whole fleet, which is why a batch of one with a value of 5 still aborts on a single failure. any_errors_fatal: true is the strict version: one failure anywhere aborts the play for all hosts, which is the right choice for operations such as a database schema migration where continuing on the others is worse than stopping.

Health checks inside the play are what make serial protective. A rolling update without a health check between batches is a slower way to break everything. The until and retries pattern above waits for a real readiness signal before the batch is considered done; since a host that never becomes healthy fails the task, max_fail_percentage and serial stop the wave.

Check mode and diff mode are the review tools before a wave. --check --diff (-C -D) asks each module to report what it would do without doing it. Its limits matter: a module must support check mode, tasks that depend on an earlier task's real effect can report wrong things, and command and shell tasks are skipped in check mode unless you set check_mode: false on a read-only one. So a clean check run is evidence, not proof. Use it in the pull-request pipeline against a staging copy, and mark read-only discovery tasks with check_mode: false so later tasks have real data.

Order matters in a fleet: order: inventory is the default, and sorted, reverse_sorted, reverse_inventory and shuffle are available. Shuffle is useful when inventory is ordered by rack or zone and you want a failure to be spread, sorted when you want the same canary every time.

Handlers and failure. Handlers run at the end of a play section and only for tasks that changed. If a play fails, pending handlers do not run unless force_handlers: true or --force-handlers is set. That is the source of a nasty incident shape: the configuration was changed, the restart handler never ran, and the service is still running the old configuration until something restarts it weeks later. Decide, per play, whether the safer failure is "run the pending handlers" or "leave everything alone", and use meta: flush_handlers when a later task depends on the restart having happened.

Idempotency, drift and the tasks that lie

Idempotency is the promise that running a task again on a system already in the desired state changes nothing and reports ok. At Mid level you learned to write for it. At platform level you need to be able to test for it and to spot the tasks that break it, because your drift reports depend on the changed count being honest.

The honest way to check is to run the play twice and require the second run to report zero changes. Automate it: a CI stage that converts a fresh instance, converts it again, and fails if the second recap shows changed= greater than zero. Molecule has this check built in as its idempotence step, and it is the single most valuable test you can give a role.

The recurring liars are worth knowing by name.

• command, shell and raw cannot know whether they changed anything, so they report changed every time unless you tell them otherwise with creates, removes or changed_when. A play full of unguarded shell tasks makes every run look like a mass change, and a drift report made from it is noise.
• Tasks that embed a timestamp or a random value in a template, such as {{ ansible_date_time }} in a header comment, rewrite the file on every run. Keep ansible_managed static.
• Downloads without a checksum fetch and replace on each run. Use the checksum parameter or force: false.
• Package tasks with state: latest are idempotent in form and not in practice: they change whenever upstream publishes. Use state: present with a version for anything you want stable and a separate, scheduled, reviewed "patch" play for upgrades.
• Registered output used to decide changed without a matching failed_when, which produces silent passes.

- name: Check the schema version (read-only)
  ansible.builtin.command: /opt/app/bin/schema-version
  register: schema
  changed_when: false
  check_mode: false

- name: Migrate if behind
  ansible.builtin.command: /opt/app/bin/migrate
  when: schema.stdout | int < target_schema | int

Drift detection as a scheduled job

With honest idempotency you can run the whole estate in check mode on a schedule and treat a non-zero changed count as a drift alert. The shape is: run --check --diff from a scheduled pipeline against production with a read-only credential where your platform allows it, parse the recap (the JUnit callback, enabled through callbacks_enabled, writes CI-readable results, and log_path keeps a file log), and open a ticket for each host that reports change. Two cautions. Check mode is not perfect, as already noted. And a drift report that nobody acts on trains people to ignore it, so route it to the owner of the role that drifted, not to a shared inbox.

Since 2.21, register can take a mapping of names to expressions and the _task object is available inside register and conditionals. That is useful for trimming what you keep from a noisy module result, which matters when results flow into a callback or a run database.

- name: Probe the service
  ansible.builtin.command: /usr/local/bin/probe
  register:
    probe_rc: _task.result.rc
    probe_out: _task.result.stdout | trim
  changed_when: false

_task is a reserved name in 2.21, so a variable called _task in an older playbook is now a conflict to fix.

Upgrades and migrations as a recurring programme

The engine under your platform changes twice a year, and the community package on top of it every six months. A platform team that treats upgrades as an emergency will always be behind; one that treats them as a programme does a small amount of work every release. Start with the support calendar, because it decides your deadlines.

Each core release is maintained for three release cycles, roughly eighteen months. The community package maintains only its latest major. The enterprise platform runs behind the community engine: AAP 2.7 supports ansible-core 2.16, 2.18 and 2.20 in its supported execution environments, with 2.16 as the default. The practical consequence is a compatibility range: if your collections must also run on the commercial platform, they must work on several core versions at once, and your CI should test the oldest and the newest.

What changed, and what each change costs

The releases since 2.19 removed or changed things that stale tutorials still teach. The ones that have broken real fleets:

2.19 (July 2025), the templating overhaul. Conditionals must evaluate to a boolean, so when: my_string fails with Conditional result (True) was derived from value of type 'str'... Conditionals must have a boolean result. Embedded templates inside expressions are rejected, so when: "{{ a }} == 'b'" becomes when: a == 'b'. Jinja native mode is always on. A become timeout became an unreachable error, so ignore_errors no longer hides it and ignore_unreachable does. The play_hosts magic variable is deprecated in favour of ansible_play_batch. The ssh connection's password handling changed: the password_mechanism option defaults to ssh_askpass, so sshpass is no longer needed for password authentication, and ansible_ssh_password_mechanism: sshpass restores the old behaviour.

2.20 (November 2025). The control node requires Python 3.12 or newer and targets need Python 3.9 or newer. The smart transport value, Galaxy v2 support and several long-deprecated keys were removed. INJECT_FACTS_AS_VARS is deprecated: it still defaults to True and will default to False in 2.24, at which point ansible_os_family as a bare variable stops existing and only ansible_facts['os_family'] works. The validate_argspec play keyword arrived as a tech preview.

2.21 (May 2026). The paramiko connection plugin was removed, so a connection: smart or persistent setting no longer falls back to it, and only local, psrp, ssh and winrm remain in core. Network devices use ansible.netcommon.network_cli, which carries its own choice of SSH library. The interpreter discovery modes auto_legacy and auto_legacy_silent are gone, leaving auto, auto_silent or an explicit path. The PSRP default negotiate_service changed from WSMAN to host. apt_key and apt_repository are deprecated in favour of ansible.builtin.deb822_repository. Failure inferred from a non-zero rc alone is deprecated for module authors, with runtime warnings planned from 2.22.

The migration method

The same sequence works for each release.

1. Read the porting guide for every version you skip, for the core and for the community package, before changing anything. The 2.19 guide is the long one.
2. Turn warnings into a backlog. Deprecation warnings are the engine telling you what will break. Keep deprecation_warnings on (the default), collect them from CI runs, and fix them in the order of their removal version. The settings deprecated for removal in 2.23 include DEFAULT_JINJA2_NATIVE, DEFAULT_JINJA2_EXTENSIONS, DEFAULT_MANAGED_STR (set an ansible_managed variable instead of the ini key) and DEFAULT_UNDEFINED_VAR_BEHAVIOR. Running ansible-config list shows which are flagged on your version.
3. Use the escape hatch only as a bridge. ALLOW_BROKEN_CONDITIONALS=True turns the non-boolean conditional error into a warning. It lets you upgrade the engine first and fix conditionals second, with a dated ticket to remove the setting. A platform that leaves it on permanently has deferred the work, not avoided it.
4. Lint, syntax-check, then check-mode in staging, then canary. ansible-playbook --syntax-check and ansible-lint find the mechanical breakage; a --check --diff run against a staging copy finds behaviour changes; a canary batch with serial finds the rest.
5. Upgrade the EE, not the laptops. The engine version a team sees should come from the execution environment image or the CI container, pinned by tag, so the upgrade is one image change with a rollback (point the tag back) rather than forty pip install commands.
6. Migrate targets too. A managed host needs Python 3.9 or newer for modern core. An old RHEL 8 host with only the platform Python 3.6 cannot run current modules until a newer Python is installed, which is often a change the operating-system team must own.

# find non-boolean conditionals early: run the suite with the escape hatch off (the default)
ANSIBLE_ALLOW_BROKEN_CONDITIONALS=False ansible-playbook site.yml --syntax-check
# then list what is about to be removed, from a real run's deprecation output
ansible-playbook site.yml --check 2>&1 | grep -i "deprecat"

A caution about the first command: --syntax-check does not evaluate conditionals, so it will not catch a non-boolean when. Only a real or check-mode run against representative data does. That is an argument for a staging inventory that exercises every branch.

Testing and policy as a platform

When other teams write playbooks on your platform, your leverage is what you can check automatically. The aim is for the platform to refuse the defects you would otherwise find in an incident, while keeping the refusals cheap to understand.

The layers, in order of cost

Static checks first. ansible-playbook --syntax-check catches YAML and structural errors. ansible-lint catches style and correctness problems and supports profiles that increase in strictness: min, basic, moderate, safety, shared and production. A platform usually enforces production for the shared library and a lower profile for team repositories during adoption, and raises it on a schedule.

ansible-lint --profile production

Argument validation. A role can declare its interface in meta/argument_specs.yml, and Ansible validates the arguments when the role runs, with a clear error for a missing or mistyped one. That turns a role's documentation into an enforced contract. Since 2.20, a play can do the same for its own variables with the validate_argspec keyword (tech preview), which reads a playbook meta file; for site.yml that file is site.meta.yml. Give your platform roles argument specs first, because these are the roles other teams call with arguments you did not anticipate.

argument_specs:
  main:
    short_description: Deploy one application release
    options:
      app_version:
        type: str
        required: true
        description: Release tag to deploy.
      health_path:
        type: str
        default: /healthz

Converge and idempotence tests. Molecule creates an instance, runs the role, runs it again and checks that nothing changed, then destroys the instance. It works with the Podman, Docker or a delegated driver. Containers are fast for the many roles whose behaviour does not need systemd or a kernel; for roles that manage services, kernel settings or firewalls, use a virtual machine, because a container that passes can hide a role that fails on real hosts.

Collection tests. If your platform ships its own collection, ansible-test sanity, units and integration run from inside the collection tree (ansible_collections/<namespace>/<name>) and are the standard way to check modules and plugins.

Check mode against a real copy. The last layer is the one that reflects your estate: run the change in --check --diff against a staging inventory that mirrors production's shape, in the pull-request pipeline, and attach the diff to the review.

Policy gates and how they fail

A gate is useful when people trust it. Three rules keep it that way. Fail only on findings the author can fix (the same rule that applies to a container scanner). Give every failure an actionable message or a link to the rule. And provide an exception path with an expiry: a .ansible-lint ignore entry or a # noqa with a ticket reference, reviewed periodically, is better than a pipeline that teams learn to bypass.

Multi-tenancy, the automation platform and cost

Command-line Ansible has no concept of a tenant. If two teams share a control node, they share its credentials, its filesystem and its log. The separation you get is whatever your repositories, inventories and credentials provide, and it is worth being plain about that, because "each team has its own repo" is not tenancy when all of them run on one machine with one SSH key.

What a platform adds

Red Hat's Ansible Automation Platform (AAP) and its upstream project AWX exist to add the missing layer. In AAP 2.5 and later, the platform gateway provides single sign-on, role-based access control and the front door to the API. The automation controller provides job templates, workflows, schedules, credentials, inventories and a PostgreSQL database that records every job and its events. Automation mesh runs execution and hop nodes over an overlay network, so jobs can reach segmented or remote networks without opening inbound paths. A private automation hub serves certified collections and execution environments inside your network, the Event-Driven Ansible controller runs rulebook activations, and everything executes inside execution environments. Installation in AAP 2.7 is containerised or through the OpenShift operator only; the RPM installer existed up to AAP 2.6.

The features that matter to a platform owner are not the user interface but the separation they enforce. Credentials are injected, not shared: a job template is granted a credential, the user who launches it never sees the secret, and external credential plugins can pull it from a secret manager at launch. Organisations, teams and role-based access decide who can read or run a project, inventory or template. Execution environments are chosen per template, so one team's dependency needs do not change another's. Instance groups and container groups isolate execution capacity, so that a noisy team cannot starve a critical one. And every job is recorded, which answers the audit question that the command line cannot.

A note that matters for accuracy: AWX, the upstream project, has had no release since July 2024 (24.6.1), when the project paused releases for a refactor. Do not present it as actively released when designing a platform, and check the project's state before you choose it as a foundation. The enterprise platform's core versions also lag the community engine, so collections often have to work on core 2.16 through 2.21 at once.

Choosing where the control plane lives

What it costs

ansible-core and the community package are free software under the GPLv3 or later. AWX is free. AAP is a Red Hat subscription priced per managed node and also sold through cloud marketplaces; prices are negotiated, so do not assume a figure in a design document. The costs that surprise people are not the licence: they are the infrastructure for the controller, execution nodes and PostgreSQL, the engineering time to run upgrades and the mesh, and the cost of a run that is slower than it should be. A forty-minute deployment run forty times a day is an hour of queue that somebody pays for.

If your employer is in a region with data-residency requirements, put the controller, its database and its job history in the same region as the managed estate, and keep credentials and execution inside it. A controller in another jurisdiction that stores job output with hostnames, IP addresses and command output can become a compliance question on its own.

Observability and audit

A platform without run history cannot answer the first question of every incident: what changed, on which host, and when. Because Ansible has no state store of its own, you assemble observability from the pieces it gives you.

Logs. Set log_path so every run is written to a file, and ship that file. Use callback_result_format = yaml in the default callback for human-readable output in logs (the old community YAML callback is not needed). Keep no_log in mind: logs are also a place where secrets end up.

Callbacks. Callback plugins receive every event. The built-in junit callback emits CI-readable results; ansible.posix.profile_tasks, timer and profile_roles measure time; community.general.opentelemetry exports traces over OTLP, which connects a play's tasks to the same trace backend as your services (see the OpenTelemetry guide); and there are callbacks for Splunk, Logstash and Elastic. ARA Records Ansible is a third-party run database that gives a browsable history of playbook runs. Enable callbacks with callbacks_enabled.

Debugging aids. ANSIBLE_KEEP_REMOTE_FILES=1 leaves the module payload on the target. ANSIBLE_DEBUG=1 prints engine internals. Since 2.19 the display_traceback setting accepts error, warning, deprecated, deprecated_value, always and never (default never), which replaces the old habit of running with -vvv just to see a traceback.

In a platform. The controller records job events through its API, keeps an activity stream of who changed what, and exposes metrics in Prometheus format from the controller's metrics endpoint, which fits the approach in the Prometheus guide.

The metrics that are worth alerting on are few: the failure and unreachable rate per run, run duration against its own baseline (a play that takes twice as long is usually a dependency problem), time since the last successful run per host group (this is your drift and stale-automation alarm), and queue wait when a platform is in use. Tie each alert to a runbook in the incident section below.

Incident playbooks

Incidents with Ansible have a recognisable shape: something is wrong with many hosts at once, and a run is either the cause or the only fast way to fix it. The playbooks below are written for the first ten minutes, when the aim is to stop the damage and learn what happened, not to be elegant.

A bad change is rolling through the fleet

1. Stop the run. Cancel the job or kill the ansible-playbook process. A run interrupted between batches is far better than one that finishes. If a platform is in use, cancel at the controller and confirm that no slices or workflow siblings are still running.
2. Freeze the pipeline. Disable the schedule and the merge-triggered deploys that could start a second run.
3. Establish the boundary. Use the run log and the recap to list which hosts got the change and which did not. With serial this is the batches completed; without it, the answer may be "all of them".
4. Decide roll back or roll forward. If the last good commit exists and the change is safe to undo (a template, a package pin), run the previous revision with --limit on the affected hosts, in batches. If the change was a one-way operation (a migration, a removed package), roll forward with a targeted fix.
5. Keep the evidence. Save the log, the commit, the EE image digest and the inventory revision before you re-run anything.

Hosts are UNREACHABLE

The recap line is UNREACHABLE! => {... "unreachable": true}, and the message says which step failed. Read it before guessing.

Since 2.19 a become timeout counts as unreachable, so a play that used ignore_errors to tolerate a slow sudo now needs ignore_unreachable. A sudden wave of unreachable hosts across unrelated groups is usually the control node's own network, a rotated key, or a changed bastion, not thirty independent failures. Check what all the failures have in common before opening thirty tickets.

A play "hangs"

Read -vvv to see which connection step it is on. The usual causes are a prompt nobody can answer (a password, a host key confirmation, a package manager lock), a long-running command with no timeout, and a dead connection that is waiting for TCP to give up. Defences you can already have in place are task_timeout and timeout, SSH ServerAliveInterval through ssh_args, and async for operations you expect to be slow. If stdin and stdout are captured by a CI harness, ERROR: Ansible requires blocking IO on stdin/stdout/stderr indicates non-blocking file handles; run under a proper terminal or pipe the output through cat.

Everything broke after an engine or collection update

Identify what changed first: the EE tag, a collection version, the controller's Python. Compare ansible --version and ansible-galaxy collection list between the last good run and the failing one. Roll the EE tag back, which is why the EE is pinned, then reproduce in staging. Typical signatures are couldn't resolve module/action 'x.y.z' (a missing or renamed collection, or a module that was removed or moved to a new namespace), a flood of non-boolean conditional errors (engine crossed 2.19), and a collection skipped for incompatibility with the core version.

A secret was exposed

Rotate first, then investigate. The value may already be in logs, caches or artefacts you cannot recall. Then find where it went: search CI logs and the run database for the value, check whether no_log was missing from the task, and whether a verbose run was stored. Then fix the cause (a lint rule for no_log, a move to an external secret store) and verify the control node was not itself the target.

The control node is compromised or lost

Treat every credential it held as exposed: rotate the SSH keys on the managed hosts, the become passwords, the cloud tokens and the vault passwords, in that order of blast radius. Rebuild the control node from code (this is the test of whether you can) and review the repository history for unreviewed commits. A platform that can rebuild its control plane from an image and a secret store in an hour is a platform that survives this day.

Where Ansible stops

Every tool has a region where another choice is better, and knowing where that is marks a senior engineer. Ansible is a general-purpose automation engine; its strengths are breadth and low ceremony, and its weaknesses follow from having no state and no resident agent.

Provisioning and lifecycle of cloud resources. Ansible has cloud modules, and they work, but it does not track the resources it created. There is no state file and no plan that shows a deletion before it happens. Tools built for this (see the Terraform guide) record what exists and compute a difference. The standard split is that Terraform provisions and Ansible configures, joined through the cloud.terraform collection (a terraform_output lookup and an inventory plugin) or through cloud inventory plugins driven by tags.

Continuous reconciliation. If your requirement is that a cluster's state is continuously corrected to match git, use a controller that watches and reconciles. For Kubernetes this is GitOps, covered in the Argo CD guide, and Ansible is the wrong shape for it. Ansible can apply manifests through kubernetes.core.k8s and install charts through its helm module, and that is fine for bootstrapping or one-off tasks, but it is not a reconciliation loop. See also the Kubernetes and Helm guides for the tools that own those concerns.

Immutable infrastructure. If hosts are rebuilt rather than patched, long-lived configuration of running machines is the wrong tool. Use Ansible during an image build, or avoid it.

Real-time or transactional control. Ansible is not a scheduler, a workflow engine with guarantees, or a messaging system. It runs tasks in sequence across hosts with best-effort error handling. If you need exactly-once, retryable, long-lived orchestration, use a system designed for it, or call Ansible as one step of one.

Windows estates at very large scale. Windows management works through ansible.windows, community.windows and microsoft.ad over WinRM, PSRP or SSH, but it is slower per task than POSIX over SSH, and the double-hop credential problem shapes the design. It is viable; size the controller and the expectations accordingly. Windows cannot be a control node natively; use WSL, a VM or a container.

Event response. Event-Driven Ansible (ansible-rulebook) connects event sources (webhooks, Kafka, Alertmanager) to actions. It is suitable for bounded remediation, such as restarting a failed service on an alert, and the production home for it is the automation platform's EDA controller. Keep remediation narrow, idempotent and logged: auto-remediation with broad privileges is a fast way to convert an incident into an outage.

Running Ansible as a platform

Once other teams depend on your Ansible, you are running a product. The technical material above is necessary; what makes it a platform is the operating model around it.

Decide what the platform guarantees

Write a short service description. It should state which core and collection versions are supported and for how long, how a team requests a new collection, how long a change takes from merge to production, who is on call for the control plane, and what the platform will not do. Teams accept constraints they can read; they route around constraints they discover.

Provide paved roads

The thing that changes a team's behaviour is a template that is easier than the alternative. A good starter includes a repository skeleton with ansible.cfg, .ansible-lint, requirements.yml, a pipeline that runs lint, syntax check, Molecule and a check-mode diff, a rolling-deploy play with serial and a health gate, and a documented way to fetch secrets. A team that copies it is safe by default. Publish shared roles with argument specs as a versioned collection, and deprecate on a schedule rather than deleting.

[defaults]
inventory = inventories/prod
roles_path = roles
collections_path = collections
stdout_callback = default
callback_result_format = yaml
callbacks_enabled = ansible.posix.profile_tasks
log_path = logs/ansible.log
forks = 25
retry_files_enabled = False
host_key_checking = True
interpreter_python = auto_silent

[ssh_connection]
pipelining = True

The file above collects settings from this guide. Two details are deliberate: collections_path is singular (the plural key is not the current name), and interpreter_python = auto_silent is correct only after you have decided that unpredictable interpreter discovery on your targets is acceptable; for a strict platform, set ansible_python_interpreter explicitly per group instead.

Govern change, not people

Put the governance in repository rules and pipelines, not in a committee. Protected branches, CODEOWNERS on sensitive paths, required status checks, and deploy credentials available only to the deploy job. Define change classes: a role edit in a team's own repository merges with one review and a check-mode run; a change to shared roles, group_vars/all or the EE needs a platform reviewer; a change that touches credentials or the control plane needs two.

Own the lifecycle

Keep a simple inventory of what you run: control nodes or controller, EE images and their tags, collections and versions, and the teams using each. Review it on the same calendar as the core support dates. Rebuild EEs on a schedule, rotate the automation keys on a schedule, and test the rebuild of the control plane on a schedule. A platform is what you do repeatedly.

Measure the platform itself

Four numbers tell you if the platform is working: the time from merge to applied change for a typical change, the change failure rate (runs that needed a rollback or an incident), the age of the oldest unmerged drift (how long a host has differed from git), and the share of playbooks on supported versions. Publish them. The last is the one that finds abandoned automation.

Teach, and make it findable

Each team should be able to reach the beginner and mid-level practices in these guides without asking you. Link from your platform documentation to the Terraform guide for provisioning, Docker for EEs and images, GitHub Actions for the pipelines, and Vault for secrets, because the platform is only as good as the practices around it.

The review checklist

Use this for pull requests that touch shared automation and for design documents that add a new use of Ansible. Each question has a reason in an earlier section.

Trust and credentials

• Does any task run as root that could run as a narrower user, and is become scoped to tasks that need it?
• Is every secret fetched at run time or held in a vault with an environment-specific ID, and does every task that touches one have no_log: true?
• Is host_key_checking left on, and do different environments use different keys?
• Does the change add a collection, role or plugin, and was it reviewed and pinned?

Safety of the change

• Does the play use serial, a canary and a health gate, or does it explain why it does not?
• Are max_fail_percentage and any_errors_fatal chosen deliberately?
• What happens if the run stops after each task: is the half-finished state recoverable?
• Are handlers safe to run late, and is force_handlers decided?

Correctness

• Is the second run clean (zero changed)? Are all command and shell tasks guarded with creates, removes or changed_when?
• Are conditionals boolean, without embedded templates, and do facts use ansible_facts['...']?
• Are modules referred to by fully qualified collection name, and do any use deprecated modules such as apt_key?
• Do roles declare argument_specs for their inputs?

Scale and operations

• Are facts limited or cached appropriately, and are long tasks async or throttled where they hit shared backends?
• Does the change respect forks and the control node's memory?
• Is there a check-mode diff in the pull request, and a rollback path?
• Will the change show up in logs and metrics so that it can be found later?

Versions and lifecycle

• Does it work on every core version in the supported range, including the platform's?
• Does it use anything deprecated with a removal date inside the support window?
• Is the EE or tag that the change was tested on the one that will run?

The complete picture

Put the pieces together on one concrete scenario: a platform team supports an ML-platform organisation whose GPU nodes, inference gateways and feature-store databases run on virtual machines in a Gulf region, configured by Ansible, with the provisioning done elsewhere.

A change to the driver role arrives as a pull request. The pipeline runs ansible-lint --profile production, checks the role's argument spec, runs Molecule's converge and idempotence steps, and posts a --check --diff against a staging copy of the GPU inventory. A CODEOWNERS rule requires a platform reviewer because the role is shared. On merge, the pipeline builds or selects the execution environment by pinned tag, whose core version and collection versions are recorded in the run.

The deploy runs on the controller in the same region as the nodes, with the SSH key and the vault password injected from the secret manager by the job's identity, so engineers never hold them. The play uses serial: [1, "10%", "50%", "100%"], drains a node from the scheduler, applies the change, waits for a real readiness signal with until, and puts the node back. max_fail_percentage stops the wave on the first unhealthy canary. Long driver installs run as async tasks with a timeout, and throttle: 5 protects the internal package mirror.

Afterwards, the run's log, the commit, the EE digest and the inventory revision are stored together. A scheduled check-mode job reports any node whose state differs from git and routes the report to the role owner. A weekly job rebuilds the EE, and a calendar entry tracks the security-only date of the core release in use. If the change goes wrong, the playbook in the previous section stops it at the canary, and the log says what the canary saw.

Every line of that scenario is a section of this guide. Nothing in it is exotic; what makes it work is that each control is the default rather than a thing someone remembered.

Where the series leaves you

Across the three levels, you went from running a first ad hoc command to being able to say what happens to a fleet when one of your tasks is wrong. At Beginner you learned the nouns: control node, inventory, module, playbook, idempotency. At Mid level you learned to use the tool well: roles, collections, vault, templating, testing, production patterns. At this level you learned to own it: the trust model, scale limits, rolling safety, the supply chain, upgrades, observability, incidents, and the line where another tool is the better answer.

The habits that matter most are the ones that will still be true after the next release. Treat the repository as the privileged surface. Make the safe path the default path. Pin what you run and know when its support ends. Test for the second run. Stop a bad change at the canary. Write down where state lives. And when a problem is not Ansible-shaped, say so early and use the right tool.

From here, the most useful next steps are practical. Take your platform through the review checklist and fix the top three findings. Run one game day for a bad rollout and one for a lost control node. Put the core and collection support dates in a shared calendar. And read the porting guide for the next release before it ships, not after. For the neighbouring topics, continue with Terraform for provisioning, Kubernetes and Helm for workloads, and Argo CD for continuous reconciliation.

Sources

• Ansible documentation home
• Releases and maintenance
• ansible-core 2.21 porting guide
• ansible-core 2.20 porting guide
• ansible-core 2.19 porting guide
• Ansible 14 porting guide
• ansible-core 2.21 changelog
• ansible-core 2.20 changelog
• ansible-core 2.19 changelog
• Configuration reference
• Interpreter discovery
• Using variables and precedence
• Building an execution environment
• ansible.builtin collection index
• Windows guide
• Windows remote management
• Red Hat Ansible Automation Platform lifecycle
• AWX releases
• ansible-core on PyPI

Part three of three. A cumulative review of the entire series, organised by the kind of question a senior interview asks: system-design prompts built around Ansible, incident-response questions, and how to argue a trade-off to people who do not care about YAML. About fifty minutes. The fast review comes first, the long-form answers after it.

This guide targets ansible-core 2.21 (the Ansible community package 14). Where an answer depends on a version, it says so, because a senior candidate who quotes a removed feature as current gives away that their experience is old.

Part one: the model in one page

Ansible is an agentless automation engine. A control node reads inventory and playbooks, packages each module, runs it on the target over SSH (or WinRM or PSRP), and reads the result. Nothing resident on the target, no state store, no scheduler. The trade is simplicity and reach against continuous enforcement and central state.

The points to have ready, in the order an interviewer tends to probe:

• Idempotency is the contract: a second run on a converged system reports ok and changes nothing. command, shell and raw break it unless guarded with creates, removes or changed_when.
• Push and pull. ansible-playbook pushes from a control node. ansible-pull makes each host clone a repository and run the playbook on itself.
• Trust boundary. Whoever can change what the control node runs can run code as root on every managed host. That includes the repository, the inventory, group_vars, collections and the execution environment.
• Scale limits in order: forks (controller CPU and memory), round trips (pipelining, SSH multiplexing), fact gathering, the slowest host under the linear strategy, then the backends your tasks hit.
• Blast radius is controlled with serial, max_fail_percentage, any_errors_fatal, health checks between batches and check mode.
• Versions. Current core is 2.21. The 2.19 release made conditionals strictly boolean and changed the templating trust model. 2.20 requires Python 3.12 or newer on the control node. 2.21 removed the paramiko connection plugin.

The facts a senior is expected to know cold

Part two: system-design prompts

These are the questions that decide a senior hiring loop. The interviewer wants to see you ask about constraints, choose, and name the failure modes of your own choice. The answers below are models for structure, not scripts.

Design 1: "Configure ten thousand Linux servers across three regions"

Clarify first. How fast do changes need to land (minutes or days)? Are hosts long-lived or rebuilt? Is there a network segment that cannot be reached from a central point? Are there data-residency rules that keep credentials and logs inside a region? What is the mix of operating systems?

A strong answer. Split the estate into regional control planes, one per region, each with its own controller or control nodes, credentials and logs, so that a failure or compromise is contained and data stays in region. Put inventory in dynamic plugins with caching, so that the cloud API is not hammered, and group hosts by tags that the provisioning layer owns. Use pinned execution environments as the runtime so that every region runs the same engine and collections. Tune for scale in order: raise forks to what the controller's memory allows, turn on pipelining and SSH multiplexing, limit or cache facts, set task_timeout so a hung host cannot hold a fork. Use job slicing or --limit by group to fan a job across workers. Roll changes with serial and a canary, and gate waves on real health checks. Consider ansible-pull for the part of the fleet that is homogeneous and self-configuring, because it removes the central bottleneck.

What a weak answer misses. The weak answer says "raise the forks" and stops. It does not say where memory limits forks, it has no plan for a region losing connectivity to the control node, and it has no answer for credentials and audit across regions.

Follow-up you should expect: "Which breaks first?" The honest answer is the controller's memory, through forks, and after that the package mirror or API your tasks depend on, so concurrency controls such as throttle matter more than more forks.

Design 2: "Give forty teams a safe way to use Ansible"

Clarify. Are the teams trusted equally? Do they need to run against shared hosts? Is there a compliance requirement for audit? Who is on call?

A strong answer. Name the problem: command-line Ansible has no tenancy, so forty teams on one control node share credentials and logs. Offer the options: an automation platform (AAP, or an AWX-class controller) for RBAC, injected credentials and job history; or a CI-based model where each team's pipeline runs an approved execution environment with credentials scoped to that team. Whichever you choose, define the paved road: a template repository with ansible.cfg, .ansible-lint on the production profile, a pipeline that runs lint, Molecule and a check-mode diff, and a rolling-deploy play with serial and a health gate. Enforce governance in the repository: protected branches, CODEOWNERS on shared roles and group_vars, required checks. Keep a small number of execution environments and rebuild them on a schedule. Publish a service description with supported versions and a request path for new collections. Mention honestly that upstream AWX has had no release since July 2024, so a self-hosted choice has that risk, and that commercial AAP lags community core (it supports 2.16, 2.18 and 2.20 in its supported EEs), so shared collections must work on several core versions.

What a weak answer misses. Treating it as a training problem. The platform's controls must make the unsafe path harder than the safe one.

Design 3: "Zero-downtime rolling deploy of a stateful service"

A strong answer. Use serial with a list (for example one host, then ten percent, then half, then all). In the play, drain the host from the load balancer with delegate_to where the load balancer API lives, apply the change, restart through a handler that is flushed with meta: flush_handlers so the restart has happened before the health check, wait for a real readiness signal with until and retries, and only then return the host to rotation. Set max_fail_percentage or any_errors_fatal on purpose: for a schema migration, one failure should abort everything. Use run_once: true for the one-time migration step, on one host, before the rolling part. Put the check-mode diff in the pull request. Explain what happens when the run is interrupted on the third host: the play must leave the system serving traffic, which it does because each host is drained before it is changed and restored after.

The trap inside this question. Handlers do not run after a failure unless force_handlers is set, so a config change can be applied with the restart never happening. Say it unprompted.

Design 4: "Migrate our Ansible estate from core 2.17 to 2.21"

A strong answer. Do not jump in one step. Read the porting guides for 2.18 to 2.21 for both the core and the community package. Note the breaking changes that matter: control-node Python 3.12 or newer and target Python 3.9 or newer (2.20); strictly boolean conditionals and templating trust changes (2.19); Galaxy v2 removal (2.20); INJECT_FACTS_AS_VARS deprecated, flipping to False in 2.24; paramiko removal and the interpreter discovery mode changes (2.21). Build the target EE, run lint and syntax checks, then run the suites in check mode against staging, then canary production. Use ALLOW_BROKEN_CONDITIONALS=True only as a temporary bridge with a dated ticket to remove it. Treat deprecation warnings as a backlog sorted by removal version. Plan for the targets too: hosts whose only Python is too old need a newer Python first, and that is a change another team may own. Keep the old EE tag so rollback is a tag change.

Design 5: "Design drift detection for the fleet"

A strong answer. Ansible has no state, so drift is found only by running. Run --check --diff on a schedule against the fleet with a read-only credential where possible, collect the recap through a callback such as junit or the controller's API, and route each host's differences to the owner of the responsible role. Make it honest by requiring idempotence tests on roles, since command and shell tasks would otherwise report change every time. State the limits: check mode is best-effort, some modules cannot report accurately, and read-only discovery tasks need check_mode: false to give later tasks real data. Alternatively, ansible-pull on a timer for continuous convergence, with a separate signal that each host is still pulling.

Design 6: "Secrets for automation that touches production"

A strong answer. Distinguish what must be encrypted at rest in git from what must be fetched at run time. Ansible Vault with vault IDs per environment for low-churn values; lookups against a secret manager (HashiCorp Vault through community.hashi_vault, or the cloud providers' secret stores) for anything rotated or audited; credentials injected by the controller so engineers never see them. Use an executable vault password script so the vault password comes from the store, and the ssh private_key option (added in 2.19) so a key can be used without a file on disk. Use no_log: true on tasks that handle secrets and lint for it. Explain that Vault is a file format, not a secrets platform: it cannot revoke one person's access, rotate a value or audit a read.

Part three: incident-response questions

Incident questions test whether you stop the damage first and understand it second. A strong answer has a recognisable order: stop, contain, establish what is affected, decide roll back or forward, preserve evidence, then fix the cause.

"A playbook is rolling a bad config through production right now. Go."

Stop the run at the controller or kill the process, and freeze any schedule or merge trigger that could start another. Use the run log and recap to list the hosts that received the change; with serial this is the completed batches. Decide whether to roll back by running the last good revision against the affected hosts in batches, or roll forward with a fix if the change was one-way. Preserve the log, commit, EE digest and inventory revision before re-running. Then fix the cause: no canary, no health gate, or no review. What a weak answer does: re-runs the play with more verbosity and makes it worse.

"Forty hosts report UNREACHABLE after a routine run."

First, read the actual message, since it says which step failed: a timeout is a network path question, Permission denied is a user or key question, Host key verification failed means something changed about the host's identity and deserves suspicion before acceptance, and Premature end of stream waiting for become success points at sudo. Then ask what the forty have in common: a subnet, a bastion, a key that was just rotated, a host group. Forty independent failures are rare; one shared cause is common. Test with plain ssh -vvv from the control node to remove Ansible from the picture. Remember that from 2.19 a become timeout is an unreachable error, so ignore_errors does not hide it.

"The same playbook takes twice as long as last week."

Compare what changed: inventory size, fact gathering, a slow dependency, a new callback, a collection update, or a host that is hanging. Enable a profiling callback such as ansible.posix.profile_tasks and read the three slowest tasks. A long task on many hosts is usually a package mirror or API being hit concurrently, which calls for throttle. A single slow host under linear holds the whole batch, which calls for a task timeout or strategy: free. If fact gathering dominates, limit it with gather_subset or use gathering = smart with a cache, accepting staleness.

"A service was reconfigured but never restarted, and nobody noticed for three weeks."

Explain the mechanism: handlers run only for tasks that changed, at the end of the section, and do not run after a failure unless force_handlers is set. The config was changed, a later task failed, the restart never ran, and the next run saw the file already correct and reported ok, so the handler was never notified again. The fix is structural: flush handlers before the tasks that depend on the restart, decide the force_handlers policy per play, and add a verification task that checks the running state, not the file.

"A vault password, or a cloud key, appeared in a CI log."

Rotate first. The value may be in caches, artefacts and downstream systems that you cannot enumerate. Then find the path: a task without no_log, a debug of a registered variable, a verbose run that was stored. Fix the cause with lint for missing no_log, a move to lookups against a secret store, and a policy that verbose logs are not retained. Check whether the exposed credential was used.

"The control node is gone, or may be compromised."

Treat every credential it held as exposed and rotate in order of blast radius: SSH keys to managed hosts, become passwords, cloud tokens, vault passwords. Rebuild the control plane from code and an image, not from memory. Review the repository history for unreviewed commits, since the boundary includes the repository. The measure of a mature platform is whether the rebuild took an hour.

"After the upgrade, everything is failing with conditional errors."

The engine crossed 2.19. The message is Conditionals must have a boolean result, and the fix is to write explicit conditions such as my_string | length > 0, my_value | bool or a comparison. The escape hatch ALLOW_BROKEN_CONDITIONALS=True turns the error into a warning, which is acceptable as a bridge with an expiry date. Roll the EE tag back if the blast radius is large, fix in staging, and add a check-mode run to the pipeline so this class of error is found before production.

Part four: arguing a trade-off to leadership

Senior interviews include a question like "convince me". The audience is a director or a finance lead who does not want a lecture on idempotency. The method is to state the decision, the options, the cost of each in terms they care about, your recommendation, and what you would measure to know if it was wrong.

"Why should we pay for Red Hat AAP when Ansible is free?"

Do not argue that the software is better. Argue about what the subscription buys that you would otherwise build and operate: role-based access control, credential injection, job history and audit, support for the execution environments and certified content, and a vendor to escalate to. Then argue the cost side honestly: the subscription is priced per managed node, so the bill grows with the estate, and the platform lags the community engine, so some collections need to work on older core versions. The recommendation depends on facts about the company: regulated estate with audit requirements, where AAP is easier to justify; small team with everything in CI, where a pipeline with a pinned execution environment may be enough. Offer to decide with data: how many teams, how many managed nodes, what the audit requirement is. Note that an AWX-class self-hosted controller is free but has had no release since July 2024, which is a risk to name.

"Why do we need to spend a quarter on upgrades?"

Speak in terms of support dates and risk. Each core release is maintained for about eighteen months: security-only fixes start after the next release and end-of-life follows. Staying on a release past its end of life means running an engine with no security fixes under every production change. The upgrade work is smaller when done every release than when deferred, because the breaking changes (2.19 was the big one) accumulate. Propose a standing programme instead of a project: one engineer-week per release, a calendar of support dates, and a metric, the share of pipelines on a supported version. The cost of the alternative is an emergency upgrade done in a hurry after a vulnerability or a collection that no longer supports the old core.

"Can't we just give every team admin access to the control node? It is faster."

Name what it trades away: one team's mistake or compromise becomes everyone's, credentials are shared and cannot be attributed, and audit becomes impossible. Offer the faster alternative: a paved road where a new team gets a repository template and a pipeline in an hour, with credentials scoped to them. Speed and safety are not opposed here; the unsafe path is only faster because the safe path was not built.

"Why not replace Ansible with Terraform, or the other way round?"

They solve different problems. Terraform tracks what exists and plans a difference, so it suits provisioning resources. Ansible configures hosts and orchestrates ordered operations and has no state of its own. The common arrangement is that Terraform provisions and Ansible configures, joined through the cloud.terraform collection or cloud inventory plugins. If the estate is moving to immutable images and Kubernetes, the amount of Ansible shrinks, and the honest answer is to say so and plan for that. Link the decision to the Terraform guide for detail.

"What is the ROI of the idempotence and lint gates?"

The gates cost minutes per change. They prevent two kinds of cost: incidents (a role that changes on every run makes drift reports useless and hides real changes) and review time (reviewers stop arguing about style that a machine settles). Propose measuring the change failure rate and the time from merge to applied change before and after, and be willing to drop a gate that does not move the numbers.