The Complete HashiCorp Vault Guide

This is part one of three. It covers everything you need to do real work with HashiCorp Vault, not a teaser. By the end you can start a server, store and version secrets, write a policy that gives a person or an application exactly the access it needs, log in as that person, hand out short-lived tokens, and read the error messages Vault gives you when something is wrong. Mid-level and Senior take the same topics further; nothing here is thrown away.

The guide targets Vault 2.1.1, released on 16 September 2026. The 2.0 release in April 2026 was the first major version since 1.0, and it changed a few details that older blog posts and tutorials still get wrong. Where that matters, the text says so and shows the current form.

Each section ends with a Try it task. Do them as you go. They take a few minutes each, and Vault only makes sense once you have watched a request get refused with "permission denied" and then fixed it yourself.

What Vault is, and the problem it solves

Vault is a server that stores, generates and hands out secrets, and decides who is allowed to see each one. A secret is anything that grants access or must stay private: a database password, a cloud access key, an API token for a payment provider, a TLS private key, an encryption key. Vault puts all of them behind one HTTP API, requires every caller to prove who they are, checks a written policy on every request, and records who asked for what.

To see why that matters, look at what teams did before.

The first habit is secrets in source code. A developer pastes a database password into a config file, commits it, and pushes. The password is now in every clone and in the history forever. Deleting the line in a later commit removes nothing, because Git remembers. Automated scanners crawl public repositories for exactly these strings within minutes of a push.

The second habit is secrets in environment variables and CI settings. That is better than source code, but the values are still long-lived, shared by everyone who can edit the pipeline, and rarely rotated. When an engineer leaves, nobody knows which of forty variables they had seen. When something leaks, nobody can tell which value leaked, from where, or when.

The third habit is a shared spreadsheet or chat message, which needs no further comment.

Underneath these habits sit four problems that Vault is designed to address together:

1. Sprawl. Secrets live in dozens of places, so nobody can list them or change them in one go.
2. Long life. A password created in 2021 and never changed is a standing invitation. The longer a secret lives, the more people have touched it and the more places it has been copied.
3. No audit trail. When a secret is read from a file, nothing records the read. After an incident you cannot answer "who had access?".
4. No fine-grained control. Anyone who can read the file can read every secret in it.

Vault's answer is to centralise secrets, put identity and policy in front of every read, record every request in an audit log, and, where it can, stop storing secrets at all. That last idea is the most interesting one. For many systems, Vault can create a credential on demand, for example a fresh database user that expires in an hour, so there is no long-lived password to leak. Later guides in this series cover that in depth; this guide concentrates on the foundations.

That diagram is the whole product. A client logs in through an auth method and receives a token. Policies attached to the token decide which paths the token may touch. Each path leads to a secrets engine, which does the real work. Everything Vault keeps on disk is encrypted. The rest of this guide unpacks each box.

A note on licensing, because you will meet the question at work. Since Vault 1.15 (August 2023) the source code is under the Business Source License 1.1 rather than the older MPL 2.0 license. The free edition you download is called Vault Community Edition, and that is what this guide uses. There is also OpenBao, a separate community fork of the last MPL-licensed code, governed by the Linux Foundation. It is a different product: do not assume its commands and features match Vault's. Finally, HashiCorp is now part of IBM, and you will see the product marketed as "IBM Vault". The command is still vault and the documentation still lives at developer.hashicorp.com/vault.

The mental model: paths, tokens, policies and the seal

Vault has a small vocabulary. Learn these nouns once and every later feature is a variation on them.

Everything is a path

Vault is an HTTP API server, and every feature is reached at a URL of the form /v1/<path>. The CLI is a convenience wrapper that builds those requests for you. A secret stored at secret/myapp/config is really an HTTP request to /v1/secret/data/myapp/config (we will explain the extra data later). Creating a database user is a request to a path under database/. Even Vault's own administration is a path: sys/ is the system backend, and it is where you turn features on, read the health of the server and manage policies.

This design has a useful consequence. Because every action is a read or a write to a path, access control is also about paths. A policy does not say "this person is a database admin". It says "this token may read the path database/creds/app". You will find that once you think in paths, permissions become concrete and testable.

Secrets engines: what lives behind a path

A secrets engine is a component mounted at a path prefix that stores, generates or encrypts data. Mounting an engine at secret/ means every request under secret/ is handed to that engine. You can mount several engines, and you can mount the same kind of engine at different prefixes.

Engines fall into three families:

• Static secrets. You put a value in, you get the same value out. The KV (key-value) engine is the main example and is what you will use first.
• Dynamic secrets. Vault creates a credential when asked and deletes it when its lease ends. The database, AWS, GCP, Azure, Kubernetes, SSH and PKI (certificates) engines work this way.
• Encryption as a service. The Transit engine encrypts and decrypts data for your application without ever giving the application the key.

Auth methods: how callers prove who they are

An auth method verifies an identity and maps it to a set of policies. Auth methods always live under auth/, for example auth/userpass/. A person might log in with a username and password, or through a company single sign-on system (OIDC, LDAP). An application might log in with approle, or with its Kubernetes service account, or with a cloud identity. The method you pick depends on who the caller is, and each method ends the same way: Vault returns a token.

Tokens: the credential you actually carry

The token is the core credential. Every request to Vault, except a few unauthenticated system endpoints, carries a token in the X-Vault-Token header (or as Authorization: Bearer <token>). When you log in, Vault gives you a token; when you use the CLI, it stores that token in a file, ~/.vault-token, and sends it for you.

Tokens you will see begin with a prefix that tells you the type: hvs. for a service token, hvb. for a batch token and hvr. for a recovery token. Before Vault 1.10 the prefixes were s., b. and r., so an old tutorial showing s.abc123 is not wrong, just old. Service tokens are the ordinary kind: stored by Vault, renewable, revocable. Batch tokens are lightweight and cannot be renewed or revoked, which suits very high-volume short jobs.

Every token has a TTL (time to live). When it expires, it stops working. The system default and the system maximum are both 768 hours, which is 32 days; you can tighten them per mount and per role. Tokens form a hierarchy: a token is a child of the token that created it, and revoking a parent revokes all of its children. A token with no parent is an orphan; tokens issued by logging in through an auth method are orphans.

A token also has an accessor, a reference you can use to look up, renew or revoke the token or check its capabilities without knowing the token itself. The accessor cannot be used to authenticate, so it is safe to log.

Policies: what a token may do

A policy is a small document, written in HCL (HashiCorp Configuration Language) or JSON, that lists paths and the capabilities allowed on each. The capabilities are create, read, update, patch, delete, list, sudo and deny. Policies are deny by default: a path not mentioned is forbidden. If two policies disagree, deny always wins.

Two policies exist from the start. default is attached to every token unless you opt out; it grants a few harmless things such as looking up your own token. You can edit it, but not delete it. root is immutable and grants everything.

The root token, and why you should not keep it

The root token carries the root policy and can do anything. You get one at initialisation and you should treat it like the master key to a building: use it once to set things up, then revoke it. Day-to-day work uses tokens with narrow policies. From Vault 2.0 onwards, even generating a replacement root token with vault operator generate-root requires an existing token as well as key shares, which is one more reason to plan your administrative accounts rather than lean on root.

Sealed and unsealed

This is the concept newcomers find strangest. When a Vault server starts, it can reach its storage but cannot read it, because the data is encrypted and the key is not in memory. The server is sealed. Until someone unseals it, it answers almost nothing, and every data request returns an error that says Vault is sealed.

Why design it this way? Because the storage (a disk, or a cloud volume) is treated as untrusted. If someone steals the disk, or a backup, they get ciphertext. The encryption happens in a layer Vault calls the barrier, which uses AES-256-GCM. The key hierarchy has three levels:

1. Your data is encrypted with an encryption key (the keyring).
2. The keyring is encrypted with the root key.
3. The root key is protected by the unseal key, which by default is split into pieces using Shamir's Secret Sharing.

With the default setup, vault operator init creates five key shares and requires any three of them to unseal. You give the shares to different people, so no single person can unseal the server alone. Production systems usually avoid the manual step with auto-unseal, where a cloud key management service (AWS KMS, Azure Key Vault, GCP Cloud KMS and others) protects the root key. In that mode init issues recovery keys instead, which cannot decrypt anything and only authorise sensitive operations.

Leases, mounts and audit

Three more nouns complete the vocabulary. A lease is metadata attached to a dynamic secret (and to service tokens): a lease ID, a TTL and a renewable flag. When the lease ends, Vault revokes the credential. You can revoke leases one at a time or by prefix. A mount is simply an engine or auth method attached to a path; you "enable" one to mount it. An audit device is a log sink (a file, syslog or a socket) that records every request and response, with sensitive values hashed so the log does not itself leak secrets.

Installing Vault and checking the setup

Install the single vault binary. The same file is both the server and the command line client.

On macOS, use the HashiCorp tap. The formula in Homebrew's core repository stopped receiving updates after the license change, so a plain brew install vault can give you something stale or nothing at all.

brew tap hashicorp/tap
brew install hashicorp/tap/vault
vault version

On Ubuntu or Debian, add HashiCorp's signed package repository and install:

wget -O - https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update && sudo apt install vault

On Red Hat family systems the package repository is added with yum-config-manager (RHEL, CentOS, Amazon Linux) or fetched with wget (Fedora), and then installed with yum or dnf; the exact lines are on the official install page at developer.hashicorp.com/vault/install. On Windows, download the zip from releases.hashicorp.com, place vault.exe in a folder on your PATH, and note that Chocolatey and winget packages exist but are maintained by the community, not by HashiCorp. You can also download the zip for any platform from releases.hashicorp.com/vault/2.1.1/ and verify it against the published SHA256 checksums.

If you prefer not to install anything, Docker works for a disposable server:

docker run --rm -p 8200:8200 -e VAULT_DEV_ROOT_TOKEN_ID=root hashicorp/vault:2.1.1 server -dev -dev-listen-address=0.0.0.0:8200

Since Vault 2.0 the official container runs as the vault user, and since 2.0.2 the binary no longer has the cap_ipc_lock capability, so a container cannot lock memory. That matters for real deployments, where you set disable_mlock = true; a throwaway dev container does not care.

Verify the install:

vault version

Vault v2.1.1 (abcdef123456), built 2026-09-16T10:00:00Z

The commit hash and build date on your machine will differ. What matters is the version number. Two optional extras are worth doing now. vault -h lists every command, and vault -autocomplete-install adds tab completion to your shell (restart the shell afterwards).

Starting a development server

Vault has a built-in development mode that gives you a working server in one command. It exists for learning and testing. It is not safe for real use, and the next paragraph explains why.

vault server -dev

In dev mode Vault stores everything in memory, so all data vanishes when you stop it. It starts already unsealed with a single unseal key. It listens on plain HTTP at 127.0.0.1:8200, with no TLS. It enables a KV version 2 engine at secret/. And it prints a root token in the terminal. Each of those conveniences would be a serious flaw in production, which is why the mode has "dev" in the name.

Leave that terminal running. The output includes lines like these:

WARNING! dev mode is enabled! In this mode, Vault runs entirely in-memory
and starts unsealed with a single unseal key. The root token is already
authenticated to the CLI, so you can immediately begin using Vault.

You may need to set the following environment variables:

    $ export VAULT_ADDR='http://127.0.0.1:8200'

The unseal key and root token are displayed below in case you want to
seal/unseal the Vault or re-authenticate.

Root Token: hvs.xxxxxxxxxxxxxxxxxxxxxxxx

Open a second terminal for the client. The CLI needs two environment variables. VAULT_ADDR tells it where the server is, and this one is required because the CLI defaults to https://127.0.0.1:8200, while the dev server speaks plain HTTP. VAULT_TOKEN supplies the token:

export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_TOKEN='hvs.xxxxxxxxxxxxxxxxxxxxxxxx'
vault status

Paste the real root token printed by your server. The status output looks like this:

Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    1
Threshold       1
Version         2.1.1
Storage Type    inmem
Cluster Name    vault-cluster-...
HA Enabled      false

Read it line by line. Initialized true means the storage has been set up. Sealed false means the server can read its data. Total Shares 1 and Threshold 1 show the dev-mode shortcut of a single unseal key. Storage Type inmem confirms the memory-only storage. HA Enabled false means this is a single node rather than a cluster.

If you prefer a fixed token so that you do not have to copy a new one on every start, pass it yourself:

vault server -dev -dev-root-token-id=root

Now the token is simply root. Convenient on a laptop, and exactly what you must never do anywhere else.

The other path is the web interface. Open http://127.0.0.1:8200 in a browser and sign in with the token. The UI is a pleasant way to browse, but the CLI is what you will use in scripts and what interviewers ask about, so this guide shows the CLI.

Your first secrets: the KV engine

The development server already has a KV (key-value) engine mounted at secret/. KV is the simplest engine and the one you will use most in the beginning. It stores named sets of key-value pairs at paths you choose.

There are two versions. KV version 1 stores one value per path with no history. KV version 2 keeps versions of each secret, lets you delete and recover them, and stores metadata. Version 2 is what the dev server mounts and what you should use by default. One detail trips up almost everyone, so here it is early: in version 2 the HTTP API inserts data/ (and metadata/) into the path. The CLI hides that for you, the policies you write do not.

Check what is mounted:

vault secrets list

Path          Type         Accessor              Description
----          ----         --------              -----------
cubbyhole/    cubbyhole    cubbyhole_a1b2c3d4    per-token private secret storage
identity/     identity     identity_e5f6a7b8     identity store
secret/       kv           kv_c9d0e1f2           key/value secret storage
sys/          system       system_03a4b5c6       system endpoints used for control, policy and debugging

Write your first secret. The -mount flag names the engine, and the last argument is the path inside it:

vault kv put -mount=secret myapp/config username=app password=s3cr3t

======= Secret Path =======
secret/data/myapp/config

======= Metadata =======
Key                Value
---                -----
created_time       2026-10-01T09:30:12.123456Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

The printed Secret Path is secret/data/myapp/config, which shows the data/ segment that the CLI added. Note version 1. Read it back:

vault kv get -mount=secret myapp/config

======== Secret Path ========
secret/data/myapp/config

======= Metadata =======
Key                Value
---                -----
created_time       2026-10-01T09:30:12.123456Z
version            1

====== Data ======
Key         Value
---         -----
password    s3cr3t
username    app

To pull out a single value without the decoration, which is what scripts want, use -field:

vault kv get -mount=secret -field=password myapp/config

s3cr3t

Now change a value. There is a trap here. kv put replaces the whole secret with exactly the keys you give it, so this command silently drops username:

vault kv put -mount=secret myapp/config password=new-password

The result is version 2 with only one key. To change one key and keep the rest, use kv patch:

vault kv patch -mount=secret myapp/config password=another-password

That produces version 3, changing only password. Because the earlier put had already dropped username, it is still gone, and patch cannot bring it back. The lesson is to reach for patch whenever you mean "update this one field", and to use put only when you intend to replace everything.

Each write made a new version, and you can read an older one:

vault kv get -mount=secret -version=1 myapp/config
vault kv metadata get -mount=secret myapp/config

The metadata command shows every version, when it was created and whether it was deleted or destroyed. KV version 2 keeps the latest 10 versions by default. You can change that per secret, and you can make old versions expire:

vault kv metadata put -mount=secret -max-versions=5 -delete-version-after=720h myapp/config

Listing works on a path prefix:

vault kv list -mount=secret myapp

Keys
----
config

A trailing slash on a listed key means it is a folder rather than a secret.

Three ways to get rid of a secret

Version 2 distinguishes between removing and erasing:

vault kv delete -mount=secret myapp/config                       # soft delete the latest version
vault kv undelete -mount=secret -versions=3 myapp/config         # bring it back
vault kv destroy -mount=secret -versions=1 myapp/config          # permanent, cannot be undone
vault kv metadata delete -mount=secret myapp/config              # every version and the metadata

A soft delete marks a version as deleted but keeps the data, so undelete can restore it. Destroy wipes the data of the given versions for good. kv metadata delete removes the secret entirely. There is also vault kv rollback, which writes an old version's contents as a new version:

vault kv rollback -mount=secret -version=1 myapp/config

If you want to use a KV engine at another path, mount it yourself. The type name kv-v2 gives a version 2 engine, and so does -version=2 kv; a plain kv without the version flag gives the older version 1 engine:

vault secrets enable -path=team-a kv-v2
vault kv put -mount=team-a payments/stripe api_key=sk_test_example

Policies: deciding who may do what

So far you have acted as root, which bypasses every check. The real value of Vault appears the moment you stop doing that. A policy is how you describe the access a person or application should have, and it is usually short.

Here is a policy for an application called myapp. Save it as myapp.hcl:

path "secret/data/myapp/*" {
  capabilities = ["create", "read", "update", "patch", "delete"]
}

path "secret/metadata/myapp/*" {
  capabilities = ["list", "read"]
}

Read it as two rules. The first says: on any path beneath secret/data/myapp/, the token may create, read, update, patch and delete. The second says: on secret/metadata/myapp/..., it may list and read. Anything else is forbidden, because policies deny by default.

Why two paths? This is the KV version 2 detail promised earlier. The engine exposes the actual values under data/ and the bookkeeping (version list, key listing) under metadata/. A policy that mentions only secret/myapp/* matches nothing and gives a permanent "permission denied". Vault also has delete/, undelete/ and destroy/ paths for the matching operations, so grant those deliberately, not by accident.

Write the policy to Vault, and list what exists:

vault policy write myapp myapp.hcl
vault policy list
vault policy read myapp

Success! Uploaded policy: myapp

The command vault policy fmt myapp.hcl tidies formatting, and vault policy delete myapp removes it.

Path matching rules

Paths in a policy follow a few rules that are worth knowing precisely:

• An exact path, such as secret/data/myapp/config, matches that path only.
• A trailing *, as in secret/data/myapp/*, matches everything beneath the prefix. The asterisk is only allowed at the end.
• A + matches exactly one path segment, so secret/data/+/shared matches secret/data/teamA/shared and secret/data/teamB/shared, but not secret/data/teamA/x/shared.
• When several rules match a path, Vault uses the most specific one. A deny capability always overrides everything else.

path "secret/data/+/shared" {
  capabilities = ["read"]
}

The capabilities, and how they map to HTTP

Capabilities correspond to HTTP verbs and to the CLI commands you run. read is an HTTP GET. create and update are POST or PUT, and Vault picks between them depending on whether the path already holds data. delete is DELETE. list is the LIST verb used by vault kv list. patch is an HTTP PATCH, used by vault kv patch. sudo is needed for a few protected administrative paths, and deny refuses outright.

A subtle example: reading a dynamic credential such as database/creds/app needs only read, even though that request creates a new database user. Vault models it as a read of a path that generates a value.

Testing a policy before you trust it

Policies are code, and you should test them. Create a token that carries only your policy and use it:

vault token create -policy=myapp -ttl=1h

Key                  Value
---                  -----
token                hvs.CAESI...
token_accessor       8Gk2...
token_duration       1h
token_renewable      true
token_policies       ["default" "myapp"]
identity_policies    []
policies             ["default" "myapp"]

Notice that the token has both myapp and default. Now try it, using a variable to keep this token separate from your root token:

VAULT_TOKEN=hvs.CAESI... vault kv get -mount=secret myapp/config
VAULT_TOKEN=hvs.CAESI... vault kv get -mount=secret other/config

The first works. The second fails:

Error reading secret/data/other/config: Error making API request.

URL: GET http://127.0.0.1:8200/v1/secret/data/other/config
Code: 403. Errors:

* 1 error occurred:
	* permission denied

Read that message carefully, because you will see it constantly. The URL shows the real path (secret/data/other/config), the Code: 403 is the HTTP status, and permission denied means the token authenticated fine but its policies do not allow this path. A missing or bad token gives a different text, which we will compare in the errors section.

Two more tools shorten the guessing. You can ask Vault what a token may do on a path, and you can print the policy a command would need:

vault token capabilities hvs.CAESI... secret/data/myapp/config
vault kv get -output-policy -mount=secret myapp/config

The first prints the capabilities (for example create, delete, patch, read, update). The second prints a ready-made policy stanza for the command, which is a clever way to start writing a policy: run the command you want to allow with -output-policy, and paste the result.

Authentication: how people and applications log in

Using the root token to do everything defeats the point. Real users and applications log in through an auth method. Each method proves an identity in its own way and ends by issuing a token that carries policies.

List what is enabled:

vault auth list

Path      Type     Accessor               Description
----      ----     --------               -----------
token/    token    auth_token_a1b2c3d4    token based credentials

The token method is always present and cannot be removed. It is what you have been using.

Userpass: a good first method for learning

The userpass method stores usernames and passwords inside Vault. It is simple, which makes it ideal for practising, though most companies prefer single sign-on through OIDC or LDAP for people.

vault auth enable userpass
vault write auth/userpass/users/alice password=change-me-please token_policies=myapp

Two points about that command. First, the word write is generic. In Vault you configure almost everything by writing to a path, and the path here is auth/userpass/users/alice. Second, token_policies lists the policies the login token will carry. Now log in as Alice:

vault login -method=userpass username=alice

Password (will be hidden):
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                    Value
---                    -----
token                  hvs.CAESI...
token_accessor         9Lm3...
token_duration         768h
token_renewable        true
token_policies         ["default" "myapp"]
identity_policies      []
policies               ["default" "myapp"]
token_meta_username    alice

The CLI saved this token in ~/.vault-token. One surprise here: if VAULT_TOKEN is still exported as root, the environment variable takes priority over the saved token, and you are still root. Run unset VAULT_TOKEN to use the token you just earned. Then check who you are:

vault token lookup

The output lists the token's policies, TTL, creation time, and whether it is renewable. Alice can now read secret/data/myapp/config and nothing else. That is the security model in action.

The same login through the HTTP API, with no CLI involved, is a POST to the login path. The response contains the token at .auth.client_token:

curl --request POST --data '{"password": "change-me-please"}' \
  http://127.0.0.1:8200/v1/auth/userpass/login/alice

Logging in with a token you already have

Since tokens are the universal credential, vault login with no method takes a token and stores it:

vault login

You can also log in by passing the token value, but that exposes it in your shell history, so prefer the prompt. To discard the stored token, delete ~/.vault-token.

Other methods you will meet

You do not need these yet, but recognise the names. AppRole is for machines and automation: a role ID plus a secret ID. Kubernetes lets a pod log in with its service account. JWT/OIDC connects to single sign-on providers and to CI systems such as GitHub Actions and GitLab, so that a pipeline can log in with a short-lived identity token instead of a stored password. AWS, GCP and Azure methods let a workload log in with its cloud identity. LDAP, GitHub, Okta and TLS certificates cover other sources. The mid-level guide builds AppRole, Kubernetes and JWT step by step.

The principle behind all of them is the same. The best credential for a workload is one it already has for another reason, such as its Kubernetes service account or its cloud identity. A method that needs you to distribute a password to the client has just moved the secret-management problem one step, and later guides give it a name: the secret zero problem.

Tokens, TTLs and leases in practice

Tokens are not forever, and that is deliberate. A leaked token that expires in an hour is a small problem. A leaked token that lives for a month is a large one. Understanding TTLs is the difference between a Vault that feels helpful and one that surprises you at 3 a.m.

Create a token yourself, with an explicit policy and lifetime:

vault token create -policy=myapp -ttl=1h
vault token create -policy=myapp -ttl=10m -use-limit=3

The second token expires in ten minutes and stops working after three uses. Short lifetimes and use limits are common defences for one-off jobs.

TTL, renewal and maximum TTL

A token has a TTL (how long until it expires) and a maximum TTL. While a token is renewable and under its maximum, you can extend it:

vault token renew
vault token renew -increment=30m hvs.CAESI...

Renewing resets the TTL but never beyond the maximum. When the maximum is reached, the token dies and the client must log in again. The system default TTL and maximum are both 768 hours; you can change them per mount, for example vault auth tune -default-lease-ttl=1h -max-lease-ttl=24h userpass/. A periodic token works differently: each renewal resets it to its period, so a long-running service can keep one alive indefinitely as long as it renews on time. An explicit_max_ttl is a hard cap that not even renewal can pass.

If you request a longer TTL than the maximum allows, Vault does not fail; it caps the value and prints a warning such as "TTL of ... exceeded the effective max_ttl ...; TTL value is capped accordingly". Read warnings like that, they tell you the value you asked for is not the value you got.

Looking at and revoking tokens

vault token lookup
vault token lookup hvs.CAESI...
vault token lookup -accessor 9Lm3...
vault token revoke hvs.CAESI...
vault token revoke -accessor 9Lm3...

The accessor-based versions are what you use when you know there is a token to cancel but you must not see the token itself, for example when going through an audit log. Revoking a token also revokes all child tokens and every lease it created, so revoke high in the hierarchy when dealing with an incident.

Leases: the same idea for dynamic secrets

When Vault creates a dynamic secret, it attaches a lease. Suppose you later enable the database engine and read a credential. The response contains a lease_id, a duration and a flag saying whether you can renew. The commands are symmetrical with tokens:

vault lease lookup <lease_id>
vault lease renew -increment=1h <lease_id>
vault lease revoke <lease_id>
vault lease revoke -prefix database/creds/app

The last command is the emergency button: it revokes every outstanding credential beneath a prefix, and Vault removes the matching database users. A team that can run that command in seconds has a much better incident story than one that must rotate a shared password across twenty services.

A quick look at dynamic secrets

You do not need a database to grasp the idea. The flow is: an administrator configures the engine with a connection and a role that holds a SQL template, and an application reads database/creds/<role>. Vault connects, creates a unique user with a short life, and returns the username and password. When the lease ends, Vault drops that user. No human ever sees or stores the password, and every application instance has a different one. The mid-level guide shows the full setup. For now, hold on to the contrast with KV, where you store a secret and it stays the same until somebody changes it.

Talking to Vault over HTTP

Every CLI command is an HTTP request in disguise. Knowing this demystifies Vault, and it is how your applications will talk to it, using a client library or plain HTTP.

The API lives at /v1/. A token goes in the X-Vault-Token header. To read the secret from earlier:

curl --header "X-Vault-Token: $VAULT_TOKEN" $VAULT_ADDR/v1/secret/data/myapp/config

The JSON response has the structure that catches people out:

{
  "request_id": "0f3a...",
  "lease_id": "",
  "renewable": false,
  "lease_duration": 0,
  "data": {
    "data": {
      "password": "s3cr3t",
      "username": "app"
    },
    "metadata": {
      "created_time": "2026-10-01T09:30:12.123456Z",
      "version": 1
    }
  }
}

There are two data keys. The outer one is Vault's standard response envelope, used by every engine. The inner one is KV version 2's wrapper around your key-value pairs, alongside metadata. With jq you pull the password with .data.data.password. KV version 1 has no inner wrapper.

To write, send a POST with the same inner data wrapper:

curl --header "X-Vault-Token: $VAULT_TOKEN" --request POST \
  --data '{"data": {"username": "app", "password": "s3cr3t"}}' \
  $VAULT_ADDR/v1/secret/data/myapp/config

Other endpoints you will want early are the health check and seal status, neither of which needs a token:

curl $VAULT_ADDR/v1/sys/health
curl $VAULT_ADDR/v1/sys/seal-status

The health endpoint's status code carries meaning, which is what load balancers and monitors rely on: 200 means the node is active and unsealed, 429 an unsealed standby, 501 not initialised, and 503 sealed.

Remember the trick from earlier: any CLI command plus -output-curl-string prints the equivalent request rather than running it.

vault kv get -mount=secret -output-curl-string myapp/config

curl -H "X-Vault-Request: true" -H "X-Vault-Token: $(vault print token)" http://127.0.0.1:8200/v1/secret/data/myapp/config

Since Vault 2.0, request paths must be clean. A path containing //, /./ or /../ is rejected. If your script builds URLs by joining strings, such as adding a trailing slash to a mount and then a leading slash to a path, it will break on 2.x with a clear refusal. Strip the stray slashes.

Audit logs: proving who did what

A secrets server that cannot tell you who read a secret is only half a security tool. Audit devices record every request and response. You enable one much like an auth method:

vault audit enable file file_path=/tmp/vault-audit.log
vault audit list -detailed

On a dev server, /tmp is fine. In real deployments the log goes to a protected directory, or to syslog, or to a socket that feeds a central logging system. Each line is a JSON object describing one request or response. The entries contain the path, the operation, the client's identity and the result. Secret values and tokens are not written in plain text: Vault replaces them with an HMAC-SHA256 hash, so you can correlate "the same value was used twice" without the log becoming a second copy of your secrets.

Do a read and look at the log:

vault kv get -mount=secret myapp/config
tail -n 2 /tmp/vault-audit.log | jq .

You will see a request entry and a response entry. The request shows "operation": "read" and "path": "secret/data/myapp/config", plus the display_name of the token holder (for example userpass-alice), and the values in the response are hashed strings beginning with hmac-sha256:.

There is a rule to learn now, because it explains a famous class of outages. If at least one audit device is enabled and none of them can write, Vault stops answering requests. It fails closed: no audit record means no request is served. A full disk that holds the audit log can therefore take down every application that depends on Vault. This is why production guides say to run at least two audit devices, to watch the disk and to alert on audit failures.

Configuration: from dev mode to a real server

A development server needs no configuration file. A real server reads one. Even if you do not run production yourself, you will read these files in your job, so know what the pieces are.

A minimal single-node configuration using Integrated Storage (Raft) looks like this:

ui            = true
disable_mlock = true
api_addr      = "https://vault.example.com:8200"
cluster_addr  = "https://vault.example.com:8201"

storage "raft" {
  path    = "/opt/vault/data"
  node_id = "vault-1"
}

listener "tcp" {
  address       = "0.0.0.0:8200"
  tls_cert_file = "/opt/vault/tls/tls.crt"
  tls_key_file  = "/opt/vault/tls/tls.key"
}

Go through it piece by piece.

• ui = true serves the web interface on the same port as the API.
• disable_mlock controls whether Vault asks the operating system to keep its memory out of swap. With Integrated Storage, the setting has no default since Vault 1.20, so the server refuses to start until you set it explicitly to true or false. In containers it is normally true, and you then disable swap on the host or encrypt it.
• api_addr is the address that clients should use, and standby nodes use it to redirect clients to the active node. cluster_addr is the address nodes use to talk to each other, on port 8201. Raft requires it.
• storage "raft" puts the encrypted data on local disk and lets several Vault nodes form a cluster. path is where the data lives and node_id names this node. Every production cluster needs three or five nodes, and the operator guide later in the series covers it.
• listener "tcp" is the network endpoint. The certificate and key files give it TLS. The setting tls_disable = true exists, but it belongs only in a dev setting.
• An optional seal "awskms" { ... } (or azurekeyvault, gcpckms, ocikms, transit) block enables auto-unseal.

Run it, point the CLI at it, and initialise:

vault server -config=/etc/vault.d/vault.hcl
export VAULT_ADDR='https://vault.example.com:8200'
vault operator init

Unseal Key 1: 8kQ...
Unseal Key 2: 3vP...
Unseal Key 3: Zx1...
Unseal Key 4: mT9...
Unseal Key 5: Lw7...

Initial Root Token: hvs.xxxxxxxxxxxxxxxxxxxxxxxx

Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.

That output appears exactly once. Vault does not keep the unseal keys or show them again. Give each share to a different trusted person, and store the initial root token only long enough to set up real administrators. Then unseal by running the next command three times, each time with a different share:

vault operator unseal

After the third share, vault status shows Sealed false. On a server with auto-unseal you skip this step, because init produces recovery keys and the server unseals itself on start.

If Vault does not start, or you want to check a config file before using it, vault operator diagnose -config=/etc/vault.d/vault.hcl reports problems such as unreadable certificates, a missing cluster_addr or a storage path it cannot write.

The installer packages on Linux create a vault system user, a systemd service and a data directory. The service runs vault server against /etc/vault.d/vault.hcl, and you manage it with sudo systemctl start vault and sudo journalctl -u vault. Two security habits from the official hardening guide are worth memorising at this stage: run Vault as an unprivileged user, and never leave the root token around.

If you run Vault on Kubernetes, the official Helm chart deploys it. The default chart values start a single node with file storage, which HashiCorp itself says is not production-ready; a real install turns on high availability with Raft (server.ha.enabled=true and server.ha.raft.enabled=true). The chart version at the time of writing deploys a slightly older Vault than 2.1.1 unless you pin server.image.tag.

Environment variables configure the client, and you will set them in every terminal:

Reading the errors Vault gives you

Most beginner frustration with Vault comes from five or six messages. Each one has a cause that you can identify quickly once you know what the message is telling you.

The distinctions matter. permission denied and missing client token both use HTTP 403, yet they point in opposite directions. The first says you are known but not allowed; the second says you never identified yourself. no handler for route is the same story at another level: the path did not reach any engine at all, so the problem is not policy but a typo or an engine that was never enabled.

Follow a fixed order when a read fails, and you will rarely be stuck for long:

1. Is the server reachable, and does vault status say Sealed false?
2. Is the token what you think it is? Run vault token lookup and check policies and TTL. Check that VAULT_TOKEN is not overriding the token you logged in with.
3. Is the path right? Run vault secrets list and check that the mount exists and that the KV version matches your path style.
4. What does the policy allow? Use vault token capabilities <path>, and use vault read sys/internal/ui/resultant-acl to see the effective policy of your current token.
5. Still unclear? Use -output-curl-string to see the real HTTP request, and read the audit log.

One error worth singling out is this one from the KV CLI: preflight capability check returned 403, please ensure client's policies grant access to path "secret/". The CLI asks Vault which KV version a mount uses, and it needs read on sys/internal/ui/mounts/secret to do so. The default policy normally covers this. If a custom setup leaves it out, either add that read or call the explicit API path.

Putting it all together

Here is a small end-to-end project that uses every idea in this guide. Suppose a small team builds a service called orders and needs a database password and an API key. You are the Vault administrator, and two people will use the secrets: a developer, Dina, who may read and update them, and a read-only auditor, Omar.

Start a dev server in one terminal with vault server -dev -dev-root-token-id=root. In another, set up the client:

export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_TOKEN='root'
vault audit enable file file_path=/tmp/vault-audit.log
vault secrets enable -path=orders kv-v2

Store the secrets under a path that reflects the service:

vault kv put -mount=orders prod/database username=orders_app password=Pa55-from-admin
vault kv put -mount=orders prod/payments api_key=sk_test_example

Write two policies, one for each role:

path "orders/data/prod/*" {
  capabilities = ["create", "read", "update", "patch"]
}

path "orders/metadata/prod/*" {
  capabilities = ["list", "read"]
}

path "orders/data/prod/*" {
  capabilities = ["read"]
}

path "orders/metadata/prod/*" {
  capabilities = ["list", "read"]
}

Upload the policies and create both users:

vault policy write orders-dev orders-dev.hcl
vault policy write orders-audit orders-audit.hcl
vault auth enable userpass
vault write auth/userpass/users/dina password=dina-demo-pass token_policies=orders-dev token_ttl=1h
vault write auth/userpass/users/omar password=omar-demo-pass token_policies=orders-audit token_ttl=1h

Now act as each person. Log in as Dina, rotate the API key with patch, and list the secrets:

unset VAULT_TOKEN
vault login -method=userpass username=dina
vault kv patch -mount=orders prod/payments api_key=sk_test_rotated
vault kv list -mount=orders prod

Then log in as Omar. He can read, but any write is refused:

vault login -method=userpass username=omar
vault kv get -mount=orders -field=api_key prod/payments
vault kv put -mount=orders prod/payments api_key=nope

The last command fails with permission denied, and that is the right result. Finally, look at the audit log to see the trail and to confirm it recorded Omar's refused write with his identity:

grep '"operation":"update"' /tmp/vault-audit.log | jq -r '.auth.display_name + "  " + .request.path + "  " + (.error // "ok")' | tail -n 3

Finish with cleanup that shows the lifecycle: look at Dina's token, revoke it, and watch her next request fail.

vault login -method=userpass username=dina
vault token revoke -self
vault kv list -mount=orders prod

You have just run the full loop. Secrets were stored in a central place, two identities received least-privilege access, every action was logged, and a token was revoked on demand. Replace the two human users with AppRole or Kubernetes login for applications, and the shape of a real deployment appears.

What you can now do, and what comes next

You can now explain what Vault is and the problem it solves, and trace a request from login through token and policy to a secrets engine. You can install it, run a development server, and tell dev mode apart from a real deployment. You can store, version, patch, delete and restore secrets in KV version 2, and you know why its paths contain data/. You can write a policy, test it with a real token, and read permission denied as a statement about policy rather than about passwords. You can log in through userpass, control token lifetimes, renew and revoke tokens, call the HTTP API directly, turn on an audit log, and read a basic server configuration.

Some facts from this guide are worth keeping in your head as a short list. The CLI defaults to HTTPS, so VAULT_ADDR matters. kv put replaces and kv patch merges. KV version 2 policies need data/ and metadata/. Root tokens are for setup, not for daily use. A sealed server is a healthy server waiting for keys. And if audit devices are enabled but cannot write, Vault stops serving requests.

What comes next depends on where you work:

• Mid-level covers machine authentication (AppRole, Kubernetes, JWT for CI), dynamic database credentials, Transit encryption, PKI certificates, response wrapping, and Vault Agent and the Vault Secrets Operator for delivering secrets to applications.
• Senior covers running Vault as a platform: Raft clusters, auto-unseal, upgrades, backup and restore, monitoring, multi-tenancy and incident playbooks.
• Neighbouring guides in this catalogue fit naturally. Vault is often deployed with Kubernetes and configured as code with Terraform, and the applications that use it often run in Docker containers.

For teams in the Gulf and Egypt, one practical point recurs in security reviews: where secrets and their encryption keys physically live. A self-managed Vault lets you keep the cluster, the audit logs and the auto-unseal key in a cloud region inside your data-residency boundary, which is often the reason a regulated employer prefers it to a managed secrets product in another jurisdiction. Raise that question early when you design a deployment.

Two last reminders about the ecosystem. Older tutorials may show Vault 1.x details (tokens starting s., unauthenticated rekey endpoints, or the HCP Vault Secrets service, which has reached end of life). And the quickest way to stay correct is to read the release notes at the start of any new project.

Sources

• Vault documentation
• Install Vault
• Vault release notes
• Important changes in Vault
• Seal and unseal concepts
• Tokens
• Policies
• KV version 2 secrets engine
• Audit devices
• Server configuration
• Integrated Storage (Raft) configuration
• Production hardening
• Health API
• CLI command reference
• Deploying Vault on Kubernetes with Helm

Part one of three. Vault interviews at the beginner level test whether you understand a small model well enough to reason with it, not whether you memorised commands. If you can explain how a request travels from login to secret, and why a server can be sealed, you can answer most of what follows. This guide gives you the short answer, the model, the commands to have ready, the traps, and model answers to the questions that come up most.

The guide targets Vault 2.1.1. Some interviewers still think in Vault 1.x terms, so where the current answer differs from old advice, the text says so. Saying "that changed in 2.0, here is the current behaviour" is a strong move, provided you are right.

The thirty-second answer

What is Vault? Vault is a server that stores and generates secrets, such as passwords, API keys, certificates and encryption keys, and controls who can reach each one. Clients log in, receive a token, and policies attached to that token decide which paths they may read or write, with every request written to an audit log.

Why would a team use it? It replaces secrets scattered through code, environment variables and chat with one audited, access-controlled service. It can also create short-lived credentials on demand, such as a database user that expires after an hour, so there is less to steal and less to rotate.

Say those two answers aloud until they take under thirty seconds. They give the interviewer a base to build on, and every later answer connects back to them.

The model in one picture

Below the picture sits the barrier. Everything Vault writes to its storage backend is encrypted with AES-256-GCM, and the storage is treated as untrusted. On startup Vault is sealed: it can see the encrypted data but not decrypt it, until enough unseal key shares (or an auto-unseal key service) release the root key.

Memorise the nouns, because the questions are built from them:

• Path. Every feature sits at /v1/<path>. Policies, mounts and permissions are all about paths.
• Secrets engine. A component mounted at a path. Static (KV), dynamic (database, cloud, PKI) or encryption as a service (Transit).
• Auth method. Verifies identity at a path under auth/: userpass, AppRole, Kubernetes, JWT/OIDC, LDAP, cloud identity.
• Token. The credential on every request. Service tokens start with hvs., batch tokens with hvb.. Tokens have a TTL, may have a parent, and have an accessor.
• Policy. HCL that lists paths and capabilities (create, read, update, patch, delete, list, sudo, deny). Deny by default; deny always wins.
• Lease. The life span attached to dynamic secrets and service tokens. Leases can be renewed and revoked, including by prefix.
• Seal and unseal. The server-level lock. Shamir splits the unseal key into five shares with a threshold of three by default.
• Audit device. A log of every request and response, with secret values hashed.

Commands to have ready

An interviewer who asks you to "show me" on a whiteboard wants fluent basics. Know these without looking.

vault server -dev                                   # throwaway, in-memory, unsealed
export VAULT_ADDR='http://127.0.0.1:8200'           # dev server is HTTP; the CLI defaults to HTTPS
vault status
vault secrets list
vault secrets enable -path=kv kv-v2
vault kv put -mount=secret myapp/config username=app password=s3cr3t
vault kv get -mount=secret -field=password myapp/config
vault kv patch -mount=secret myapp/config password=new
vault kv metadata get -mount=secret myapp/config
vault kv delete -mount=secret myapp/config
vault kv undelete -mount=secret -versions=2 myapp/config
vault kv destroy -mount=secret -versions=1 myapp/config
vault policy write myapp myapp.hcl
vault auth enable userpass
vault write auth/userpass/users/alice password=... token_policies=myapp
vault login -method=userpass username=alice
vault token create -policy=myapp -ttl=1h
vault token lookup
vault token revoke <token>
vault operator init
vault operator unseal

And a policy to write from memory:

path "secret/data/myapp/*" {
  capabilities = ["create", "read", "update", "patch", "delete"]
}

path "secret/metadata/myapp/*" {
  capabilities = ["list", "read"]
}

The two paths are not a detail; they are a classic test of whether you have really used KV version 2.

The classic traps, and why they happen

Interviewers like traps because they reveal whether you have used the tool. Each trap below comes with its reason, since the reason is what gets you the marks.

Trap 1: the KV version 2 data/ path. The KV v2 engine exposes values at secret/data/<path> and metadata at secret/metadata/<path>. The CLI adds data/ for you, so commands work, but a policy written against secret/myapp/* matches nothing and everything returns permission denied. The reason is that v2 separates data from metadata so that it can version and list without leaking values.

Trap 2: kv put replaces. vault kv put writes a new version containing only the keys you supply; all other keys are gone from the latest version. The older version still exists, so you can roll back, but it surprises people who expected a merge. The fix is vault kv patch.

Trap 3: HTTP versus HTTPS. The CLI defaults to https://127.0.0.1:8200. A dev server speaks plain HTTP. Forgetting VAULT_ADDR gives http: server gave HTTP response to HTTPS client.

Trap 4: the environment variable beats the saved token. After vault login, the token lives in ~/.vault-token. If VAULT_TOKEN is still exported with a root token, the variable wins and you keep acting as root, which hides policy mistakes.

Trap 5: dev mode is not a deployment. It stores data in memory, runs unsealed, serves HTTP and prints a root token. Everything that makes it convenient makes it unsafe.

Trap 6: the root token is a leftover. Initialisation produces a root token for setup. Leaving it in use or in a file defeats least privilege. Revoke it, and generate a new one (with a quorum of key holders) only when truly needed. Since Vault 2.0, generate-root and rekey also require a token by default, where before 2.0 they were unauthenticated.

Trap 7: audit fails closed. If an audit device is enabled and none can write, Vault refuses requests. A full disk can therefore cause an outage, and the lesson is to run two audit devices and monitor them.

Trap 8: a sealed server after every restart. With Shamir keys, a restart leaves the node sealed until enough shares arrive. People see "Vault is sealed" (HTTP 503) and think it is broken. It is healthy and locked. Production avoids the manual step with auto-unseal.

Common interview questions

What problem does Vault solve, and what did teams do before?

Teams used to keep secrets in source code, environment variables, CI settings and shared documents. Those places have no central control, no audit trail, rarely rotate, and give everyone with file access every secret. Vault centralises secrets, requires identity for each request, applies least-privilege policies, logs every request, and can issue short-lived dynamic credentials so there is nothing long-lived to leak. A strong answer adds that Vault does not remove the need for good habits: a secret that is read and pasted into a chat is still exposed.

Explain the path from a login to a secret.

The client authenticates through an auth method, say userpass or Kubernetes. Vault verifies the identity and issues a token, with the policies configured for that identity attached. The client sends the token with each request. Vault resolves the policies, checks whether the requested path and operation are allowed, and if so routes the request to the secrets engine mounted at that prefix. The request and response are written to the audit log. Every step can be inspected: vault token lookup shows the token, vault token capabilities shows what it can do on a path, and the audit log shows the history.

What are the main kinds of secrets engines?

Static engines, mostly KV, store values you put in. Dynamic engines, such as database, AWS, GCP, Azure, Kubernetes, SSH and PKI, create credentials on request with a lease and remove them when the lease ends. Encryption-as-a-service engines, mainly Transit, encrypt and decrypt data for an application without handing over the key. At a beginner level you are expected to use KV and to describe dynamic secrets in principle.

What is the difference between KV version 1 and version 2?

Version 1 stores one value per path with no history. Version 2 keeps versions (ten by default), supports soft delete, undelete and permanent destroy, stores metadata, allows rollback and supports check-and-set. The API paths differ: version 2 uses data/ and metadata/, so policies have to name those. allowed_parameters and denied_parameters do not work with KV v2. The default in dev mode is v2, and v2 is what most teams use.

What does it mean that Vault is sealed?

When Vault starts, its storage is encrypted and the decryption key is not in memory, so the server is sealed and cannot read or write secrets. Three layers of keys exist: the encryption key protects data, the root key protects the encryption key, and the unseal key (or an auto-unseal service) protects the root key. By default vault operator init splits the unseal key into five shares, and any three unseal the server. The design means a stolen disk or backup is useless without the shares. Auto-unseal, using a cloud KMS or an HSM, removes the manual step; in that mode init issues recovery keys, which cannot decrypt anything.

How do you unseal a Vault, and what if you lose the keys?

Run vault operator unseal and enter a different key share each time until the threshold is reached. If you lose enough shares that fewer than the threshold remain, you cannot unseal, and because Vault does not store the keys anywhere else, the encrypted data is unrecoverable. That is why shares go to separate custodians, and why production usually relies on auto-unseal with recovery keys stored by process. With auto-unseal you do not need shares on restart, but you must protect access to the KMS key.

What is a token, and what kinds exist?

A token is the credential sent on each request, in the X-Vault-Token header. Service tokens (hvs.) are stored, renewable, revocable and can have children. Batch tokens (hvb.) are encrypted blobs that Vault does not store, which makes them cheap at high volume, but they cannot be renewed or revoked. Every non-root token has a TTL and a maximum TTL. Revoking a parent revokes its children. An accessor lets you look up or revoke a token without the token value.

What is a policy, and how does path matching work?

A policy is a set of rules that give capabilities on paths. Everything is denied unless a rule allows it. A trailing * matches anything under a prefix, a + matches exactly one path segment, and the most specific rule wins. deny overrides everything. You attach policies to identities through auth methods, for example through token_policies on a userpass user. A strong answer mentions the KV v2 data/ path and shows a two-path policy.

Why should you not use the root token every day?

It bypasses every policy, so everything done with it is unrestricted and, if stolen, total. Best practice is to use it for initial setup, create scoped administrative identities, and revoke it. If a root token is needed again, it is regenerated with vault operator generate-root, which requires a quorum of unseal or recovery key holders and, from Vault 2.0, also a token.

What is a lease?

A lease is the metadata that goes with a dynamic secret or service token: an ID, a TTL and whether it can be renewed. When the lease ends, Vault revokes the credential, for example by deleting the database user it created. You can renew a lease up to its maximum, revoke a single lease, or revoke all under a prefix with vault lease revoke -prefix, which is an incident-response tool.

What is the difference between a dynamic secret and a static one?

A static secret, such as a KV entry, is a value someone stored, and it stays the same until changed. A dynamic secret is generated when requested, is unique to the requester, and expires. Dynamic secrets shrink the exposure window, make the audit trail more precise (each credential maps to one requester) and remove the burden of rotation. They need Vault to have privileges on the target system, so the root credential Vault holds has to be protected and rotated.

How do applications authenticate to Vault?

Not with a human password. They use machine identities: AppRole (a role ID and a secret ID), Kubernetes service accounts, cloud identities for AWS, GCP and Azure, or JWT tokens issued by a CI platform. The aim is to use an identity the workload already has, because any method that needs a pre-shared secret leaves a "secret zero" problem: how do you deliver the first secret safely? Mentioning response wrapping for delivering an AppRole secret ID shows awareness.

What is an audit device and why does it matter?

An audit device records each request and response to a file, syslog or socket. Secret values are HMAC-hashed so the log does not reveal them. It matters for security investigations and compliance. Vault fails closed, so when audit devices are enabled but none can write, Vault stops serving requests. Good practice is at least two devices and alerts on audit write failures.

How does Vault differ from a cloud secrets manager?

Cloud secrets managers, such as AWS Secrets Manager, store secrets well inside one cloud. Vault is cloud-neutral, has a rich model of dynamic secrets, encryption as a service, a PKI and a unified policy language across providers, and you can run it where you choose. The cost is operating it: clustering, upgrades, unsealing and backups. For a single-cloud team with simple needs, the managed option can be the right answer, and saying so shows judgment. Vault can also push secrets into those managers through Secrets Sync, which is an Enterprise feature.

What changed in Vault 2.0 that a beginner should know?

The main items are these. sys/rekey and sys/generate-root now require a token, where before 2.0 they needed only key shares. Request paths with //, /./ or /../ are rejected. The official container image runs as the vault user, and from 2.0.2 it cannot lock memory, so container deployments set disable_mlock = true. The release also adopted IBM's support lifecycle. The HTTP API is still /v1 and the CLI is largely unchanged. Do not claim more than that unless you have checked the release notes.

Is Vault open source?

Since version 1.15 (August 2023) the source is under the Business Source License 1.1, which is source-available with restrictions on competing commercial use, not an OSI open source license. The free edition is Vault Community Edition. The community fork OpenBao continues the last MPL 2.0 code under the Linux Foundation. Show you know they are different products.

How would you give a CI pipeline access to a secret?

The best answer avoids a stored long-lived token. Use the JWT or OIDC auth method so that the CI platform proves its identity with a short-lived signed token, bound to a repository or branch, and Vault returns a token with a short TTL and a narrow policy. At beginner level it is acceptable to say you would create a dedicated token or AppRole with a minimal policy and a short TTL, while noting that JWT is the better design.

What would you check if a read returns permission denied?

Work through it in order. First, who am I: vault token lookup (is it the token I expect, and has it expired; is VAULT_TOKEN overriding my login?). Second, what do my policies allow on that path: vault token capabilities <token> <path>, or vault read sys/internal/ui/resultant-acl. Third, is the path right, including data/ for KV v2, and is the engine mounted at that prefix? Fourth, is a namespace involved? Only then change the policy, adding the smallest rule that fixes it.

How do you keep secrets out of your shell history?

Do not pass a secret as a command-line argument. Read it from a file or from standard input, and keep shell history disabled or filtered for sessions that handle secrets. The CLI can read a value from stdin when you pass - for it. In automation, fetch values with a script that never echoes them.

Questions to ask back

At the end of an interview you can ask informed questions, and doing so signals experience. Try: "Is the cluster auto-unsealed, and who holds the recovery keys?", "Which auth methods do applications use, and is there any long-lived AppRole secret?", "Are there two audit devices, and does anyone alert on audit failures?", "Do you use dynamic secrets, or mostly KV?" and "What is your upgrade process, and have you read the 2.x breaking changes?"

What a strong answer covers that a weak one misses

A weak answer lists commands. A strong answer explains the model: identity, token, policy, path, engine. A weak answer says "Vault encrypts secrets". A strong answer says the barrier encrypts storage, the server is sealed until unsealed, and the threat model assumes the storage is untrusted. A weak answer says "use the root token". A strong answer says you revoke it after setup. A weak answer ignores lifetimes; a strong answer talks about TTL, leases and revocation as the main defence. And a strong answer admits limits: "I have used KV and userpass, and I know dynamic secrets in principle but have not run them in production."

Sixty-second self-test

Answer each one aloud, without notes, in about ten seconds.

1. In one sentence, what is Vault?
2. Name the four boxes from login to secret.
3. What do the prefixes hvs. and hvb. mean?
4. Why does a KV v2 policy mention data/ and metadata/?
5. What is the difference between kv put and kv patch?
6. What does "sealed" mean, and how many shares unseal by default?
7. Why can a full disk stop Vault from serving requests?
8. What environment variable must you set for a dev server, and why?
9. Why should the root token be revoked after setup?
10. Name one thing that changed in Vault 2.0.

If any answer takes more than fifteen seconds, reread the matching section of the detailed guide, then the tips guide. The next part of the series, at mid level, turns these ideas into production patterns: AppRole, Kubernetes and JWT auth, dynamic database credentials, Transit and PKI. If you want to see the neighbouring tools, Kubernetes and Terraform are the most common companions.

Part one of three. Most Vault errors at this level come from four mistakes: forgetting that the CLI defaults to HTTPS, writing policies without the KV version 2 data/ segment, confusing which token is actually in use, and treating dev mode as if it were a real server. Start with the table, then read the longer notes on each mistake, the habits worth building on day one, and the checklist at the end.

Common errors at this level

The mistakes beginners actually make

Each mistake below has the same shape: what you see, what is really happening, and what to do. Read them with a terminal open, and cause each one deliberately at least once. A mistake you have made on purpose is a mistake you recognise at 6 p.m.

Mistake 1: a policy that never matches

Symptom. You wrote a policy for secret/myapp/*, attached it to a user, and every read still says permission denied. Root can read the secret, so the secret exists.

Cause. KV version 2 serves values under secret/data/myapp/... and metadata under secret/metadata/myapp/.... The CLI adds data/ to your command, so the command looks like secret/myapp/config, but the request that Vault checks against policy uses the real path. Your rule matched nothing.

Fix. Write the real paths. Give read on secret/data/myapp/*, and list on secret/metadata/myapp/* if the user needs to list keys.

path "secret/data/myapp/*" {
  capabilities = ["read"]
}

path "secret/metadata/myapp/*" {
  capabilities = ["list"]
}

A faster way to get the right path is vault kv get -output-policy -mount=secret myapp/config, which prints the stanza the command needs. Another is -output-curl-string, which shows the actual URL.

Mistake 2: acting as root without noticing

Symptom. You created a limited user and logged in, yet everything is allowed. A colleague tries the same steps and gets permission denied.

Cause. The environment variable VAULT_TOKEN takes priority over the token that vault login saved in ~/.vault-token. Your shell still has the root token exported from the dev-server setup.

Fix. Run unset VAULT_TOKEN, then vault token lookup and read the policies line. Make it a habit to run vault token lookup whenever a permission result surprises you, in either direction. A policy test made with root proves nothing.

Mistake 3: treating dev mode as a real server

Symptom. You built a proof of concept on vault server -dev, restarted the laptop, and every secret is gone. Or a colleague points an application at it, and the application talks HTTP with a root token.

Cause. Dev mode keeps everything in memory, starts unsealed, serves plain HTTP and prints a root token. Those properties are the reason it is easy to start, and the reason it is unsafe to keep.

Fix. Keep dev mode for learning and tests. For anything a teammate will rely on, write a configuration file with real storage (Integrated Storage, Raft), TLS on the listener and a proper initialisation. If you need a long-lived sandbox, run a real single-node server with a config file and accept the unseal step.

Mistake 4: kv put when you meant kv patch

Symptom. You updated one field of a secret and the other keys are gone from the latest version.

Cause. vault kv put writes a new version that contains exactly the keys you passed. It does not merge.

Fix. Use vault kv patch to change selected keys. If you already made the mistake, the previous version is still there: vault kv get -version=N to look, then vault kv rollback -version=N to restore it, then patch properly.

vault kv patch -mount=secret myapp/config password=new-password
vault kv rollback -mount=secret -version=1 myapp/config

Mistake 5: confusing delete, undelete and destroy

Symptom. A secret was "deleted" and then needed again, or was "deleted" and somebody claims it is gone though the data is still there.

Cause. KV v2 has layers. kv delete is a soft delete of the latest version: recoverable with kv undelete. kv destroy permanently removes the data of given versions. kv metadata delete removes all versions and the metadata.

Fix. Use soft delete for normal cleanup. Use destroy only when you need the data gone, for example after a leak, and note that destroying versions does not undo the leak itself: rotate the real credential at its source.

Mistake 6: passing secrets on the command line

Symptom. A secret is visible in history, in ps output on a shared machine, and in the terminal scrollback.

Cause. Anything you type as an argument is stored by your shell and visible to other users of the machine for the duration of the command.

Fix. Read values from a file or from standard input, and keep history off for sensitive sessions. The CLI accepts - for a value to read it from stdin. When you only need to show a value to a script, use -field and keep it in a variable that is never echoed.

Mistake 7: losing the unseal keys and the root token

Symptom. The init output scrolled away, or was pasted into a chat, or the only copy is in a notes file.

Cause. Vault prints the unseal key shares and the initial root token once, and it does not store them. Any person who holds enough shares can unseal the server, so gathering them in one place defeats the point.

Fix. Before you finish setup, distribute each share to a different person through a secure channel, and record who holds which. Use the root token to create real administrative identities, then revoke it. For production, use auto-unseal and keep recovery keys with separate custodians.

Mistake 8: ignoring TTLs until they bite

Symptom. A script that worked all week fails every morning with permission denied.

Cause. The token it logs in with has a TTL. The default is long (768 hours), but a role or mount may set it much shorter, and once it is over, the token is invalid. The error does not say "expired".

Fix. Check vault token lookup for ttl and expire_time. Make the script log in when it starts, or renew before expiry. Do not solve the problem by creating a token that never expires; use a periodic token or a login method that the workload can repeat.

Mistake 9: editing a policy file and expecting Vault to notice

Symptom. You changed myapp.hcl, but behaviour is the same.

Cause. Vault keeps policies in its own storage. A local file is only input to vault policy write.

Fix. Re-upload with vault policy write myapp myapp.hcl, then confirm with vault policy read myapp. Keep policy files in version control and upload them from a pipeline, so the repository and Vault cannot drift apart.

Mistake 10: turning off TLS checks to silence an error

Symptom. A certificate error appears, and a forum post suggests VAULT_SKIP_VERIFY=true. It works, so it stays in the shell profile.

Cause. The client does not trust the CA that signed the server certificate. Often the server uses an internal CA, or the machine sits behind a proxy that re-signs traffic.

Fix. Give the client the CA with VAULT_CACERT=/path/to/ca.pem. If a TLS-inspecting proxy sits in between, add its CA to the trust path instead. Skipping verification is a dev-only convenience, because it lets anyone on the network impersonate Vault.

Mistake 11: expecting old tutorials to match version 2

Symptom. A tutorial from 2022 shows s. tokens, an unauthenticated vault operator rekey, or HCP Vault Secrets, and the commands behave differently.

Cause. Vault moved on. Tokens have used the hvs. prefix since 1.10. From 2.0, sys/rekey and sys/generate-root require a token. Request paths with //, /./ or /../ are rejected. HCP Vault Secrets reached end of life on 1 July 2026. Containers from 2.0.2 cannot lock memory, so they need disable_mlock = true.

Fix. Check the date and the version on a tutorial before you follow it. Start with vault version, then the release notes at developer.hashicorp.com/vault/docs/updates/release-notes.

Habits worth adopting on day one

Think in paths before you type

Before each command, say the path aloud: "I am reading secret/data/myapp/config." If you cannot say the path, you cannot write the policy for it. This habit makes KV v2 paths second nature and makes permission denied easy to debug.

Always check who you are

Run vault token lookup at the start of each session and whenever a result surprises you. It takes one second and ends the whole class of "I thought I was logged in as someone else" problems. Keep a shell prompt or an alias that shows whether VAULT_TOKEN is set.

Write the smallest policy that works

Begin with no access and add paths until the job succeeds. Avoid * on the whole mount. Avoid sudo unless the documentation requires it. Write one policy per role or application, named for what it is for, such as orders-dev or orders-readonly. The -output-policy flag shows what a command needs.

Test a policy with a real token

A policy is code, and code needs tests. Create a token with only that policy, try one allowed request and one forbidden request, and check that each gives the result you expect. Include the forbidden test, because it is the one that proves least privilege.

TOKEN=$(vault token create -policy=myapp -ttl=10m -field=token)
VAULT_TOKEN=$TOKEN vault kv get -mount=secret myapp/config
VAULT_TOKEN=$TOKEN vault kv get -mount=secret other/config

Prefer short lives

Give tokens the shortest TTL that lets the work finish. Use -ttl and -use-limit for one-off jobs. A short TTL limits the damage of a leak and forces the habit of logging in, which you want anyway.

Turn on audit before you need it

An audit device records history from the moment you enable it, not before. On any shared Vault, enable a file device at the start and check that lines appear. On real systems run at least two, and alert when writes fail, because a blocked audit device stops the whole server from answering.

Keep the root token away

Use it to bootstrap, then revoke. Create named administrators with their own policies. If you ever need root again, use vault operator generate-root, which needs a quorum of key holders (and, from 2.0, a token). Write the procedure down before the day you need it.

Keep secrets out of code, history and logs

Read secret values from files or stdin. Do not echo them in scripts. Mask them in CI logs. Never put a Vault token in a repository. If a secret appears in a commit, treat it as leaked: rotate it at the source, then clean the history, in that order.

Put policies and configuration in version control

Policy files, auth method setup and mount lists belong in Git, applied by a pipeline or by Terraform with the hashicorp/vault provider. Then a review can see who asked for access, and you can rebuild Vault's configuration after a disaster. See the Terraform guide for the tool that commonly does this.

Learn the API through the CLI

Run commands with -output-curl-string and read the request. Applications call the HTTP API, so the habit pays off when you write your first integration. It also teaches you the paths that policies refer to.

Read the release notes before you upgrade

Vault 2.0 was a major release, and it tightened several defaults. Before you adopt a new version, read the important-changes page, test on a copy, and take a snapshot first. Do not learn about a breaking change from a production alert.

Know where your data lives

For teams serving Gulf or Egyptian employers, residency and key custody are real review questions. Decide which region holds the cluster, the snapshots and the audit logs, and which key service protects auto-unseal. Write the answer down in the design document, early.

Practice cards

Try each card without notes. They take about ten minutes apiece.

Card 1: the path detective. Write a KV v2 secret at secret/team/app/db. Write a policy that lets a user read it but not list siblings. Test with a token. Then change it to allow listing under team/app. Explain the two paths you used.

Card 2: the lockout. Start a dev server, export a root token, create a limited user, log in, and notice you are still root. Fix it and prove the fix with vault token lookup.

Card 3: the version history. Write a secret, change it three times with put and patch, read version 2, roll back to version 1, soft-delete it, undelete it, then destroy version 1. Predict the result of each command before you run it.

Card 4: the expiring token. Create a token with -ttl=2m. Use it, wait, use it again. Read the error. Then create one with -use-limit=2 and exhaust it.

Card 5: the audit trail. Enable a file audit device. Perform one allowed and one forbidden request as a limited user. Find both in the log, and identify the user, path and outcome of each.

Card 6: the broken restart. Write a real configuration with Raft storage. Initialise with three shares and a threshold of two. Stop the server, start it, and read the status. Unseal with two shares. Explain what each line of vault status tells you.

Quick diagnosis sequence

When something fails, follow the same order every time, and you will seldom need to guess:

1. Reachability. vault status. Is the address right, is it reachable, is it sealed?
2. Identity. vault token lookup. Which token, which policies, what TTL?
3. Path. vault secrets list. Is the engine mounted, and is the path right for the KV version?
4. Policy. vault token capabilities <path> and vault read sys/internal/ui/resultant-acl.
5. Evidence. -output-curl-string, and the audit log.

Checklist

Tick these before you say you are comfortable with the basics.

• I can explain the path from login to token to policy to secrets engine.
• I set VAULT_ADDR with the right scheme, and I know the CLI defaults to HTTPS.
• I can start a dev server, and I can say four reasons it is not safe for production.
• I use -mount with vault kv commands and I know the data/ and metadata/ paths.
• I know kv put replaces and kv patch merges.
• I can delete, undelete, destroy and roll back a secret version.
• I wrote a policy, uploaded it, and tested it with a real token in both directions.
• I can log in with userpass, and I checked vault token lookup afterwards.
• I set -ttl and -use-limit on tokens, and I can renew and revoke them.
• I enabled an audit device and found my own requests in it.
• I understand sealed and unsealed, the five-share and three-threshold default, and what auto-unseal changes.
• I can say why the root token is revoked after setup.
• I can read the common errors in the table and name the layer each one belongs to.
• I know what changed in 2.0: authenticated rekey and generate-root, clean paths only, and containers without mlock.

If you ticked all of them, the mid-level guide is next. If three or more are empty, redo the matching Try it tasks in the detailed guide rather than moving on; the later topics assume these foundations.

This is part two of three. Beginner got you to the point where you can start a server, store secrets in KV, write a small policy and read a "permission denied". This level is about using Vault well where it matters: at work, with real applications, real pipelines and a cluster that other people depend on. The gap between "it works on my dev server" and "it survives a Monday morning" is almost entirely made of things this guide covers: how tokens and leases really behave, how machines prove who they are without a password in a config file, how credentials get from Vault into a pod, what to log, what to monitor, and what to check before you upgrade.

The guide targets Vault 2.1.1, released on 16 September 2026. Version 2.0 (April 2026) was the first major release since 1.0, and it tightened several defaults. Where an older tutorial will mislead you, the text says so and shows the current form.

Where this picks up

Each section starts with the problem it solves and ends with a Try it you can run against a dev server (started with vault server -dev -dev-root-token-id=root and export VAULT_ADDR=http://127.0.0.1:8200) or, where a cluster is required, against a scratch one.

Tokens, precisely

Everything in Vault is a token, eventually. Even when you log in with OIDC or AppRole, the result of the login is a Vault token, and every later request carries it in the X-Vault-Token header (or as Authorization: Bearer). If you can predict how a token behaves, most "why did my job suddenly get a 403" questions answer themselves.

Service tokens versus batch tokens. The default is a service token, prefixed hvs.. It is written to storage, which is what lets it be renewed, revoked, given children, looked up by accessor and given its own cubbyhole. A batch token, prefixed hvb., is an encrypted blob that Vault does not store. It cannot be renewed or revoked individually, has no children and cannot be a root token. The trade is cost: a service token costs a storage write and a lease entry for every login, and a batch token costs neither. Think about a CI system that logs in four thousand times a day, or a fleet of short functions. Service tokens there create storage and lease pressure you will pay for later. Batch tokens are the tool for high-volume, short-lived work. You pick the type on the role, for example token_type=batch on an auth role.

The hierarchy. A token created by another token is its child, and revoking the parent revokes every descendant. This is a feature when you want a clean teardown and a hazard when a long-lived "parent" token is rotated and suddenly takes a whole application's tokens with it. An orphan token has no parent, and tokens issued by a login through a non-token auth method are orphans already, which is why an AppRole login does not die when the admin who configured it logs out. You can create an orphan explicitly with vault token create -orphan when you hold the right permission.

TTL and max TTL. Every non-root token has a time-to-live. The system default and the system maximum are both 768 hours (32 days), controlled by default_lease_ttl and max_lease_ttl, and you can tune them per mount and per role. The effective TTL is the lowest of what you ask for, what the role allows, what the mount allows and the system ceiling. A role that says token_ttl=20m and token_max_ttl=1h gives you a token that lives 20 minutes and can be renewed until it is an hour old, after which it cannot be renewed again no matter what. Renewal asks for more time; it cannot exceed the max TTL.

Two variations are worth knowing. A periodic token has a period rather than a max TTL: each renewal resets its TTL to the period, so the token can live indefinitely as long as something keeps renewing it. That suits long-running daemons that renew themselves. explicit_max_ttl is the opposite, a hard cap that no renewal can pass, useful when policy says "no credential lives longer than a day, period".

Accessors. Every service token has an accessor, a reference that lets you look up the token's metadata, renew it, revoke it and check its capabilities without knowing the token itself. An accessor cannot be used to authenticate. This matters for operations: your audit log records accessors, so an engineer can revoke a token that appeared in a log without ever seeing the secret value.

vault token create -policy=dev -ttl=20m -format=json | tee /tmp/t.json
ACCESSOR=$(jq -r .auth.accessor /tmp/t.json)
vault token lookup -accessor "$ACCESSOR"        # ttl, policies, creation_time, orphan, renewable
vault token capabilities "$(jq -r .auth.client_token /tmp/t.json)" secret/data/myapp/config
vault token revoke -accessor "$ACCESSOR"

Token types at a glance.

The prefixes changed in Vault 1.10 from s., b. and r.. If you see a token starting with s. in an old runbook, it is from before that.

Policies beyond the first one

Beginner showed the shape: a path, a list of capabilities, deny by default. Real policy work is about the rules for which path block wins, how to avoid writing one policy per team, and how to keep mistakes out of production.

Path matching. A policy path may end in *, which matches any suffix (the glob is only valid at the end), and may contain +, which matches exactly one path segment. secret/data/+/shared matches secret/data/team-a/shared but not secret/data/team-a/sub/shared. When several paths match, the most specific one wins, and among the same specificity the rule is that capabilities from the exact match are used rather than merged with shorter globs. If two policies attached to one token both mention the same path, capabilities are unioned, with one exception: deny always wins over everything else. That is why deny is the tool for carving a hole in an otherwise generous grant.

The KV v2 prefixes. KV v2 splits one logical secret across several API paths, and the policy has to name the right one. Reading and writing use secret/data/.... Listing and metadata use secret/metadata/.... Soft delete, undelete and permanent destroy use secret/delete/..., secret/undelete/... and secret/destroy/.... Forgetting metadata/ is the classic reason vault kv list says "permission denied" while vault kv get works.

# Read and write the app's own secrets, nothing else
path "secret/data/payments/*" {
  capabilities = ["create", "read", "update", "patch"]
}

# Let the app list its own prefix (for discovery), but not delete history
path "secret/metadata/payments/*" {
  capabilities = ["list", "read"]
}

# Nobody with this policy may permanently destroy anything
path "secret/destroy/payments/*" {
  capabilities = ["deny"]
}

# Dynamic database credentials: reading the path creates a user
path "database/creds/payments-ro" {
  capabilities = ["read"]
}

Note the last block. Reading database/creds/<role> is a read capability even though the call creates a database user. Policies are about HTTP verbs on paths, not about what the engine does in response.

Templated policies. Writing one policy per service does not scale. Policies can use identity templates so one policy adapts to the caller:

path "secret/data/teams/{{identity.entity.metadata.team}}/*" {
  capabilities = ["create", "read", "update", "delete"]
}
path "secret/metadata/teams/{{identity.entity.metadata.team}}/*" {
  capabilities = ["list", "read"]
}

Each caller's entity carries a team metadata value and sees only its own prefix. This is the standard way to emulate tenancy in Community Edition, where namespaces are not available.

Parameter constraints. For engines that take parameters, a policy can restrict them with allowed_parameters, denied_parameters and required_parameters. They are useful on, for example, auth/approle/role/* or pki/issue/*. They do not work on KV v2, because the parameters of a KV v2 write are nested inside a data object. Two behaviour changes bite here. From 1.21, list comparison in these constraints changed from "exact match" to "contains all", so the rule matches individual elements. And from 2.0.4, Vault lowercases the policies field before checking denied_parameters, closing a bypass with a mixed-case policy name.

Strict parsing. Since 1.21, duplicate attributes in HCL policies and config files are errors, where 1.20 only warned. If an old policy file has the same key twice, Vault now refuses it. There is a temporary escape hatch, the environment variable VAULT_ALLOW_PENDING_REMOVAL_DUPLICATE_HCL_ATTRIBUTES, but the fix is to delete the duplicate.

The sudo capability and default. Some system paths need sudo on top of the normal capability, for instance listing root-protected resources or changing certain mounts. Grant it on the narrowest path possible. The built-in default policy is attached to every token unless you opt out; you may edit it, but you cannot delete it, and it is better to keep it small and add your own. The root policy is immutable.

Testing policies. Two cheap tools avoid shipping a wrong policy. First, vault policy fmt normalises the file and catches syntax errors. Second, -output-policy on any CLI command prints the policy that command needs, which is the quickest way to write a least-privilege policy for a script: run the command with the flag and paste the output.

vault policy fmt payments-app.hcl
vault policy write payments-app payments-app.hcl
vault kv put -output-policy -mount=secret payments/db password=x
# path "secret/data/payments/db" { capabilities = ["create"] }

Identity: entities, aliases and groups

Tokens say what a caller may do. The identity system says who the caller is. Without it, the same engineer logging in through LDAP on Monday and OIDC on Tuesday is two unrelated principals, and your audit trail and policy attachment fragment accordingly.

An entity represents a person or a workload. An alias ties an entity to one login on one auth mount, so one entity can have an alias on auth/oidc/ and another on auth/ldap/. A group collects entities, and policies can be attached to entities and groups as well as to the auth role. A token produced by a login inherits the policies of its role plus those of the entity and its groups. There are two kinds of group. Internal groups are defined inside Vault with explicit members. External groups map to something in the identity provider, such as an LDAP group or an OIDC groups claim, so membership follows the directory and you never maintain lists by hand.

vault write identity/entity name=alice metadata=team=payments
vault write identity/group name=payments-devs type=internal policies=payments-app member_entity_ids=$ENTITY_ID
vault read identity/entity/name/alice

Why this matters beyond tidiness: clients are counted per entity for Enterprise and HCP Vault Dedicated billing, and your audit log can attribute actions to an entity rather than to a throwaway token. A workload that logs in a thousand times as a thousand unrelated identities is a thousand clients; the same workload mapped to one entity is one. Design your auth roles so a service's identity is stable.

Machine authentication and the secret-zero problem

A person can type a password or click through SSO. A program cannot. Every application that uses Vault has to present something to get its first token, and whatever that something is must itself be protected. This is called the secret-zero problem: you used a secret to fetch secrets. The whole art of machine auth is to make that first credential short-lived, single-use, narrowly scoped or, best of all, absent, because the platform the workload runs on already vouches for it.

Ranked from best to worst for most teams:

1. Platform identity. Kubernetes service accounts, AWS IAM, GCP and Azure identities, and OIDC tokens from a CI system. Vault verifies a signed statement from a platform it trusts. No secret is shipped to the workload at all.
2. AppRole with wrapped, single-use secret IDs. For workloads with no platform identity to use, such as a VM in a data centre that was provisioned by a pipeline.
3. Static tokens in config or CI variables. Avoid. They do not expire in practice, they leak into logs and they cannot be attributed.

AppRole. An AppRole has two halves. The RoleID is like a username: not secret, fine to bake into an image or a config file. The SecretID is like a password: issued per instance or per deployment, with a TTL and a use limit. Both are needed to log in.

vault auth enable approle

vault write auth/approle/role/web \
    token_policies=web \
    token_ttl=20m token_max_ttl=1h \
    secret_id_ttl=10m secret_id_num_uses=1

# Non-secret half: give to the workload's config
vault read -field=role_id auth/approle/role/web/role-id

# Secret half: generated by the deploy system, wrapped so only the workload can open it
vault write -f -wrap-ttl=120s auth/approle/role/web/secret-id

# The workload logs in with both
vault write auth/approle/login role_id=$ROLE_ID secret_id=$SECRET_ID

The pattern that makes this safe is response wrapping. With -wrap-ttl=120s, Vault does not return the SecretID to whoever asked. It stores it in a single-use cubbyhole and returns only a wrapping token. The deployer passes the wrapping token to the workload, which calls vault unwrap. Because a wrapping token can be used exactly once, two things follow: if an attacker unwraps it first, the legitimate workload's unwrap fails and you know you were intercepted; and the wrapping token is worthless after two minutes even if it leaks into a log. A SecretID with secret_id_num_uses=1 adds a second lock, since a replayed SecretID fails.

# In the workload, at start-up
SECRET_ID=$(VAULT_TOKEN=$WRAPPING_TOKEN vault unwrap -field=secret_id)

Who is allowed to mint SecretIDs? The identity that can write to auth/approle/role/web/secret-id is effectively able to impersonate the role. Give that path to the deployment system only, and nothing else. Consider binding the role with secret_id_bound_cidrs and token_bound_cidrs so a stolen credential only works from known networks.

Kubernetes authentication and getting secrets into pods

On Kubernetes every pod already has a cryptographically signed identity: its service account token. The Kubernetes auth method lets Vault verify that token with the cluster's API server and map the service account name and namespace to a Vault role.

vault auth enable kubernetes
vault write auth/kubernetes/config kubernetes_host=https://$KUBERNETES_SERVICE_HOST:443

vault write auth/kubernetes/role/app \
    bound_service_account_names=app \
    bound_service_account_namespaces=prod \
    token_policies=app \
    ttl=1h

A pod in namespace prod running as service account app can now log in and receive a one-hour token with the app policy. A pod in staging, or one running as default, cannot. The two bound_ fields are the security boundary, so be strict: a * in bound_service_account_namespaces means any namespace can assume the role.

That gets you a token. How the secret reaches the application is a separate choice, and there are three supported options:

The decision that matters is whether secret values may exist as Kubernetes Secret objects. A Kubernetes Secret is only base64-encoded in etcd unless you configure encryption at rest, and anyone with get secrets RBAC in the namespace can read it. VSO's model is convenient because it yields ordinary Secrets that every tool understands, and it costs you that exposure. The injector and CSI routes keep values in pod memory or files on a memory-backed volume. Neither is universally better; pick on the threat model and on how much your applications want to read files versus environment variables.

A VSO setup for a static secret looks like this:

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultConnection
metadata:
  name: vault
  namespace: prod
spec:
  address: https://vault.example.com:8200
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  name: app
  namespace: prod
spec:
  vaultConnectionRef: vault
  method: kubernetes
  mount: kubernetes
  kubernetes:
    role: app
    serviceAccount: app
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: app-config
  namespace: prod
spec:
  vaultAuthRef: app
  mount: secret
  type: kv-v2
  path: payments/config
  refreshAfter: 60s
  destination:
    name: app-config
    create: true
  rolloutRestartTargets:
    - kind: Deployment
      name: payments

The operator authenticates as the app service account, reads secret/payments/config, writes the Secret app-config, re-reads every 60 seconds, and restarts the payments Deployment when the value changes. That last field is what turns secret rotation from a manual chore into a non-event.

Installation details to get right: the Helm chart (hashicorp/vault, currently 0.34.1) deploys Vault 2.0.4 by default, so pin server.image.tag if you want 2.1.1. The default chart mode is standalone with file storage and is not production-ready; use server.ha.enabled=true with server.ha.raft.enabled=true. The operator is a separate chart, hashicorp/vault-secrets-operator. For how Helm releases and GitOps tie in, see Helm and Argo CD, and for the cluster itself see Kubernetes.

CI/CD without stored tokens: JWT and OIDC auth

The second most common place a long-lived Vault token hides is a CI variable. It is pasted once, never rotated, readable by anyone who can edit the pipeline, and it leaks the first time a job prints its environment. Modern CI systems make it unnecessary: each job can obtain a short-lived, signed JSON Web Token that states who the job is (which repository, branch and workflow), and Vault's JWT auth method can verify that signature against the CI system's published keys.

For GitHub Actions the configuration looks like this:

vault auth enable jwt

vault write auth/jwt/config \
    oidc_discovery_url=https://token.actions.githubusercontent.com \
    bound_issuer=https://token.actions.githubusercontent.com

vault write auth/jwt/role/ci \
    role_type=jwt \
    user_claim=repository \
    bound_audiences=https://github.com/ORG \
    bound_claims='{"repository":"ORG/REPO"}' \
    token_policies=ci \
    token_ttl=10m

Read the role as a sentence. "Accept a token from GitHub's issuer, whose audience is our organisation and whose repository claim is exactly ORG/REPO; treat the repository as the user; hand back a ten-minute token with the ci policy." The security of the whole arrangement sits in bound_claims. A role that omits it accepts tokens from any repository that can request one for that audience, which on a public platform means strangers. Bind to the repository at minimum, and for production secrets also to a branch or environment claim (for example ref or environment) so a pull request from a fork cannot reach them.

In a workflow, the hashicorp/vault-action step does the login and fetch. GitLab CI offers id_tokens together with a secrets:vault keyword, and Jenkins and Bitbucket have equivalent plugins or patterns. The pipelines themselves are covered in GitHub Actions and GitLab CI; from Vault's side what matters is identical. An Okta provider with group fetching was also added to the JWT method in 2.1.1, using fetch_groups=true, for teams who map groups from there.

permissions:
  id-token: write     # lets the job request an OIDC token
  contents: read
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: hashicorp/vault-action@v3
        with:
          url: https://vault.example.com:8200
          method: jwt
          role: ci
          secrets: |
            secret/data/deploy/registry token | REGISTRY_TOKEN

Note the secret path: secret/data/..., because this action talks to the API directly and uses the real KV v2 path.

Vault Agent and Vault Proxy

Not every application can log in to Vault, renew its token, fetch secrets and handle rotation itself, and you would not want to write that code forty times. Vault ships two client-side daemons that take the work off the application.

Vault Agent handles three jobs: auto-auth (logging in by whatever method you configure and keeping the token renewed), templating (rendering secrets into files), and process supervision (starting your program with secrets in its environment). Vault Proxy does auto-auth too, but its other jobs are API proxying and caching: an application talks to a local listener with no token at all, and the Proxy adds the token. The old Agent "API proxy" function is deprecated with removal scheduled, so anything new that wants proxy behaviour uses Vault Proxy.

An Agent configuration that logs in as a Kubernetes workload, renders a file and supervises the app:

auto_auth {
  method "kubernetes" {
    config = { role = "app" }
  }
  sink "file" {
    config = { path = "/home/vault/.token" }
  }
}

template {
  source      = "/etc/tpl/db.ctmpl"
  destination = "/etc/secrets/db.env"
}

env_template "DB_PASS" {
  contents = "{{ with secret \"secret/data/app\" }}{{ .Data.data.password }}{{ end }}"
}

exec {
  command                   = ["./app"]
  restart_on_secret_changes = "always"
}

Run it with vault agent -config=agent.hcl. The template block renders a file from a Consul-Template-style source, and Agent re-renders it when the secret changes or its lease nears expiry. The env_template plus exec pair injects the value as an environment variable and restarts the process when it changes, which is the route for legacy applications that only read environment variables. The template for the file source might look like:

{{ with secret "database/creds/payments-ro" }}
DB_USER={{ .Data.username }}
DB_PASS={{ .Data.password }}
{{ end }}

What a template does with dynamic secrets is worth understanding. Agent reads database/creds/payments-ro, which creates a database user with a lease. It keeps that lease alive by renewing it and, when the lease reaches its max TTL, reads the path again to obtain a fresh user and re-renders the file. The application sees a file that changes; Vault sees one lease at a time.

Vault Proxy runs with vault proxy -config=proxy.hcl and uses api_proxy { use_auto_auth_token = true }, a cache {} block and a listener. Static KV secret caching in Proxy is Enterprise-server dependent, so check your edition before relying on it.

Dynamic secrets and leases

A static password in KV is a liability with a long half-life: it is copied, cached, shared in chat and rarely rotated because rotation is painful. A dynamic secret inverts this. Nothing exists until a client asks; Vault creates a fresh credential on the target system, attaches a lease, and deletes the credential when the lease ends. No human knows it, it is unique per requester, and a leak has a built-in expiry.

The database secrets engine is the canonical example. You give Vault a privileged account on the database, then define roles that describe the SQL to run for a new user.

vault secrets enable database

vault write database/config/pg \
    plugin_name=postgresql-database-plugin \
    connection_url="postgresql://{{username}}:{{password}}@db:5432/app" \
    allowed_roles=app \
    username=vault password=initial-password

# Immediately change the password so that no human knows it any more
vault write -f database/rotate-root/pg

vault write database/roles/app \
    db_name=pg default_ttl=1h max_ttl=24h \
    creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";"

vault read database/creds/app

The last command prints a username like v-token-app-xxxx, a password, and a lease_id. Three practices separate a good setup from a fragile one. First, rotate the root credential with database/rotate-root/<name>, as above, so the password Vault uses is known only to Vault. Second, keep roles least-privilege; the SQL in creation_statements is yours, and "GRANT ALL" is easy to write. Third, choose TTLs to match how long a connection pool really keeps connections, because a credential revoked under a live pool produces errors that look like database outages.

When an account must exist permanently, for example a service user that other systems already reference, use a static role, where Vault rotates the password on a schedule but the user stays:

vault write database/static-roles/svc db_name=pg username=svc rotation_period=24h
vault read database/static-creds/svc

From 2.1, rotation_period and rotation_schedule are mutually exclusive: choose an interval or a cron-style schedule, not both.

Leases in practice. Every dynamic secret, and every service token, has a lease: an ID, a TTL and a renewable flag. You can inspect, renew and revoke them:

vault lease lookup database/creds/app/AbC123...
vault lease renew -increment=1h database/creds/app/AbC123...
vault lease revoke database/creds/app/AbC123...
vault lease revoke -prefix database/creds/app        # incident button: kill every credential from that role

-prefix revocation is the most valuable operational tool in the whole engine. If a credential from a role is suspected leaked, revoking the prefix deletes every outstanding database user it created. Applications holding them will fail, then fetch new ones on their next attempt. You can also set a per-mount ceiling so no engine's leases outlive your policy: vault secrets tune -default-lease-ttl=1h -max-lease-ttl=24h database/.

Cost. Leases are not free. Vault keeps each one in memory and storage, and expiry work runs in the background. Hundreds of thousands of long-lived leases slow start-up and expiration. The mitigation is the same as for tokens: short TTLs, renew only what you must, use batch tokens where possible, and on Enterprise add lease-count quotas (sys/quotas/lease-count/...). Watch vault.expire.num_leases in metrics.

Transit: encryption as a service

Sometimes the right move is not to store a secret in Vault but to let Vault protect data that lives elsewhere. The Transit engine performs cryptographic operations and never hands out the keys. An application sends plaintext, Vault returns ciphertext, and the application stores that ciphertext in its own database. The application never holds the key, so a database dump without Vault access is useless.

vault secrets enable transit
vault write -f transit/keys/orders

vault write transit/encrypt/orders plaintext=$(echo -n "4111111111111111" | base64)
# Key        Value
# ciphertext vault:v1:8SDd3WHDOjf7mq69CyCqYjBXAiQQAVZRkFM13ok481zoCmHnSeDX9vyf7w==

vault write transit/decrypt/orders ciphertext=vault:v1:8SDd3W...

Plaintext must be base64-encoded, and what you get back from decrypt is base64 too. The v1 in the ciphertext is the key version, which is what makes rotation painless.

Key rotation and rewrap. Rotate the key with vault write -f transit/keys/orders/rotate. New encryptions use version 2; old ciphertexts still decrypt because earlier versions are retained. To move stored data to the new version without the application ever seeing plaintext, send the old ciphertext to transit/rewrap/orders, and store what comes back. Once everything has been rewrapped you can retire old versions by raising min_decryption_version:

vault write -f transit/keys/orders/rotate
vault write transit/rewrap/orders ciphertext=vault:v1:8SDd3W...     # returns vault:v2:...
vault write transit/keys/orders/config min_decryption_version=2

Setting min_decryption_version too early makes data encrypted under older versions undecryptable, so make sure a rewrap sweep has completed and is verified first. There is also min_encryption_version and a setting that marks a key as non-deletable; leave deletion protection on.

When Transit is the right tool. Encrypting individual fields (card numbers, national IDs, health data) when you want central key control and audit. It suits data-residency conversations common in Gulf and Egyptian deployments, because you can show which keys live in a region-local cluster. It is a poor fit for encrypting large blobs: every byte passes through Vault over HTTP. For bulk data, the usual pattern is envelope encryption: ask Transit for a data key (transit/datakey/...), encrypt locally with the plaintext key and store the wrapped key next to the data. Also note 2.0.2 set a maximum RSA key modulus of 8192 bits, so audit any larger keys before upgrading.

PKI: short-lived certificates

Internal TLS fails in the same boring way everywhere: a certificate issued for two years expires on a weekend, because nobody owns the renewal. The PKI engine turns Vault into a certificate authority that issues certificates in milliseconds, so you can issue them with lifetimes of hours or days and automate renewal rather than remember it.

vault secrets enable pki
vault secrets tune -max-lease-ttl=87600h pki

vault write pki/root/generate/internal common_name=example.com ttl=87600h

vault write pki/roles/web \
    allowed_domains=example.com allow_subdomains=true max_ttl=72h

vault write pki/issue/web common_name=api.example.com ttl=24h

The role is the control point: it says which names may be requested (allowed_domains, allow_subdomains) and how long a certificate can live (max_ttl). Applications get only pki/issue/web through policy, so they can request *.example.com certificates and nothing else. Short lifetimes change what revocation means. A certificate valid for 24 hours rarely needs a CRL entry; you stop issuing and wait. When you do need to revoke, /pki/revoke and /pki/revoke-with-key accept an RFC 5280 revocation reason as of 2.1.1.

In a real deployment, do not keep the root CA online. The standard layout is an offline or tightly held root, an intermediate in Vault that signs leaf certificates, and leaf certificates consumed by services. Generating the root inside the same Vault that issues leaf certificates, as the example does, is for learning. The PKI engine also speaks ACME, and in 2.0 and 2.1 added external CA support (Enterprise), PKCS#12 and JKS output for Java estates, and automatic DNS-01 fulfilment for Route53, Azure DNS, Google Cloud DNS and RFC 2136. For Kubernetes, cert-manager can use Vault as an issuer, and Agent or VSO's VaultPKISecret can renew certificates for workloads directly.

Audit devices

An audit device is a sink that records every request and every response Vault handles. When something goes wrong, or when a reviewer asks who read a secret and when, this is the only complete answer. Enable it from day one, since you cannot reconstruct events that were never logged.

vault audit enable file file_path=/var/log/vault/audit.log
vault audit enable syslog tag="vault" facility="AUTH"
vault audit list -detailed

The types are file, syslog and socket. Every value that could be sensitive, including tokens and secret data, is hashed with HMAC-SHA256 before being written, so the log proves that a request happened without disclosing what it contained. To correlate a known value with a log line, hash it with the audit device's salt through sys/audit-hash.

Two behaviours surprise people. The first is that Vault fails closed: if at least one audit device is enabled and none of them can write, Vault refuses to serve requests. A full disk or a dead syslog endpoint becomes a full outage. That is why the hardening guidance is to run at least two devices on different sinks and to alert on audit failures before they cascade. The second is that audit logs can carry a lot of data and must be shipped off the box, rotated and retained; treat them as a SIEM feed, not a file someone reads with tail. In 2.0 audit entries can carry an extra supplemental_audit_data key.

# A request header you want to see unhashed in the log, for correlating with a load balancer
vault write sys/config/auditing/request-headers/X-Forwarded-For hmac=false

How a real cluster is put together

The dev server hides everything that makes Vault a service: durable storage, a seal, TLS, a second and third node. To work with a production cluster without being surprised by it, you need the moving parts and what each one does.

One node is active and handles all requests. The others are standbys that replicate its data and take over if it fails; a standby that receives a request forwards it to the active node (or redirects the client using api_addr). Everything the active node persists passes through the barrier, which encrypts it with AES-256-GCM, so the storage layer and the network between nodes are treated as untrusted. In Community Edition, standbys do not serve reads. Enterprise performance standbys do, which is one reason large read-heavy estates buy it.

Integrated Storage (Raft) is the recommended backend. Vault keeps its data inside its own nodes, replicated by the Raft consensus protocol, and you need no external database. Raft needs a quorum, a majority of voting nodes, to accept a write.

Always use an odd number, spread nodes across availability zones, and never run a single node in production: one disk failure is then data loss. The official guidance for a standard deployment is five. Raft cannot be combined with a separate ha_storage. A minimal production node configuration:

ui            = true
cluster_name  = "prod"
disable_mlock = true
api_addr      = "https://vault-1.example.com:8200"
cluster_addr  = "https://vault-1.example.com:8201"

storage "raft" {
  path    = "/opt/vault/data"
  node_id = "vault-1"

  retry_join { leader_api_addr = "https://vault-2.example.com:8200" }
  retry_join { leader_api_addr = "https://vault-3.example.com:8200" }
}

listener "tcp" {
  address       = "0.0.0.0:8200"
  tls_cert_file = "/opt/vault/tls/tls.crt"
  tls_key_file  = "/opt/vault/tls/tls.key"
}

telemetry {
  prometheus_retention_time = "24h"
  disable_hostname          = true
}

seal "awskms" {
  region     = "eu-central-1"
  kms_key_id = "alias/vault-unseal"
}

Each setting earns its place. disable_mlock must be stated explicitly with Raft since 1.20 (there is no default, and Vault will not start without a value), and true is the right choice with Raft because its BoltDB storage is memory-mapped and mlock cannot hold it; pair it with disabled swap. It is also mandatory in practice for containers: since 2.0.2 the official image no longer carries the cap_ipc_lock capability, so a container cannot call mlock at all. cluster_addr is required with Raft, and omitting it gives "Cluster address must be set when using raft storage". Port 8200 is the API and UI; 8201 carries cluster traffic, Raft RPC and request forwarding, so both must be open between nodes. The retry_join blocks let a starting node find the cluster. From 2.1.1 concurrent retry-join workers are capped at 20, and extras fail with too many concurrent raft retry joins in progress, which matters only if you list very many targets or start many nodes at once.

Seal and unseal. On start, Vault is sealed: it can read storage but cannot decrypt it. Three layers of keys are involved. Data is encrypted with the encryption key (the keyring); the keyring is encrypted with the root key (older documents call it the master key); the root key is protected either by Shamir shares or by an auto-unseal device. With Shamir, vault operator init splits the unseal key into shares (the defaults are -key-shares=5 -key-threshold=3) and humans must enter three of five after every restart. That is fine for a demonstration and painful for a cluster whose pods are rescheduled at 03:00.

Auto-unseal hands protection of the root key to a KMS or HSM: awskms, azurekeyvault, gcpckms, ocikms, alicloudkms, another Vault's transit engine, or pkcs11 (Enterprise). Nodes then unseal themselves on start. The price is a new dependency: if the KMS is unreachable or the node loses IAM access to the key, Vault stays sealed and returns 503. Keep the key in the same region as the cluster, give only the Vault nodes access, and monitor KMS errors. With auto-unseal, init produces recovery keys instead of unseal keys. They cannot decrypt anything; they authorise sensitive operations such as generating a root token or rekeying. Distribute them PGP-encrypted to separate people.

vault operator init -recovery-shares=5 -recovery-threshold=3
vault operator raft list-peers
vault operator members
vault operator diagnose -config=/etc/vault.d/vault.hcl

vault operator diagnose validates a configuration before you start, catching TLS problems, unreadable files, permission issues and a disabled listener TLS (tls_disable = true produces a warning). Run it in your deployment pipeline against the config you are about to roll out.

TLS. Terminate TLS in Vault itself (tls_cert_file, tls_key_file) rather than at a load balancer that re-encrypts badly, and keep the minimum at TLS 1.2 (the default of tls_min_version). Your clients must trust the issuing CA: set VAULT_CACERT, never VAULT_SKIP_VERIFY=true. If a corporate proxy re-signs traffic, the error is x509: certificate signed by unknown authority, and the fix is to trust that proxy's CA on the client, not to disable verification. Certificate and key reloads happen on SIGHUP, so a renewed certificate does not require a restart.

The load balancer. Point it at /v1/sys/health, which reports role through the status code, so one URL answers "should I send traffic here":

A load balancer that treats only 200 as healthy sends everything to the leader, which is what you want for CE. Query parameters such as standbyok=true or perfstandbyok=true change which codes count as healthy. A node returning 474 with the log line local node not active but active cluster node not found almost always has a wrong api_addr or cluster_addr, or port 8201 blocked.

Autopilot, snapshots and the backups you must test

A cluster can lose a node gracefully and still lose everything through a bad upgrade, an accidental mass deletion or a corrupt write. The defence is backups, and Raft makes them simple.

vault operator raft snapshot save /backups/vault-$(date +%F).snap
vault operator raft snapshot inspect /backups/vault-2026-10-01.snap
vault operator raft snapshot restore /backups/vault-2026-10-01.snap

A snapshot is a consistent point-in-time copy of Vault's data, including secrets, policies, auth configuration, leases and the encrypted keyring. It does not include your config file and it is useless without the ability to unseal (the Shamir shares, or access to the same KMS key). Store snapshots off the cluster, encrypt them at rest and restrict who can read them, since a snapshot is as sensitive as the cluster. Enterprise adds automated snapshots (sys/storage/raft/snapshot-auto/config/...). In Community Edition, run snapshot save from a scheduled job using a token with a narrow policy on sys/storage/raft/snapshot, and alert if it stops producing files.

An untested backup is a hope. The habit that separates teams: restore into a scratch cluster on a schedule and verify a known secret can be read. The upgrade procedure requires the same rehearsal, so you lose nothing by building it once.

Autopilot manages Raft membership. It is on by default and brings new nodes in as non-voters until they have stabilised, then promotes them. Dead-server cleanup is off by default, which means a permanently lost node stays in the peer list and counts towards quorum arithmetic until you remove it by hand. To enable automatic cleanup, be deliberate about thresholds:

vault operator raft autopilot state
vault operator raft autopilot set-config \
    -cleanup-dead-servers=true \
    -dead-server-last-contact-threshold=10m \
    -min-quorum=3
vault operator raft remove-peer vault-3        # manual removal of a node that is gone for good

min-quorum is required once cleanup is on. Set it to the number of nodes you need to preserve, so Autopilot can never "clean up" your way below a working cluster.

Upgrading, and what 2.x changed

Vault upgrades are simple mechanically and dangerous in their defaults, because each release tightens behaviour that scripts relied on. The official procedure has four beats: read the release notes and the change tracker; back up (snapshot plus config); rehearse on an isolated copy restored from a snapshot; then upgrade node by node, which means stop the service, replace the binary, start it, run vault status and unseal if needed. The usual order is standbys first and the active node last, followed by vault operator step-down so leadership moves to a node already on the new version. Storage is migrated forward, so rollback means restoring the snapshot, which is why the snapshot is not optional. On Helm, the Vault pods are a StatefulSet with the OnDelete update strategy, so you delete pods one at a time, standbys first; pin server.image.tag so the chart upgrade does not also pick an image.

Below is the checklist for moving from a 1.x cluster to 2.x, expressed as the symptoms you would otherwise debug in production.

1. Callers of rekey and generate-rootFrom 2.0, sys/rekey, sys/generate-root and the DR operation-token endpoint need a Vault token as well as the key shares. Break-glass runbooks that assumed shares alone now return 403. The compatibility lever is the top-level config key enable_unauthenticated_access, but the better fix is updating the runbook.
2. Scripts that build paths by concatenationVault rejects //, /./ and /../ in request paths. "$MOUNT/" + "/foo" style code breaks. Clean the path in the script or SDK call.
3. Containers and mlock2.0.2 removed cap_ipc_lock from the image. Set disable_mlock = true and disable or encrypt swap.
4. Templated policies with wildcardsA rendered + or * is refused from 2.0.1.
5. Large RSA keysThe maximum RSA modulus is 8192 bits from 2.0.2 (CVE-2026-39829). Audit Transit and PKI keys.
6. Tools inside the imageUBI images lost gnupg, openssl and procps in 2.0.4. A debug script that shelled out to openssl in the container breaks.
7. Azure authValues in auth/azure/config now win over AZURE_* environment variables.
8. Duplicate HCL attributesErrors since 1.21. Fix the file rather than setting the temporary override variable.

Older changes also remain relevant when you jump several versions: from 1.20, disable_mlock has no default with Raft; /sys/internal/counters/tokens was removed (it returns 403 "unsupported path"); from 1.21, list comparison in allowed_parameters and denied_parameters changed to "contains all". Removed features you may still have mounted: the Active Directory secrets engine (use the LDAP engine with schema=ad), Centrify auth, and some legacy fields. The deprecation table in the docs lists which built-in plugins are "pending removal", where Vault refuses to mount them; VAULT_ALLOW_PENDING_REMOVAL_MOUNTS=true is an emergency override only.

Also know the licence context, because it comes up in every procurement conversation. Since 1.15 the source is under the Business Source License 1.1; the free build is Vault Community Edition. OpenBao is a separate, community-maintained fork of the last MPL-licensed code, and its flags and features are not Vault's, so do not mix documentation. HCP Vault Secrets has been decommissioned; HCP Vault Dedicated is the managed Vault offering that remains.

Observability: metrics, logs and alerts

Vault exposes what it is doing, and most incidents announce themselves in these numbers several minutes before users notice. Metrics come from GET /v1/sys/metrics, in Prometheus format when asked. The endpoint needs a token with read on sys/metrics, or the listener can open it with telemetry { unauthenticated_metrics_access = true } when it sits on a trusted network. In the telemetry stanza, prometheus_retention_time defaults to 24h and must be non-zero for Prometheus to work, disable_hostname = true is recommended so metric names stay stable when nodes are replaced, and other sinks (statsd, dogstatsd, Stackdriver and more) are available. Collect them with Prometheus; for wider tracing and logging pipelines see OpenTelemetry.

scrape_configs:
  - job_name: vault
    metrics_path: /v1/sys/metrics
    params:
      format: ["prometheus"]
    scheme: https
    authorization:
      credentials_file: /etc/prometheus/vault-token
    static_configs:
      - targets: ["vault-1.example.com:8200", "vault-2.example.com:8200", "vault-3.example.com:8200"]

The metrics worth an alert, and why:

Also keep the server log as JSON (log_format = "json") and ship it alongside the audit log. Combine them in your alerting: a sudden wave of 403s in the audit log after a deploy means a policy change or a rotated role; a wave of 429s means a rate-limit quota is biting.

Terraform, and managing Vault as code

Clicking and typing CLI commands does not survive a team. The hashicorp/vault Terraform provider manages mounts, policies, auth backends, roles and more, which means a reviewable pull request is how access changes, and a fresh cluster can be rebuilt from the repository. The broader Terraform practices (state, modules, workspaces) are in Terraform; this section covers the Vault-specific points.

provider "vault" {
  address = "https://vault.example.com:8200"
  # Token comes from VAULT_TOKEN in the environment, never from this file
}

resource "vault_mount" "kv" {
  path    = "secret"
  type    = "kv"
  options = { version = "2" }
}

resource "vault_policy" "payments_app" {
  name   = "payments-app"
  policy = file("${path.module}/policies/payments-app.hcl")
}

resource "vault_auth_backend" "kubernetes" {
  type = "kubernetes"
}

resource "vault_kubernetes_auth_backend_role" "payments" {
  backend                          = vault_auth_backend.kubernetes.path
  role_name                        = "payments"
  bound_service_account_names      = ["payments"]
  bound_service_account_namespaces = ["prod"]
  token_policies                   = [vault_policy.payments_app.name]
  token_ttl                        = 3600
}

There is one trap above all others. Anything Terraform reads from Vault can end up in state, and state is a file that many people and systems can read. A vault_kv_secret_v2 data source that reads a password puts the plaintext in the state file. Terraform's ephemeral values keep such data out of state for providers that support them, but check that the provider version you use supports ephemeral resources before depending on it. The safer design is to let Terraform manage the shape of Vault (mounts, policies, roles) and let applications fetch secrets themselves at run time, so Terraform never touches values.

Also decide who owns Terraform's own Vault credential. Use the JWT method from the CI job that runs terraform apply, with a policy that can write sys/policies/acl/*, sys/mounts/* and sys/auth/* but not read application secrets. Separation of "can change configuration" from "can read data" is one of the cleanest controls Vault gives you.

Debugging beyond "permission denied"

Beginner gave you the list of common errors. A mid-level engineer needs a method, since the same 403 can come from five different causes and the fix differs.

A few specific patterns. "no handler for route" means the path is wrong or the engine is not mounted there; run vault secrets list. "preflight capability check returned 403" appears when the KV CLI helper cannot read sys/internal/ui/mounts/<path> to detect the KV version, so either grant that read or use the explicit API path. 403 with a valid-looking token in a namespace setup often means VAULT_NAMESPACE is unset or wrong. HTTP 431 (Enterprise, 2.0+) means a token header exceeds max_token_header_size (8 KB by default), typical with very large JWTs; raise it on the listener. 429 "rate limit quota exceeded" is a quota, not an outage; the Go client honours Retry-After, and your own scripts should back off rather than retry hot. After upgrading to 2.x, a 403 from sys/rekey or sys/generate-root is the new authentication requirement.

When a login works but secret access fails, the question is always whether the token the login produced carries the policy you think. Look at the role's token_policies, then at the entity and group policies, then at default. When a dynamic secret request fails, the error usually comes from the target system, so check the engine's configuration path and the target database's own logs. vault read database/config/pg shows the connection details Vault uses.

Scaling and performance without guessing

Vault is not a database and does not scale like one. Writes all go to the active node, and every node's data set has to fit in memory comfortably because Raft's BoltDB is memory-mapped. Knowing which resource is the limit tells you what to change.

Requests. Most of the time cost of a request is authentication and policy evaluation plus a storage round trip. Reads from KV are cheap; dynamic secret creation is slow, because it includes a round trip to the target system. A deploy that starts two hundred pods, each requesting database credentials, is a thundering herd on both Vault and the database; stagger it, or let a few shared roles serve many pods via leases. Clients that cache and renew (Agent, Proxy, VSO) cut request volume dramatically compared with code that logs in and reads every time.

Entry size. A Raft entry defaults to a max_entry_size of 1 MiB, so a single KV value above that fails. Vault is for secrets, not documents; store large files elsewhere and keep keys or references in Vault.

Leases and tokens. These are the usual culprits when a cluster slows down over months. Symptoms include a slow start-up (Vault loads leases), expiration backlogs and growing storage. Keep TTLs short, revoke by prefix when a role is decommissioned, and use batch tokens for high-volume authentication. Watch vault.expire.num_leases and vault.token.count.

Rate limiting. A rate-limit quota protects the cluster from a misbehaving client:

vault write sys/quotas/rate-limit/global rate=500 interval=1s
vault write sys/quotas/rate-limit/ci path=auth/jwt rate=50

The second line limits a single path, which stops a runaway CI loop from starving everyone. Lease-count quotas are Enterprise-only. Reaching a quota returns HTTP 429, so clients must handle it.

Network and region. Put the cluster close to its main clients. Cross-region calls for every secret read add latency you will feel under load, and some Gulf and Egyptian employers have residency requirements that decide the region for you. Multi-region is an Enterprise feature (performance replication for local reads, DR replication for failover); with Community Edition the pattern is a separate cluster per region plus snapshot and restore for recovery.

What to do first when it is slow. Check the metrics before changing anything: is it request rate (handle_request), storage (raft.commitTime), leases (num_leases), or the audit device backing up (audit latency, a slow syslog)? An audit device that blocks will slow every request, as every request waits for it to write.

Putting it all together

Here is one service's complete path through everything above, in the order you would build it. A payments API runs in Kubernetes namespace prod, uses PostgreSQL, deploys from GitHub Actions and must encrypt card references.

1. Platform. A three-node Raft cluster across three zones, disable_mlock = true, TLS on the listener, auto-unseal with a region-local KMS key, two audit devices, Prometheus scraping with the alerts from the observability section, and a nightly snapshot save shipped off-cluster with a weekly restore rehearsal.
2. Configuration as code. Terraform manages the mounts, policies and auth roles, applied by a CI job that logs in with JWT and has no read access to application data.
3. Identity. The Kubernetes auth role payments is bound to service account payments in prod. The GitHub role ci is bound to ORG/payments and to the production environment claim.
4. Policy. A payments-app policy gives read on secret/data/payments/*, list on secret/metadata/payments/*, read on database/creds/payments-ro, and update on transit/encrypt/orders and transit/decrypt/orders. Nothing else, and no sudo.
5. Secrets. Static configuration sits in KV v2. Database access is dynamic with a one-hour TTL. Card numbers are encrypted with Transit, and the ciphertext is stored in PostgreSQL.
6. Delivery. VSO syncs payments/config and database credentials into Secrets with rolloutRestartTargets so a rotation restarts the Deployment. Alternatively, an injector sidecar writes files if Kubernetes Secret objects are not allowed.
7. Certificates. Internal mTLS certificates come from a PKI role with a 72-hour maximum lifetime, renewed automatically.
8. Operations. Upgrades follow the checklist: snapshot, rehearse, standbys first, step-down, active last.

The skeleton for the application's policy, which ties the pieces together:

path "secret/data/payments/*"      { capabilities = ["read"] }
path "secret/metadata/payments/*"  { capabilities = ["list", "read"] }
path "database/creds/payments-ro"  { capabilities = ["read"] }
path "transit/encrypt/orders"      { capabilities = ["update"] }
path "transit/decrypt/orders"      { capabilities = ["update"] }

Transit's encrypt and decrypt endpoints use update (a POST), not create or read; this catches people who write the obvious read.

Where you are now

You can now explain how a token's lifetime is decided, write policies that adapt to the caller and test them, give a workload an identity without a stored password, choose between VSO, the injector and CSI on a real trade-off, issue dynamic database credentials and revoke them in bulk, encrypt data with Transit and rotate keys with rewrap, issue short-lived certificates, and keep an audit trail that cannot silently stop. You also know what a production cluster is made of, what to monitor, how to back it up and what to check before moving from 1.x to 2.x.

The Senior guide takes the same ground as an owner of the platform: failure modes of the Raft cluster, scaling limits and what breaks first, the trust model and the supply chain, multi-tenancy with namespaces, upgrade and migration programmes, incident playbooks, and when Vault is the wrong tool. To see the neighbours in this catalogue, read Kubernetes, Helm, Terraform, Argo CD, Docker and Prometheus.

Sources

• Vault documentation
• Release notes
• Important changes
• Deprecation notices
• Upgrade guide
• Tokens
• Policies
• Seal and unseal
• Production hardening
• AppRole auth
• Kubernetes auth
• JWT/OIDC auth
• Database secrets engine
• Transit secrets engine
• PKI secrets engine
• KV v2
• Audit devices
• Vault Agent and Vault Proxy
• Vault Secrets Operator
• Helm chart
• Integrated storage (Raft)
• Telemetry
• Health API
• Raft commands

Part two of three. This is a cumulative review of Beginner and Mid-level Vault, organised as the scenario and trade-off questions a hiring panel asks someone who already runs Vault at work. The Senior level covers platform ownership. Read the model answers aloud; an answer you can say in two minutes is one you understand.

The target is Vault 2.1.1. Interviewers in 2026 increasingly probe the 2.x changes, so those are flagged.

Where this picks up

The two-minute mental model

Be able to say this without notes. Vault is an HTTP API. A client authenticates with an auth method and receives a token. The token carries policies, which allow or refuse paths. Paths are routed to secrets engines mounted at prefixes. Everything Vault stores is encrypted by the barrier before it reaches storage. On start-up Vault is sealed and cannot read its own data until it is unsealed.

A strong candidate then adds one sentence that shows depth: "Storage is untrusted, which is why it passes through the barrier, and why Vault fails closed when it cannot audit." A weak candidate stops at "it's a password manager for servers".

Scenario questions

"A service starts getting 403 after a deploy. What do you do?"

A strong answer is a method, in order. First reproduce with the exact identity: log in as the role the service uses and run the failing call, because testing with your own admin token proves nothing. Second, print the request with -output-curl-string to confirm the path and verb, which exposes missing data/ segments on KV v2 and wrong namespaces. Third, look at the token: vault token lookup for expiry and policies, vault token capabilities TOKEN PATH for the specific path, and vault read sys/internal/ui/resultant-acl for the merged view. Fourth, run the command with -output-policy to see the policy it requires and diff that against what was granted. Fifth, read the audit log for the request by accessor. Sixth, ask what changed in the deploy: a role renamed, a service account or namespace changed so the auth role's bound_ fields no longer match, a policy edited in Terraform, or a token that simply expired because the app does not renew.

What a weak answer misses: it jumps to "add * to the policy". What the best answers add: "after a 2.x upgrade, I also check for a policy template that now renders a wildcard, because from 2.0.1 that gives permission denied."

"Explain how you would give a Kubernetes application access to a database password."

Start with the goal: no static password anywhere a human can read it. Use the Kubernetes auth method, with a role bound to the exact service account and namespace. Use the database secrets engine so the application receives a unique credential with a one-hour lease, and Vault removes it when the lease ends. Then choose delivery. Vault Secrets Operator syncs to a Kubernetes Secret, refreshes it and restarts the Deployment on change; the injector writes a file into the pod through a sidecar; the CSI provider mounts a volume. The trade-off you must state: VSO produces Kubernetes Secret objects, which anyone with get secrets can read and which are only base64 in etcd unless encryption at rest is configured. Files in a memory-backed volume avoid that, at the cost of the application reading files rather than environment variables.

A strong answer also raises connection pools. A credential is valid for an hour, but the app's pool may hold connections for longer, and when Vault revokes the user, those connections break. So either the application handles re-authentication, or TTLs and pool lifetimes are aligned.

"How do you get the first credential to a workload that has no platform identity?"

This is the secret-zero question. The answer is AppRole with response wrapping. The RoleID is not secret and lives in configuration. The SecretID is generated by the deployment system with -wrap-ttl=120s and secret_id_num_uses=1, so the orchestrator hands over only a single-use wrapping token. The workload unwraps it once. Two properties follow: if an attacker unwraps first, the real workload's unwrap fails and you detect the interception, and a leaked wrapping token is worthless after two minutes. Also mention CIDR binding and that the identity allowed to mint SecretIDs is effectively able to impersonate the role, so that path is restricted to the deployment system.

The follow-up is "why not just use a token in an environment variable?" Answer: it does not expire in practice, it cannot be attributed to a workload, it appears in process listings, crash dumps and CI logs, and rotating it means touching every consumer.

"CI needs secrets. Show me a design that has no long-lived token."

Use JWT or OIDC auth. The CI platform issues each job a short-lived signed token whose claims say which repository, ref and environment it is. Vault verifies the signature against the platform's published keys. The role has bound_audiences and, critically, bound_claims restricting the repository and, for production secrets, the branch or environment. The resulting Vault token lasts ten minutes, and for a busy pipeline is a batch token so nothing accumulates in storage.

The trap to name unprompted: a role without bound_claims accepts tokens from any repository that can request one for that audience. A strong candidate explains what a pull request from a fork could reach.

"The cluster is sealed after a node restart. Walk me through it."

First find out which kind of seal. With Shamir, three of five key holders must enter shares, so the question is whether they are reachable; that operational burden is itself an argument for auto-unseal. With auto-unseal, a sealed node means Vault cannot reach the KMS or HSM: check IAM permissions on the key, network path, key policy, region, and whether the key was disabled or scheduled for deletion. journalctl -u vault shows the seal error. Check the health endpoint: 503 means sealed. Do not restart repeatedly, since each start re-attempts and adds noise; fix the dependency and the node unseals itself.

Also say what you would check for the other nodes and the cluster: does vault operator raft list-peers show quorum, and is an active node present? And mention the recovery keys: with auto-unseal they cannot decrypt the root key, they only authorise operations such as generating a root token or rekeying.

"You need to upgrade from 1.21 to 2.1. What is your plan?"

Read the release notes and the important-changes page. Snapshot the cluster and back up the config. Restore the snapshot into an isolated cluster, upgrade that first and run a smoke suite. Then search for the known 2.x behaviour changes. sys/rekey and sys/generate-root now need a token, so break-glass runbooks change. Paths with //, /./ or /../ are rejected, so scripts that concatenate strings break. Containers lost cap_ipc_lock in 2.0.2, so disable_mlock = true. Templated policies may not render wildcards. RSA keys above 8192 bits are refused. UBI images lost openssl in 2.0.4. Then upgrade standbys first, active last with vault operator step-down. Rollback is restoring the snapshot because storage is migrated forward.

A strong answer explicitly says what the smoke test contains: login from CI, a KV read, a database credential, a certificate, an audit line written.

"Vault gets slow over a few months. Why?"

Suspect leases and tokens before anything else. A CI role issuing service tokens with long TTLs, or an application that fetches a fresh dynamic credential per request, builds hundreds of thousands of leases. Symptoms are slow start-up and unseal, expiration backlog and growing storage. Look at vault.expire.num_leases and vault.token.count. Fixes: shorter TTLs, batch tokens for bursty automation, client-side caching via Agent or Proxy, revoking unused prefixes, and lease-count quotas on Enterprise. Then check the other usual suspects: storage commit time (vault.raft.commitTime) for slow disks, and a slow audit device, because every request waits for it.

Trade-off questions

Static KV secrets or dynamic secrets?

Dynamic secrets give unique, short-lived credentials that no human knows, so a leak has an expiry and revocation is a single prefix call. They cost more moving parts: the target system must be reachable by Vault, applications must cope with credentials that change, and the lease count grows. Static KV fits third-party API keys that Vault cannot rotate and configuration that is rotated by a human process. A mature setup uses both: dynamic where the backend supports it, KV v2 with versioning where it does not, and static roles where an account must persist but its password should rotate.

Vault Secrets Operator, Agent Injector, or CSI?

VSO is HashiCorp's current recommended native route: declarative custom resources, refresh, rollout restarts, and it works with anything that reads a Secret. Its cost is secrets stored as Kubernetes Secrets. The injector keeps values in files within the pod and needs no Kubernetes Secret, but adds a sidecar to every pod, which costs memory and complicates Jobs. CSI integrates with an existing Secrets Store CSI setup and mounts files. A sensible answer asks about the threat model and whether applications read env vars or files before choosing.

AppRole or platform authentication?

Platform identity wins whenever it exists because no secret is delivered to the workload. AppRole is the fallback for places with no verifiable identity, and its safety depends on wrapping, single-use SecretIDs and CIDR binding. Saying "AppRole is fine everywhere" is a weak answer.

Shamir seal or auto-unseal?

Auto-unseal removes the human from every restart and is essential in Kubernetes, where pods are rescheduled at any time. It adds a dependency: the KMS. If the key or the access to it is lost, Vault cannot unseal, so key protection, region placement and monitoring matter. Shamir has no external dependency and suits a small, attended or air-gapped installation. The key decision point is who will be awake at 03:00.

Batch tokens or service tokens?

Batch tokens are cheap: no storage write, no lease, good for CI and high-volume short jobs, but no renewal, revocation by ID, children or cubbyhole. Service tokens are persisted and flexible, which costs storage and lease load. Use batch for bursty automation, service for long-running applications and humans.

Namespaces or path prefixes for tenancy?

Namespaces (Enterprise and HCP) give genuine isolation: separate mounts, policies and tokens per tenant. In Community Edition you emulate tenancy with path prefixes, templated policies and per-team auth roles. That works but is convention, not a boundary, and a single mistake in a policy crosses tenants. Mention that namespace cost is per client, so avoid creating distinct identities for what is one workload.

Open-source Vault, Enterprise, or OpenBao?

State the licence facts precisely. Since 1.15 the source is under the Business Source License, and the free build is Vault Community Edition. OpenBao is a separate fork of the last MPL-licensed code; it is not the same product and its features should not be assumed to match. Enterprise adds namespaces, replication, performance standbys, Sentinel policies, Seal HA and more; HCP Vault Dedicated is the managed offer that remains since HCP Vault Secrets ended on 1 July 2026. A candidate who says "Vault is open source" without the licence nuance is out of date.

What a strong answer covers that a weak one misses

Commands to have ready

Quick-fire questions

What is the difference between a lease and a token TTL? A token TTL is how long the credential to Vault lives. A lease applies to what Vault issued (a database user, a cloud key, also service tokens), and revoking the lease deletes that thing on the target system.

Why does vault kv list fail when vault kv get works? The policy grants secret/data/... but not secret/metadata/..., which listing needs.

What does + mean in a policy path? Exactly one path segment. * matches any suffix and is only valid at the end.

What happens when two policies disagree? Capabilities are unioned, except deny, which always wins.

What is an accessor? A reference to a token that can look up, renew, revoke and check capabilities but cannot authenticate. Audit logs record accessors.

Why is reading database/creds/app a read capability? Policies describe HTTP verbs on paths, not the engine's effect. A GET creates the user.

What is the default TTL? Both default and maximum are 768 hours (32 days), set by default_lease_ttl and max_lease_ttl, and tunable per mount and role.

What are the health status codes you should know? 200 active, 429 unsealed standby, 472 DR secondary, 473 performance standby, 474 standby that cannot reach the active node, 501 not initialised, 503 sealed, 530 removed from the cluster.

How many nodes and why? Three or five voters across zones, odd numbers, because Raft needs a majority. Five tolerates two failures; three tolerates one.

What changed for rekey in 2.0? It and generate-root need a Vault token as well as the shares. enable_unauthenticated_access restores the old behaviour if you must.

Why must containers set disable_mlock = true now? From 2.0.2 the official image no longer carries cap_ipc_lock, so the container cannot call mlock. With Raft, the setting must be explicit since 1.20 anyway.

Why does Transit use update capability? Encrypt and decrypt are POST requests to transit/encrypt/<key> and transit/decrypt/<key>, which map to update, so a policy with only read fails.

How do you rotate a Transit key safely? Rotate, rewrap stored ciphertext through transit/rewrap, verify, and only then raise min_decryption_version.

What does Vault do when no audit device can write? It refuses requests. It fails closed.

Common follow-up probes

An interviewer often asks "and what if that fails?" after each answer. Prepare the next layer.

"What if the KMS is down during a rolling restart?" Nodes that are already unsealed keep serving; a node that restarts stays sealed until the KMS returns. So roll one node at a time, verify it unseals before continuing, and never restart all three together. This is also why a failed first node is a stop signal for the rollout.

"What if the active node loses contact with the followers?" Raft elects a new leader when a quorum can communicate. The old leader steps down if it loses quorum. Clients see a brief window of errors and retry. Load balancer health checks on /v1/sys/health should move traffic to the new active node, so a check that treats only 200 as healthy sends no traffic to standbys, which is intended for Community Edition.

"What if someone reads a secret they should not have?" Use the audit log to find the request by accessor and path, revoke the token with vault token revoke -accessor, revoke the dynamic leases by prefix, rotate the affected static secrets (a new KV version does not make the old one safe; the old value was still exposed), and tighten the policy. The hard question is always "was it already copied", which is why short TTLs matter.

"What if Terraform's state contains a secret?" Treat it as compromised to the extent state is readable. Rotate the secret, stop reading values in Terraform, and let the application fetch it at run time instead.

"How would you test a policy change before applying it?" Write it to a scratch path or a dev cluster, create a token with it, and run vault token capabilities and the real commands with -output-policy for comparison. Also run vault policy fmt. In Terraform, review the diff as code.

Sixty-second self-test

Answer without looking back.

1. Name the three token prefixes and what each is for.
2. Write the path a policy needs to read a KV v2 secret app/db mounted at secret/.
3. What makes a wrapping token safer than just passing the SecretID?
4. Why must a JWT role have bound_claims?
5. What is the quorum for a five-node Raft cluster?
6. Name two things an audit device hashes, and what happens when all devices fail.
7. Which command revokes every credential a database role issued?
8. What changed about sys/generate-root in 2.0?
9. Why does rotate-root matter for the database engine?
10. A standby returns 474. What do you check?

Check yourself: hvs., hvb., hvr. (service, batch, recovery); secret/data/app/db; single use, short life, and tamper detection; without it any repository that can mint a token for the audience is accepted; three; tokens and secret data (HMAC-SHA256), and Vault refuses requests; vault lease revoke -prefix database/creds/<role>; it needs a token as well as the shares; so no human knows the password Vault uses; api_addr, cluster_addr and port 8201 connectivity.

Sources

• Vault documentation
• Important changes
• Tokens
• Policies
• Health API
• Production hardening

Part two of three. This file is the working companion to the Mid-level guide: the errors you will actually meet when Vault is part of your job, the practices that pay off most in order of leverage, practice cards to drill, and operational recipes you can paste into a runbook. It assumes Vault 2.1.1 and calls out where 2.x behaves differently from the tutorials you may have learned from.

Common errors at this level

Each entry gives the symptom, the cause and the fix. The causes are more useful than the fixes, because the same cause hides behind several messages.

permission denied on list, but get works

Symptom. vault kv get -mount=secret app/db works; vault kv list -mount=secret app returns Code: 403 ... permission denied. Cause. KV v2 splits a secret across API paths. Reading uses secret/data/app/db; listing uses secret/metadata/app. The policy grants only data/. Fix. Add path "secret/metadata/app/*" { capabilities = ["list", "read"] }. Verify with vault token capabilities TOKEN secret/metadata/app.

preflight capability check returned 403

Symptom. The KV CLI says preflight capability check returned 403, please ensure client's policies grant access to path "secret/". Cause. The vault kv helper reads sys/internal/ui/mounts/<path> to learn whether the mount is KV v1 or v2, and the token cannot. Fix. Grant read on that internal path, or bypass the helper with the explicit API path and vault read secret/data/app/db.

Reading a database credential is refused although "it only reads"

Symptom. vault read database/creds/app returns 403. Cause. People write list or nothing, assuming a read is harmless. The endpoint requires read, even though it creates a database user as a side effect. Fix. path "database/creds/app" { capabilities = ["read"] }. Scope to the exact role, never database/creds/*.

The app worked for an hour, then every request fails

Symptom. Healthy at first, 403 after exactly the token's TTL. Cause. The application logged in once and never renewed, or the token hit its max TTL (renewal cannot pass it). Often the role says token_ttl=1h with nothing to renew. Fix. Put Vault Agent or Vault Proxy in front (they renew and re-authenticate), or make the client re-login on 403. For daemons that really must run for days, use a periodic token. Check with vault token lookup and read ttl, expire_time and renewable.

Database connections die when a lease ends

Symptom. Bursts of password authentication failed or role does not exist every hour. Cause. Vault revoked the dynamic user at lease end, but the application's connection pool still held connections and tries to open new ones with the old credential. Fix. Make the pool's maximum connection lifetime shorter than the lease TTL, and have the client fetch a new credential when it reconnects. With Agent or VSO the file or Secret changes and the app must reload it or restart (restart_on_secret_changes in Agent, rolloutRestartTargets in VSO).

no handler for route

Symptom. * 1 error occurred: * no handler for route "secret/data/x". route entry not found. Cause. Either the path is misspelled or the engine is not mounted there, for example it was mounted at kv/ in production and secret/ in your dev server. Fix. vault secrets list and compare. Use mount names from configuration, not from memory.

The x509 error

Symptom. x509: certificate signed by unknown authority. Cause. The client does not trust the CA that signed Vault's certificate. Very common behind a proxy that re-signs TLS traffic, in a container that lacks your corporate CA, or with a private CA. Fix. Give the client the CA: export VAULT_CACERT=/path/ca.pem. Do not set VAULT_SKIP_VERIFY=true in anything but a disposable dev setup, because it removes the only protection the token has in transit.

Vault is sealed (503) after a restart

Symptom. Health returns 503 and clients get Code: 503. Errors: * Vault is sealed. Cause. After a restart a node is sealed. With Shamir, someone has to unseal it. With auto-unseal, the node cannot reach or use the KMS key. Fix. Shamir: vault operator unseal until the threshold is reached. Auto-unseal: read journalctl -u vault for the KMS error and fix IAM, network or the key's state.

The container will not start: mlock and permissions

Symptom. A Raft start refusing because disable_mlock is unset, a memory-lock failure, or mkdir /opt/vault/data/...: permission denied. Cause. Three separate changes converge here. Since 1.20, disable_mlock has no default with Raft and must be set explicitly. Since 2.0.2 the official image has no cap_ipc_lock, so mlock cannot work in a container. Since 2.0 the image runs as the vault user, so a volume owned by root cannot be written. Fix. disable_mlock = true (and disable or encrypt swap), plus chown vault:vault with mode 0750 on the data volume, or fsGroup in Kubernetes.

403 on rekey or generate-root after upgrading to 2.0

Symptom. An old break-glass runbook suddenly returns 403. Cause. From 2.0 these endpoints need a Vault token as well as key shares. Fix. Update the runbook to authenticate first. Only if you must keep the old behaviour, set enable_unauthenticated_access = ["rekey", "generate-root"] at the top level of the server configuration, and record why.

Scripts fail with path errors after 2.0

Symptom. Requests with a double slash are rejected. Cause. Vault now refuses paths containing //, /./ or /../, including URL-encoded forms. Fix. Normalise paths in the script. Build them as "${MOUNT%/}/foo" rather than concatenating a trailing slash with a leading one.

A templated policy suddenly denies everything

Symptom. After 2.0.1, callers of a path such as secret/data/{{identity.groups.names.x.metadata.region}}/* get permission denied. Cause. The metadata value contains * or +, and Vault refuses to render wildcards into a policy path. Fix. Use literal values in the metadata, and create one policy block per region if you need sets.

The policy file will not load

Symptom. An error about duplicate attributes. Cause. Since 1.21, duplicate HCL attributes are errors (1.20 only warned). Fix. Delete the duplicate key. VAULT_ALLOW_PENDING_REMOVAL_DUPLICATE_HCL_ATTRIBUTES exists as a stopgap and should be removed from your environment as soon as the file is fixed.

too many concurrent raft retry joins in progress

Symptom. Nodes fail to join on 2.1.1. Cause. Concurrent retry_join workers are capped at 20. Fix. Stagger node start-up or reduce the number of join targets. For three to five nodes you should not be near the cap.

HTTP 431 on large tokens (Enterprise)

Symptom. Requests carrying a big JWT fail with 431. Cause. max_token_header_size defaults to 8 KB since 2.0. Fix. Raise it in the listener: max_token_header_size = 16384.

The practices that pay off most

In order of leverage. If you can only do four things this quarter, do the first four.

1. Remove every long-lived tokenReplace tokens in CI variables and config files with JWT or OIDC login, Kubernetes auth, or cloud IAM auth. This one change removes the most common incident cause: a leaked credential that never expires.
2. Enable two audit devices and alert on failuresYou cannot investigate what you did not log, and Vault fails closed if every device fails. Two devices on different sinks, with alerts on the failure metrics.
3. Automate unseal and test the snapshotAuto-unseal removes the 03:00 human step. A scheduled restore into a scratch cluster proves the backup is real.
4. Prefer dynamic credentials with short TTLsDatabase and cloud engines turn a leak into a minutes-long problem, and lease revoke -prefix gives you an incident button.
5. Scope every policy to an exact pathOne policy per workload, no * where a name will do, no sudo unless a system path demands it. Use -output-policy to derive them.
6. Put Vault configuration in TerraformMounts, policies and roles in version control, reviewed like code, with a separate break-glass path that the pipeline cannot edit.
7. Use an agent or operator, not hand-written clientsAgent, Proxy and VSO handle login, renewal and rotation so your applications do not reimplement them badly.
8. Monitor the four signalsUnsealed, active node present, audit failures, lease and token counts.

Practice cards

Drill these until the answer is automatic.

Operational recipes

Recipe: least-privilege policy from a command

vault kv put -output-policy -mount=secret payments/db password=x
vault read -output-policy database/creds/payments-ro
vault write -output-policy transit/encrypt/orders plaintext=$(echo -n x | base64)

Paste each output into a policy file, merge, run vault policy fmt, and write it. You now have a policy derived from the real calls instead of a guess.

Recipe: find out why a token is refused

vault token lookup                                    # policies, ttl, renewable
vault token capabilities secret/data/payments/db      # verdict for one path
vault read sys/internal/ui/resultant-acl              # merged view across policies
vault kv get -output-curl-string -mount=secret payments/db

Recipe: a JWT role for CI, tightened

vault write auth/jwt/role/ci-prod \
    role_type=jwt user_claim=repository \
    bound_audiences=https://github.com/ORG \
    bound_claims='{"repository":"ORG/payments","environment":"production"}' \
    token_type=batch token_policies=ci-prod token_ttl=10m

The environment claim is what stops a branch build from reaching production credentials. Check the exact claim names your CI platform emits by decoding one real token, rather than copying this one.

Recipe: a wrapped SecretID handoff

WRAP=$(vault write -f -wrap-ttl=120s -field=wrapping_token auth/approle/role/web/secret-id)
# Transport $WRAP to the workload, then on the workload:
SECRET_ID=$(VAULT_TOKEN=$WRAP vault unwrap -field=secret_id)
vault write auth/approle/login role_id="$ROLE_ID" secret_id="$SECRET_ID"

Recipe: nightly snapshot with a narrow policy

path "sys/storage/raft/snapshot" {
  capabilities = ["read"]
}

vault operator raft snapshot save "/backups/vault-$(date +%F).snap"
find /backups -name 'vault-*.snap' -mtime +14 -delete

Run it from a scheduler with a token from a role bound to that one policy, copy the file off-cluster and alert if no new file appears. Encrypt the destination; a snapshot holds everything.

Recipe: a safe rolling upgrade

vault operator raft snapshot save pre-upgrade.snap
vault operator raft list-peers          # identify leader and followers
# On each follower in turn: stop, replace the binary, start, then
vault status                            # Sealed false, new Version
vault operator raft list-peers          # node is back and voting
# Finally on the leader:
vault operator step-down
# ...then upgrade the former leader the same way

Do one node at a time, verify before moving on, and stop the whole procedure if a node fails to unseal. On Helm, pin server.image.tag, and with the OnDelete strategy delete pods one at a time, standbys first.

Recipe: Vault Agent template for a dynamic credential

{{ with secret "database/creds/payments-ro" }}
export DB_USER="{{ .Data.username }}"
export DB_PASS="{{ .Data.password }}"
{{ end }}

Agent keeps the lease alive, and when it nears its max TTL fetches a new credential and re-renders the file. Combine with exec and restart_on_secret_changes = "always" so the process picks up the change.

Recipe: a Prometheus alert set

groups:
  - name: vault
    rules:
      - alert: VaultSealed
        expr: vault_core_unsealed == 0
        for: 1m
      - alert: VaultNoActiveNode
        expr: sum(vault_core_active) == 0
        for: 2m
      - alert: VaultAuditFailure
        expr: increase(vault_audit_log_request_failure[5m]) > 0

Metric names in Prometheus format use underscores in place of dots. Adjust to what /v1/sys/metrics?format=prometheus actually emits on your version, and add lease and token growth rules once you know your baseline. See Prometheus for writing the rules well.

Recipe: a health check the load balancer can trust

curl -s -o /dev/null -w "%{http_code}\n" https://vault.example.com:8200/v1/sys/health
# 200 active, 429 standby, 503 sealed, 501 not initialised
curl -s -o /dev/null -w "%{http_code}\n" "https://vault.example.com:8200/v1/sys/health?standbyok=true"

Reading an incident the Vault way

When something breaks at work, the order in which you look matters more than how clever you are. The following sequence works for most Vault-related incidents, and each step rules out a whole class of causes.

Two habits make this fast. First, keep a small synthetic probe outside the cluster that logs in through a real auth method, reads a known secret and reports latency. It answers "is Vault working for a client" in a way that "the process is running" never does. Second, keep the commands above in a runbook that the on-call engineer has actually run on the scratch cluster, because the first time you type vault operator raft list-peers should not be during an outage.

A word on the human side. Decide in advance who may revoke in an incident, who holds the recovery keys, and how to reach them. The 2.0 change that made rekey and generate-root need a token makes this planning more important: if the only people who can produce a token are asleep or have left the company, the key shares alone no longer get you in.

Habits worth adopting now

A short checklist

Before you call a Vault integration finished:

• The workload authenticates with a platform identity, or a wrapped single-use SecretID.
• The role's bound claims, service account and namespace are exact, with no *.
• The policy names exact paths, with metadata/ added where listing is needed, and was derived with -output-policy.
• TTLs are short and the client renews or re-authenticates; the database pool lifetime is shorter than the lease.
• Rotation is handled: Agent restarts, VSO rolloutRestartTargets, or an application reload.
• Two audit devices are on and alerted; metrics are scraped; an external probe does a real login and read.
• A snapshot is taken on a schedule, stored off-cluster and restored into a scratch cluster at least monthly.
• disable_mlock = true, TLS is on, the data directory belongs to vault, nodes have explicit node_ids.
• The upgrade checklist for 2.x has been walked, including rekey callers, path concatenation, wildcard templates, RSA sizes and image tooling.

Sources

• Vault documentation
• Important changes
• Production hardening
• Troubleshooting tutorial
• Telemetry

This is part three of three. It takes every topic from Beginner and Mid one level further, where each one gains a failure mode, a scaling limit or an ownership question, and it adds the work you carry when Vault is not your tool but your team's product.

Up to now the questions were about getting a secret out of Vault. From here they change: what happens to forty teams when the leader loses its disk, who can produce a root token at three in the morning, why a single chatty CI system can exhaust the lease table, and how you upgrade a cluster that every deployment in the company depends on without anyone noticing. Vault is unusual among infrastructure tools because it sits on the critical path of everything else and is also the most attractive target in your estate. A platform owner has to be good at both availability and paranoia, and the two often pull in opposite directions.

This guide is written against Vault 2.1.1 (released 16 September 2026). Vault 2.0 shipped on 14 April 2026 and was the first major version since 1.0, but the number reflects IBM's versioning and support lifecycle rather than a rewrite: the HTTP API is still /v1/... and the CLI is recognisably the same. The breaking changes that matter are specific and small, and they are covered where they bite.

Where this picks up

I am starting with the architecture, because every failure mode in this guide is easier to predict once you can draw the cluster from memory.

The architecture you actually operate

You know the one-line model: clients authenticate, receive a token, and policies decide which paths the token may touch. As an operator you need a second model, which is about where state lives and who is allowed to change it.

A Vault cluster is a set of server processes that share one storage backend. Everything Vault persists passes through the barrier, an AES-256-GCM layer between the logical core and storage, so storage is treated as untrusted. There are three layers of keys. Your data is encrypted by an encryption key held in a keyring. The keyring is encrypted by the root key (older documents call it the master key; use "root key" today). The root key is protected either by the Shamir unseal key, split into shares, or by an auto-unseal device such as a cloud KMS or an HSM. A node that can read storage but cannot yet decrypt the keyring is sealed, and it answers almost nothing.

At any moment exactly one node is active. It is the only node that writes, and it holds the leader lock. The others are standbys: unsealed, healthy, and forwarding requests to the active node over the cluster port, or redirecting clients to its api_addr. In Community Edition a standby does no work of its own. In Enterprise, performance standbys can serve reads locally, which changes the scaling story considerably and is discussed later.

The consequences follow directly. Because reads and writes on Community Edition all land on the active node, vertical headroom on that node is your throughput ceiling, and adding standbys buys you availability, not capacity. Because the active node also runs lease expiration and credential rotation, a burst of expirations competes with client traffic for the same CPU. Because the barrier key never leaves memory in plaintext form, a restart of an auto-unsealed node depends on its KMS being reachable at that moment, which quietly makes your cloud KMS a dependency of your secrets platform.

What each failure looks like from the outside

It is worth learning the health endpoint's vocabulary, because it is how your load balancer, your alerting and your runbooks all describe a node. GET /v1/sys/health answers 200 when the node is active, 429 when it is an unsealed standby, 473 for a performance standby, 472 for a disaster recovery secondary, 474 for a standby that cannot find the active node, 501 when uninitialised, 503 when sealed and 530 when the node has been removed from the cluster. Each can be remapped with a query parameter such as standbyok, perfstandbyok or sealedcode. A load balancer that treats everything except 200 as unhealthy sends traffic to exactly one node, which is correct for Community Edition; a balancer that accepts standbys forces every request through an extra forwarding hop. Neither is wrong, but you should choose deliberately and know that 474 is the code that means your api_addr or cluster_addr is wrong or port 8201 is blocked.

Integrated storage, quorum and Autopilot

Integrated Storage, which is Raft, is the recommended backend and the only one this guide treats as a default. Consul is still a supported HA backend, but it means running and securing a second distributed system for the sake of one more failure domain. Other backends (file, PostgreSQL, S3, DynamoDB and so on) are community-supported rather than HashiCorp-supported, and Raft cannot be combined with a separate ha_storage.

Raft is a consensus protocol. A write is committed once a quorum of voting nodes has written it to their logs, and quorum is ceil((N+1)/2). That arithmetic is the most useful thing to have memorised:

The official guidance for a standard production deployment is five servers across availability zones, which tolerates two failures, and you should use odd numbers. Going from five to seven rarely improves availability but does add replication work on every write, because the leader waits on a larger majority. Two voters is worse than one in practical terms: you pay for two nodes and lose the cluster if either fails. A single node is acceptable for a lab and never for production.

The failure modes of Raft you will meet

Loss of quorum. With five voters, losing three means no leader can be elected and no write succeeds. This is a real outage, and recovery is surprisingly manual. If the dead nodes are gone for good, you recover with a peers.json file (the documented Raft recovery path) or, more commonly, restore a snapshot into a fresh cluster. The mistake is to start inventing your own procedure in the middle of the incident. Write down, in advance, which of those two paths your team uses and who is authorised to do it.

Slow disks and noisy neighbours. Raft is latency-sensitive because every write waits for fsync on a majority. The metric to watch is vault.raft.commitTime; a healthy cluster shows low milliseconds, and a cluster on overcommitted storage shows spikes that turn into leader elections. Follower freshness is visible in vault.raft.leader.lastContact. The storage stanza exposes performance_multiplier (default 5, range 1 to 10), which scales Raft timeouts: a higher value makes the cluster more tolerant of slow networks at the cost of slower failure detection. Treat it as a patch for an environment you cannot fix, and put Raft data on its own fast SSD rather than tuning around a bad disk.

Memory, not just disk. Integrated Storage uses a memory-mapped database, so the working set should fit in RAM. This is also why disable_mlock is set to true with Raft. Size nodes for the data set plus headroom for lease and token tables.

The 1 MiB entry limit. max_entry_size defaults to 1048576 bytes. A KV secret, or a large entry in the mount table, that exceeds it fails. A team storing a JSON blob of certificates in a single KV value finds this out in production. Enforce a size convention in your onboarding and reject the anti-pattern in review rather than raising the limit.

Joining and dead-server cleanup. New nodes join with vault operator raft join, or automatically with retry_join. Since 2.1.1 the number of concurrent retry_join workers is capped at 20, and surplus attempts fail with too many concurrent raft retry joins in progress, so a configuration that lists dozens of targets needs trimming. Autopilot, which is on by default, stabilises new servers: a new node joins as a non-voter and is promoted only after it has been healthy for server_stabilization_time (default 10s). Dead-server cleanup is opt-in, because cleanup_dead_servers defaults to false. That default is conservative and correct for a newcomer, but an operator who owns an autoscaling group should turn it on deliberately and pair it with min_quorum, which is required when cleanup is enabled, so that Autopilot can never remove so many peers that it shrinks the cluster below a safe size.

vault operator raft autopilot get-config
vault operator raft autopilot set-config \
  -cleanup-dead-servers=true \
  -dead-server-last-contact-threshold=10m \
  -min-quorum=3
vault operator raft autopilot state
vault operator raft list-peers

Other Autopilot defaults worth knowing: dead_server_last_contact_threshold of 24h, last_contact_threshold of 10s and max_trailing_logs of 1000. Enterprise adds automated upgrades (autopilot_upgrade_version), redundancy zones and non-voting members (retry_join_as_non_voter), which let you add read-scaling followers without enlarging the quorum.

Snapshots are the backup, and the backup is worth nothing until restored

vault operator raft snapshot save backup.snap produces a point-in-time copy of the whole data set, and snapshot inspect tells you what is inside. snapshot restore replaces the cluster's state, with -force for a snapshot taken under a different seal. Enterprise adds automated snapshots (sys/storage/raft/snapshot-auto/config/<name>) to cloud object storage, which is the route to take if you have a licence. On Community Edition you write that job yourself: a scheduled task with a dedicated, least-privilege token that calls snapshot save, ships the file to encrypted object storage in a separate account, and alerts when the file is absent.

Two properties make snapshot discipline different from ordinary database backups. First, the snapshot is encrypted by the barrier, so restoring it elsewhere requires the matching seal: the same KMS key for auto-unseal, or the Shamir shares. A snapshot without access to its seal is a large, useless file. Second, a snapshot contains everything: tokens, leases, identity, policies. Restoring it resurrects revoked credentials if they were alive when the snapshot was taken, so the post-restore procedure in a security incident includes revoking and rotating, not merely restoring.

# On a scratch cluster, never on production
vault operator raft snapshot save /tmp/prod-$(date +%F).snap
vault operator raft snapshot inspect /tmp/prod-$(date +%F).snap

# Restore into the isolated copy
vault operator raft snapshot restore -force /tmp/prod-2026-09-30.snap
vault kv get -mount=secret canary/restore-test

The seal, the key ceremony and the root token problem

The seal is where Vault's confidentiality guarantee becomes an organisational process. At the lowest level it is an encryption setup; at the level you own, it is a question of which humans and which cloud roles can cause Vault to open, and whether any one of them can do so alone.

Shamir versus auto-unseal, as a risk decision

With the default Shamir seal, vault operator init splits the unseal key into shares (the CLI default is -key-shares=5 -key-threshold=3). Every restart needs a threshold of shareholders to be awake and present. That is excellent for separation of duties and dreadful for availability, because an unattended node restart at 03:00 leaves the node sealed until three people respond. It also means the unseal key shares are long-lived secrets that have to be stored by people, and people leave companies.

Auto-unseal moves the protection of the root key to a KMS or HSM: awskms, azurekeyvault, gcpckms, ocikms, alicloudkms, transit, or pkcs11 with Enterprise. Restarts then need nobody. You trade human availability for a cloud dependency, and you change what init produces: instead of unseal keys you receive recovery keys (default 5 shares with a threshold of 3). Recovery keys cannot decrypt the root key. They authorise privileged operations such as generating a root token or rekeying. That distinction is the part interviewers like, and it has a practical corollary: with auto-unseal, the cloud IAM policy on the KMS key is the real unseal key. Whoever can Decrypt with that key from a machine that can read the Raft data can open the cluster. Review that policy as seriously as you would the recovery key custody.

For Gulf and Egyptian employers with data-residency constraints, the seal is also a residency decision: a KMS key in a region outside the jurisdiction holds the key that protects data stored inside it. Keep the key in the same region as the cluster, and confirm with legal whether a multi-region key counts as leaving the country. Enterprise also offers Seal HA, where two or three auto-seals are configured at once so the loss of one KMS does not block a restart, and where online seal migration becomes possible; a Shamir seal cannot take part in it.

Rekey and generate-root after 2.0

The operator commands that change key custody were unauthenticated before Vault 2.0: anyone who held enough key shares could use them. From 2.0, sys/rekey, sys/generate-root and sys/replication/dr/secondary/generate-operation-token require a Vault token as well as the key shares. That is a deliberate defence in depth, and it changes your break-glass process: the person who holds the recovery keys also needs a token with the right capabilities, which raises the question of what you do when Vault is sealed and no token can exist. The escape hatch is the top-level config key enable_unauthenticated_access, which accepts a list containing "rekey", "generate-root" and "generate-operation-token". Think carefully before using it. Enabling it restores the old behaviour for exactly the operations you list, and your threat model should decide whether the availability it buys in a disaster is worth the larger attack surface on every other day.

# Only if your break-glass design genuinely needs unauthenticated generate-root
enable_unauthenticated_access = ["generate-root"]

The commands themselves are unchanged. The rekey flow uses vault operator rekey -init -key-shares=5 -key-threshold=3, then each shareholder supplies a share; the rotation of the encryption key (keyring) is a different command, vault operator rotate, which does not change who can unseal. Confusing rekey with rotate is a classic slip, so make the runbook name both and say which one you want.

The root token deserves its own policy. Only three things create one: the initial token from vault operator init, another root token, or vault operator generate-root with a quorum of key holders. Best practice is to revoke the initial root token once bootstrap is finished and to generate a new one only for an emergency, using a quorum, then revoke it again. The audit log will show the generate-root ceremony, and the procedure should require two people so no single person can create an unobserved god token.

Scaling: which component breaks first

Vault rarely fails by running out of raw requests per second. The components that give way first are slow-moving: tables that grow for months, then tip over during an unrelated event. Knowing the order lets you instrument the right thing before it hurts.

First, the lease table. Every dynamic secret and every service token has a lease held in memory and in storage, and the active node's expiration manager walks them. A fleet of autoscaled pods that each request database credentials with a one-hour TTL, every time they start, produces lease churn proportional to your deployment rate, not your traffic. A CI system that logs in for every job with a service token adds a token and a lease per job, and the tokens live until their TTL ends. The gauge to watch is vault.expire.num_leases, and the token count is vault.token.count. When either climbs steadily without a plateau, something is requesting and never revoking. The cheap fixes come first: shorten TTLs to the real need, make clients revoke on shutdown, and reuse credentials through Vault Agent or VSO caching rather than logging in and fetching on every job step.

Second, batch tokens for volume. A batch token (prefix hvb.) is an encrypted blob that is not stored: no storage write at creation, no lease, no expiration work. It cannot be renewed or revoked individually, cannot have children and cannot be a root token, which is the price of being cheap. For high-volume short jobs, such as thousands of serverless invocations or ephemeral CI runners, batch tokens move load off the write path entirely. The decision rule is simple: if the workload never needs to renew or revoke its own token, and will be done within minutes, use batch tokens; otherwise service tokens. You opt in on the role, for example token_type=batch on an auth role.

Third, the single writer. All writes go through the leader and through Raft consensus. KV writes, token creation, lease creation and rotation all compete. When the commit time climbs, the answer is almost never "add nodes"; it is to remove writes. Stop using Vault as a general key-value database for frequently changing data, stop issuing a fresh credential per request, and look at who is creating service tokens unnecessarily. Enterprise performance replication and performance standbys add capacity for reads and for locality, but they do not make the leader faster at writes.

Fourth, quotas as a fuse. Rate-limit quotas (sys/quotas/rate-limit/<name>) exist in Community Edition and cap requests per second for a path, a mount or a namespace; a tripped quota returns HTTP 429 with the message rate limit quota exceeded. Lease-count quotas (sys/quotas/lease-count/<name>) are Enterprise. The right design is a global rate-limit quota as a last-resort fuse, plus narrower ones on the mounts you know will be abused, such as the login path of an auth method used by thousands of ephemeral clients.

# A global fuse: no more than 500 requests per second in total
vault write sys/quotas/rate-limit/global rate=500 interval=1s

# A narrower one on a noisy login path
vault write sys/quotas/rate-limit/ci-login path=auth/jwt/ rate=50 interval=1s

# Enterprise: bound the number of leases a mount may hold
vault write sys/quotas/lease-count/db path=database/ max_leases=10000

Quota responses include Retry-After and the Go API client honours it, which is useful: well-behaved clients slow down instead of hammering. The clients that do not are the ones to find in the audit log.

Fifth, the things that cannot be tuned. Entry-size limits (1 MiB by default per Raft entry), the mount and namespace table, and the cost of very large identity stores are structural. For the mount table the answer is architectural: do not create a mount per microservice, because every mount is an entry in a single table. Enterprise can raise the table limit with max_mount_and_namespace_table_entry_size, but having to is a sign the design needs to consolidate.

Revocation storms and leases that will not die

A revocation at scale is itself a load event. vault lease revoke -prefix database/creds/app is the right incident tool, and it is also capable of generating thousands of database DROP ROLE statements in a burst. If the database is slow, revocations fail and leases become irrevocable, which stay in the table until they are force-revoked. Enterprise has remove_irrevocable_lease_after for that case. The lesson is to rehearse a prefix revoke on a staging copy and rate-limit the operation, rather than learning its cost during an incident.

Observability you need before the incident

The Prometheus endpoint is GET /v1/sys/metrics with the prometheus format requested; it needs a token with read on sys/metrics unless the listener sets telemetry { unauthenticated_metrics_access = true }. In the telemetry stanza, prometheus_retention_time (default 24h) must be non-zero for Prometheus to work, and disable_hostname = true keeps metric names stable across node replacements.

telemetry {
  prometheus_retention_time = "30s"
  disable_hostname          = true
}

listener "tcp" {
  address       = "0.0.0.0:8200"
  tls_cert_file = "/opt/vault/tls/tls.crt"
  tls_key_file  = "/opt/vault/tls/tls.key"
  telemetry {
    unauthenticated_metrics_access = false
  }
}

The alerts a platform owner should have, in rough order of priority:

1. vault.core.unsealed equal to 0 on any node: a node is sealed.
2. vault.core.active showing zero active nodes for more than a short period: no leader.
3. vault.audit.log_request_failure or vault.audit.log_response_failure above zero: an audit device is failing and Vault may be about to refuse requests.
4. vault.raft.leader.lastContact and vault.raft.commitTime trending up: storage or network trouble ahead of an election.
5. vault.expire.num_leases and vault.token.count growth against a baseline: the slow leak.
6. TLS certificate expiry on the listener and the Raft peers, measured by a blackbox probe, because a certificate that expires silently takes out a whole cluster at once.
7. On Enterprise, vault.core.license.termination_time_epoch: the licence clock.

If you use the OpenTelemetry or Prometheus guides in this catalogue, Prometheus covers the scrape and alert mechanics, and this is the list of what to put into them.

The security model, and how to defend it

Vault's threat model assumes that storage and the network are untrusted and that the Vault process's memory is protected. Everything you do as a hardening owner either enforces the second half of that sentence or protects the human and automation credentials that can reach the API.

Host and process hardening

The official production-hardening guidance reduces to a short list that is easy to audit. Run Vault as an unprivileged vault user with minimal write access. Use TLS 1.2 or above end to end, preferring 1.3, and set HSTS where an HTTP front-end exists. Disable swap and core dumps, because a core dump of the Vault process is a plaintext secret dump (LimitCORE=0 in the systemd unit). Prefer single-tenant hosts: bare metal first, then a VM, then a container, with no other workloads on the node. Firewall to ports 8200 and 8201 from known sources only. Keep NTP correct, because expiry and TLS both depend on the clock. Restrict access to the storage directory, pass secrets on standard input and avoid shell history, and keep Vault upgraded.

Extended hardening adds systemd sandboxing (ProtectSystem=full, PrivateTmp=yes, NoNewPrivileges=yes), SELinux or AppArmor profiles, raised ulimits and, on Enterprise, an administrative namespace. There is also deny_slash_in_templated_paths = true, which blocks a class of templated-policy bypass where an identity value containing a slash expands into extra path segments.

mlock, containers and the 2.0.2 change

Memory locking (mlock) stops the operating system from swapping Vault's memory to disk. Two recent changes make this a decision you must now record rather than inherit. Since Vault 1.20, disable_mlock is required to be set explicitly with integrated storage, and it has no default: Vault refuses to start until you choose true or false. And since 2.0.2 the official container image no longer carries the cap_ipc_lock capability, so Vault in a container cannot call mlock() at all. The practical rule: on Raft, in containers, set disable_mlock = true, and compensate by disabling swap on the node or encrypting it. Memory-mapped Raft data is the reason mlock is pointless for this storage backend in any case.

disable_mlock = true
cluster_name  = "prod-ruh-1"
api_addr      = "https://vault-0.vault.example.com:8200"
cluster_addr  = "https://vault-0.vault.example.com:8201"

Container image changes that break debug habits

Since 2.0 the image runs as the vault user by default, which is good, and breaks volume mounts whose ownership assumed root. Since 2.0.4 the UBI images no longer contain gnupg, openssl or procps, so a debug script or an init container that ran openssl inside the Vault image fails. The fix is a separate tooling container or a debug pod with the tools, never adding packages to the Vault image itself, which breaks your supply-chain story. The Docker guide's senior level covers image pinning by digest, which you should apply to the Vault image too.

Policy as a security boundary

Policies are deny-by-default and path-based, and deny always wins. Four details trip up senior reviews:

1. Globs only at the end. A * is valid only as the final character of a path; + matches exactly one segment. Write policy paths with + where you mean "any tenant name" and reserve * for the tail.
2. Templated policies and wildcards. As of 2.0.1, identity templates may not render + or * into a policy path. A path such as kv/data/{{identity.groups.names.x.metadata.region}}/*, where the rendered metadata contains a wildcard, now returns permission denied. This closes an injection route where someone sets metadata to *. Audit your metadata for stored wildcards before upgrading.
3. Parameter constraints on lists. Since 1.21, allowed_parameters and denied_parameters compare lists on a "contains all" basis instead of exact match. Policies that relied on an exact list comparison change behaviour. The environment variable VAULT_LEGACY_EXACT_MATCHING_ON_LIST reverts it temporarily. And allowed_parameters and denied_parameters do not work with KV v2 at all, so do not try to constrain values there.
4. Lowercase policy names. From 2.0.4, Vault lowercases the policies field before it checks a denied_parameters constraint, closing a bypass that used a mixed-case policy name. If you relied on that constraint to stop a role from granting itself a policy, upgrade.

Duplicate attributes in HCL policies and configuration are errors from 1.21. A policy written years ago with the same key twice now fails to load. The temporary escape hatch is VAULT_ALLOW_PENDING_REMOVAL_DUPLICATE_HCL_ATTRIBUTES; use it only to buy time to fix the files.

# Read own secrets, list own metadata, nothing else
path "kv/data/payments/*"     { capabilities = ["create", "read", "update", "patch", "delete"] }
path "kv/metadata/payments/*" { capabilities = ["list", "read"] }

# Templated per-entity path; the rendered value must never contain * or +
path "kv/data/users/{{identity.entity.name}}/*" { capabilities = ["read", "create", "update"] }

# Explicit deny survives any other grant
path "kv/data/payments/prod-master-key" { capabilities = ["deny"] }

Audit devices are a fail-closed dependency

Audit devices (file, syslog, socket) record every request and response with sensitive values hashed using HMAC-SHA256. If at least one audit device is enabled and none can write, Vault refuses requests. It fails closed, because an unlogged action is treated as worse than no action. That behaviour is the right one for a security product, and it means the audit pipeline is part of your availability design. A full disk on a file audit device, or a socket device whose listener has died, looks exactly like a Vault outage.

The hardening guidance is to run at least two audit devices: if one breaks, the other keeps Vault serving, and the metrics vault.audit.log_request_failure and vault.audit.log_response_failure tell you which device to fix. Write the file device to a dedicated partition with log rotation, ship it with an agent to your SIEM, and alert on the failure counters above zero. Because values are HMACed, correlating an entry with a known secret uses sys/audit-hash: hash the candidate value with the device's salt and search the log for the digest. Headers such as X-Forwarded-For can be made visible with vault write sys/config/auditing/request-headers/X-Forwarded-For hmac=false, which helps attribute requests behind a load balancer. Since 2.0, audit entries can also carry a supplemental_audit_data key.

vault audit enable -path=file-primary file file_path=/var/log/vault/audit.log
vault audit enable -path=socket-siem  socket address=siem.internal:9090 socket_type=tcp
vault audit list -detailed

# Does this candidate value appear in the log? Hash it with the device's salt.
vault write sys/audit-hash/file-primary input="s3cr3t-candidate"

Identity and secret zero for an entire organisation

At the Mid level you learned each auth method separately. As a platform owner the question is how they fit together so that no team invents its own.

Secret zero is the first credential a workload uses to prove who it is, and every secrets system has to get it there somehow. The design principle is to use an identity the platform already attested. A Kubernetes pod has a service account token signed by the cluster, so auth/kubernetes needs no extra secret. An EC2 instance, an Azure VM or a GCP service account has a platform identity the cloud auth methods verify. A GitHub Actions or GitLab CI job receives a short-lived signed JWT, which auth/jwt validates against the issuer's discovery URL. In each case the workload never holds a credential that you issued and someone had to ferry to it.

Where no platform identity exists, AppRole is the fallback, and it needs discipline. The role_id is semi-public, like a username. The secret_id is the secret, so deliver it response-wrapped with a short -wrap-ttl and limit its use with secret_id_num_uses=1 and a short secret_id_ttl, so a leaked secret ID is either already used or already expired. The delivery is performed by a trusted orchestrator that holds only the narrow capability to create wrapped secret IDs for one role, never a broad token. If the wrapped token is found already unwrapped, the delivery was intercepted and you know it, which is the security property of response wrapping.

# The orchestrator's only job: mint a single-use, wrapped secret_id
vault write -f -wrap-ttl=120s auth/approle/role/web/secret-id

# The workload unwraps it once, then logs in
VAULT_TOKEN=<wrapping_token> vault unwrap
vault write auth/approle/login role_id="$ROLE_ID" secret_id="$SECRET_ID"

Entities, aliases and what they cost

Every successful login creates or refreshes an entity alias, which belongs to an entity: Vault's record that two logins belong to the same person or workload. You attach policies to entities and groups rather than to individual roles, and identity templates such as {{identity.entity.name}} turn one policy into a per-tenant policy. The financial angle matters for Enterprise and HCP Vault Dedicated: they are billed by clients, meaning unique entities and non-entity tokens in a month. Logins through auth methods that do not produce a stable entity count as separate clients each time. The levers are to make sure a long-lived workload logs in as the same entity, to avoid creating a unique token per pipeline run where an entity-backed login would be stable, and to watch the numbers via vault operator usage, the UI dashboard or sys/internal/counters/activity. Two other 2.x billing endpoints are sys/billing/overview and sys/billing/certificates.

Humans

Humans should log in through OIDC or SAML (SAML is Enterprise) against your identity provider with groups mapped to Vault external groups, so joiner-mover-leaver happens in one place. JWT auth gained an Okta provider with group fetching (fetch_groups=true) in 2.1.1. Give humans read-mostly policies, use response wrapping for any handoff, and set MFA on privileged paths. Enterprise 2.x adds TOTP self-enrolment (enable_self_enrollment) and SCIM 2.0 provisioning, which was still beta in 2.0.0. Enterprise control groups are now called access workflows in the UI; the idea of requiring approval from named people before a sensitive read is unchanged.

The 2.x series also adds machinery for agents: Agentic IAM became generally available in 2.1.0 on Enterprise, with an Agent Registry and ceiling policies, plus the OAuth Resource Server feature. Note the 2.1.0 immutability rule: in OAuth Resource Server profiles, issuer_id and unique_id_claim cannot be changed after creation, so delete and recreate a profile to change them. Treat these as Enterprise-only and verify current behaviour against the release notes before you design around them.

Multi-tenancy, cost and chargeback

A platform serves several tenants: teams, business units or customers. The question is how strongly they are isolated, and the answer depends on your edition.

Enterprise and HCP: namespaces. A namespace is a "Vault within Vault" with its own mounts, auth methods, policies, tokens and identity, and namespaces nest. Selection is by VAULT_NAMESPACE or the X-Vault-Namespace header. The tenant gets delegated administration without touching anyone else's data, and you can set quotas per namespace. An administrative namespace (administrative_namespace_path) lets operators use a particular namespace to manage others. In 2.1.1 a vault secrets move or vault auth move with a leading slash placed the mount in the root namespace, which was fixed, so pin your patch level if you move mounts between namespaces.

Community Edition: convention and templates. With no namespaces you emulate tenancy. Give every team a path prefix (kv/team-a/*), a dedicated auth role per workload, and policies templated from identity so a tenant's policy is generated rather than hand-written. Then add a per-team rate-limit quota so a noisy neighbour cannot starve the rest. The limitations are honest ones: tenants share the mount table, the root namespace's administrators can see everything, and delegated self-service is weaker. If your organisation requires hard separation of tenants, that is one of the clearest reasons to pay for Enterprise, or to run separate clusters per regulatory boundary.

Do not create a cluster per team as a reflex. Each cluster is another seal, another backup job, another upgrade, another on-call surface and, on a licensed product, another cost. Use namespaces or prefixes, and keep separate clusters for genuine regulatory boundaries such as a Saudi cluster and an Egyptian one when data must not leave either country.

Cost levers

Community Edition is free under the Business Source License (BUSL 1.1, in effect since 1.15 in August 2023; the source was MPL 2.0 before that). Enterprise and HCP Vault Dedicated are billed by clients. The documentation does not publish prices and this guide will not guess them. The levers you control: map workloads to entities so each counts once, avoid creating unique non-entity tokens by the thousand, reduce the number of clusters, and use the usage reports to charge teams by their own client counts. Note that HCP Vault Secrets has been decommissioned (end of sale 30 June 2025, end of life 1 July 2026), so do not plan a migration to it; HCP Vault Dedicated remains the managed offering, and VSO's HCPVaultSecretsApp resource is no longer a path to teach.

Upgrades and migrations

Vault's storage is forward-migrated: a newer version may change the on-disk format, and an older binary cannot read it. That single fact shapes the whole upgrade discipline, because rollback means restoring a snapshot, not reinstalling the previous binary.

The official procedure is short. Read the release notes and the "important changes" page for every version between yours and the target. Take a snapshot and back up the configuration. Rehearse on an isolated copy restored from that snapshot. Then upgrade node by node: stop the service, replace the binary, start it, verify with vault status, and unseal if you use Shamir. The conventional rolling order is standbys first and the active node last, followed by vault operator step-down so leadership moves to an upgraded node. That ordering comes from long-standing practice and older documentation, so confirm it against the current upgrade page before you bake it into a runbook. Enterprise Autopilot can automate the sequence with autopilot_upgrade_version.

The 1.x to 2.x checklist

Turn the breaking changes into a checklist you can run against the estate, before the maintenance window rather than in it:

1. Callers of rekey and generate-root. Anything scripted that used key shares alone now also needs a token, unless you deliberately set enable_unauthenticated_access.
2. Path construction. From 2.0, requests whose paths contain //, /./ or /../, or non-canonical URL encodings, are rejected. Search scripts and SDK wrappers for string concatenation such as "$MOUNT/" + "/foo".
3. Container disable_mlock. Since 2.0.2, containers cannot lock memory. Set disable_mlock = true, and on Raft it had to be explicit from 1.20 anyway.
4. Templated policies with wildcards. From 2.0.1, rendered + or * in an identity template yields permission denied.
5. RSA keys above 8192 bits. CVE-2026-39829 caps the RSA modulus at 8192 bits from 2.0.2, so audit Transit and PKI keys.
6. Azure auth. Values in auth/azure/config now win over AZURE_* environment variables.
7. Tools inside the image. UBI images from 2.0.4 lack gnupg, openssl and procps.
8. HCL duplicates and list parameters. Both became strict in 1.21.
9. Enterprise token headers. The 8 KB default max_token_header_size returns HTTP 431 for large JWTs; raise it on the listener if needed.
10. Deprecations. Check the official table: the Vault Agent API proxy is pending removal (use Vault Proxy), the Active Directory secrets engine is removed (use the LDAP secrets engine with schema=ad), and the Vault Service Broker is pending removal.

Pending-removal built-ins log an error and Vault refuses to mount them, with VAULT_ALLOW_PENDING_REMOVAL_MOUNTS=true as an emergency override that treats them as merely deprecated. The override is a bridge, not a place to live.

Two known problems from the 2.0 line are worth recognising. The Enterprise Docker image could fail with unable to set CAP_SETFCAP effective capability: Operation not permitted, which has the workaround SKIP_SETCAP=1 or an upgrade to 2.0.2 or later, and external Enterprise plugins could fail signature checks with openpgp: key expired, fixed in 2.0.2. The broader lesson is to skip x.0.0 releases for production and move to the first patches, as the fix list here illustrates.

Kubernetes and Helm upgrades

The official Helm chart (0.34.1 at the time of writing) deploys Vault 2.0.4 by default, so pin server.image.tag to the version you have rehearsed. In the chart's HA mode the Vault pods are a StatefulSet with an OnDelete update strategy, meaning changing the image does nothing until you delete pods yourself, one at a time, standbys first. That is the safe behaviour, and it means a Helm upgrade can silently leave your cluster on an old version if nobody deletes the pods. Put the pod-by-pod procedure in a script and a checklist. The default chart mode is standalone with file storage, which is not production-ready; set server.ha.enabled=true and server.ha.raft.enabled=true. See Helm and Kubernetes for the mechanics, and Argo CD if you drive the upgrade from Git, taking care that sync does not delete pods in parallel.

server:
  image:
    tag: "2.1.1"
  ha:
    enabled: true
    replicas: 5
    raft:
      enabled: true
      config: |
        ui = true
        disable_mlock = true
        listener "tcp" {
          tls_disable     = 0
          address         = "[::]:8200"
          cluster_address = "[::]:8201"
          tls_cert_file   = "/vault/userconfig/vault-tls/tls.crt"
          tls_key_file    = "/vault/userconfig/vault-tls/tls.key"
        }
        storage "raft" {
          path = "/vault/data"
          retry_join {
            leader_api_addr = "https://vault-0.vault-internal:8200"
          }
        }
        service_registration "kubernetes" {}

Storage and seal migrations

Migrating storage backends offline uses vault operator migrate -config=migrate.hcl, which copies all data from one backend to another while the cluster is stopped; the usual destination is Raft. Seal migration, for example from Shamir to a cloud KMS or between KMSes, is done by adding the new seal configuration with the disabled marker on the old seal and running vault operator unseal -migrate. Enterprise Seal HA (two or three auto-seals, enabled with enable_multiseal = true) allows online migration between seals. Verify the exact stanza names against the seal documentation for your version before rehearsing, because this area has changed in recent releases. Whatever the route, the same rule applies: snapshot first, rehearse on a copy, and keep the old seal's key material until the new configuration has survived a full restart.

Integrating Vault into the delivery platform

The integration points are where a secrets platform either becomes invisible infrastructure or a bottleneck that teams route around.

Kubernetes. There are three delivery mechanisms and the choice should be made once for the organisation. The Vault Secrets Operator (VSO), a separate Helm chart (hashicorp/vault-secrets-operator), uses CRDs (VaultConnection, VaultAuth, VaultStaticSecret, VaultDynamicSecret, VaultPKISecret) to sync Vault data into native Kubernetes Secrets, handle drift and trigger rollout restarts; it is HashiCorp's current recommended native path. The Agent Injector is a mutating webhook that adds an Agent sidecar (vault.hashicorp.com/agent-inject* annotations) writing files under /vault/secrets. The CSI provider mounts secrets as a volume through the Secrets Store CSI Driver. The trade-off is that VSO's secrets land in etcd as Kubernetes Secrets, which means etcd encryption and RBAC on Secrets now matter, while the injector and CSI keep secrets off etcd at the cost of more moving parts per pod. Vault can also serve as a KMS provider for etcd encryption.

Terraform. The hashicorp/vault provider configures Vault as code: mounts, policies, auth backends and roles. Doing so through a pipeline with a scoped identity, reviewed in pull requests, is the single most effective governance control you can apply. The caveat is that secrets read by Terraform data sources end up in state; Terraform 1.10 and later offer ephemeral resources that keep values out of state, but verify the Vault provider's ephemeral support for each resource before depending on it. See Terraform and Pulumi for the state implications, and Ansible if you configure hosts.

CI/CD. Use JWT/OIDC auth from the CI system (GitHub Actions through hashicorp/vault-action, GitLab CI through id_tokens and secrets:vault, Jenkins, Bitbucket), with bound_claims tight enough that only the intended repository and ref can assume a role, and 10-minute token TTLs. No long-lived token belongs in a CI variable. GitLab CI and Jenkins cover the pipeline side.

Observability. The stack in Prometheus, OpenTelemetry and the ELK Stack handles metrics, traces and log shipping. Do not send audit logs to a system with lower access controls than Vault itself, because the log is a map of your secrets.

Certificates. The PKI engine issues short-lived certificates with an ACME server, and Enterprise adds EST, CMPv2 and SCEP, plus PKI External CA (2.0 and 2.1), which obtains certificates from public CAs over ACME with automatic DNS-01 fulfilment through Route53, Azure DNS, Google Cloud DNS and RFC 2136 in 2.1.0. Short-lived certificates turn revocation into a non-problem: the certificate expires before anyone can exploit it. The 2.1.0 release also added PKCS#12 and JKS bundle output, and 2.1.1 added RFC 5280 revocation reasons on revocation endpoints.

Workload Identity Federation and Secrets Sync. 2.0 added WIF for Secrets Sync, which pushes secrets to AWS Secrets Manager, Azure Key Vault, GCP Secret Manager, GitHub and Vercel without long-lived static cloud credentials. It is an Enterprise or HCP feature; confirm the edition before designing around it.

Governance and incident playbooks

Governance is a set of rules the platform enforces rather than relies on memory for. The rules that matter most: all Vault configuration lives in a Git repository applied by a pipeline; policy changes are reviewed by a second person; the root token does not exist between incidents; audit logs are immutable and retained for the period your regulator requires; and every tenant has an owner recorded next to its mount or namespace.

The playbooks below are written as short decision sequences. Each one should exist in your runbook store with names filled in.

Playbook: the cluster is sealed

1. Run vault status on every node. Is it one node or all?
2. If all, and you use auto-unseal, check the KMS: key disabled, policy changed, region outage, network path or credentials to the KMS. Fix those first; the nodes unseal on the next restart or via vault operator unseal when needed.
3. If you use Shamir, assemble the threshold of shareholders. Nothing else works.
4. After unsealing, verify the leader with vault operator raft list-peers and check the audit devices.
5. Write the post-incident note naming which dependency was missing from your design document.

Playbook: no leader, writes failing

1. vault operator raft list-peers from any reachable node. How many voters are healthy against quorum?
2. Bring dead nodes back if their disks are intact. A returning node rejoins cleanly; a replaced node joins as a non-voter and is promoted by Autopilot.
3. If a quorum cannot be restored, choose in advance between the documented peers-file recovery and a snapshot restore into a new cluster. Declare which, record the time, and do not mix them.
4. After recovery, revoke credentials that were issued between the snapshot and the failure if you restored from a snapshot.

Playbook: a credential leaked

1. Identify its type. A token: vault token revoke -accessor <accessor> (the accessor lets you revoke without exposing the token). A dynamic secret: vault lease revoke <lease_id> or, when many are involved, vault lease revoke -prefix <path>. A static KV value: rotate it at the source, then write a new version, because KV history retains the old one until destroyed with vault kv destroy -versions=<n>.
2. Use the audit log to find every read of the secret, starting from the HMAC of the leaked value with sys/audit-hash.
3. If an AppRole secret ID leaked, rotate the role and investigate who held the wrapping token.
4. If root credentials for a backend were involved, vault write -f database/rotate-root/<name> for each connection.
5. If the compromise is broad, treat it as a storage incident: rotate the encryption keyring with vault operator rotate, and plan a rekey.

Playbook: Vault is up but everything is slow

1. Check vault.raft.commitTime and disk latency. A sick volume is a common cause.
2. Check vault.expire.num_leases and whether a revocation storm is under way.
3. Check audit device latency, because a slow socket audit sink slows every request.
4. Use vault debug -duration=2m -output=debug.tar.gz to collect pprof profiles, metrics and host information for analysis or a support case.
5. Look for one client dominating the audit log, then add a rate-limit quota to its path.

Playbook: a certificate expired

Listener, Raft peer and client-facing certificates fail differently. An expired listener certificate makes clients fail with x509: certificate has expired or is not yet valid. Reload certificates with SIGHUP after replacing the files; Vault picks up the new listener certificates without a restart. Put expiry in your monitoring, and issue the listener certificates from a PKI you operate with short lifetimes renewed by an agent, or the expiry will arrive at the worst possible time.

Where Vault stops

Being honest about limits is part of the senior job, and the right recommendation is sometimes another tool.

Vault is not a general configuration store or database. Frequently changing or large data belongs elsewhere; the 1 MiB entry limit and the single writer are telling you so.

Cloud-native secret managers (AWS Secrets Manager, Azure Key Vault, Google Secret Manager) are simpler when one team runs one workload in one cloud: no cluster to operate, IAM as the access model, native rotation. Vault earns its place when you need a consistent secrets and identity layer across clouds, on-premises and Kubernetes, dynamic credentials for many backends, a PKI or encryption-as-a-service, or a single audit surface. A reasonable pattern is Vault as the source of truth and Secrets Sync pushing to the cloud stores for workloads that can only read there.

Kubernetes-only teams with no dynamic secret needs may be well served by a sealed-secrets or external-secrets approach against a cloud manager. Bring Vault in when dynamic credentials or cross-cluster identity appear.

Licensing and the fork. Since 1.15 the source is under BUSL 1.1. OpenBao is the community fork of the last MPL-licensed code, governed under the Linux Foundation and OpenSSF. It is a different product: do not assume its flags, features or release cadence match Vault's, and evaluate it on its own documentation. The decision between CE, Enterprise, HCP Vault Dedicated and OpenBao is a legal and procurement one as well as a technical one, so involve the right people early. Also note the IBM context: the product is marketed as IBM Vault, the docs remain at developer.hashicorp.com/vault and the binary is still vault.

Enterprise capabilities you will miss on CE. Namespaces, performance and DR replication, performance standbys, Sentinel policies (EGP and RGP), access workflows (control groups), Seal HA, automated snapshots, non-voters and redundancy zones, SCIM, FIPS 140-3 builds and seal wrap. CE's answer for multi-region is snapshot and restore into a standby cluster, which has a recovery point measured in your snapshot interval.

Encryption of huge data sets. Transit encrypts payloads you send it, which makes it a key service, not a bulk cipher. For large objects, use the datakey endpoint to get a data key, encrypt locally, and store the wrapped key next to the data.

Running Vault as a platform for other teams

Everything above is the machinery. A platform is machinery plus a product relationship with its users, and that relationship is where most Vault programmes succeed or fail.

Define the service. Write down what you offer: which auth methods, which secrets engines, the supported delivery path on Kubernetes, the TTL defaults, the onboarding lead time and the availability target. A platform with no stated contract is judged against an imagined one. Keep the catalogue narrow, because each mounted engine is something you must patch and explain.

Self-service through code. Tenants request access with a pull request to a repository containing Terraform for their prefix or namespace, role and policy. You review the parts that matter (paths, TTLs, bound claims) and the pipeline applies it. The review becomes cheap because the shape is templated, and the history is your audit trail for who was granted what and why.

Defaults that do the right thing. Set default_lease_ttl and max_lease_ttl per mount instead of living with the system 768-hour defaults, and cap roles with explicit token_max_ttl. Turn on user lockout, which is enabled by default for AppRole, LDAP and userpass, and leave it on. Make short-lived dynamic credentials the path of least resistance and static KV the path that requires a justification.

Ownership metadata. Each mount, namespace or prefix records an owning team and an on-call contact. When a policy looks wrong, or a lease count spikes, you know whom to ask. Periodically review unused roles and stale entities.

Communication. Announce maintenance windows, publish incident summaries, and keep the breaking-change list from the release notes in front of your users before upgrades. Most Vault "outages" that users report after an upgrade are the changes in the checklist above reaching their scripts.

Sustainability. A platform team of one is a single point of failure in the same sense as a single Vault node. Rotate the on-call, cross-train on the recovery ceremony, and ensure the recovery keys and KMS ownership do not depend on one person's employment.

The review checklist

Use this for design documents and for pull requests that touch Vault configuration. Every line is a question to answer, not a box to tick.

Availability

• How many voters, in how many zones, and what is the quorum?
• What happens during a KMS outage, a certificate expiry, and an audit device failure?
• Is there a restore drill, and when did it last succeed?
• Are health checks using /v1/sys/health with the right status code mapping?

Security

• Is the root token revoked, and is the generate-root procedure written and two-person?
• Who can Decrypt with the seal key, and who holds recovery keys?
• Are there at least two audit devices, and do alerts fire on failure counters?
• Are policies free of sudo, broad globs on sys/, and wildcard template output?
• Is TLS 1.2 or better enforced, swap disabled, core dumps off, disable_mlock set deliberately?

Scaling

• Are TTLs as short as the workload allows, and do clients revoke on exit?
• Are batch tokens used for high-volume short jobs?
• Are there rate-limit quotas, global and per tenant?
• Are KV values small, and is Vault kept out of the hot path of high-volume data?

Tenancy and cost

• What is each tenant's isolation, owner and quota?
• Do long-lived workloads log in as one entity?
• Is client count reported monthly?

Change management

• Is configuration in Git, applied by a pipeline with a scoped identity?
• Is the 1.x to 2.x checklist complete for this change?
• Is the Helm image tag pinned, and has the rollout order been rehearsed?
• Is the rollback plan a restore rather than a downgrade?

The complete picture

Across three levels the picture is one system viewed at three distances. At Beginner it was an API that stores secrets. At Mid it was a set of auth methods and engines that you wire to applications, with leases and policies controlling lifetime and reach. At Senior it is a cluster with a quorum, a seal and a single writer, a platform with tenants and a contract, and a security boundary whose weakest link is usually organisational: who can open it, who can reconfigure it, and who notices when something goes wrong.

The same ideas recur at every level. Short-lived credentials beat long-lived ones, because expiry is a control that needs no vigilance. Identity the platform has already attested beats a secret you ferry. Everything that can be code should be code, because review and history are free security. Everything that can fail should have been rehearsed, because Vault sits under everything else and its failure modes are therefore everyone's failure modes. And everything that changes between versions, such as the 2.0 rekey authentication, the path cleaning, the mlock rules and the stricter HCL, should be in a checklist before it is in an incident.

Where the series leaves you

You can now draw a Vault cluster and say what fails first, run the seal and key ceremonies without a single person holding the keys, scale by removing load rather than adding nodes, design tenancy for your edition, upgrade across a major version with a rehearsal and a real rollback plan, and argue honestly about when to choose another tool.

What to practise next, in order of return: run a full restore drill every quarter; move one more class of static credential to dynamic or platform-attested access each month; publish a per-team client and lease report; and read each release's "important changes" page the week it ships. For neighbouring topics, Kubernetes and Helm cover where Vault's workloads run, Terraform and Argo CD cover delivering its configuration, Prometheus covers watching it, and Docker covers the image it ships in. The interview and tips files in this level turn everything here into questions to rehearse and habits to keep.

Sources

• Vault documentation
• Release notes
• Important changes
• Deprecation plans
• Upgrade guidance
• Vault CHANGELOG on GitHub
• Seal and unseal concepts
• Tokens
• Policies
• Production hardening
• Server configuration
• Integrated storage (Raft) configuration
• Telemetry configuration
• Integrated storage internals
• operator raft commands
• Health API
• Audit devices
• Namespaces
• Replication
• Vault on Kubernetes with Helm
• Vault Secrets Operator
• Vault Agent and Vault Proxy
• AppRole auth method
• JWT/OIDC auth method
• PKI secrets engine
• Transit secrets engine
• Troubleshooting tutorial

Part three of three. A senior Vault interview is rarely about syntax. It tests whether you can design a secrets platform, reason about what fails first, respond to an incident without inventing procedure on the spot, and defend a trade-off to someone who controls the budget. About fifty minutes. Design prompts first, then incident questions, then the conversations with leadership, then a rapid-fire round.

This file targets Vault 2.1.1. Where a question depends on the 2.0 changes, the answer says so, because interviewers in 2026 use them to tell people who have run the current version from people who read about 1.x.

How to answer a design prompt

Senior interviewers mark you on the order of your thinking more than on the final diagram. A structure that works for almost every Vault prompt:

1. Clarify the constraints. Number of workloads, clouds, regions, compliance regime, who is on call, which edition, what the consequence of an outage is.
2. State the trust model. What are we defending, from whom, and what do we assume is already compromised?
3. Draw availability first. Nodes, quorum, seal, load balancer, backup.
4. Then identity and policy. How does a workload prove who it is, and what can it do?
5. Then lifecycle. TTLs, rotation, revocation.
6. Then operations. Observability, upgrades, tenancy, cost.
7. Name the trade-off you made and what would make you choose otherwise.

System-design prompts

Design a Vault platform for forty teams across two regions

What the interviewer listens for: a cluster design, a tenancy model, a recovery story and an honest edition discussion.

A strong answer. Start by asking whether the two regions must both serve traffic and whether data may cross a border. Suppose it is Riyadh and Cairo with residency rules. Then the defensible design is a separate cluster per jurisdiction, not one cluster stretched across both. A stretched Raft cluster puts a wide-area latency into every commit and makes the loss of one region threaten quorum. Within a region, run five voters across three availability zones: quorum is three, two failures are tolerated, and writes stay fast because the zones are close. Use auto-unseal with a KMS key in the same region, with a key policy restricted to the Vault nodes' role. Put a load balancer in front that health-checks /v1/sys/health and sends traffic only to the active node, because on Community Edition standbys add no capacity.

For tenancy, the answer depends on the edition. On Enterprise, use one namespace per team, with delegated administration and a quota per namespace. On Community Edition, use a path prefix per team, a dedicated auth role per workload and identity-templated policies, with a rate-limit quota per team. Say plainly that CE tenancy is enforced by policy only, and that if an auditor demands hard separation, that is the argument for Enterprise or for another cluster.

For access, the workloads authenticate through platform identity: Kubernetes auth from clusters, JWT auth from CI with tight bound_claims, cloud auth from VMs. Humans use OIDC against the company identity provider with groups mapped to Vault external groups. For lifecycle, dynamic database credentials with short TTLs, PKI for internal certificates, KV v2 only for what must be static.

For recovery, automated snapshots shipped off-cluster, a quarterly restore drill, and for cross-region resilience either Enterprise DR replication within a jurisdiction or snapshot-and-restore into a warm standby on CE. Close with operations: Prometheus metrics with the alerts that matter, two audit devices, configuration as code through the Terraform provider, and a written upgrade procedure.

Where weak answers go wrong: drawing one cluster across two continents, forgetting the seal as a dependency, describing namespaces without noting they are Enterprise, and omitting the restore drill.

Follow-ups to expect. What if the regulator also forbids the snapshot leaving the country? Keep the snapshot bucket and its encryption key in the same jurisdiction. What if one team needs its own seal? That is a separate cluster, since the seal belongs to the cluster, not the namespace.

Design the secret-delivery path for Kubernetes

A strong answer. Name the three mechanisms and choose. The Vault Secrets Operator uses CRDs (VaultConnection, VaultAuth, VaultStaticSecret, VaultDynamicSecret, VaultPKISecret), syncs into native Kubernetes Secrets, handles drift and can trigger a rollout restart when a secret changes. It is HashiCorp's current recommended native path and it is the simplest for application teams, because the app reads an ordinary Secret. The cost is that secrets now live in etcd, so you need etcd encryption at rest (Vault itself can be the KMS provider) and tight RBAC on Secrets. The Agent Injector keeps secrets off etcd by writing files into the pod from a sidecar, at the cost of a sidecar per pod and annotation-driven configuration. The CSI provider mounts a volume through the Secrets Store CSI Driver and suits teams who already use CSI.

Then authentication: auth/kubernetes with roles bound to a service account name and namespace, and short TTLs. Explain why you would not use one broad role per cluster. Finish with failure behaviour: what do pods do when Vault is down? With VSO the last synced Secret remains, so running pods keep working and only new rollouts of dynamic credentials suffer; with an injector, a sidecar that cannot authenticate blocks a pod from starting. Say which failure mode the business prefers.

The trade-off to name: operational simplicity and developer experience (VSO) against keeping secrets out of etcd (injector, CSI).

Design CI/CD access to secrets with no long-lived credentials

A strong answer. Use the CI system's OIDC token. GitHub Actions and GitLab CI both issue a short-lived signed JWT per job. Configure auth/jwt with the issuer's discovery URL and bound_issuer, create a role per repository or per application, and set bound_claims so only the expected repository, and for production only a protected branch or environment, can assume it. Token TTL around ten minutes. The pipeline requests exactly the paths it needs, nothing more.

Why this is better than a CI variable holding a token: nothing long-lived exists to steal; the audit log attributes each access to a repository and a run; revocation is automatic by expiry. The weak point is an over-permissive claim binding, for example a role bound to an organisation rather than a repository, which any repository in the organisation can use. Mention forks: pull requests from forks must not get a role that can read production secrets.

A senior detail: for very high job volumes, use batch tokens on the role so you do not write a token and lease per job, and mention entity counting for billing.

Design dynamic database credentials for a busy service

A strong answer. The database secrets engine creates a user per request with a TTL. For a fleet of autoscaled pods, a credential per pod start is fine; a credential per request is not, because it multiplies leases and database connections. Use Vault Agent or VSO to fetch and renew the credential once per pod. Pick default_ttl long enough to cover a process lifetime with renewals and max_ttl to bound risk. Rotate the root credential Vault uses with database/rotate-root/<name> so no human knows it.

Discuss failure modes honestly. If the database is slow, revocations fail and leases can become irrevocable; if Vault is down, running pods keep their credentials until TTL, so TTL defines how long an outage is survivable. Rate-limit or lease-count quota the path so a crash loop cannot create ten thousand database users. For accounts that must exist with a fixed name, use a static role with rotation_period.

Design the backup and recovery strategy

A strong answer. Use Raft snapshots as the backup primitive. Take them on a schedule with a dedicated low-privilege identity; write to encrypted object storage in a separate account and region (inside the jurisdiction when required); alert if the newest snapshot is older than the interval. Emphasise that a snapshot is encrypted by the barrier, so a restore needs the same seal: keep the KMS key alive and keep the recovery material. Then define the recovery time and point objectives and show how each choice meets them: on Enterprise, DR replication gives a warm standby with a small recovery point; on CE the recovery point is your snapshot interval.

Then the part that separates seniors from everyone else: the restore drill. A scheduled job restores the latest snapshot into an isolated cluster, unseals it and reads a canary secret, and pages someone if it fails. And the security corollary: a restore resurrects tokens and leases that were alive at snapshot time, so a post-compromise restore includes revocation and rotation.

Design the encryption service for a payments application

A strong answer. Transit as encryption as a service: the application sends plaintext and receives ciphertext prefixed vault:v1:; keys never leave Vault. Create a key per data domain, rotate it with transit/keys/<name>/rotate, and use rewrap to move existing ciphertext to the latest key version without exposing plaintext to the client. Control old versions with min_decryption_version. For large objects, do not send them through Transit; request a data key, encrypt locally and store the wrapped key alongside the data. The key-size limit worth knowing from 2.0.2: CVE-2026-39829 caps RSA keys at 8192 bits.

Note performance: Transit operations are requests to the active node on CE, so calling Transit per row in a batch job is a load problem. Cache data keys or batch requests.

Incident-response questions

Incident questions test calm and sequence. A strong answer states the first three commands, what each result would mean, and the decision point. It names dependencies other than Vault itself.

"Every application is getting 503 from Vault. What do you do?"

The code 503 means sealed. Run vault status on each node. If every node is sealed at once, something they share changed. With auto-unseal, that is almost always the KMS: a disabled key, a changed key policy, a network path change or a regional outage. Check the KMS audit trail for denied Decrypt calls. Fix access, then unseal by restarting the service or running the unseal command as appropriate. If it is one node, check whether a standby has taken leadership and whether the load balancer is routing correctly. Afterwards, verify audit devices are healthy and the leader is stable with vault operator raft list-peers. The post-incident item: the KMS dependency should have been in the design document and in an alert.

"Writes fail but some reads still work."

Suspect loss of Raft quorum. vault operator raft list-peers shows voters and their state. With five voters, three must be up. Restore dead nodes if their disks survive. If a quorum cannot return, use the pre-decided recovery path, either the documented peers-file recovery or restoring the most recent snapshot into a new cluster. Explain the data-loss implication of the snapshot route: anything written after the snapshot is gone. Mention Autopilot: a replaced node rejoins as a non-voter, and its promotion is automatic after stabilisation.

"A developer pushed a Vault token to a public repository."

Speed matters, so do things in order. First, revoke the token: vault token revoke -accessor <accessor> if you have only the accessor, or vault token revoke <token>; revoking a token also revokes its children and their leases. Second, read the audit log for that token's activity from the time of the leak. Third, for every secret it read, decide whether to rotate: dynamic secrets expire and can be revoked by lease; static KV values must be rotated at the source. Fourth, find out why a long-lived token existed at all, and replace it with platform-attested auth. Finally, add secret scanning to the repository. Note that Git history still contains it, so assume it was harvested even if the commit was deleted.

"Vault is slow and getting slower over weeks."

A slow drift points to growth, not a spike. Look at vault.expire.num_leases and vault.token.count. A steady climb means something requests credentials and never revokes them, usually CI creating service tokens per job or pods fetching database credentials on every start without releasing them. Find the client in the audit log, shorten TTLs, make the client revoke its token at exit, move bulk jobs to batch tokens, and add a lease-count or rate-limit quota. Also check vault.raft.commitTime for storage latency and audit sink latency. Use vault debug to capture pprof data if the cause is unclear.

"We upgraded to 2.x and our break-glass script stopped working."

The likely cause: sys/rekey, sys/generate-root and the DR operation-token endpoint require a Vault token from 2.0. A script that supplied only key shares now gets a 403. The options are to supply a token as well, which is the recommended direction, or to set enable_unauthenticated_access for the specific operations. Explain the trade-off: enabling it restores the old convenience and the old exposure. Another cause for scripts failing after 2.0: non-canonical paths with // are now rejected.

"An audit device fills the disk and Vault stops answering."

Vault fails closed when audit devices are enabled and none can write. Free or extend the audit volume first. Explain the design fix: at least two audit devices so one failure does not stop the service, log rotation with shipping to the SIEM, alerts on vault.audit.log_request_failure and vault.audit.log_response_failure, and separate disk for audit logs. Do not disable the audit device to get service back, because that is the failure the control exists to prevent; if you must, record the decision and the window.

"Someone suggests restoring last night's snapshot after a credential compromise."

Explain why a restore does not undo a compromise. The snapshot contains tokens, leases and secrets; if the attacker's credentials existed at snapshot time, they are back. Do the incident properly: revoke by prefix, rotate backend root credentials with rotate-root, rotate static secrets at their sources, rotate the keyring with vault operator rotate, and plan a rekey if unseal or recovery material may be exposed. Restore only for data loss, not as a cleaning step.

Arguing a trade-off to leadership

Leadership questions test whether you can turn technical risk into a decision someone can approve. The pattern: state the decision, the options, the cost and risk of each in business terms, your recommendation, and what you will measure afterwards.

"Why do we need Vault Enterprise, or HCP Vault Dedicated, when the open-source edition is free?"

Do not open with features. Open with the requirements the free edition cannot meet for this organisation. The list that matters: hard tenant separation through namespaces, cross-region resilience through DR and performance replication, read scaling through performance standbys, policy controls such as Sentinel, approval workflows for sensitive reads, automated snapshots and upgrades, and vendor support during an incident. Then connect each to a named risk or cost: "Without namespaces, forty teams share one administrative boundary and an auditor will ask how we prove separation." Put the cost in context: billing is by clients, so the driver is how many distinct workloads and people use the service, which we can measure and control. Offer the honest alternative: if the requirements are only availability and basic secrets, CE with a strong operating model is enough, and we should say so and spend the money elsewhere. Also mention licence terms: CE is under the Business Source License, and legal should confirm the intended use is permitted.

"Can we just use the cloud provider's secret manager and avoid running Vault?"

Agree where it is true. For a single team on a single cloud with modest needs, a cloud secret manager is cheaper to run, because IAM is the access model and there is no cluster to keep alive. The argument for Vault is scope: a consistent identity, policy and audit layer across clouds, Kubernetes and on-premises; dynamic credentials for many backends; an internal PKI; encryption as a service. Then state the cost honestly: Vault is critical infrastructure that needs an on-call rota and a restore drill, and a secrets platform that is poorly operated is worse than a managed service. Recommend based on what the organisation will actually staff. A hybrid is legitimate: Vault as the source of truth with Secrets Sync pushing to cloud managers for workloads that can only read there, which is an Enterprise or HCP feature.

"An upgrade to 2.x is a major version. Why should we take the risk now?"

Reframe the question: the version number follows IBM's lifecycle, the API is still /v1, and the breaking changes are a short, searchable list. The risk of staying is that 1.20 and 1.21 continue only as Enterprise patch lines, and the public Community Edition images stop at 1.21.4, so CE users on the 1.x line stop receiving fixes. Security fixes such as the RSA size cap (CVE-2026-39829) and the denied-parameters bypass fix landed on 2.x. Then present the plan that makes the risk small: rehearse from a snapshot, run the checklist against scripts and policies, upgrade staging first, skip the initial x.0.0 release for production and move to a patch, pin the Helm image tag, and keep the snapshot as the rollback. Ask for a window and report afterwards.

"The security team wants every secret rotated every thirty days. Platform disagrees. What is your position?"

A senior answer does not choose a side by authority; it separates secret classes. Dynamic secrets are already rotated constantly, often hourly, so the rule is met by design. Static secrets that Vault manages with static roles can rotate on a schedule through rotation_period or rotation_schedule (mutually exclusive since 2.1). Static secrets that third parties hold, such as an API key for a vendor, rotate at the vendor's pace and with human coordination, which is where a blanket rule creates outages. Propose a tiered policy: maximum age by class, automation wherever possible, and an exception process with an owner. Measure the result with the age distribution of static secrets.

"How do we know Vault is healthy, and what would you report monthly?"

Offer a small, honest scorecard: availability measured at the load balancer, restore drill result and date, time to recover in the last game day, number of static credentials remaining in CI variables, median credential lifetime, lease and token growth, client count by team, and the number of open policy exceptions. Tie each metric to a decision it informs. Avoid vanity metrics such as request counts.

"A team wants its own Vault cluster. Do we allow it?"

Ask why. If the reason is a hard regulatory boundary or a fundamentally different availability requirement, a separate cluster is justified and you help them run it to the same standard. If the reason is autonomy or fear of noisy neighbours, offer a namespace (Enterprise) or a prefix with their own quota (CE), because every additional cluster adds a seal, a backup job, an upgrade path and an on-call surface. Put the operating cost of a cluster in a table and let them decide with the numbers.

Common senior interview questions

What is the difference between recovery keys and unseal keys? With Shamir, the unseal key shares can decrypt the root key, so they open Vault. With auto-unseal, the KMS or HSM protects the root key, and init returns recovery keys, which cannot decrypt anything; they authorise operations such as generate-root and rekey. In the auto-unseal case, the KMS key policy is what actually opens Vault.

Why does disable_mlock = true appear in almost every modern config? Raft's storage is memory-mapped, so locking memory is not the defence it was. Since 1.20 the setting must be explicit with integrated storage, and since 2.0.2 the official container image lacks cap_ipc_lock, so containers cannot lock memory anyway. The compensating control is to disable or encrypt swap.

Service token or batch token? Service tokens are persisted, renewable, revocable and can have children. Batch tokens are encrypted blobs with no storage write and no lease, cannot be renewed or revoked individually, and suit high-volume short jobs. The decision turns on whether the workload needs renewal or revocation.

What does "fail closed" mean for audit? If audit devices are enabled and none can log a request, Vault refuses the request. Run at least two devices.

What does a quorum of five tolerate? Three of five must agree, so two nodes can fail. Seven tolerates three but adds write cost. Two or four voters give no better tolerance than one fewer.

How do you stop Autopilot removing too many nodes? Dead-server cleanup is off by default. If you enable it, min_quorum is required so the cluster cannot shrink below the floor you set.

How does response wrapping help with secret zero? The orchestrator asks Vault to wrap the AppRole secret ID in a single-use token with a short TTL, stored in a cubbyhole. The workload unwraps it once. If someone else unwrapped it first, the workload's unwrap fails, and you know the delivery was intercepted.

What changed for templated policies in 2.0.1? Rendered identity values containing + or * no longer match in a policy path; they yield permission denied. It closes a route where metadata could be set to a wildcard to widen access.

Why is a snapshot restore not a security response? It restores tokens and leases that existed at snapshot time.

What is Vault Proxy versus Vault Agent? Agent handles auto-auth, templating and process supervision. Proxy handles auto-auth, API proxying and caching. Agent's own API proxy function is deprecated and pending removal, so use Proxy for that role.

What is a lease and why does it matter for scale? Metadata with a TTL attached to dynamic secrets and service tokens. Leases live in memory and storage, and the active node expires them, so their count is a scaling constraint.

How do you authenticate a human? OIDC or SAML against the identity provider, groups mapped to Vault external groups, short TTLs, MFA on privileged paths. Not userpass.

What is HCP Vault Secrets' status? Decommissioned: end of sale 30 June 2025 and end of life 1 July 2026. HCP Vault Dedicated is the managed Vault offering that remains.

How does Vault count clients for billing? By unique entities and non-entity tokens per month. Stable entity-backed identities for long-lived workloads keep the count low.

Give me a safe 1.x to 2.x checklist. Rekey and generate-root callers, path construction, container disable_mlock, templated policy wildcards, RSA sizes above 8192 bits, Azure auth environment variables, tools missing from the image, strict HCL, and the deprecations table.

When would you choose OpenBao? When the BUSL licence is unacceptable and the team accepts a separate product with its own documentation and release cadence. Do not assume feature parity with Vault, and verify current details on its own pages.

The sixty-second self-test

Answer aloud, then check against the sections above.

1. Name the three things that can stop a healthy-looking cluster from serving requests, and the command that shows each.
2. How many failures does a five-node cluster tolerate, and why is four nodes no better than three?
3. Who can open your Vault if you use a cloud KMS seal?
4. What changed for sys/rekey and sys/generate-root in 2.0, and what config key reverses it?
5. Why do containers need disable_mlock = true from 2.0.2?
6. What are the first three lease-related symptoms of growth?
7. Name two ways to issue secret zero without a long-lived secret.
8. What does a snapshot not protect you against?
9. When is a namespace the wrong answer to a tenancy question?
10. What is the one metric that tells you an audit device is about to stop Vault?

If you could answer eight of these without hesitation, you can carry a platform interview. If fewer, reread the architecture, seal and scaling sections of the detailed file, and run the "Try it" drills on a scratch cluster, because interviewers can tell the difference between someone who has restored a snapshot and someone who has read about it.

Part three of three. The tips at this level are the ones that only matter once Vault is shared: traps that appear after months of growth, habits that keep a platform from becoming a liability, and checklists for the pull requests and design documents that decide whether it stays healthy. Everything here assumes Vault 2.1.1.

Production traps that only appear at scale

Each entry gives the symptom, the cause and the fix, in the order you are most likely to meet them.

The lease table that grows for months

Symptom. Latency creeps up over weeks. vault.expire.num_leases and vault.token.count rise and never plateau. Restarts take longer, because expiration has to load and walk all of them.

Cause. Something asks for credentials and never releases them. Typical offenders are CI jobs that log in with a service token per job, pods that request a database credential on every start without revoking at shutdown, and scripts that call vault login in a loop.

Fix. Find the client in the audit log, shorten the role's TTLs to what the workload really needs, make clients revoke on exit (vault token revoke -self), move bulk short-lived jobs to batch tokens, and cache through Vault Agent or the Vault Secrets Operator. Then add the gauge to a dashboard so the next leak is a trend line you see in week two, not week twelve.

The token that nobody can explain

Symptom. The client count in the usage report is far higher than the number of services.

Cause. Logins that do not map to a stable entity. Each unique non-entity token counts as a client. A pipeline that authenticates with a fresh token for every run inflates the bill and leaves no stable identity to attach policy to.

Fix. Use an auth method that creates an entity alias, make long-lived workloads log in as the same identity, and review vault operator usage and sys/internal/counters/activity monthly. Attribute counts to teams.

The audit device that becomes an outage

Symptom. Requests hang or fail across the board, and nothing is wrong with the Raft cluster.

Cause. An audit device cannot write: the disk is full, a socket listener died, or the SIEM pipeline is down. Vault fails closed.

Fix. Run at least two audit devices, alert on vault.audit.log_request_failure and vault.audit.log_response_failure being above zero, give the file device its own volume with rotation, and never point a socket device at a service that is less available than Vault.

The one-megabyte secret

Symptom. A write fails with an entry size error, or one path behaves differently from its neighbours.

Cause. The Raft max_entry_size default is 1048576 bytes. A team stored a bundle of certificates, a large JSON document or a base64 binary as one KV value.

Fix. Do not raise the limit as a first response. Split the data, or store the object in encrypted object storage and keep a pointer and a key in Vault. Put a size rule in the onboarding document and a reviewer question in the pull-request template.

The mount per microservice

Symptom. Mount table operations get slow, upgrades reveal odd behaviour, and vault secrets list scrolls for pages.

Cause. A well-intentioned "one mount per service" convention multiplies mounts. The mount and namespace tables are single entries in storage and have size limits.

Fix. Mount per class of secret, not per consumer, and separate tenants by path prefix or namespace. Enterprise has a max_mount_and_namespace_table_entry_size override, but needing it is a design smell.

The Helm upgrade that did nothing

Symptom. helm upgrade succeeds, and vault status still reports the old version.

Cause. The Vault StatefulSet in HA mode uses an OnDelete update strategy, so Kubernetes does not replace pods when the template changes. The chart also defaults to its own pinned Vault version, which for chart 0.34.1 is 2.0.4, not the latest release.

Fix. Pin server.image.tag explicitly, and script the pod-by-pod replacement: delete one standby, wait until it is unsealed and healthy, repeat, and step the active node down last. Include a version check at the end of the script that fails loudly if any pod disagrees.

The certificate that expires on a Saturday

Symptom. Clients report x509: certificate has expired or is not yet valid; standbys lose contact with the leader.

Cause. The listener certificate, or the certificate used for Raft peer communication, expired because nothing tracked it.

Fix. Probe expiry from outside with a blackbox check and alert at thirty and seven days. Issue listener certificates from an internal PKI with short lifetimes, renew them with an agent, and reload with SIGHUP. Test the reload in staging, because not every setting reloads without a restart.

The path that worked until 2.0

Symptom. After upgrading, a script that worked for years fails on a path with a double slash.

Cause. From 2.0, Vault rejects uncleaned paths: //, /./, /../ and non-canonical URL encodings. Code that concatenates a mount and a path with an extra slash now fails.

Fix. Normalise paths in shared helper libraries, and grep for string concatenation in every tool that calls Vault. Add a test that sends a path with a trailing slash to your client wrapper.

The break-glass script that lost its authority

Symptom. A rekey or generate-root runbook returns 403 after 2.0.

Cause. sys/rekey, sys/generate-root and sys/replication/dr/secondary/generate-operation-token require a Vault token as well as key shares.

Fix. Update the runbook to include a token with the right capability, and rehearse it. If your disaster model truly needs the older behaviour, set enable_unauthenticated_access for the named operations and record the risk decision.

The policy that silently stopped matching

Symptom. Permission denied for a team whose policy did not change.

Cause. One of the tightened behaviours: an identity template that renders + or * into the path (2.0.1 onward), list parameters compared on "contains all" instead of exact (1.21), or duplicate HCL attributes now rejected (1.21).

Fix. Use vault token capabilities <token> <path> and vault read sys/internal/ui/resultant-acl to see the effective result. Remove wildcards from identity metadata, rewrite parameter constraints, and delete duplicate keys. The environment variables VAULT_LEGACY_EXACT_MATCHING_ON_LIST and VAULT_ALLOW_PENDING_REMOVAL_DUPLICATE_HCL_ATTRIBUTES buy time and nothing more.

The container that cannot lock memory

Symptom. After an image upgrade, Vault will not start, or starts with mlock errors.

Cause. Since 2.0.2 the official image does not carry cap_ipc_lock. Raft storage also requires an explicit disable_mlock since 1.20.

Fix. Set disable_mlock = true, and disable or encrypt swap on the nodes. Do not re-add the capability by rebuilding the image.

The debug script that lost openssl

Symptom. An init container or a debug habit that ran openssl, gpg or ps inside the Vault UBI image fails after 2.0.4.

Cause. Those packages were removed from the UBI images.

Fix. Use a separate tools image or an ephemeral debug container. Keep the Vault image minimal.

The join storm

Symptom. A large cluster restart logs too many concurrent raft retry joins in progress.

Cause. From 2.1.1 the number of concurrent retry_join workers is capped at 20.

Fix. Trim the list of join targets to a handful of stable peers, or use auto_join with a provider tag, and stagger node starts.

Platform-ownership habits

The difference between a Vault that stays healthy for years and one that turns into a liability is mostly habit. These are the ones that pay off, in rough order of leverage.

1. Keep the recovery path alive. A restore drill every quarter, with the real seal configuration. The drill finds the expired KMS grant, the moved bucket and the runbook that mentions a person who left. It is the single most valuable habit on this list.

2. Everything as code, applied by a pipeline. Mounts, auth methods, roles and policies live in Git, are reviewed, and are applied by automation with a scoped identity. Nobody edits production in the UI. The benefit is not just audit: it makes the review checklist below enforceable, since every change passes through a pull request.

3. Short TTLs as a default, long TTLs as an exception. Set default_lease_ttl and max_lease_ttl per mount, rather than inheriting the 768-hour system defaults. A credential that expires in an hour is a problem that solves itself.

4. No root token between incidents. Revoke the initial root token after bootstrap, write the generate-root procedure down, require two people, and read the audit log afterwards.

5. Attested identity first. Kubernetes service accounts, cloud instance identity and CI OIDC tokens before any AppRole. When AppRole is unavoidable, deliver secret IDs wrapped, single-use and short-lived.

6. Read the release notes the week they ship. The 2.0 and 2.1 lines contained changes that broke scripts, policies and images. A fifteen-minute read of the "important changes" page prevents an afternoon of debugging.

7. Publish golden paths. Give teams one module and one snippet that produce a working tenant. The fewer custom policies you receive, the fewer you must review.

8. Report usage back to teams. Monthly client count, lease count, static credential count. People fix what they can see.

9. Write the dependency diagram. KMS, load balancer, DNS, certificates, identity provider, audit sink, storage volume. Each is a way for Vault to "be down" without Vault being broken.

10. Practise the incident. Seal a node on purpose in staging, revoke a leaked credential by prefix, collect a vault debug bundle, and time each. The playbooks improve when someone has actually run them.

A nightly hygiene check

The script below reports the things a platform owner wants to hear about before an auditor does. It uses only documented read commands and needs a token with read on the relevant paths; it does not modify anything.

#!/usr/bin/env bash
set -euo pipefail

echo "== seal and HA state =="
vault status -format=json | jq '{sealed, version, ha_enabled, is_self: .is_self}'

echo "== raft peers =="
vault operator raft list-peers

echo "== autopilot health =="
vault operator raft autopilot state -format=json | jq '{healthy, failure_tolerance}'

echo "== audit devices (want at least two) =="
vault audit list -format=json | jq 'keys'

echo "== mounts with default TTLs left unset =="
vault secrets list -detailed -format=json \
  | jq -r 'to_entries[] | select(.value.config.default_lease_ttl == 0) | .key'

echo "== policies mentioning sudo =="
for p in $(vault policy list); do
  if vault policy read "$p" | grep -q 'sudo'; then echo "$p"; fi
done

The output gives you a daily summary you can forward to a channel. Extend it with the owner-metadata check for your own convention.

Review checklists

Use these for pull requests that change Vault configuration and for design documents that depend on Vault. The aim is to catch the error while it is a comment, not an incident.

Pull request checklist: policies and roles

• Does the policy grant only the paths the workload needs, using + for one segment and * only as the final character?
• For KV v2, are data/ and metadata/ paths both considered, with list only on metadata?
• Is sudo absent, unless a named reviewer approved it in the pull request?
• Is there any grant on sys/policies, sys/auth or sys/mounts that lets the holder widen its own access?
• Do templated paths use identity values that cannot contain *, + or a slash?
• Does the auth role bind tightly: service account name and namespace for Kubernetes, repository and branch claims for JWT, never just the issuer?
• Are token_ttl and token_max_ttl explicit and as short as the workload allows? Is token_type=batch appropriate?
• Is an owner recorded for the new role or mount?
• Does the file contain duplicate attributes, which are errors since 1.21?

Pull request checklist: server configuration

• Is disable_mlock set explicitly, and is swap handled?
• Are api_addr and cluster_addr correct for every node, and is port 8201 reachable between peers only?
• Is TLS enabled with a supported minimum version, and is tls_disable absent?
• Are there at least two audit devices, and are they on separate failure domains?
• Is telemetry configured with prometheus_retention_time non-zero and disable_hostname = true?
• Is the image or binary version pinned, and does the change list reference the release notes for the jump?
• Does any new retry_join list exceed the 20-worker cap on restart?
• If enable_unauthenticated_access appears, is there a recorded risk decision?

Design-document checklist

• Availability. Voter count, zones, quorum, load balancer and health-check codes. What happens during KMS outage, certificate expiry, audit failure, zone loss?
• Recovery. Snapshot schedule, location, encryption, last successful restore drill, recovery point and time objectives.
• Seal. Type, key location relative to data residency, who can Decrypt, who holds recovery keys, generate-root procedure.
• Identity. How each workload class authenticates, where secret zero comes from, how humans log in and leave.
• Lifecycle. Dynamic versus static, TTL defaults, rotation, revocation plan and its load impact.
• Scale. Expected leases and tokens, batch token use, quotas, entry sizes, which component breaks first.
• Tenancy and cost. Namespace or prefix model, quotas per tenant, client-count reporting, edition and licence consequences.
• Change. Configuration as code, upgrade and rollback procedure, the 1.x to 2.x checklist if relevant.
• Observability. Metrics, alerts, audit shipping, dashboards, who is paged.
• Exit. What would make us choose a cloud secret manager or another platform, and how would we migrate away?

Practice cards

Short drills to do on a scratch cluster. Each takes under fifteen minutes and leaves you with a command you will reach for during a real incident.

Card 1: find the leak. Create 500 dynamic credentials in a loop with a one-hour TTL. Find them with vault list sys/leases/lookup/database/creds/app, revoke them with vault lease revoke -prefix database/creds/app, and watch vault.expire.num_leases fall.

Card 2: who read this secret? Enable a file audit device, read a KV secret as two different tokens, hash the secret value with sys/audit-hash, and find both reads in the log.

Card 3: quorum by hand. Start five Raft nodes, stop two, confirm writes work, stop a third, observe the failure, and recover by restarting one.

Card 4: the seal. Configure a Transit auto-unseal from a second dev cluster, initialise, restart the first cluster, and observe that it unseals itself. Then disable the key on the second cluster and observe what a restart does.

Card 5: the 2.0 tightening. Try a rekey without a token and with one. Try a path with a double slash. Try a templated path whose metadata contains a wildcard. Write down what you saw and why each exists.

Card 6: the rolling upgrade. On a three-node Raft cluster, replace the binary one node at a time, standbys first, step the active node down, and confirm the version on each with vault status. Script it.

Card 7: the quota. Add a global rate-limit quota of 50 per second, run a load loop, read the 429 and the Retry-After header, then add a narrower quota for one path.

Card 8: the restore. Save a snapshot, add ten secrets, restore, and confirm they are gone. Explain aloud what else the restore undid.

Operational recipes

Commands worth keeping in a runbook, each with the situation it serves.

Check cluster health quickly

vault status
vault operator raft list-peers
vault operator raft autopilot state
curl -s -o /dev/null -w "%{http_code}\n" "$VAULT_ADDR/v1/sys/health"

The health code tells you the role: 200 active, 429 standby, 503 sealed, 474 standby that cannot find the active node.

Preflight a config before a restart

vault operator diagnose -config=/etc/vault.d/vault.hcl

Diagnose reports problems such as a TLS-disabled listener, unreadable storage directories and unreachable seal devices before you bring a node down, which is a far better moment to discover them.

Collect evidence during an incident

vault debug -duration=2m -output=debug.tar.gz
vault monitor -log-level=debug

Bundle the debug output and handle it as sensitive. Use trace level sparingly, and never in a shared channel.

Revoke a compromised credential

vault token revoke -accessor "$ACCESSOR"
vault lease revoke -prefix database/creds/app
vault write -f database/rotate-root/pg

Revoking by accessor avoids exposing the token itself. Rotating the backend root afterwards is part of the clean-up.

Explain a permission denied

vault token lookup
vault token capabilities "$TOKEN" secret/data/myapp/config
vault read sys/internal/ui/resultant-acl

Most denials are a missing data/ segment in a KV v2 policy, a missing list on metadata, an expired token or the wrong namespace.

Add and adjust quotas

vault write sys/quotas/rate-limit/global rate=500 interval=1s
vault read sys/quotas/rate-limit/global
vault list sys/quotas/rate-limit

Change Autopilot safely

vault operator raft autopilot set-config \
  -cleanup-dead-servers=true \
  -dead-server-last-contact-threshold=10m \
  -min-quorum=3

Always set -min-quorum when turning cleanup on.

Reload TLS without a restart

sudo systemctl kill --signal=HUP vault

SIGHUP reloads listener certificates, the log level and some other settings. Verify the new certificate with openssl s_client from a separate host, since the Vault image itself no longer ships openssl from 2.0.4.

Find out why an operation needs a particular policy

vault kv get -output-policy -mount=secret myapp/config

The -output-policy flag prints the policy the command would need, which is a good way to draft a least-privilege policy for a new workload. Its sibling -output-curl-string prints the equivalent curl command for an API-level reproduction.

The checklist to run before any change ships

1. Can I state, in one sentence, which failure mode this change could trigger and how I would recognise it?
2. Is there a fresh snapshot, and have I restored one in the last quarter?
3. Has the change been applied to staging through the same pipeline?
4. Did I read the release notes and the important-changes page for this version jump?
5. Does it leave at least two audit devices working?
6. Does it keep TTLs short and use attested identity, not a stored secret?
7. Does any policy contain sudo, a broad sys/ grant, or a wildcard in template output?
8. Is there an owner, a quota and a cost line for any new tenant?
9. Is the rollback a restore, and do I know how long it takes?
10. Who is on call for the next 24 hours, and do they know this change happened?

If you can answer all ten, ship it. If you cannot answer one, that is the thing to fix first. For where these skills connect to the rest of the platform, see the Kubernetes, Terraform and Prometheus guides.