The Complete Terraform Guide

This is part one of three, and it covers everything you need to do real work with Terraform. By the end you can install Terraform, write a configuration from scratch, read a plan the way a reviewer does, understand what the state file is and why everyone is nervous about it, use variables, outputs and loops, and create, change and destroy infrastructure with confidence. You will also know how to read the error messages that stop most beginners. Mid-level and Senior take the same ideas into teams, pipelines and production. Nothing on this page is thrown away later.

The main examples run entirely on your laptop, with no cloud account and no bill. They use two small providers that manage local files and random names, so you can make every mistake on purpose and lose nothing. One section then shows the same workflow against a real cloud, so you can see that nothing changes except the provider.

Each section ends with a Try it task. Do them as you go. Terraform only clicks once you have watched your own plan say "1 to add", applied it, broken it and watched the next plan notice.

What Terraform is, and the problem it solves

Terraform is an infrastructure as code tool. You describe the infrastructure you want (servers, networks, storage buckets, databases, DNS records, Kubernetes clusters, even a file on disk) in plain text files. Terraform works out what has to change to make the real world match that description, shows you the list of changes, and then makes them by calling the relevant APIs for you.

That sounds abstract, so start with what came before it.

The first way people build cloud infrastructure is by clicking. You open the AWS, Azure or Google Cloud console, create a storage bucket, tick a few boxes, create a virtual machine, attach it to a network, and write down what you did (or don't). This is sometimes called ClickOps, and it works well for one person on day one. It works badly for everyone after that. Six months later, nobody remembers why the bucket has public access turned on. The staging environment was built by a different person on a different afternoon, so it differs from production in twenty small ways, and one of them causes an outage. When someone asks "can we build this again in another region?", the honest answer is "roughly, over a week, with some surprises".

The second way is scripting. You write a shell script or a Python script that calls the cloud's command-line tool: create the bucket, create the machine, attach the network. Scripts are repeatable, which is an improvement, but they are imperative: they describe steps, not an end result. Run the script twice and the second run fails because the bucket already exists, or worse, creates a second machine. To make the script safe to re-run you add "if it doesn't exist, create it" checks everywhere. Then someone wants to change the machine size, and you need "if it exists but is the wrong size, resize it" logic too. Before long the script is mostly checking and very little doing, and it still cannot tell you what it is about to change before it changes it.

Terraform takes the third approach. You write down the desired end state, and Terraform does the comparison for you:

Three consequences of that design explain most of what follows in this guide, so notice them now.

The infrastructure is described in files you commit. Your .tf files live in Git next to your application code. Every change to the infrastructure is a diff that someone can review before it happens, and git log tells you who opened port 22 to the internet and why. Teams that adopt Terraform usually say this is the real win: infrastructure changes become ordinary code review.

You see the change before it happens. Terraform's plan command prints exactly what it will create, modify and destroy. You read it, and only then apply it. No script gives you that, and it is the habit that separates careful engineers from people who make headlines.

Terraform keeps a record of what it manages. To know that the bucket in your file corresponds to a particular bucket in your account, Terraform writes a state file. State is the most important and most misunderstood part of Terraform, and it gets its own section below.

Terraform works with far more than one cloud. It talks to each platform through a plugin called a provider, and there are providers for AWS, Azure, Google Cloud, Oracle Cloud, Kubernetes, GitHub, Cloudflare, Datadog, and thousands of other services on the public Terraform Registry. The workflow you learn on this page is identical for all of them.

What people in machine learning and data teams use it for:

A word on the name and the licence, because you will hear both in interviews. Terraform is made by HashiCorp, which IBM acquired in 2025. Since version 1.6 in 2023, Terraform has been released under the Business Source License rather than an open-source licence. In response, the community created OpenTofu, an open-source fork now run by the Linux Foundation. OpenTofu started from the same code and uses the same language, but it is a separate project with its own releases and features. This guide is about Terraform. Most of the beginner material transfers directly to OpenTofu, but don't assume newer features match.

HashiCorp also sells a hosted service that runs Terraform for teams. It used to be called Terraform Cloud and is now called HCP Terraform, and there is a self-hosted version called Terraform Enterprise. You don't need either to learn Terraform. Everything on this page uses the free command-line tool, usually called the Terraform CLI.

Declarative versus imperative

Every interview about Terraform starts here, and it is also the idea that explains Terraform's behaviour when it surprises you, so it is worth a section on its own.

An imperative tool takes instructions: create this, then change that, then delete the other. You are responsible for knowing the current situation and choosing the right steps. A declarative tool takes a description of the result: there should be one bucket called churn-model-artifacts, with versioning on. The tool is responsible for finding out the current situation and choosing the steps.

The property in the second row has a name worth knowing: idempotence. An operation is idempotent if doing it once and doing it ten times gives the same result. terraform apply is idempotent. When the real infrastructure already matches your files, a second apply reports "No changes" and touches nothing. This is what makes it safe to run Terraform often, including automatically in a pipeline.

The third row catches people out. In a declarative tool, removing something from the description is an instruction to remove it from the world. If you delete a resource block from your file and apply, Terraform destroys that resource. This is exactly the right behaviour, because the file says it should not exist. But it means that "tidying up" a Terraform file is a real change, and the plan is where you catch it.

Declarative does not mean Terraform can do anything you describe. It can only manage what its providers know how to create, read, update and delete, and some settings of some resources cannot be changed in place, so Terraform has to destroy the old object and create a new one. The plan tells you when that will happen, and you will learn to spot it below.

The core model: provider, resource, state, plan

Four nouns carry almost all of Terraform. Learn them precisely, because every error message and every interview question uses them.

A provider is a separate program, downloaded from a registry, that translates Terraform's generic "create this resource" into the specific API calls a platform understands. Terraform itself knows nothing about S3 or virtual machines. The AWS provider does. Every provider has an address in the form namespace/type, such as hashicorp/aws. The full form includes the registry's hostname, registry.terraform.io/hashicorp/aws, and you will see that longer form in error messages.

A resource is one infrastructure object that Terraform manages. You write it as a block with two labels: the resource type, which comes from the provider, and a local name that you choose. The combination is the resource's address, which is how you refer to it everywhere else: aws_s3_bucket.artifacts, local_file.readme. The name is only used inside Terraform. It is not the name of the bucket in AWS.

State is a JSON file in which Terraform records every resource it manages, the real-world ID behind it, and the values of its attributes as of the last run. Without state, Terraform would have no way to know that aws_s3_bucket.artifacts in your file is the bucket it created last Tuesday rather than some other bucket with a similar name.

A plan is the result of comparing three things: your configuration (the .tf files, which say what should exist), the state (which says what Terraform created last time), and the real infrastructure (which the providers read fresh at the start of every plan, in a step called refresh). The differences become a list of actions: create, update in place, replace, or destroy.

A few more words you will meet in the first hour, defined now so they don't trip you later:

• Configuration: all the .tf files in one directory, read together. Terraform merges them, and the order of files and blocks does not matter.
• Module: any directory of .tf files. The directory you run Terraform in is the root module. Mid-level covers calling other modules from it.
• Data source: a read-only lookup of something that already exists, written as a data block. It reads and never changes anything.
• Apply: the command that carries out a plan.
• Drift: a difference between the state and reality, usually because someone changed something outside Terraform.
• HCL: the HashiCorp Configuration Language, the syntax .tf files are written in.

Installing Terraform and checking your setup

Terraform is a single binary with no runtime dependencies, so installation is mostly about getting that one file onto your PATH from a trustworthy source. HashiCorp publishes official packages for every common operating system.

macOS. Use HashiCorp's own Homebrew tap. The name matters:

brew tap hashicorp/tap
brew install hashicorp/tap/terraform

Ubuntu and Debian. Add HashiCorp's signing key and package repository, then install with apt:

wget -O - https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(grep -oP '(?<=UBUNTU_CODENAME=).*' /etc/os-release || lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update && sudo apt install terraform

The first line downloads HashiCorp's public key and stores it where apt can use it to check signatures. The second line tells apt where the packages live and to trust only packages signed with that key. The third installs. It looks like a lot of ceremony, but it is what lets sudo apt upgrade keep Terraform current later.

Red Hat, CentOS, Fedora and Amazon Linux have equivalent repositories. On RHEL:

sudo yum install -y yum-utils
sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
sudo yum -y install terraform

Windows. The official route is to download the zip file for your architecture from releases.hashicorp.com/terraform, extract terraform.exe into a folder such as C:\tools\terraform, and add that folder to your PATH under System Properties, Environment Variables. Community package managers also work (winget install Hashicorp.Terraform or choco install terraform), but HashiCorp does not maintain those packages, so check the version they give you.

Any system, pinned exactly. For a build server where you want one precise version, download the zip plus the SHA256SUMS file from https://releases.hashicorp.com/terraform/<version>/, check the checksum, and unzip the binary onto your PATH. Teams that need different versions per repository often use a version manager such as tenv or tfenv. These are community tools, not HashiCorp products.

Now check the result. Three commands tell you everything:

terraform version
terraform -help
terraform -install-autocomplete

terraform version should print something like this:

Terraform v1.16.4
on darwin_arm64

The first line is the Terraform release. The second is your operating system and processor architecture, here macOS on Apple Silicon. You'll see linux_amd64 on most Linux laptops and servers. If a newer release exists, Terraform adds a line telling you your version is out of date. That check contacts HashiCorp's servers, and you can turn it off by setting the environment variable CHECKPOINT_DISABLE=1.

terraform -help lists every subcommand, with the main ones first. terraform plan -help prints the options for one command, and you should reach for it before searching the web, because it always matches the version you have. The third command adds tab completion for Terraform subcommands to bash or zsh. Open a new terminal afterwards for it to take effect.

About versions. Terraform releases a new minor version (1.14, 1.15, 1.16) every three to four months, with small patch releases (1.16.1, 1.16.2) in between. All 1.x releases follow HashiCorp's compatibility promise: a configuration that works on 1.10 keeps working on 1.16. The reverse isn't guaranteed, though. A state file written by a newer Terraform may not be readable by an older one. So everyone who works on the same project should use the same minor version, and you will pin it in your configuration shortly.

Your first project, step by step

Your first project creates two things: a random project name, and a text file that mentions it. That is deliberately small. It uses two official HashiCorp providers, random and local, which need no account and cost nothing, and it still exercises the whole workflow you will use against a real cloud.

Make an empty folder and open it in your editor:

mkdir hello-terraform && cd hello-terraform

Create one file called main.tf:

terraform {
  required_version = "~> 1.16"

  required_providers {
    local = {
      source  = "hashicorp/local"
      version = "~> 2.9"
    }
    random = {
      source  = "hashicorp/random"
      version = "~> 3.9"
    }
  }
}

resource "random_pet" "project" {
  length = 2
}

resource "local_file" "readme" {
  filename = "${path.module}/out/README.txt"
  content  = "Project ${random_pet.project.id} is managed by Terraform.\n"
}

output "project_name" {
  value = random_pet.project.id
}

Read it top to bottom before running anything.

The terraform block holds settings for Terraform itself. required_version = "~> 1.16" says this configuration needs Terraform 1.16 or any later 1.x release, and Terraform refuses to run it on anything older. required_providers lists the plugins this configuration needs, where to download each one (source) and which versions are acceptable (version). The ~> operator is called the pessimistic constraint: ~> 2.9 means "2.9 or newer, but below 3.0", which accepts bug-fix and feature releases while refusing the next major version, where breaking changes live.

The first resource block asks the random provider for a random_pet, a readable random name such as lucky-rooster, made of two words. The second asks the local provider for a file. Its filename uses path.module, a built-in value meaning "the directory this configuration lives in", so the file lands in an out folder next to your code. Its content refers to the pet name with random_pet.project.id. That reference is doing two jobs. It inserts the value, and it tells Terraform that the file depends on the pet, so the pet must be created first.

The output block prints a value after Terraform runs, and makes it available to scripts. Here it prints the generated name.

Step 1: terraform init

terraform init

init prepares the working directory. It reads required_providers, downloads each provider, and records exactly which versions it chose. The output tells you precisely what happened:

Initializing the backend...

Initializing provider plugins...
- Finding hashicorp/local versions matching "~> 2.9"...
- Finding hashicorp/random versions matching "~> 3.9"...
- Installing hashicorp/local v2.9.1...
- Installed hashicorp/local v2.9.1 (signed by HashiCorp)
- Installing hashicorp/random v3.9.1...
- Installed hashicorp/random v3.9.1 (signed by HashiCorp)

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

"Initializing the backend" refers to where state will be stored. You have not configured anything, so it is the default, a local file. The "Finding … Installing … Installed" lines show each provider constraint being resolved to a concrete version and downloaded. "Signed by HashiCorp" means the download's signature was checked. Your version numbers will be newer if new releases have come out since this was written, which is exactly what the ~> constraints allow.

init created two things in your folder. The .terraform/ directory is a local cache holding the downloaded provider binaries. It can always be recreated, so it never goes into Git. The .terraform.lock.hcl file is the dependency lock file. It records the exact provider versions chosen and checksums of their packages, so that a teammate, or your CI pipeline, running init next week gets the same versions and not whatever is newest that day. It always goes into Git.

You run init once when you start, and again whenever you add a provider, change a version constraint, or change where state is stored. Running it again when nothing changed is harmless.

Step 2: terraform fmt and terraform validate

terraform fmt
terraform validate

fmt rewrites your files into Terraform's standard layout: two-space indentation and aligned equals signs. It prints the names of files it changed, and nothing if they were already tidy. Every Terraform codebase uses it, so diffs show real changes rather than whitespace.

validate checks that the configuration is internally consistent: correct syntax, known argument names, references that point to things that exist, values of the right type. It doesn't contact any cloud API, so it is fast and needs no credentials. You should see:

Success! The configuration is valid.

Step 3: terraform plan

terraform plan

This is the command you will run most often. It compares your configuration with the state (empty so far) and with reality, and prints what it would do:

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # local_file.readme will be created
  + resource "local_file" "readme" {
      + content              = (known after apply)
      + content_base64sha256 = (known after apply)
      + directory_permission = "0777"
      + file_permission      = "0777"
      + filename             = "./out/README.txt"
      + id                   = (known after apply)
      # (other checksum attributes omitted here)
    }

  # random_pet.project will be created
  + resource "random_pet" "project" {
      + id        = (known after apply)
      + length    = 2
      + separator = "-"
    }

Plan: 2 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + project_name = (known after apply)

Every + means "will be created". The last line is the summary to read first on every plan you ever run: two to add, nothing changed, nothing destroyed. Some values say (known after apply). The pet's name does not exist until the random provider generates it during apply, so Terraform cannot show it yet, and the file's content depends on the name, so that is unknown too. The plan also shows default values you never wrote, such as separator = "-", because the provider fills them in.

A plan changes nothing. You can run it a hundred times.

Step 4: terraform apply

terraform apply

apply computes the plan again, prints it, and then stops and asks:

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

Only the full word yes is accepted. Anything else cancels. After you confirm, Terraform creates the resources in dependency order and reports each one:

random_pet.project: Creating...
random_pet.project: Creation complete after 0s [id=lucky-rooster]
local_file.readme: Creating...
local_file.readme: Creation complete after 0s [id=e3ca33a3f60921936c06ea699a33c806463ec814]

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Outputs:

project_name = "lucky-rooster"

Notice the order: the pet first, then the file, because the file refers to the pet. The value in square brackets is each resource's ID, the identifier the provider uses to find the real object again. For a cloud resource this would be something like a bucket name or an instance ID.

Look at what you have now:

cat out/README.txt
ls

The file exists with a generated name inside it. There is also a new file in your folder, terraform.tfstate. That is the state, and the section after next is about it.

Step 5: run it again

terraform plan

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration
and found no differences, so no changes are needed.

This is idempotence in practice. The pet name does not change, and the file is not rewritten. Terraform read the state, checked the real file, compared both with your configuration, and found nothing to do. A second apply would say the same.

Step 6: clean up

terraform destroy

destroy plans the removal of everything this configuration manages, shows you the plan with - symbols, and asks for yes. It removes resources in the reverse of creation order, so the file goes before the name it depends on. It is the same as terraform apply -destroy. Run terraform apply again afterwards and you get a new pet name, because the old one was destroyed with everything else.

Reading a plan like a reviewer

A plan is the one moment when you can stop a mistake before it reaches the real world, so reading it well is the single most valuable Terraform skill. At work, the plan is also what your teammates review in a pull request. People who skim plans are the people who delete production databases.

Every resource in a plan starts with a symbol and a comment line saying what will happen to it:

The replacement symbols deserve the most attention. Some settings of some resources cannot be changed on an existing object. A bucket cannot be renamed, and a virtual machine cannot move to another availability zone, for example. When you change one of those settings, the provider tells Terraform the only way to get there is to destroy the object and create a new one. For a random name or a text file that is harmless. For a database it means an empty database.

Here is what a replacement looks like. Change the file content in the first project by adding "Owner: data team." to the string, then run terraform plan:

Terraform will perform the following actions:

  # local_file.readme must be replaced
-/+ resource "local_file" "readme" {
      ~ content              = <<-EOT # forces replacement
          - Project lucky-rooster is managed by Terraform.
          + Project lucky-rooster is managed by Terraform. Owner: data team.
        EOT
      ~ content_md5          = "d7ce13a0cc8c9a04f01ef8bf1ae31421" -> (known after apply)
      ~ id                   = "e3ca33a3f60921936c06ea699a33c806463ec814" -> (known after apply)
        # (3 unchanged attributes hidden)
    }

Plan: 1 to add, 0 to change, 1 to destroy.

Read it in this order. First, the summary line: one to add and one to destroy means a replacement, not a change, even though you only edited text. Second, the resource comment: "must be replaced". Third, find the attribute marked # forces replacement. That is the reason. The local provider treats a file's content as unchangeable, so a new content means a new file. Fourth, the individual attribute changes: ~ marks a changed value, shown as old -> new, and a multi-line string shows removed lines with - and added lines with +. Unchanged attributes are hidden and counted, which keeps big plans readable.

The plan also ends with a note you will see every time:

Note: You didn't use the -out option to save this plan, so Terraform can't
guarantee to take exactly these actions if you run "terraform apply" now.

This is a real warning, not boilerplate. terraform apply on its own computes a fresh plan. If something changed between your plan and your apply (a teammate applied, or someone edited a setting in the console), the apply plan can differ from the one you reviewed. The fix is to save the plan to a file and apply that exact file:

terraform plan -out=tfplan
terraform apply tfplan

Applying a saved plan does not ask for confirmation, because the review happened when you read it. If the state changed after you saved it, Terraform refuses with Saved plan is stale, and you plan again. Treat the tfplan file as sensitive and keep it out of Git: it is binary, and it can contain secret values in readable form. Pipelines at work almost always use this two-step form.

Two more plan options are worth knowing on day one. terraform plan -destroy shows what a destroy would remove without doing it. terraform plan -detailed-exitcode changes the exit code so scripts can tell the outcomes apart: 0 means no changes, 1 means an error, and 2 means there are changes. That is how scheduled jobs detect drift.

State: Terraform's memory

Open terraform.tfstate in your editor. It is JSON, and the interesting part looks like this (shortened):

{
  "version": 4,
  "terraform_version": "1.16.4",
  "serial": 3,
  "lineage": "8f1c0a52-…",
  "outputs": {
    "project_name": { "value": "lucky-rooster", "type": "string" }
  },
  "resources": [
    {
      "mode": "managed",
      "type": "random_pet",
      "name": "project",
      "provider": "provider[\"registry.terraform.io/hashicorp/random\"]",
      "instances": [
        { "attributes": { "id": "lucky-rooster", "length": 2, "separator": "-" } }
      ]
    }
  ]
}

Each entry maps a resource address in your configuration (type random_pet, name project) to the real object (ID lucky-rooster) and records its attributes as of the last apply. serial goes up with every write. lineage is a unique ID given to this state when it was first created, which stops Terraform from mixing up two unrelated state files.

State exists because the configuration alone cannot answer the question "which real object is this block about?". Your file says there should be a bucket. Your AWS account has forty buckets. State is the link between the two, and it also lets Terraform notice that a block has been deleted from your files, since the resource is still in state but no longer in the configuration, which means it should be destroyed.

You should almost never read the JSON directly. Terraform has commands for it:

terraform state list                       # every resource address in state
terraform state show random_pet.project    # one resource's recorded attributes
terraform show                             # the whole state, human-readable

$ terraform state list
local_file.readme
random_pet.project

$ terraform state show random_pet.project
# random_pet.project:
resource "random_pet" "project" {
    id        = "lucky-rooster"
    length    = 2
    separator = "-"
}

state list is the fastest way to answer "what does this configuration manage?". Mid-level covers the commands that change state (moving and removing entries), which you should not need yet.

Drift: when reality changes behind Terraform's back

At the start of every plan Terraform refreshes: each provider reads the real objects and compares them with what state recorded. If someone changed something by hand, that difference is called drift, and the plan will propose to put things back the way your configuration says. Try it:

echo "edited by hand" > out/README.txt
terraform plan

random_pet.project: Refreshing state... [id=lucky-rooster]
local_file.readme: Refreshing state... [id=e3ca33a3f60921936c06ea699a33c806463ec814]

  # local_file.readme will be created
  + resource "local_file" "readme" {
      + content  = <<-EOT
            Project lucky-rooster is managed by Terraform.
        EOT
      ...
    }

Plan: 1 to add, 0 to change, 0 to destroy.

The "Refreshing state" lines are the provider reading reality. The local provider noticed that the file no longer has the content it wrote, so it treats the old file as gone, and Terraform plans to write it again. A cloud provider would more often show a ~ update that reverts the changed setting. Either way, the rule is the same: your configuration wins. If the manual change was actually correct, the fix is to put it in the .tf file, not to keep changing things by hand.

The rules for state

Beginners get hurt by state in four predictable ways, so learn the rules before you need them.

State can contain secrets in plain text. Any attribute a provider returns is written to state, and that includes database passwords, private keys and access tokens. Marking a value sensitive hides it from the terminal but not from the state file. Treat the state file like a password file.

Never commit state to Git. It holds secrets, and two people with two copies of state in two branches will each believe different things about what exists. Add *.tfstate and *.tfstate.* to .gitignore on day one.

Never edit state by hand. If the JSON becomes invalid, or an ID points at the wrong object, Terraform will make confident, wrong decisions. When state really needs changing, there are commands for it, covered at Mid-level.

Local state does not work for a team. The file on your laptop is only on your laptop. For shared work, state is stored in a backend: a remote store such as an S3 bucket, an Azure storage account or a Google Cloud Storage bucket, or HCP Terraform. A good backend also provides locking, which stops two people applying at the same time and corrupting the state. You will set one up at Mid-level. For now, know that terraform.tfstate in your folder is the local backend, the default, and that it is fine for learning and for one-person experiments only.

HCL in one sitting

Terraform files are written in HCL, the HashiCorp Configuration Language. It is small. You can learn enough of it to read almost any configuration in one sitting, and this section covers that much.

Blocks are the containers. A block has a type, zero or more labels in quotes, and a body in braces:

resource "local_file" "readme" {   # type "resource", two labels
  filename = "out/README.txt"       # an argument
}

The block types you will use as a beginner are terraform, provider, resource, data, variable, locals and output. Some blocks contain nested blocks, like the required_providers inside terraform or a validation inside a variable.

Arguments assign a value to a name: filename = "out/README.txt". Which argument names are allowed depends on the block. For resources, the provider's documentation on the Terraform Registry lists every argument, which ones are required, and which attributes the resource exports after creation (such as the id you used). Get into the habit of opening the resource's documentation page every time you use a resource type for the first time.

Values have types:

References read values from elsewhere in the configuration. You have already seen random_pet.project.id, which reads the id attribute of a resource. Variables are var.NAME, locals are local.NAME, and data sources are data.TYPE.NAME.ATTRIBUTE. There are also a few built-ins, such as path.module for the configuration's directory.

Interpolation puts a value inside a string with ${ }: "Project ${random_pet.project.id}". When the whole value is a single reference, write it without quotes: value = random_pet.project.id, not value = "${random_pet.project.id}". Both work, but reviewers and linters expect the first, and older examples that wrap every reference in "${ }" date from before Terraform 0.12.

Expressions and functions calculate values. The conditional expression is condition ? value_if_true : value_if_false, as in var.environment == "prod" ? 3 : 1. Terraform has a library of built-in functions, called like upper("dev"), join("-", ["a", "b"]), lower(var.name), jsonencode({...}), file("path") and merge(map1, map2). You cannot write your own functions in HCL. For experimenting, terraform console opens an interactive prompt where you can type any expression and see its value:

$ terraform console
> upper("churn-model")
"CHURN-MODEL"
> join("-", ["churn", "dev", "01"])
"churn-dev-01"
> "dev" == "prod" ? 3 : 1
1
> exit

Multi-line strings use the "heredoc" form, which you will use for file contents and scripts. With <<-EOT, the leading indentation is removed, so you can indent the text to match your code:

content = <<-EOT
  # ${var.project}
  Managed by Terraform.
EOT

Comments use # (preferred) or // for a single line and /* ... */ for several lines.

Files and layout. Terraform reads every .tf file in the directory and treats them as one configuration, so the split between files is only for humans. The convention most teams follow, and the one used in the capstone below:

Terraform ignores files in subdirectories. It only reads the directory you run it in, so modules/network/main.tf is not part of your configuration until you call it as a module. Terraform also ignores files with other extensions, which is a classic trap: a file called main.tf.txt or variables.tf.bak is silently skipped.

Variables, locals and outputs

Your first project hard-coded everything. That is fine for one experiment, but the point of infrastructure as code is to build the same thing several times with small differences: a dev copy and a prod copy, a bucket in one region and another in a second region. Three block types handle this, and they map neatly onto a function in any programming language. Input variables are the parameters, locals are the intermediate values you compute inside the function, and outputs are the return values.

Input variables

A variable block declares an input. By convention they live in variables.tf:

variable "project" {
  type        = string
  description = "Short name of the ML project, used in every resource name."
}

variable "environment" {
  type        = string
  description = "Which copy of the project this is."
  default     = "dev"

  validation {
    condition     = contains(["dev", "staging", "prod"], var.environment)
    error_message = "environment must be dev, staging or prod."
  }
}

variable "data_folders" {
  type        = list(string)
  description = "Top-level folders every project gets."
  default     = ["raw", "features", "models"]
}

Every argument inside is optional, but write type and description every time. The type makes Terraform reject a wrong value early, with a clear message, instead of failing halfway through an apply. The description is the documentation a teammate reads when they wonder what to pass. A variable with a default is optional for the caller. A variable without one is required, and Terraform will not plan until it has a value.

The validation block adds your own rule on top of the type. condition is any expression that must be true, and error_message is what the user sees when it is not. Validation runs during plan, before anything changes, so a typo like environment = "prdo" stops at the first command instead of creating a misnamed bucket.

You read a variable anywhere in the configuration as var.NAME: var.project, var.environment.

Giving variables values

There are five ways to set a variable, and when the same variable is set in more than one place, the later one in this list wins:

1. An environment variable named TF_VAR_ plus the variable name, for example export TF_VAR_environment=staging.
2. A file called terraform.tfvars in the working directory, loaded automatically.
3. A file called terraform.tfvars.json, also loaded automatically.
4. Any files ending in .auto.tfvars or .auto.tfvars.json, loaded automatically in alphabetical order.
5. The -var and -var-file options on the command line, in the order you type them, the last one winning.

A .tfvars file contains only assignments, no blocks:

project     = "churn"
environment = "dev"

Most teams keep one file per environment and choose it explicitly on the command line, which leaves no doubt about which copy you are touching:

terraform plan -var-file=prod.tfvars
terraform plan -var 'environment=staging'

If a required variable still has no value, Terraform asks for it interactively:

var.project
  Short name of the ML project, used in every resource name.

  Enter a value:

That prompt is convenient on a laptop and a problem in automation, where nobody is there to type. Pipelines pass -input=false, and then a missing value becomes an error instead of a prompt: Error: No value for required variable.

Locals

A locals block names a value you compute once and use in several places. Locals cannot be set from outside, which is exactly what you want for derived values:

locals {
  name_prefix = "${var.project}-${var.environment}"

  common_tags = {
    project     = var.project
    environment = var.environment
    managed_by  = "terraform"
  }
}

You read them as local.name_prefix (note: the block is locals, plural, and the reference is local., singular, which trips everyone once). The rule of thumb is simple. If a value comes from the person running Terraform, it is a variable. If it is built from other values, it is a local. A configuration where the same "${var.project}-${var.environment}" string appears in ten places has a local waiting to be written.

Outputs

An output block publishes a value after apply. Outputs are printed at the end of terraform apply, can be read again later with terraform output, and are how scripts and other tools get information out of Terraform, such as a bucket name for a training job or an endpoint for a smoke test.

output "name_prefix" {
  description = "Prefix used for every resource in this environment."
  value       = local.name_prefix
}

terraform output                    # all outputs, human-readable
terraform output name_prefix        # one output, quoted
terraform output -raw name_prefix   # one output, bare, for scripts
terraform output -json              # everything, for tools like jq

Outputs also matter for a reason you will meet at Mid-level: when one configuration is called as a module by another, its outputs are the only values the caller can see.

Making several of something: count and for_each

Real configurations rarely need exactly one of each thing. An ML project needs a folder or bucket for raw data, one for features and one for models, and copying the same resource block three times is how bugs get in: somebody fixes the first copy and forgets the other two. Terraform has two meta-arguments (arguments that work on any resource, whatever the provider) for making several instances from one block.

count takes a whole number and makes that many instances, numbered from zero:

resource "random_pet" "worker" {
  count  = 3
  length = 2
}

The instances are addressed as random_pet.worker[0], random_pet.worker[1] and random_pet.worker[2]. Inside the block, count.index gives the current number. count is ideal for "N identical things", and it has a second common use as an on/off switch: count = var.create_bucket ? 1 : 0.

for_each takes a map or a set of strings and makes one instance per element, addressed by key rather than number:

resource "local_file" "folder_readme" {
  for_each = toset(var.data_folders)

  filename = "${path.module}/out/${each.key}/README.md"
  content  = "# ${each.key}\n\nPart of ${local.name_prefix}.\n"
}

Inside the block, each.key is the current element (and for a map, each.value is its value). The instances are local_file.folder_readme["raw"], local_file.folder_readme["features"] and local_file.folder_readme["models"]. toset() is there because for_each refuses a plain list. A list can contain duplicates and has an order, and for_each needs unique, unordered keys.

Which one to use matters more than it looks, and the reason is what happens when the list changes:

With count, the identity of each instance is its position. Take "features" out of a three-item list and "models" moves from position 2 to position 1, so Terraform sees position 1 change from "features" to "models" and position 2 disappear. With files that is a rewrite. With buckets full of data it is a replacement and a deletion. With for_each, removing "features" removes exactly ["features"] and nothing else moves.

In a shell, quote addresses that contain brackets and quotes, or the shell will mangle them: terraform state show 'local_file.folder_readme["raw"]'.

Data sources, references and the dependency graph

So far every value came from your own configuration. Real infrastructure has to fit into what already exists: a network another team built, the ID of the latest machine image, the account you are logged in to. A data source is a read-only lookup for exactly that. It is written as a data block, it never creates or changes anything, and its result is read during plan (or during apply, if it depends on something not yet created).

The local provider has a data source that reads a file, which keeps the example offline:

data "local_file" "team" {
  filename = "${path.module}/team.txt"
}

resource "local_file" "owners" {
  filename = "${path.module}/out/OWNERS"
  content  = "Owned by: ${trimspace(data.local_file.team.content)}\n"
}

A data source is referenced with the data. prefix: data.TYPE.NAME.ATTRIBUTE. In a cloud configuration the same pattern looks like data "aws_caller_identity" "me" {}, which tells you which AWS account your credentials belong to, or a lookup of an existing network by its tags. The provider's documentation lists data sources separately from resources, under "Data Sources".

How Terraform decides the order

You never tell Terraform what order to do things in. It works it out from references. Whenever one block mentions another (random_pet.project.id inside the file's content, data.local_file.team.content inside another file), Terraform draws an edge between them in a dependency graph. It then creates things in an order that respects every edge, destroys them in the reverse order, and runs anything without an edge between them in parallel, ten operations at a time by default.

This is why the order of blocks in your files does not matter, and why you should never try to control ordering by rearranging them. References are the ordering mechanism.

Very occasionally, one thing must wait for another without any value flowing between them. The classic cloud example is a server that needs an access policy attached before it starts, where nothing in the server's arguments mentions the policy. For that case there is depends_on:

resource "local_file" "done_marker" {
  filename   = "${path.module}/out/.done"
  content    = "ready\n"
  depends_on = [local_file.folder_readme]
}

Use it sparingly. A reference is always better when one exists, because it documents why the dependency is there. An unnecessary depends_on slows plans down and can make a data source wait until apply to be read, which turns known values in your plan into (known after apply).

If you want to see the graph, terraform graph prints it in the DOT format used by Graphviz, and since Terraform 1.16 terraform graph -format=mermaid prints a Mermaid diagram you can paste into a Markdown file on GitHub.

The same workflow against a real cloud

Everything so far ran on your laptop. This section shows that moving to a real cloud changes the provider and the credentials, and nothing else. The example creates a private, versioned storage bucket for model artifacts on AWS. It costs almost nothing while empty, but it does need an AWS account, and you should destroy it at the end. If you don't have an account, read it anyway: the shape is identical on Azure, Google Cloud and Oracle Cloud.

Credentials first

A provider needs to authenticate to its API, and credentials never go in .tf files. The AWS provider looks for them in the same places the AWS command-line tool does: environment variables, the shared profile files in ~/.aws/, or a single sign-on session. The simplest setup on a laptop is a named profile:

aws configure sso          # or: aws configure, for access keys
export AWS_PROFILE=my-sandbox
aws sts get-caller-identity

If the last command prints your account ID, Terraform will find the same credentials. If it fails, Terraform will too, with this message:

Error: configuring Terraform AWS Provider: no valid credential sources for Terraform AWS Provider found.

That error always means the provider found no credentials in its environment, never that your configuration is wrong. Fix the shell, not the code.

The configuration

terraform {
  required_version = "~> 1.16"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
    random = {
      source  = "hashicorp/random"
      version = "~> 3.9"
    }
  }
}

provider "aws" {
  region = var.region

  default_tags {
    tags = {
      project    = "churn"
      managed_by = "terraform"
    }
  }
}

variable "region" {
  type        = string
  description = "AWS region for the artifact bucket."
  default     = "me-central-1"
}

resource "random_pet" "suffix" {
  length = 2
}

resource "aws_s3_bucket" "artifacts" {
  bucket = "churn-artifacts-${random_pet.suffix.id}"
}

resource "aws_s3_bucket_versioning" "artifacts" {
  bucket = aws_s3_bucket.artifacts.id

  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_public_access_block" "artifacts" {
  bucket                  = aws_s3_bucket.artifacts.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

output "bucket_name" {
  value = aws_s3_bucket.artifacts.bucket
}

Three things are new. The provider "aws" block configures the provider: which region to work in, and default_tags, which the AWS provider adds to every resource that supports tags, so nobody forgets the ownership tags. Second, bucket settings such as versioning and public-access blocking are separate resources in version 4 and later of the AWS provider, each pointing at the bucket by its ID. That is a provider design choice, and it is why reading the provider documentation beats guessing. Third, the random suffix exists because S3 bucket names are global across every AWS account in the world, and churn-artifacts is certainly taken.

The workflow is the one you already know: terraform init (which now downloads the AWS provider, a much bigger binary), terraform plan, read the plan, terraform apply, and finally terraform destroy.

What changes and what doesn't

Compared with the local project, you changed the provider and added credentials. The commands, the plan symbols, the state file, variables, for_each and the dependency graph are all the same. The state file now holds the bucket's name and ARN, and for databases it would hold connection details and sometimes passwords, which is why the state rules earlier in this guide matter more from here on.

Destroy is also more serious. terraform destroy on this configuration fails if the bucket still contains objects, because S3 refuses to delete a non-empty bucket. That refusal is a feature. The AWS provider has a force_destroy argument that empties the bucket first, and you should not set it on anything holding data you care about.

The everyday commands, grouped by what you are doing

You have now used most of the commands a beginner needs. Here they are in one place, grouped by the question you are trying to answer. Every one accepts -help.

Two of those rows replace older commands you will still see in blog posts. terraform taint marked a resource to be rebuilt on the next apply; today you pass -replace to plan or apply, which shows you the replacement in the plan before it happens. terraform refresh updated state from reality without a review step; today terraform apply -refresh-only shows you what it would record and asks first. Both old commands still exist, but the new forms are safer.

A safe daily rhythm is short enough to memorise: edit, fmt, validate, plan, read, apply. The cheapest commands come first because they fail fastest.

1. EditChange the .tf files, never the real infrastructure.
2. fmt and validateSeconds, offline. They catch syntax errors and typos before any API call.
3. planRead the summary line first, then every - and -/+.
4. applyIdeally the saved plan you just read, so what runs is what you reviewed.
5. CommitThe .tf files and the lock file, never the state.

Configuration: versions, the lock file and your repository

A Terraform project that works on your laptop has to work identically for a teammate and a pipeline. Three things make that happen, and all three should be in place before your first commit.

Pin versions with constraints. required_version pins the Terraform CLI and required_providers pins each provider. Use the pessimistic operator on both: ~> 1.16 for Terraform, and ~> 6.0 or similar for a provider. The operators are =, !=, >, >=, <, <= and ~>. The rule for ~> is that only the rightmost number you wrote may increase: ~> 1.16 allows 1.17 and 1.20 but not 2.0, while ~> 1.16.0 allows 1.16.9 but not 1.17. Leaving a provider unpinned means the next init -upgrade may bring a new major version with breaking changes.

Commit the lock file. Constraints say what is acceptable. .terraform.lock.hcl says what was actually chosen, with checksums of the provider packages. With the lock file in Git, terraform init on another machine installs exactly the same versions and refuses a package whose checksum does not match. You change it on purpose with terraform init -upgrade, and review the diff like any other code.

Ignore the right files. Add this .gitignore before the first commit:

# Local provider cache, recreated by terraform init
.terraform/

# State and its backups: may contain secrets
*.tfstate
*.tfstate.*

# Saved plans: binary, may contain secrets
tfplan
*.tfplan

# Crash logs
crash.log
crash.*.log

# Personal overrides
override.tf
override.tf.json
*_override.tf
*_override.tf.json

Notice what is not ignored: .terraform.lock.hcl, and your .tfvars files. Whether to commit .tfvars depends on what is in them. Environment settings such as region and instance sizes belong in Git, because they are part of the reviewed configuration. Anything secret does not belong in a .tfvars file at all.

Environment variables Terraform reads are worth knowing because pipelines use them:

Common errors, and how to read them

Terraform's error messages are long, and that is good news: they almost always say what is wrong, where, and often what to do. Read them in a fixed order. The first line after Error: is the category. The on main.tf line 12 line is the location, with the offending line quoted and underlined. The paragraph after that is the explanation. Beginners scroll to the bottom looking for the answer, when the answer is in the first three lines.

Error: Reference to undeclared input variable

  on main.tf line 12, in resource "local_file" "readme":
  12:   filename = "${path.module}/out/${var.projct}.txt"

An input variable with the name "projct" has not been declared. Did you mean "project"?

These are the ones you will meet in your first weeks:

When the message itself isn't enough, turn on logging for one run and read what the provider actually asked the API:

TF_LOG=DEBUG TF_LOG_PATH=./tf.log terraform plan

The log is long and noisy. Search it for error and for the resource address. Delete it afterwards, because debug logs can contain secrets.

Putting it all together

This project uses everything above to lay out a local workspace for an ML project: a folder per data stage with a README, a JSON config file a training script could read, and outputs a script can consume. It runs offline, and every line maps to a section of this guide.

terraform {
  required_version = "~> 1.16"

  required_providers {
    local = {
      source  = "hashicorp/local"
      version = "~> 2.9"
    }
    random = {
      source  = "hashicorp/random"
      version = "~> 3.9"
    }
  }
}

variable "project" {
  type        = string
  description = "Short name of the ML project."

  validation {
    condition     = can(regex("^[a-z][a-z0-9-]{2,20}$", var.project))
    error_message = "project must be 3-21 lowercase letters, digits or hyphens."
  }
}

variable "environment" {
  type        = string
  description = "dev, staging or prod."
  default     = "dev"

  validation {
    condition     = contains(["dev", "staging", "prod"], var.environment)
    error_message = "environment must be dev, staging or prod."
  }
}

variable "data_folders" {
  type        = set(string)
  description = "Data stages that each get their own folder."
  default     = ["raw", "features", "models"]
}

resource "random_pet" "run" {
  length = 2
}

locals {
  name_prefix = "${var.project}-${var.environment}"
  root        = "${path.module}/workspace/${local.name_prefix}"
}

resource "local_file" "stage_readme" {
  for_each = var.data_folders

  filename = "${local.root}/${each.key}/README.md"
  content  = <<-EOT
    # ${each.key}

    Stage folder for ${local.name_prefix}.
    Managed by Terraform. Do not edit by hand.
  EOT
}

resource "local_file" "config" {
  filename = "${local.root}/config.json"
  content = jsonencode({
    project     = var.project
    environment = var.environment
    run_name    = random_pet.run.id
    paths       = { for stage in var.data_folders : stage => "${local.root}/${stage}" }
  })
}

output "workspace_root" {
  description = "Where the workspace was created."
  value       = local.root
}

output "config_path" {
  description = "Path to the generated config, for training scripts."
  value       = local_file.config.filename
}

project     = "churn"
environment = "dev"

terraform init
terraform fmt -check
terraform validate
terraform plan -var-file=dev.tfvars -out=tfplan
terraform apply tfplan
cat "$(terraform output -raw config_path)"
terraform destroy -var-file=dev.tfvars

What you can now do, and what comes next

You can explain what Terraform is and why declarative infrastructure beats clicking and scripting, install it correctly, write a configuration with providers, resources, variables, locals and outputs, run the init, plan and apply cycle, and read a plan closely enough to catch a replacement or a destroy before it happens. You know what state is, why it holds secrets, and why it never goes in Git. You can make many resources with for_each, read existing things with data sources, let references order your resources, pin versions with a committed lock file, and go from an error message to its fix.

Mid-level builds directly on this page: remote backends with locking so a team can share state, modules for reusing configuration, separate environments with separate state, importing resources that were created by hand, refactoring with moved blocks, terraform test, and running plan and apply in CI with pull-request review. Senior covers owning Terraform for an organisation: state architecture and blast radius, secrets that never touch state, policy as code, supply-chain controls, upgrades, and when to reach for something else.

Terraform usually creates the platform that other tools run on. The natural next guides are Kubernetes, to run workloads on the cluster Terraform builds, and then Helm and Argo CD for delivering applications onto it. If you haven't containerised anything yet, the Docker guide comes first.

Sources

• Terraform documentation
• Install Terraform
• Terraform releases
• Terraform v1.16 changelog
• Terraform v1.x compatibility promises
• CLI commands overview
• terraform plan command
• Resource block reference
• The for_each meta-argument
• The lifecycle meta-argument
• Manage sensitive data
• Dependency lock file
• S3 backend
• hashicorp/local provider
• hashicorp/random provider
• hashicorp/aws provider

Part one of three. A twenty-five-minute review of the Beginner Detailed track, followed by the questions interviewers actually ask about Terraform. Mid-level reviews this plus its own material, and Senior reviews all three. Every answer below is written to be said out loud, so practise saying them rather than reading them.

The thirty-second answer

Terraform is a declarative infrastructure-as-code tool: you describe the infrastructure you want in HCL files, and Terraform compares that description with its state and with the real world, shows you a plan of what will change, and then makes those changes through provider plugins that call each platform's API. Because the description lives in Git, every infrastructure change can be reviewed, repeated and rolled forward like application code.

Then add the line that shows you have used it for real: "the plan is the review step, and the state file is the thing you protect, because it maps every block to a real object and it holds secrets in plain text."

If the interviewer asks what came before, keep it to one sentence each. Clicking in a cloud console is fast once and impossible to reproduce or review. Imperative scripts are repeatable but have to check what already exists before every step, and they cannot show you what they will change before they change it. Terraform moves that checking into the tool.

The model

The single most useful sentence in any Terraform interview is that a plan compares three things: the configuration, the state and the real infrastructure. When a candidate explains a surprising plan by naming which of the three changed, the interviewer knows they have debugged Terraform rather than read about it.

Declarative, idempotent, and what that implies

Declarative means you describe the result, not the steps. Idempotent means applying the same configuration twice leaves things exactly as the first apply did; the second run reports "No changes". Interviewers like to push on the implication people miss: in a declarative tool, deleting a block is an instruction to destroy the resource. The state still lists it, the configuration no longer does, so the plan proposes a destroy. That is correct behaviour, and it is why tidying a Terraform file is a real change that deserves a plan review.

The second follow-up is "is Terraform configuration management?" The answer is no. Terraform mainly creates and connects infrastructure (networks, machines, buckets, permissions). Ansible, Chef and Puppet configure what runs inside machines. In container setups, the second job mostly becomes "run an image", which is why Terraform plus Kubernetes is such a common pairing.

The commands to have ready

Know the order for a normal change without thinking about it: init (once, or after dependency changes), fmt, validate, plan -out=tfplan, read the plan, apply tfplan. Mentioning the saved plan is what separates someone who has run Terraform in a pipeline from someone who has only run it on a laptop.

Reading a plan

Interviewers often show a plan snippet and ask what will happen. Read it in the same order every time: the summary line, then each resource's comment line, then the attribute marked # forces replacement.

The trap they are testing is that Plan: 1 to add, 0 to change, 1 to destroy after a one-line edit is a replacement, not an update. Some attributes cannot change on an existing object (a bucket's name, a machine's availability zone, a database's engine), so the provider forces destroy-and-recreate. For stateful resources that means data loss, and the right answer is "I would stop, find the forcing attribute, and change the approach before applying".

State, in the detail they expect

State exists because a configuration cannot tell which of forty real buckets a block refers to. State records the mapping and the attributes, and it lets Terraform notice deletions from the configuration.

Four rules to state crisply:

1. State may contain secrets in plain text. Any attribute a provider returns is stored. sensitive = true only redacts terminal output.
2. Never commit state to Git. Secrets, and conflicting copies across branches.
3. Never edit it by hand. Use the state commands, or configuration blocks at Mid-level.
4. Teams need a remote backend with locking. Otherwise two applies can run at once and one overwrites the other.

If asked "what happens if you lose the state file?", the answer is that the infrastructure keeps running but Terraform forgets it manages it. The next plan would try to create everything again, and creation would fail on names that already exist or, worse, succeed and duplicate things. Recovery means importing resources back into a new state, which is why remote backends with versioning matter.

Variables, locals, outputs

Think of a configuration as a function. Variables are the parameters, set from outside. Locals are computed intermediate values that cannot be set from outside. Outputs are the return values, printed after apply and read by scripts or by a calling module.

Variable precedence, lowest to highest, is a favourite question: TF_VAR_ environment variables, then terraform.tfvars, then terraform.tfvars.json, then *.auto.tfvars files in alphabetical order, then -var and -var-file on the command line in the order given. The short version: "the command line wins, environment variables lose".

count versus for_each

count makes N instances addressed by index, [0], [1]. for_each makes one instance per key of a map or set of strings, addressed by key, ["raw"]. The reason to prefer for_each for named things is stability: remove an item from the middle of a count list and every later index shifts, so Terraform updates or replaces everything after the gap. With for_each, only the removed key is destroyed. count is still right for "N identical things" and as an on/off switch, count = var.enabled ? 1 : 0.

The error they may quote: for_each rejects a list, so wrap it in toset(). And for_each keys must be known at plan time, so keys built from attributes that only exist after apply fail with "cannot be determined until apply".

Versions and the lock file

required_version constrains the CLI and required_providers constrains each provider, usually with the pessimistic operator: ~> 6.0 means 6.0 or newer but below 7.0. The lock file records the exact versions chosen and their checksums, and it is committed so every machine installs the same providers. terraform init -upgrade moves versions within the constraints, and that diff is reviewed. .terraform/ is a cache and is never committed.

A licensing question comes up more often since 2023. Terraform has been under the Business Source License since version 1.6. OpenTofu is the open-source fork under the Linux Foundation. It shares Terraform's language and its early history, but it is a separate project, so don't describe the two as interchangeable for newer features. HCP Terraform is the new name for Terraform Cloud.

Dependencies and ordering

Expect "how does Terraform know what to create first?" The answer is references. Whenever one block uses a value from another, such as a bucket policy that mentions aws_s3_bucket.artifacts.id, Terraform records an edge in a dependency graph. It creates resources in an order that respects every edge, destroys them in the reverse order, and runs anything without an edge between them in parallel, ten operations at a time by default. The order of blocks in files, and the order of files in a directory, mean nothing.

depends_on exists for the rare case where one thing must wait for another but no value flows between them, such as an application that needs a permission attached before it starts, where nothing in its arguments mentions that permission. A strong answer adds that depends_on should be rare: a reference documents why the dependency exists, and an unnecessary depends_on on a data source delays its read until apply, which fills the plan with (known after apply) values that reviewers can't check.

A cycle, where A refers to B and B refers to A, is an error (Error: Cycle:), and the fix is to break one of the references. On AWS, the classic example is two security groups that allow traffic from each other, solved by moving the rules into separate rule resources.

Terraform in an ML team

Interviewers for MLOps roles often ask where Terraform fits in the machine-learning stack, and a concrete answer lands better than a list of clouds. Terraform builds the platform that training and serving run on: object storage for raw data, features and model artifacts, with encryption and versioning set the same way in every environment; the Kubernetes cluster and its GPU node pools; managed ML services such as SageMaker, Vertex AI or Azure ML workspaces; the IAM roles that let a training job read one bucket and nothing else; and the network that keeps a model endpoint private. It usually stops at the platform boundary. Deploying the model itself onto Kubernetes is typically the job of Helm or a GitOps tool such as Argo CD, and training runs are scheduled by an orchestrator, not by terraform apply.

Two practical points show experience. First, GPU capacity is expensive, so being able to create a node pool from a file and destroy it after a run is a cost control as much as a convenience. Second, for employers in the Gulf and Egypt, the region is frequently a data-residency decision, so it belongs in a reviewed variable rather than in someone's memory.

The traps, and why they share one cause

Nearly every beginner incident comes from forgetting that Terraform acts on the difference between three things, and that it trusts the configuration over everything else.

Common interview questions

1. What is Terraform, and why use it instead of the console or scripts?A declarative infrastructure-as-code tool. You describe the end state, and it computes and shows a plan before making API calls through providers. Compared with the console, you get review, history and reproducibility. Compared with scripts, you get idempotence and a preview of every change without writing existence checks yourself.
2. What is a provider?A separately downloaded plugin that translates Terraform's generic create, read, update and delete operations into a platform's API calls. Terraform core knows nothing about AWS or Kubernetes; the provider does. It is declared in required_providers with a source such as hashicorp/aws and a version constraint.
3. What does terraform init do?Initialises the working directory: configures the backend, downloads providers and modules into .terraform/, and creates or checks the dependency lock file. You rerun it after cloning, adding a provider, changing a version constraint or changing the backend.
4. What is the difference between plan and apply?plan computes the changes and changes nothing. apply executes them. apply without a plan file computes a fresh plan and prompts, which may differ from the one you reviewed, so in automation you save with plan -out=tfplan and apply that file.
5. What is state, and why does Terraform need it?A JSON record mapping each resource address to its real object ID and last known attributes. Terraform needs it to know which real object a block refers to, to detect resources removed from the configuration, and to track dependencies and outputs. It is sensitive, and in a team it lives in a locked remote backend.
6. Where should state be stored in a team, and why?In a remote backend such as an S3 bucket, Azure storage, Google Cloud Storage or HCP Terraform, with encryption, versioning and locking. Locking prevents two concurrent applies from corrupting it, and a shared location means everyone works from the same truth.
7. What is drift, and how does Terraform handle it?A difference between state and reality, usually from manual changes. Terraform refreshes at the start of each plan, sees the difference, and proposes to restore the configured values. If the manual change was right, you put it in the configuration. apply -refresh-only accepts reality into state without changing infrastructure.
8. What does -/+ mean in a plan?The resource must be replaced: destroyed and recreated, because a changed attribute cannot be updated in place. The attribute is marked # forces replacement. On a stateful resource that is data loss, so it is the line to stop at.
9. What is the difference between a resource and a data source?A resource is managed: Terraform creates, updates and destroys it. A data source is read-only: it looks up something that already exists, such as an account ID or a network another team built, and never changes it.
10. How does Terraform decide the order of operations?From references. Every reference from one block to another becomes an edge in a dependency graph. Terraform creates in an order that respects the edges, destroys in reverse, and runs independent operations in parallel. depends_on adds an edge when no value flows between two resources, and should be rare.
11. count or for_each?for_each for distinct named things, because instances are keyed by name and removing one doesn't shift the others. count for identical, interchangeable copies or a conditional on/off.
12. How do you pass different values per environment?Input variables with a .tfvars file per environment, chosen explicitly with -var-file. Mention that one configuration with one state is one copy of the infrastructure, so separate environments also need separate state, which is a Mid-level topic.
13. Does sensitive = true protect a secret?Only from display. The value is redacted in plan and output but still written in plain text to state and to saved plan files. Real protection comes from protecting state, and from keeping secrets out of it entirely with newer features covered at Senior.
14. What goes into Git?The .tf files, non-secret .tfvars, and .terraform.lock.hcl. Not state, not saved plans, not .terraform/, not crash logs.
15. What happens if an apply fails halfway?There is no rollback. Resources created before the failure stay and are recorded in state, and a resource that failed during creation is marked tainted so the next apply replaces it. You fix the cause and apply again.
16. How do you force Terraform to recreate one resource?terraform apply -replace=ADDRESS. The older taint command still exists, but -replace shows the replacement in the plan before it happens, so it is the current recommended form.
17. What is the lock file for?It records the exact provider versions and package checksums chosen by init, so every machine and pipeline installs identical providers and rejects tampered packages. Constraints say what is allowed; the lock file says what was chosen.
18. Terraform or OpenTofu?They share the language and early history. Terraform moved to the Business Source License in version 1.6, and OpenTofu is the open-source fork under the Linux Foundation. For most beginner work they behave the same. Newer features have diverged, so a team picks one and pins it.
19. A plan shows changes you didn't make. What do you do?Don't apply. First check the directory and that terraform state list shows what you expect. Then read which resources changed: ~ updates on untouched resources are drift from manual changes or new provider defaults. Find out who changed what and why, then either put the change into the configuration or let Terraform revert it on purpose.
20. What is terraform validate not able to catch?Anything that needs the real API: a bucket name already taken, a missing permission, a quota limit, an instance type not offered in a region. It checks syntax, references and types offline. Those other errors only appear during plan or apply.

Sixty-second self-test

• Give the thirty-second answer without notes, including why state matters.
• Name the three inputs a plan compares.
• Say what init creates, and which of those you commit.
• Read -/+ and # forces replacement aloud and say what you would do next.
• Give the variable precedence order, lowest to highest.
• Explain why for_each beats count for named resources, with an example.
• Say why sensitive = true is not a secret-management strategy.
• Explain how to apply exactly the plan you reviewed.
• Describe what happens after a failed apply, and after a lost state file.
• Replace one resource the modern way, and name the command it replaces.

Part one of three. Most beginner Terraform trouble comes from three misunderstandings: you forget that the configuration always wins over reality, you forget that state is the only link between your files and the real objects, or you run the right command in the wrong place. Start with the table, then read the longer explanations, the habits and the practice cards.

Common errors at this level

The wrong directory is the most expensive mistake

Terraform acts on the directory you run it in, with the state configured for that directory. It has no idea which project you meant. Beginners with several folders open (a dev copy, a prod copy, an old experiment) run apply in the wrong one, and the plan dutifully shows what that folder's configuration would do to that folder's state.

The symptom is a plan that makes no sense for the change you just made: resources you have never heard of, a destroy count you didn't expect, or "no changes" after an edit you know you saved. The fix is a habit, not a command. Look at the prompt before every plan, and read the resource names in the plan before every apply. When scripts are involved, use terraform -chdir=path/to/project plan so the directory is written into the command rather than depending on where the shell happens to be.

A related trap: a new folder that hasn't been init-ed yet, or a folder where somebody deleted terraform.tfstate, will plan to create everything from scratch. If that "everything" already exists in your cloud account, the apply either fails on duplicate names or, for resources without unique names, silently creates duplicates. If a plan wants to create things you are sure already exist, stop and find out where the state went.

Configuration beats reality, every time

The second most common surprise is a teammate's manual fix disappearing. Someone raises a limit in the cloud console during an incident. The next terraform apply, for a completely unrelated change, sets it back, because the configuration says the old value and Terraform's job is to make reality match the configuration.

The rule to adopt is that anything Terraform manages is changed only through Terraform. If an emergency forces a manual change, the follow-up task is to put the same change in the .tf file and apply it, so the next plan shows no changes for that resource. And whenever a plan shows ~ updates you didn't write, read them: they are drift, and somebody somewhere made a change you should know about before you undo it.

Reading plans properly

A plan is text, and it is tempting to scroll to the bottom and type yes. Build a reading routine instead, because it takes twenty seconds and it is the only safety net you have.

1. Summary line first"Plan: 2 to add, 1 to change, 0 to destroy". Does it match the size of the edit you made? Any destroy should have a reason you can name.
2. Every destroy and replaceSearch the plan for will be destroyed and must be replaced. For a replacement, find # forces replacement to see which attribute caused it.
3. Every update you didn't writeA ~ on a resource you didn't touch is drift or a changed default. Understand it before you overwrite it.
4. OutputsChanged outputs are what scripts and other configurations consume. A changed bucket name ripples outwards.
5. Apply what you readterraform plan -out=tfplan, then terraform apply tfplan, so a late change by a teammate can't slip in between.

-auto-approve has a place in automation, after a saved plan has been reviewed by a human. On your own laptop, against anything that matters, the prompt is doing you a favour.

State hygiene from day one

State is where beginner mistakes turn into outages or leaks. Five small habits prevent almost all of it.

Write the .gitignore before the first commit, so *.tfstate, *.tfstate.*, .terraform/ and saved plan files can never be added by accident. If state has already been committed, removing it in a later commit doesn't remove it from Git history, so any secrets it held should be treated as exposed and rotated.

Don't open terraform.tfstate in an editor to "fix" something. Read state with terraform state list, terraform state show ADDRESS and terraform show. Changing it is a Mid-level topic with dedicated commands and configuration blocks.

Don't copy state files between folders or machines to "share" them. Two copies diverge the moment either person applies. The moment a second person needs to run Terraform on something, it is time for a remote backend with locking, and that is the first thing Mid-level teaches.

Don't delete terraform.tfstate to "start fresh" while the resources still exist. Terraform forgets them, but they keep running and billing. Run terraform destroy first if you really want them gone.

Keep terraform.tfstate.backup in mind as a last-resort safety net for local state, not as history. It holds exactly one previous version.

Versions: pin, lock, commit

Unpinned versions cause the "it worked yesterday" class of problems. A provider without a version constraint is upgraded to whatever is newest the next time anyone runs init -upgrade, including a new major version with renamed or removed arguments. The fix takes three lines per provider: a source, a version = "~> X.Y" constraint, and a committed lock file.

Pin the CLI too, with required_version = "~> 1.16", because state written by a newer Terraform may not be readable by an older one. If two people on a team run different minor versions, the newer one can upgrade the state format and leave the other unable to plan. A version manager such as tenv or tfenv (community tools) makes switching painless when you work on several projects.

When you do upgrade a provider, do it on purpose: change the constraint if needed, run terraform init -upgrade, commit the lock file change in its own pull request, and read the plan for surprises. Provider major versions ship an upgrade guide on the Terraform Registry, and reading it takes less time than debugging the first Unsupported argument error.

Secrets: what beginners get wrong

Three misunderstandings leak most secrets. First, a password written as a literal in main.tf is in Git forever, even after you delete the line. Second, sensitive = true on a variable or output only hides it on screen; the value is still in state and in saved plans. Third, terraform.tfvars holding a password gets committed along with the harmless variables.

At beginner level, the practical rules are simple. Keep secrets out of .tf and committed .tfvars files. If you must pass one, use TF_VAR_name in your shell or a .tfvars file that is listed in .gitignore. Mark the variable sensitive = true so it doesn't end up in terminal logs. And treat state as the secret it is. Mid-level and Senior cover the proper tools: remote state with encryption and tight access, secret managers read at apply time, and newer Terraform features that keep values out of state entirely.

Read error messages from the top

Terraform's errors look long, which makes beginners scroll straight to the bottom or paste the whole thing into a search engine. The useful part is at the top, and it has a fixed shape. The line after Error: names the category, such as Reference to undeclared input variable or Unsupported argument. The on main.tf line 12, in resource "local_file" "readme": line gives the exact file, line and block, and the next line quotes your code with the problem underlined. The paragraph after that explains the rule you broke, and often ends with a suggestion such as Did you mean "project"?.

Read those three parts in order and most errors fix themselves in under a minute. When they don't, the problem is usually outside your configuration, in credentials, networking, permissions or quotas, and then the provider's message (often quoted from the cloud API) is the thing to read carefully. For one stubborn run, set TF_LOG=DEBUG and TF_LOG_PATH=./tf.log, search the log for error and the resource address, and delete the log afterwards because it can contain secrets.

One more thing worth knowing early: an error during apply is not a rollback. Anything created before the failure stays created and is recorded in state. Fix the cause and run apply again; the next plan continues from where the last one stopped.

The habits that pay off most

Small things worth knowing

terraform console is the fastest way to test an expression. Instead of editing, planning and reading, type jsonencode({ a = 1 }) or cidrsubnet("10.0.0.0/16", 8, 2) and see the answer immediately. It reads values from your real state, so random_pet.project.id works too.

terraform plan -help beats a web search. It always matches the version you have installed, while blog posts may describe flags that were renamed or deprecated years ago, such as taint instead of -replace, or terraform refresh instead of apply -refresh-only.

Quote addresses with brackets in your shell. terraform state show 'local_file.folder_readme["raw"]' works; without quotes, zsh and bash try to interpret the brackets and quotes themselves.

Outputs are an interface. Use terraform output -raw NAME in scripts, not grep on the apply log. The -raw form prints the value with no quotes and no trailing formatting.

Name resources for humans. The local name in resource "aws_s3_bucket" "artifacts" appears in every plan, error and state listing. artifacts tells a reviewer something; bucket1 does not. Changing it later looks like a destroy and create unless you use the refactoring tools from Mid-level, so choose well the first time.

Tag everything that supports tags. On AWS, default_tags in the provider block adds ownership and environment tags to every resource. Tags are how a finance team finds out who owns the GPU machine nobody turned off.

Choose the region on purpose. For teams in the Gulf and Egypt, data-residency rules often decide the region before latency does. Make the region a variable with a description, so the choice is visible in review. Remember that newer regions such as AWS me-central-1 must be enabled on the account first.

Practice cards

Short, self-contained exercises. Each takes a few minutes, runs offline with the local and random providers, and ends in something you can check on screen.

1. Prove idempotenceApply the first project, then run terraform apply twice more. Read "No changes" and confirm the random name did not change.
2. Watch drift get revertedEdit a Terraform-managed file by hand, plan, and find the refresh lines and the planned fix. Apply and check the content is back.
3. Cause a replacement on purposeChange a random pet's length. Find -/+, # forces replacement, and every resource that follows because it refers to the pet.
4. Delete a blockRemove a resource block and plan. Read the destroy. Put it back and plan again to see "No changes".
5. count versus for_eachBuild three files both ways, remove the middle item from the list, and count the actions in each plan.
6. Precedence puzzleSet one variable with TF_VAR_, terraform.tfvars and -var at once, predict the winner, then check with terraform console.
7. Stale planSave a plan, apply a different change, then try to apply the saved plan and read the refusal.
8. Lock file checkChange a provider constraint to a version the lock file doesn't satisfy, run init, read the error, then restore it.

A checklist before every apply

• I am in the right directory, and terraform state list shows the resources I expect.
• terraform fmt -check and terraform validate pass.
• I saved the plan with -out and read the summary line.
• Every destroy and every replacement has a reason I can say out loud.
• Any ~ update I didn't write has been explained.
• No secrets are in .tf files or committed .tfvars.
• .gitignore covers state, .terraform/ and plan files, and the lock file is committed.
• I will apply the saved plan file, not a fresh one.

A starter setup to copy

A folder you can clone for every new experiment. It encodes the habits above so you don't have to remember them.

.terraform/
*.tfstate
*.tfstate.*
tfplan
*.tfplan
crash.log
crash.*.log

terraform {
  required_version = "~> 1.16"

  required_providers {
    random = {
      source  = "hashicorp/random"
      version = "~> 3.9"
    }
    local = {
      source  = "hashicorp/local"
      version = "~> 2.9"
    }
  }
}

variable "project" {
  type        = string
  description = "Short project name, used in every resource name."
}

variable "environment" {
  type        = string
  description = "dev, staging or prod."
  default     = "dev"

  validation {
    condition     = contains(["dev", "staging", "prod"], var.environment)
    error_message = "environment must be dev, staging or prod."
  }
}

terraform init
terraform fmt -check && terraform validate
terraform plan -var-file=dev.tfvars -out=tfplan
terraform apply tfplan

Six details carry the lesson of this page: a .gitignore written first, pinned CLI and provider versions with a committed lock file, typed and validated variables, an explicit variables file per environment, a saved plan that you read, and applying exactly that file.

Mid-level tips take each of these further: remote state with locking, splitting environments into separate state, modules and their versioning, importing and refactoring without destroying anything, and running plan and apply safely from CI.

This is part two of three. Beginner got you to the point where terraform plan and terraform apply do what you expect on a single configuration that you run from your laptop. This level is about what changes when Terraform is part of a team's daily work: state that several people and a pipeline share, modules other teams consume, refactors that must not destroy a production database, and a CI job that is trusted to apply. Most expensive Terraform incidents are not syntax errors. They happen when someone's picture of what Terraform will do is slightly wrong, and the plan output was too long for anyone to read carefully. The aim here is to make your picture exact.

Where this picks up

Everything here is written against Terraform 1.16, the current minor release at the time of writing. Where a feature arrived recently the release is named, because your team's pinned version may be older. Each section starts with the problem it solves and ends with a Try it you can run against a sandbox account in a few minutes.

How a plan is actually computed

Beginner described plan as "Terraform shows you what it will change". That is true, but it hides three separate inputs and four separate phases. Once you can name them, you can predict almost every surprising plan before you run it.

Terraform compares three things. The configuration is the .tf files in the root module plus every module it calls. The state is Terraform's record of which real objects it manages, keyed by resource address. The real infrastructure is what the cloud API reports right now. A plan is the difference between the configuration and reality, as seen through the state.

The phases run in this order:

Load. Terraform reads every .tf file in the directory (file names and order do not matter; they are merged), and resolves module calls from the .terraform/modules cache that init filled.

Graph. Every resource, data source, variable, local, output, module call and provider configuration becomes a node. Every reference between them becomes an edge: if aws_iam_role_policy_attachment.x mentions aws_iam_role.worker.name, there is an edge from the role to the attachment. That is the whole ordering mechanism. Terraform does not run blocks in file order, and it does not guess. It walks the graph, and any two nodes with no path between them may run at the same time, up to -parallelism concurrent operations (the default is 10). depends_on adds an edge the references did not express, which is why you need it only for hidden dependencies, such as an IAM policy that must exist before a service can use it but is never referenced by that service.

Refresh. For every resource in state, the provider calls the cloud API and reads the object's current attributes. This is where drift (changes made outside Terraform, say in the console) is discovered, and it is also where most of the plan's wall-clock time goes on a large configuration, because it is at least one API call per managed object.

Diff. For each resource, Terraform hands the provider the prior state, the refreshed state and the new configuration, and the provider returns the planned new state. The provider, not Terraform core, decides whether a change can be made in place or requires replacement. An attribute the provider marks as requiring replacement (in the AWS provider, changing the availability_zone of a subnet, for example) turns an update into destroy-and-create. That is why the plan annotates the specific attribute with # forces replacement: read that line before anything else.

Plan. The result is a list of actions per resource address: create (+), update in place (~), destroy (-), replace (-/+ or +/-), and read (<=) for data sources that can only be read during apply.

Unknown values

Some attributes are only known after the object exists: a generated ID, an assigned IP address, an ARN. During planning, Terraform represents these as unknown, which the plan prints as (known after apply). Unknowns propagate. If a subnet's id is unknown, then anything built from it is unknown too.

Most of the time unknowns are harmless, because the apply phase fills them in. They become a problem in exactly one kind of place: anything that decides how many resources exist. count and for_each must be fully known during planning, because Terraform has to decide the set of resource addresses before it can plan changes to them. If you derive for_each keys from an attribute that does not exist yet, planning stops with the error covered in the for_each section below.

The provider is a separate program

A provider is its own executable that Terraform starts as a child process and talks to over gRPC. That has two practical consequences. First, most "Terraform bugs" are provider bugs: the message Provider produced inconsistent final plan or Provider produced inconsistent result after apply means the provider returned a value during apply that did not match what it promised during plan. The fix is a provider upgrade or an upstream bug report, not a change to your configuration. Second, the provider version decides the schema: which arguments exist, which force replacement, and what the defaults are. Two people running the same configuration with different provider versions can get different plans. That is what the dependency lock file, covered later, prevents.

terraform plan -out=tfplan                     # save exactly what was planned
terraform show tfplan                          # re-read it as text
terraform show -json tfplan | jq -r '.resource_changes[] | select(.change.actions != ["no-op"]) | "\(.change.actions | join(",")) \(.address)"'

The last command prints one line per changing resource, such as delete,create aws_instance.web. On a plan with hundreds of lines of attribute diffs, that summary is the first thing to read, because it tells you where the destroys are.

State in depth

State is the part of Terraform that surprises people most, because it is invisible when things go well. It deserves a precise description.

The state file is a JSON document. Its top level holds a version (the state format), the terraform_version that last wrote it, a serial that increases on every write, a lineage (a unique ID created when the state was first made), the root module outputs, and a resources list. Each resource entry records its address, its provider, and one or more instances with every attribute the provider returned, plus the dependencies Terraform recorded for ordering destroys.

Three of those fields matter operationally.

serial and lineage protect you from overwriting newer state with older state. When you run terraform state push, Terraform refuses to replace a state with a different lineage, or with a lower serial, unless you force it. That check is the only thing standing between a stale local copy and a production state, which is why forcing it is almost never right.

The attribute values are complete, and that includes secrets. A database password you passed to aws_db_instance, a generated private key from the tls provider, an access key created by aws_iam_access_key: all of them are stored in plaintext in state. Marking a variable sensitive = true hides it from the terminal and the plan output. It does not remove it from state. So the state file is a secret, and its storage must be treated like one.

terraform_version records who wrote it last. Terraform can read state written by an older 1.x version, but downgrading is not guaranteed, because the format can move forward. If one engineer upgrades their CLI and applies to a shared state, everyone else may be forced to upgrade too. That is why teams pin required_version and upgrade together.

Remote backends, properly

A backend decides where state lives and how it is locked. The default local backend writes terraform.tfstate in the working directory, which fails as soon as two people or a pipeline are involved. For AWS, the current recommended form is the S3 backend with S3-native locking, which became generally available in Terraform 1.11:

terraform {
  backend "s3" {
    bucket       = "acme-tfstate-me-central-1"
    key          = "platform/network/prod/terraform.tfstate"
    region       = "me-central-1"
    encrypt      = true
    use_lockfile = true
  }
}

With use_lockfile = true, Terraform writes a lock object next to the state (<key>.tflock) using a conditional write, and deletes it when the operation finishes. Before 1.11, locking required a separate DynamoDB table set with dynamodb_table. That argument is now deprecated and prints a Warning: Deprecated parameter telling you to use use_lockfile instead. During a migration you can set both, so old and new clients respect the same lock, and then drop DynamoDB once everyone is on 1.11 or later.

The bucket must exist before init, because the backend does not create it. Create it once, by hand or from a small separate bootstrap configuration, with versioning enabled (so every previous state is recoverable), default encryption with a KMS key, public access blocked, and a bucket policy that only admits the roles that run Terraform. The identity running Terraform needs s3:ListBucket on the bucket, s3:GetObject and s3:PutObject on the state key, and s3:GetObject, s3:PutObject and s3:DeleteObject on the .tflock key.

The region matters for more than latency. If your employer is subject to data-residency rules, such as Saudi Arabia's PDPL or sector rules from a Gulf central bank, the state file is a copy of your infrastructure's configuration, and sometimes of its credentials, so it belongs in the same jurisdiction as the data it describes. AWS me-central-1 (UAE) and me-south-1 (Bahrain), Google Cloud's Doha and Dammam regions, and Azure's UAE and Qatar regions are the usual choices. Oracle Cloud users in Saudi Arabia can use the oci backend added in 1.12, which stores state in OCI Object Storage.

Changing a backend is an init operation. If you edit the backend block, the next command fails with Error: Backend configuration changed. Run terraform init -migrate-state to copy the existing state to the new location, or terraform init -reconfigure to start fresh and ignore the old one. The first is almost always the one you want. The second on an existing configuration makes Terraform believe it manages nothing, so the next plan proposes to create everything again.

Locking and what to do when it sticks

Every operation that might write state takes the lock first. If someone else holds it, you see:

Error: Error acquiring the state lock

Error message: ...
Lock Info:
  ID:        6f0c3c7e-5b0a-2a1e-9d3b-1f8e2c4a7b90
  Path:      acme-tfstate-me-central-1/platform/network/prod/terraform.tfstate
  Operation: OperationTypeApply
  Who:       runner@ci-7f9c
  Version:   1.16.4
  Created:   2026-09-29 08:14:03.51 +0000 UTC

Read the Who, Operation and Created lines before doing anything. An apply started by CI two minutes ago is running: wait for it. In pipelines, add -lock-timeout=5m so a job waits for the lock instead of failing immediately. A lock from a laptop yesterday, where the process was killed, is stale. Only then, and only after confirming no process is still running, release it with the ID from the message:

terraform force-unlock 6f0c3c7e-5b0a-2a1e-9d3b-1f8e2c4a7b90

Never use -lock=false against shared state. It does not make the other operation go away; it lets two writers race, and the loser's changes to state are overwritten while the real resources stay changed.

Reading and editing state directly

These commands inspect state without changing it:

terraform state list                                  # every address
terraform state list 'module.network'                 # filter by prefix
terraform state show 'aws_subnet.private["a"]'        # one instance, human-readable
terraform state show -json 'aws_subnet.private["a"]'  # machine-readable (1.16+)
terraform state pull > backup-$(date +%F).tfstate     # a local copy, before any surgery

The editing commands (state mv, state rm, state replace-provider and state push) still exist, but for renames, moves and forgetting resources the configuration blocks covered in the refactoring section are preferred, because they are reviewed in a pull request and applied through the normal plan. Keep the manual commands for recovery work, and always state pull a backup first.

Splitting state by blast radius

A single state for "all of our infrastructure" works for a month. Then plans take eight minutes, every change to a DNS record also refreshes the production database, two teams queue behind one lock, and a mistake in one file can propose destroying resources that have nothing to do with it. The fix is to split the configuration into several root modules, each with its own state, along two axes.

By environment. Development, staging and production never share a state. A plan against staging should not be able to see, let alone change, a production resource.

By layer and rate of change. Networking changes rarely and breaks everything when it goes wrong. Data stores change rarely and hold data you cannot recreate. Compute and application infrastructure change weekly. Each layer gets its own state, so the fast-moving parts do not put the slow, dangerous parts at risk every time someone applies.

A common repository layout that follows this looks like:

infra/
├── modules/                 # reusable modules, versioned
│   ├── network/
│   ├── artifact-store/
│   └── eks-cluster/
└── live/                    # root modules: one directory = one state
    ├── staging/
    │   ├── network/
    │   ├── data/
    │   └── compute/
    └── prod/
        ├── network/
        ├── data/
        └── compute/

Each directory under live/ has its own backend key, such as platform/network/prod/terraform.tfstate, and can be planned independently with terraform -chdir=live/prod/network plan.

Passing values between states

Once states are split, the compute layer needs the network layer's subnet IDs. There are two ways to get them.

The first is the terraform_remote_state data source, which reads another state's root outputs directly:

data "terraform_remote_state" "network" {
  backend = "s3"
  config = {
    bucket = "acme-tfstate-me-central-1"
    key    = "platform/network/prod/terraform.tfstate"
    region = "me-central-1"
  }
}

locals {
  private_subnet_ids = data.terraform_remote_state.network.outputs.private_subnet_ids
}

It works, but it has a cost: the reading configuration needs read access to the whole of the other state, including any secrets in it, and the two configurations are now coupled through output names.

The second is to look the values up from the provider, using tags or names as the contract:

data "aws_vpc" "main" {
  tags = { Name = "prod-main" }
}

data "aws_subnets" "private" {
  filter {
    name   = "vpc-id"
    values = [data.aws_vpc.main.id]
  }
  tags = { Tier = "private" }
}

This needs only read permission on the cloud API, and it keeps working if the network layer is later managed by a different tool. Parameter stores (AWS SSM Parameter Store, for example) are a third option: the producing state writes a parameter and the consuming state reads it. Prefer provider lookups or a parameter store for anything crossing a team boundary, and keep terraform_remote_state for states the same team owns.

Modules other teams can consume

At Beginner level a module was a way to avoid repeating yourself. At work, a module is an interface: other people call it, pin a version of it, and expect it not to change underneath them. Designing that interface well is most of the job.

A module's interface is its variable blocks (inputs), its output blocks, and the providers it needs. Everything else is implementation, which you should be able to change without breaking callers.

Inputs with real types and validation

A loosely typed input such as variable "config" { type = any } pushes errors from plan time into apply time, or into a confusing message from deep inside the module. Give inputs precise types. For groups of settings, use an object type with optional() attributes and defaults, so callers set only what they care about:

variable "name" {
  type        = string
  description = "Bucket name prefix. Lowercase letters, digits and hyphens."

  validation {
    condition     = can(regex("^[a-z0-9-]{3,40}$", var.name))
    error_message = "name must be 3-40 characters of lowercase letters, digits and hyphens."
  }
}

variable "lifecycle_rules" {
  type = object({
    noncurrent_days = optional(number, 30)
    archive_after   = optional(number)       # null means "never archive"
  })
  default     = {}
  description = "Retention for old artifact versions."
}

variable "kms_key_arn" {
  type        = string
  default     = null
  description = "Customer-managed KMS key. When null, S3-managed keys are used."
}

The validation block runs during planning, before any API call, and prints your error_message next to the offending line of the caller's code. That turns "the bucket creation failed with an InvalidBucketName from AWS" into a clear message in the caller's own file.

Terraform 1.15 added two small features that help maintain module interfaces over time. A deprecated = "message" attribute on a variable or output block warns callers who still use it, so you can phase an input out over a few releases instead of breaking it in one. And output blocks can now declare a type, which documents and enforces what you promise to return.

Outputs as a contract

Output what callers need to connect other things to your module: IDs, ARNs, names, endpoints. Don't output entire resource objects (value = aws_s3_bucket.this), because that makes every attribute of the provider's schema part of your interface, and a provider upgrade can then break your callers.

output "bucket_name" {
  value       = aws_s3_bucket.this.bucket
  description = "Name of the artifact bucket."
}

output "bucket_arn" {
  value       = aws_s3_bucket.this.arn
  description = "ARN, for IAM policies that grant access."
}

Providers belong to the caller

A reusable module declares which providers it needs, with a minimum version, in required_providers. It should not contain provider "aws" { region = ... } blocks. The root module configures providers and passes them in. That way the same module can be called for two regions or two accounts:

provider "aws" {
  region = "me-central-1"
}

provider "aws" {
  alias  = "dr"
  region = "eu-central-1"
}

module "artifacts" {
  source = "../../../modules/artifact-store"
  name   = "acme-ml-artifacts"
}

module "artifacts_dr" {
  source    = "../../../modules/artifact-store"
  name      = "acme-ml-artifacts-dr"
  providers = { aws = aws.dr }
}

A module that contains its own provider block also cannot be used with count or for_each, and removing a module call that has its own provider configuration fails, because Terraform needs that configuration to destroy the resources. Both problems disappear when providers come from the caller.

Versioning and sources

Local paths (./modules/x) are fine inside one repository, where the module and its callers change in the same pull request. As soon as another repository calls your module, it needs a version. Two common ways:

# Git, pinned to a tag. The double slash separates the repo from the subdirectory.
module "vpc" {
  source = "git::https://github.com/acme/terraform-modules.git//network?ref=v2.3.1"
}

# A private registry (HCP Terraform, Terraform Enterprise, or another registry).
module "eks" {
  source  = "app.terraform.io/acme/eks-cluster/aws"
  version = "~> 4.2"
}

The version argument works only with registry sources. For Git, the ref is the version, and it can be a tag or a full commit SHA. A tag is readable; a SHA cannot be moved by anyone who can push tags. Until 1.15, source and version had to be literal strings. Since 1.15 they may use variables and locals, but the value must be known when terraform init runs, because that is when modules are downloaded.

Use semantic versioning for module releases and mean it: a patch release fixes behaviour, a minor release adds optional inputs, and a major release is the only place where an input is removed or where upgrading can cause a resource to be replaced. Callers then pin with ~> 4.2 and take minor updates without fear.

count, for_each and keys you can know

Both meta-arguments create several instances from one block, but they address those instances differently, and that difference decides what happens when the list changes.

count addresses instances by position: aws_subnet.private[0], [1], [2]. Remove the first element of the input list and every later element shifts down one index. Terraform sees that index 0 now has different arguments (probably forcing replacement), index 1 likewise, and index 2 no longer exists. Removing one subnet replaces all three.

for_each addresses instances by key: aws_subnet.private["a"], ["b"], ["c"]. Remove "a" and only "a" is destroyed. That is why for_each over a map is the default choice for anything that has an identity, and count is kept for "zero or one" toggles:

variable "azs" {
  type    = map(number)                 # az suffix -> subnet index
  default = { a = 0, b = 1, c = 2 }
}

resource "aws_subnet" "private" {
  for_each          = var.azs
  vpc_id            = aws_vpc.main.id
  availability_zone = "me-central-1${each.key}"
  cidr_block        = cidrsubnet(aws_vpc.main.cidr_block, 4, each.value)
  tags              = { Name = "private-${each.key}", Tier = "private" }
}

resource "aws_flow_log" "vpc" {
  count                = var.enable_flow_logs ? 1 : 0   # a toggle, not a collection
  vpc_id               = aws_vpc.main.id
  traffic_type         = "ALL"
  log_destination      = aws_s3_bucket.logs.arn
  log_destination_type = "s3"
}

for_each accepts a map or a set of strings. A list is rejected with the "for_each" argument must be a map, or set of strings, and you have provided a value of type tuple (or list of string when the value comes from a variable typed list(string)); wrap it in toset(). Sensitive values are also rejected as for_each arguments, because the keys appear in resource addresses, which are printed everywhere.

The plan-time rule

The keys of for_each and the value of count must be known during planning. This fails:

resource "aws_route53_record" "service" {
  for_each = toset([for lb in aws_lb.service : lb.dns_name])   # dns_name is unknown until the LB exists
  # ...
}

with:

Error: Invalid for_each argument

  on dns.tf line 2, in resource "aws_route53_record" "service":
   2:   for_each = toset([for lb in aws_lb.service : lb.dns_name])

The "for_each" set includes values derived from resource attributes that
cannot be determined until apply, and so Terraform cannot determine the full
set of keys that will identify the instances of this resource.

The fix is almost always to build the keys from things you wrote yourself (variable keys, static names) and put the unknown attributes in the values:

resource "aws_route53_record" "service" {
  for_each = aws_lb.service                     # keys come from the LB's own for_each keys: known
  zone_id  = var.zone_id
  name     = "${each.key}.internal.acme.example"
  type     = "CNAME"
  ttl      = 300
  records  = [each.value.dns_name]              # unknown values are fine here
}

Running terraform apply -target=aws_lb.service first and then a full apply also works, but it is a workaround for a one-off, not a design. A configuration that always needs two applies will eventually be run once by someone who does not know that.

Switching from count to for_each without replacing anything

Changing count to for_each changes every address, from [0] to ["a"], so Terraform would destroy and recreate everything. Tell it the instances moved:

moved {
  from = aws_subnet.private[0]
  to   = aws_subnet.private["a"]
}
moved {
  from = aws_subnet.private[1]
  to   = aws_subnet.private["b"]
}
moved {
  from = aws_subnet.private[2]
  to   = aws_subnet.private["c"]
}

The next plan shows each subnet has moved to its new address with no other changes. That is the signal the refactor is safe.

Lifecycle rules and custom conditions

The lifecycle block changes how Terraform treats one resource when its configuration changes. Each argument exists for a specific failure you will meet sooner or later.

create_before_destroy = true. By default a replacement destroys the old object before creating the new one, which means downtime and, for anything with a unique name, is sometimes the only order that works. With this flag Terraform creates the replacement first, repoints dependants, then destroys the old one. Use it for launch templates, certificates, and anything behind a load balancer. The new object needs a name that can coexist with the old one, which is why these resources usually use name_prefix rather than name. The flag also propagates: Terraform applies the same ordering to the resources this one depends on, because replacing them destroy-first would break the create-first guarantee.

prevent_destroy = true. Any plan that would destroy or replace this resource fails with Error: Instance cannot be destroyed. Put it on state stores, databases and artifact buckets. It is a guard against mistakes in configuration, not an access control: it does not stop someone deleting the block and the flag together, and it lives in configuration rather than state.

ignore_changes = [tags["LastScanned"], desired_count]. Terraform will not try to revert differences in these attributes. Use it when something else legitimately owns an attribute, such as an autoscaler managing an ECS service's desired_count or a node group's desired_size or a security scanner stamping a tag. ignore_changes = all exists, but it turns Terraform into a create-only tool for that resource, and it is almost always a sign that the resource belongs to someone else.

replace_triggered_by = [terraform_data.model_version]. Replace this resource whenever the referenced resource or attribute changes, even though none of this resource's own arguments did. A common MLOps use is recreating an inference endpoint configuration when the model version changes:

resource "terraform_data" "model_version" {
  input = var.model_version
}

resource "aws_sagemaker_endpoint_configuration" "serve" {
  name_prefix = "fraud-model-"
  # production_variants { ... }

  lifecycle {
    create_before_destroy = true
    replace_triggered_by  = [terraform_data.model_version]
  }
}

destroy = false is new in Terraform 1.16. When a plan would destroy a resource with this setting (a replacement, or terraform destroy), Terraform removes it from state instead of deleting the real object. The documentation adds a caveat worth repeating: apart from create_before_destroy, lifecycle arguments are not stored in state, so if you delete the whole resource block, the setting goes with it and Terraform destroys the object. For the common case of "stop managing this resource", use the removed block in the next section.

Preconditions, postconditions and check blocks

Validation on variables checks inputs. Two other mechanisms check assumptions about the world:

data "aws_ami" "gpu" {
  most_recent = true
  owners      = ["amazon"]
  filter {
    name   = "name"
    values = ["Deep Learning Base OSS Nvidia Driver GPU AMI (Ubuntu 22.04)*"]
  }
}

resource "aws_instance" "trainer" {
  ami           = data.aws_ami.gpu.id
  instance_type = var.instance_type

  lifecycle {
    precondition {
      condition     = data.aws_ami.gpu.architecture == "x86_64"
      error_message = "The selected AMI is not x86_64; g-family instances need an x86_64 image."
    }
    postcondition {
      condition     = self.root_block_device[0].volume_size >= 100
      error_message = "Training hosts need at least 100 GB of root disk for datasets and checkpoints."
    }
  }
}

A precondition is evaluated before the resource is planned and blocks the plan if it fails. A postcondition is evaluated after the provider has planned or applied it, and can refer to the resource itself as self. Both fail the run, which makes them contracts: use them for things that must be true, and put a sentence in error_message that tells the reader what to fix.

A check block is the non-blocking version. It can contain its own scoped data source, and its assertions produce warnings rather than errors:

check "mlflow_health" {
  data "http" "mlflow" {
    url = "https://mlflow.internal.acme.example/health"
  }

  assert {
    condition     = data.http.mlflow.status_code == 200
    error_message = "MLflow tracking server did not return 200 on /health."
  }
}

Checks run at the end of every plan and apply, so they catch drift in the things your infrastructure is supposed to deliver, not just in its attributes. HCP Terraform's health assessments also run them on a schedule.

Refactoring without destroying anything

Configurations get reorganised: resources move into modules, get better names, and occasionally get handed to another team or adopted from something created by hand. In old Terraform, all of that meant terraform state mv, state rm and terraform import commands run by one person against live state, with no review and no record. Terraform now has configuration blocks for all three, which go through a pull request and show up in the plan like any other change.

moved: renames and moves

moved {
  from = aws_s3_bucket.artifacts
  to   = module.artifacts.aws_s3_bucket.this
}

On the next plan, Terraform updates the state address and reports # aws_s3_bucket.artifacts has moved to module.artifacts.aws_s3_bucket.this. If the resource's arguments also differ between the old and new blocks, you see those changes as well, which is how you catch a move that is not quite a move. Since 1.8 you can also move between compatible resource types when the provider supports it.

Keep moved blocks for as long as anyone might still be applying an older state. Inside a published module, that means at least until the next major version. Removing a moved block too early means a caller who skipped a release gets a destroy-and-create.

removed: stop managing without destroying

removed {
  from = aws_instance.legacy_jupyter

  lifecycle {
    destroy = false
  }
}

Delete the resource block, add this, and the plan says the instance will be removed from state but not destroyed. After the apply, the removed block can be deleted. With destroy = true, or with no lifecycle block, it is simply an explicit, reviewable destroy.

import: adopt what already exists

Someone created the model-artifacts bucket in the console last year. You want Terraform to manage it without recreating it.

import {
  to = aws_s3_bucket.legacy_artifacts
  id = "acme-legacy-artifacts"
}

If you have not written the resource block yet, let Terraform draft it:

terraform plan -generate-config-out=generated.tf

Terraform writes a resource "aws_s3_bucket" "legacy_artifacts" block with every attribute it read, and the plan shows 1 to import, 0 to add, 0 to change, 0 to destroy. The generated file is a starting point, not a finished resource: it includes computed and default values you should delete, and it hard-codes values you would normally reference. Tidy it, run plan again until it shows only the import and no changes, then apply. After that the import block has done its job and can be removed.

Import blocks accept for_each, which makes adopting twenty buckets one block instead of twenty commands, and since 1.12 they accept an identity object instead of id for providers that support resource identity. Since 1.16, import blocks are also allowed inside child modules, so a module can adopt the resources it is meant to manage. For finding what exists in the first place, 1.14 added terraform query, which runs list blocks from *.tfquery.hcl files against a provider and can emit import and resource blocks with -generate-config-out. It is covered in the Senior level, since it is mostly useful for large adoption projects.

Environments: workspaces or separate roots

Every team eventually needs the same infrastructure in development, staging and production. Terraform offers two mechanisms, and the choice matters more than it seems.

CLI workspaces give one configuration several named states in the same backend. terraform workspace new staging creates one, terraform workspace select -or-create prod switches (creating it if needed), and terraform.workspace is the current name inside the configuration. With the S3 backend, non-default workspaces are stored under an env:/ prefix in the same bucket.

locals {
  sizes = {
    dev     = "t3.medium"
    staging = "g5.xlarge"
    prod    = "g5.2xlarge"
  }
  instance_type = local.sizes[terraform.workspace]
}

That is convenient for near-identical copies, such as a short-lived copy of a stack per feature branch. It is weak as environment isolation, and HashiCorp's own documentation says so. Every workspace shares one backend, so the credentials that can read dev state can read prod state. Nothing in the directory tells you which workspace is selected, so terraform apply in the wrong one is a single forgotten command away. And differences between environments end up as terraform.workspace conditionals scattered through the code.

Separate root modules (the live/staging/... and live/prod/... directories from earlier) give each environment its own backend configuration, its own provider credentials, and its own variable files. They call the same versioned modules, so the logic is shared, but everything that differs is explicit:

module "cluster" {
  source  = "app.terraform.io/acme/eks-cluster/aws"
  version = "4.2.0"                    # prod pins exactly; staging tracks ~> 4.2

  name              = "prod"
  gpu_instance_type = "g5.2xlarge"
  gpu_max_size      = 8
}

This layout also lets production lag deliberately behind staging on module versions, which is how a module upgrade gets tested before it reaches production.

HCP Terraform workspaces are a different thing that happens to share the name: each one is a separate unit with its own state, variables, runs and permissions, much closer to a separate root than to a CLI workspace. Tools such as Terragrunt exist mainly to remove the repetition that separate roots bring (the backend block and provider setup copied into every directory). They are worth a look once you have more than a handful of roots, but the layout itself is the important decision.

Providers, versions and the lock file

A plan is only reproducible if everyone runs the same Terraform version and the same provider versions. Three mechanisms control that, and they do different jobs.

required_version constrains the Terraform CLI. With required_version = "~> 1.16", any 1.x release from 1.16 onwards is accepted, and a colleague on 1.14 gets Error: Unsupported Terraform Core version instead of silently writing state in an older format. Version managers such as tenv or tfenv read a .terraform-version file in the repository and switch automatically, which removes the "works on my machine" class of plan differences.

required_providers version constraints say which provider versions the configuration is compatible with. In a root module, pin to a major version with ~>:

terraform {
  required_version = "~> 1.16"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"            # any 6.x, never 7.0
    }
    random = {
      source  = "hashicorp/random"
      version = "~> 3.6"
    }
  }
}

In a reusable module, state only a minimum (>= 6.0), because a module that pins tightly makes it impossible for callers to upgrade until you release. Let the root module decide the exact range.

The dependency lock file, .terraform.lock.hcl, records the exact version init selected for each provider, plus checksums of the provider packages. The next init, on any machine, installs that exact version and verifies the checksum, even if a newer version matching the constraint has been released. Commit it. Without it, two engineers running init a week apart can get different provider versions, and a provider release you never reviewed reaches production the day it is published.

Upgrading is then a deliberate act:

terraform init -upgrade                   # pick the newest versions the constraints allow, rewrite the lock file
terraform plan                            # review the effect of the new provider on your resources
git diff .terraform.lock.hcl              # the upgrade is visible in review

The lock file across operating systems

The checksums in the lock file are per platform. If everyone on the team uses an Apple Silicon Mac, the lock file records only darwin_arm64 hashes, and the Linux CI runner fails with:

Error: Failed to install provider

Error while installing hashicorp/aws v6.0.0: the current package for
registry.terraform.io/hashicorp/aws 6.0.0 doesn't match any of the checksums
previously recorded in the dependency lock file

Record every platform your team and CI use when you create or upgrade the lock file:

terraform providers lock \
  -platform=linux_amd64 \
  -platform=linux_arm64 \
  -platform=darwin_arm64 \
  -platform=windows_amd64

In CI, run terraform init -lockfile=readonly. Init then fails instead of modifying the lock file, so a missing entry is caught in review rather than papered over by the pipeline. Automated dependency tools such as Renovate and Dependabot understand Terraform and open pull requests that update both the constraint and the lock file.

Download once, not every run

Each init downloads providers into .terraform/providers. The AWS provider alone is several hundred megabytes, so a repository with ten root modules downloads it ten times. Set a shared cache directory and Terraform links providers from it:

export TF_PLUGIN_CACHE_DIR="$HOME/.terraform.d/plugin-cache"
mkdir -p "$TF_PLUGIN_CACHE_DIR"

In CI, persist that directory between runs with the CI system's cache feature. Organisations that restrict internet access from build runners can go further with terraform providers mirror and a provider_installation block in the CLI configuration file, which the Senior level covers.

Secrets, sensitive values and ephemeral values

Terraform touches secrets constantly: database passwords, API tokens for providers, keys it generates. The mid-level mistake is believing that sensitive = true protects them. It does not, and the difference matters.

sensitive = true on a variable or output redacts the value in plan and apply output, where it shows as (sensitive value). The value is still written to state and to saved plan files in plaintext. terraform output -raw db_password prints it. It is a display setting.

Ephemeral values (Terraform 1.10 and later) are never written to state or to a plan. There are three ways to produce one. A variable with ephemeral = true accepts a value for this run only. An ephemeral resource block, such as ephemeral "aws_secretsmanager_secret_version", reads a secret fresh during each phase. And a module output can be marked ephemeral = true to pass such a value upward.

Ephemeral values can only be used in places that do not persist: provider configuration, locals, other ephemeral blocks, and write-only arguments. Write-only arguments (Terraform 1.11 and later, and they need provider support) are resource arguments, conventionally suffixed _wo, whose value is sent to the API and then forgotten:

ephemeral "aws_secretsmanager_secret_version" "db" {
  secret_id = "prod/mlflow/db"
}

resource "aws_db_instance" "mlflow" {
  identifier          = "mlflow-prod"
  engine              = "postgres"
  instance_class      = "db.t4g.medium"
  allocated_storage   = 50
  username            = "mlflow"
  password_wo         = jsondecode(ephemeral.aws_secretsmanager_secret_version.db.secret_string)["password"]
  password_wo_version = 1              # bump to push a new password; write-only values are not diffed
  skip_final_snapshot = false
}

Because Terraform never stores a write-only value, it cannot tell when it has changed. The companion _wo_version argument is how you tell it: increase the number and the new value is sent.

Two further rules keep secrets out of places they should not be. Never put secrets in committed .tfvars files. Supply them from the environment (TF_VAR_db_password), from a secret store read by an ephemeral resource, or from HCP Terraform sensitive variables. And treat saved plan files like state: a tfplan created with -out contains every non-ephemeral value in cleartext, so it should never become a long-lived, widely readable CI artifact. The Vault guide covers generating short-lived database credentials that Terraform can read ephemerally.

Testing Terraform code

Testing infrastructure code has layers, and each catches a different class of mistake at a different cost. Run the cheap ones on every commit and the expensive ones where they earn it.

Static checks. terraform fmt -check -recursive fails on unformatted files. terraform validate checks syntax, references and types without touching the cloud; run terraform init -backend=false first in CI so it needs no backend credentials. Since 1.15, validate also checks the backend block. TFLint adds provider-aware rules, such as an instance type that does not exist. Scanners such as Checkov or Trivy look for insecure settings: a public bucket, an unencrypted volume, a security group open to 0.0.0.0/0.

terraform test is Terraform's built-in test framework. Test files end in .tftest.hcl and usually live in a tests/ directory next to the module. Each file contains run blocks, which execute a plan or an apply of the module and then evaluate assert blocks against the result:

mock_provider "aws" {}

variables {
  name = "unit-test-artifacts"
}

run "versioning_is_enabled" {
  command = plan

  assert {
    condition     = aws_s3_bucket_versioning.this.versioning_configuration[0].status == "Enabled"
    error_message = "Artifact buckets must have versioning enabled."
  }
}

run "public_access_is_blocked" {
  command = plan

  assert {
    condition     = aws_s3_bucket_public_access_block.this.block_public_acls && aws_s3_bucket_public_access_block.this.restrict_public_buckets
    error_message = "Artifact buckets must block public access."
  }
}

run "rejects_bad_names" {
  command = plan

  variables {
    name = "Not_A_Valid_Name"
  }

  expect_failures = [var.name]
}

command = plan makes a run fast and free: nothing is created. mock_provider "aws" {} replaces the real provider with one that returns generated values, so the test needs no credentials at all and can run on every pull request. expect_failures turns a validation rule into a test: the run passes only if that variable's validation fails. When a mocked value needs to be specific (an ARN in a particular format, say), override_resource and override_data blocks set it.

terraform test                                 # every file under tests/
terraform test -filter=tests/artifact_store.tftest.hcl -verbose
terraform test -junit-xml=test-results.xml     # for CI test reporting

Runs without command = plan default to apply: they create real resources, assert against them, and destroy everything at the end in reverse order. Keep those for a small number of integration tests against a dedicated test account, run on merge or nightly rather than on every commit. Go-based Terratest is the older alternative for integration testing, and it is still common when tests need to do things like send HTTP requests to what was created.

Terraform in a CI/CD pipeline

Running Terraform from laptops works until two people apply at once, someone applies an unmerged branch, or nobody can say who changed production last Tuesday. The standard answer is a pipeline with two halves: plan on every pull request so reviewers see exactly what will change, and apply only after merge, from the pipeline, with credentials no human holds day to day.

Four details make this safe rather than merely automated.

Non-interactive mode. Set TF_IN_AUTOMATION=1, which trims output that assumes a human at a terminal, and pass -input=false (or set TF_INPUT=0), so a missing variable fails the job instead of hanging it waiting for keyboard input.

Short-lived credentials through OIDC. Instead of storing an AWS access key as a CI secret, the CI system issues a signed identity token for each job, and the cloud exchanges it for temporary credentials for a specific role. GitHub Actions, GitLab CI and HCP Terraform all support this with AWS, Azure and Google Cloud. Use two roles: one that can read everything and write only the state lock, for plans on pull requests, and one with write access, which only the main branch can assume.

Apply what was reviewed. terraform apply with no plan file makes a fresh plan, which can differ from the one reviewers approved if anything changed in between. Save the plan with -out=tfplan and apply that exact file. If state changed after the plan was saved, Terraform refuses with Error: Saved plan is stale, which is the behaviour you want: re-plan and review again.

Exit codes that mean something. terraform plan -detailed-exitcode exits 0 when there are no changes, 1 on error, and 2 when there are changes. That lets a pipeline skip the apply stage when nothing would change, and it is the basis of drift detection below.

A GitHub Actions workflow that puts this together for one root module:

name: terraform prod/data

on:
  pull_request:
    paths: ["live/prod/data/**", "modules/**"]
  push:
    branches: [main]
    paths: ["live/prod/data/**", "modules/**"]

permissions:
  id-token: write          # needed for OIDC
  contents: read
  pull-requests: write     # only needed if you add a step that comments the plan

env:
  TF_IN_AUTOMATION: "1"
  TF_INPUT: "0"
  TF_PLUGIN_CACHE_DIR: ${{ github.workspace }}/.terraform-plugin-cache
  WORKDIR: live/prod/data

concurrency:
  group: terraform-prod-data   # one run at a time per state
  cancel-in-progress: false

jobs:
  terraform:
    runs-on: ubuntu-latest
    environment: ${{ github.event_name == 'push' && 'production' || 'plan' }}
    steps:
      - uses: actions/checkout@v7

      - uses: hashicorp/setup-terraform@v4
        with:
          terraform_version: "1.16.4"
          terraform_wrapper: false

      - uses: aws-actions/configure-aws-credentials@v6
        with:
          role-to-assume: ${{ github.event_name == 'push' && secrets.TF_APPLY_ROLE || secrets.TF_PLAN_ROLE }}
          aws-region: me-central-1

      - uses: actions/cache@v6
        with:
          path: .terraform-plugin-cache
          key: tf-plugins-${{ hashFiles('live/prod/data/.terraform.lock.hcl') }}

      - run: mkdir -p "$TF_PLUGIN_CACHE_DIR"
      - run: terraform fmt -check -recursive
      - run: terraform -chdir="$WORKDIR" init -lockfile=readonly
      - run: terraform -chdir="$WORKDIR" validate
      - run: terraform -chdir="$WORKDIR" plan -lock-timeout=5m -out=tfplan

      - name: Plan summary
        run: terraform -chdir="$WORKDIR" show -no-color tfplan > plan.txt

      - name: Apply
        if: github.event_name == 'push'
        run: terraform -chdir="$WORKDIR" apply -lock-timeout=5m tfplan

The environment: production line is where the human approval lives: in GitHub, an environment can require a named reviewer before any job using it starts, and the apply role's trust policy can require that the token came from that environment. The concurrency group ensures two merges queue rather than race for the lock. And because plan and apply happen in the same job on main, the saved plan never becomes a stored artifact with secrets in it. The GitHub Actions guide covers environments, OIDC trust and reusable workflows in more depth.

Drift detection

Someone changes a security group in the console during an incident and forgets to tell anyone. The next unrelated apply either reverts their fix or fails. A scheduled plan finds drift before that happens:

terraform plan -detailed-exitcode -lock=false -input=false > /dev/null
case $? in
  0) echo "no drift" ;;
  2) echo "drift or unapplied changes detected"; exit 2 ;;
  *) echo "plan failed"; exit 1 ;;
esac

A read-only plan is one of the few places where -lock=false is reasonable, because it writes nothing. Run it nightly for each root module and alert on exit code 2. To bring state in line with a legitimate manual change without touching the resource, use terraform apply -refresh-only, which updates state from reality and shows you what it is recording. The older terraform refresh command does the same thing without showing you anything first, and is deprecated.

Other tools implement the same pipeline shape. Atlantis runs plan and apply from pull request comments on your own infrastructure. HCP Terraform runs them remotely with a web UI, policy checks and an audit trail, and is priced by resources under management. The mechanics you just learned are the same underneath all of them.

Terraform, Kubernetes and the MLOps stack

In an MLOps platform, Terraform usually provisions the foundations: the network, a Kubernetes cluster with CPU and GPU node groups, object storage for datasets and model artifacts, a database for the experiment tracker, and IAM roles that let workloads reach those stores. The question is where Terraform should stop.

The pattern that holds up is a hand-off. Terraform creates the cluster and the cloud resources around it. It may also install the small set of cluster-level components that must exist before anything else can be deployed, such as the Argo CD controller itself. Everything that runs on the cluster (model servers, pipelines, the MLflow deployment) is delivered by a GitOps tool such as Argo CD, usually packaged with Helm.

There are three reasons for the split. The first is the chicken-and-egg problem: the kubernetes and helm providers need the cluster's endpoint and credentials to be configured, and if the cluster is created in the same state, those values are unknown during the first plan. The second is rate of change: applications deploy many times a day, and a Terraform run that also refreshes the whole cluster is too slow and too risky for that. The third is ownership: Kubernetes controllers constantly modify the objects they manage, and Terraform sees those modifications as drift.

When a separate state does need to configure the cluster (creating namespaces and service accounts for a team, for instance), have it look the cluster up rather than share state, and get short-lived tokens through the cloud CLI:

data "aws_eks_cluster" "main" {
  name = "prod"
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.main.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.main.certificate_authority[0].data)

  exec {
    api_version = "client.authentication.k8s.io/v1beta1"
    command     = "aws"
    args        = ["eks", "get-token", "--cluster-name", data.aws_eks_cluster.main.name]
  }
}

resource "kubernetes_namespace" "ml_serving" {
  metadata {
    name   = "ml-serving"
    labels = { team = "ml-platform" }
  }
}

The exec block runs aws eks get-token whenever the provider needs a token, so nothing long-lived is stored, and the token never enters state.

GPU capacity is where cost goes, so it deserves explicit configuration. Put GPU nodes in their own node group with a taint (so only workloads that tolerate it land there), a minimum size of zero, and ignore_changes on the desired size, because the cluster autoscaler or Karpenter owns that number once the group exists:

resource "aws_eks_node_group" "gpu" {
  cluster_name    = aws_eks_cluster.main.name
  node_group_name = "gpu-g5"
  node_role_arn   = aws_iam_role.node.arn
  subnet_ids      = [for s in aws_subnet.private : s.id]
  instance_types  = ["g5.xlarge"]
  ami_type        = "AL2023_x86_64_NVIDIA"

  scaling_config {
    min_size     = 0
    desired_size = 0
    max_size     = 4
  }

  taint {
    key    = "nvidia.com/gpu"
    value  = "present"
    effect = "NO_SCHEDULE"
  }

  lifecycle {
    ignore_changes = [scaling_config[0].desired_size]
  }
}

The Kubernetes guide picks up from here: taints and tolerations, GPU scheduling, and how model servers such as KServe request accelerators.

Keeping large configurations fast

A plan that takes ten minutes is not only annoying. People stop reading plans they had to wait for, and they start reaching for -target to get their work done, which is how state and configuration drift apart. The cost of a plan is dominated by refresh, which makes at least one API call per resource, so the levers are mostly about doing fewer of those calls.

Split the state. This is the largest lever by far, and it is the one described earlier. A state of 150 resources refreshes in seconds; one of 3,000 takes minutes and runs into API rate limits.

Tune parallelism, in both directions. -parallelism=n controls how many graph operations run at once, defaulting to 10. Raising it helps when a plan is slow because of many independent resources with slow APIs. Lowering it helps when the plan fails with throttling errors from the cloud API. Set it for a whole pipeline with TF_CLI_ARGS_plan="-parallelism=20". Since 1.14, Terraform builds respect a container's CPU quota, so a runner limited to one CPU may run fewer operations concurrently than it did on older versions.

Skip refresh while iterating, never before applying. terraform plan -refresh=false plans against state as recorded, with no API reads. It is fast and useful while you are writing a module, but it can miss drift entirely, so it does not belong in the pipeline that applies.

Cache providers and modules. TF_PLUGIN_CACHE_DIR removes provider downloads from every init. Persist it between CI runs.

Avoid -target as a speed tool. -target restricts planning to one resource and its dependencies. It exists for recovering from mistakes, and Terraform prints a warning every time you use it for that reason. A configuration you can only apply with -target is telling you the state should be split.

Terraform 1.17, currently in beta, adds a -minimal-refresh option that refreshes only the resources with proposed changes. It is worth watching, but it is not in the current release.

Debugging beyond the error message

Most Terraform errors are clear once you read the whole message, including the file and line it points to. When they are not, four tools cover nearly everything.

terraform console evaluates expressions against the current configuration and state. It is the fastest way to find out why a for expression produces the wrong shape or what a data source returned:

terraform console
> var.azs
> [for k, s in aws_subnet.private : s.cidr_block]
> cidrsubnet("10.20.0.0/16", 4, 2)
> exit

Since 1.16, terraform console -scope=module.network evaluates expressions as if you were inside that module, so you can inspect its locals and variables directly.

The JSON plan. terraform show -json tfplan produces a structured document with resource_changes (the action and before/after values for each address), prior_state, and configuration. When a plan is long or confusing, filter it with jq rather than scrolling. For example, to see only the attributes that force a replacement:

terraform show -json tfplan | jq '.resource_changes[] | select(.change.actions | index("delete") and index("create")) | {address, replace_paths: .change.replace_paths}'

The dependency graph. When Terraform reports Error: Cycle: aws_security_group.a, aws_security_group.b, two resources reference each other. terraform graph -format=mermaid (1.16) or terraform graph | dot -Tsvg > graph.svg shows the edges. The usual fix for security groups is to move the rules into separate aws_vpc_security_group_ingress_rule resources so neither group references the other.

Logs. TF_LOG sets the log level (TRACE, DEBUG, INFO, WARN, ERROR, or JSON), and TF_LOG_PATH writes it to a file instead of the terminal. TF_LOG_PROVIDER and TF_LOG_CORE set the level separately for providers and for Terraform itself. At DEBUG, provider logs usually include the HTTP requests sent to the cloud API and the responses, which is how you find out which API call failed and why:

TF_LOG_PROVIDER=DEBUG TF_LOG_PATH=./tf-debug.log terraform plan
grep -n -i -E 'error|denied|throttl' tf-debug.log | head

Debug logs can contain credentials and request bodies. Delete them after use and never attach them to a public issue without reading them first.

Putting it all together

The pieces combine into one small, production-shaped project: an ML artifact store for a team training models in the UAE region, built as a tested module, called from a production root with its own state, and applied from CI.

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 6.0"
    }
  }
}

resource "aws_s3_bucket" "this" {
  bucket_prefix = "${var.name}-"

  lifecycle {
    prevent_destroy = true
  }
}

resource "aws_s3_bucket_versioning" "this" {
  bucket = aws_s3_bucket.this.id
  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "this" {
  bucket = aws_s3_bucket.this.id
  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm     = var.kms_key_arn == null ? "AES256" : "aws:kms"
      kms_master_key_id = var.kms_key_arn
    }
  }
}

resource "aws_s3_bucket_public_access_block" "this" {
  bucket                  = aws_s3_bucket.this.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

resource "aws_s3_bucket_lifecycle_configuration" "this" {
  bucket = aws_s3_bucket.this.id
  rule {
    id     = "expire-old-versions"
    status = "Enabled"
    filter {}
    noncurrent_version_expiration {
      noncurrent_days = var.lifecycle_rules.noncurrent_days
    }
  }
}

moved {
  from = aws_s3_bucket.bucket          # the name used in v1 of this module
  to   = aws_s3_bucket.this
}

terraform {
  required_version = "~> 1.16"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
  }
  backend "s3" {}                       # values from backend/prod.hcl at init
}

provider "aws" {
  region = "me-central-1"
  default_tags {
    tags = { ManagedBy = "terraform", Stack = "prod/data" }
  }
}

module "artifacts" {
  source = "git::https://github.com/acme/terraform-modules.git//artifact-store?ref=v2.1.0"
  name   = "acme-ml-artifacts"
  lifecycle_rules = {
    noncurrent_days = 90
  }
}

output "artifact_bucket" {
  value = module.artifacts.bucket_name
}

bucket       = "acme-tfstate-me-central-1"
key          = "platform/data/prod/terraform.tfstate"
region       = "me-central-1"
encrypt      = true
use_lockfile = true

cd live/prod/data
terraform init -backend-config=backend/prod.hcl
terraform providers lock -platform=linux_amd64 -platform=darwin_arm64
terraform plan -out=tfplan
terraform apply tfplan

Each decision in that project maps to a section of this guide:

Where you are now

You can explain how a plan is built from configuration, state and refresh, and predict replacements and unknown values before they surprise you. You keep state remote, locked, encrypted and split by blast radius, and you can recover from a stuck lock without making things worse. You design modules as versioned interfaces, use for_each with keys known at plan time, and refactor through moved, removed and import blocks instead of state surgery. You know the difference between sensitive and ephemeral values, test modules without a cloud account, and run Terraform from a pipeline that applies only what was reviewed.

Senior takes this further, with scale, security and ownership in view. It covers the trust model of providers and state, and how to design IAM so a compromised pipeline is contained. Policy as code comes next, with Sentinel, OPA and the plan JSON. After that come large-scale import with terraform query, state and module layouts for dozens of teams, HCP Terraform and Stacks, and air-gapped provider mirrors. The level ends with upgrade strategy across a whole estate, the incidents that happen only at scale, and where Terraform stops and another tool should take over.

Sources

• Terraform documentation
• Terraform CLI commands
• terraform plan command
• terraform force-unlock command
• terraform query command
• resource block reference
• The lifecycle meta-argument
• The for_each meta-argument
• import block reference
• The terraform_data resource
• Manage sensitive data
• Dependency lock file
• S3 backend
• Tests
• Upgrade guides
• Terraform v1.x compatibility promises
• Terraform v1.16 changelog
• Terraform v1.15 changelog
• Terraform v1.14 changelog
• Terraform v1.11 changelog
• hashicorp/setup-terraform GitHub Action

Part two of three. This is a review of Beginner and Mid-level Terraform for interviews where you are expected to have used Terraform at work, not just finished a tutorial. Mid-level interviews are mostly scenarios: something has gone wrong, or two approaches are on the table, and the interviewer wants to hear how you reason. Read the model first, then the scenarios, then the trade-offs, and finish with the self-test. It takes about forty minutes.

Where this picks up

The model in one screen

Terraform compares the configuration with real infrastructure, using state to know which real object each resource address refers to. A provider plugin reads each object, decides whether a change can happen in place or needs replacement, and Terraform orders the resulting actions by a dependency graph built from references. Independent branches run in parallel, ten at a time by default.

Four facts carry most mid-level answers:

• State holds every attribute in plaintext, including passwords and generated keys. sensitive = true hides values from output, not from state.
• The provider decides replacement. Read # forces replacement in the plan before anything else.
• count and for_each must be known at plan time, because they define resource addresses.
• The lock file pins providers. required_providers says what is allowed; .terraform.lock.hcl says what was chosen, with checksums.

Commands to have ready

terraform init -backend-config=backend/prod.hcl   # partial backend config per environment
terraform init -migrate-state                     # after changing the backend block
terraform init -upgrade                           # take newer providers within constraints
terraform init -lockfile=readonly                 # CI: fail rather than edit the lock file
terraform providers lock -platform=linux_amd64 -platform=darwin_arm64
terraform plan -out=tfplan -lock-timeout=5m
terraform plan -detailed-exitcode                 # 0 none, 1 error, 2 changes
terraform apply tfplan                            # apply exactly what was reviewed
terraform apply -replace='aws_instance.web[0]'    # force one replacement
terraform apply -refresh-only                     # accept drift into state, reviewably
terraform state list
terraform state show 'aws_subnet.private["a"]'
terraform state pull > backup.tfstate            # before any state surgery
terraform force-unlock LOCK_ID
terraform show -json tfplan                       # for jq, policy tools, CI summaries
terraform test -junit-xml=results.xml
terraform console -scope=module.network           # 1.16+

Scenario questions

These are the questions where the interviewer is listening for your order of investigation, not just the fix.

"The plan wants to destroy and recreate our production database. What do you do?"

Stop, and do not apply. Find the attribute marked # forces replacement in the plan, or run terraform show -json tfplan and read replace_paths for that resource. Then work out which of three causes it is. The most common is that a configuration change touched an attribute the provider cannot change in place, such as an identifier or an engine setting. Revert it, or find an in-place path the provider supports. The second is that the address changed: a rename, a move into a module, or a switch from count to for_each. The plan shows a destroy at the old address and a create at the new one, and the fix is a moved block. The third is a provider upgrade that changed a default or a schema. Check the provider's upgrade guide and pin back if needed. Close by mentioning prevention: prevent_destroy = true on stateful resources, so a plan like this fails instead of waiting for someone to approve it.

"terraform apply fails with Error acquiring the state lock. What now?"

Read the lock info before acting. Who, Operation and Created tell you whether it is a live CI run or a crashed session from yesterday. If it is live, wait, and give pipelines -lock-timeout so they queue instead of failing. If it is stale, confirm the process is really gone, which may mean asking the person or checking the CI job, then terraform force-unlock with the lock ID. The weak answer is -lock=false. That lets two writers race on one state, and the last to finish overwrites the other's record while both sets of real changes remain.

"The plan takes twelve minutes. Where do you look first?"

Refresh. Every managed resource costs at least one API call per plan, so time scales with the size of state. Check terraform state list | wc -l. If it is in the thousands, the real fix is splitting the state by environment and layer, which also reduces blast radius and lock contention. Short-term levers are tuning -parallelism (up if APIs are slow but not throttling, down if you see rate-limit errors), caching providers with TF_PLUGIN_CACHE_DIR so init is not part of the problem, and checking whether the runner has enough CPU, since from 1.14 Terraform respects container CPU quotas. -refresh=false is fine while developing, but not in the pipeline that applies, because it hides drift.

"CI fails with a checksum mismatch in the lock file, but it works on my laptop."

The lock file records provider checksums per platform. If it was created on an Apple Silicon Mac, it has only darwin_arm64 hashes, and the Linux runner's package does not match any of them. Run terraform providers lock with every platform the team and CI use, commit the result, and use init -lockfile=readonly in CI so a gap fails in review rather than being silently fixed by the pipeline. A related cause is a provider mirror serving a different package, which the same error message describes.

"Someone changed a security group in the console during an incident. How do you handle it?"

First decide whether the change is correct. If it is, put it into the configuration so the next apply does not revert it. Run a plan and expect no changes; if there is a remaining difference, terraform apply -refresh-only records reality into state after showing you what it will record. If the change was wrong, a normal apply reverts it. Then prevent the next surprise: a scheduled plan -detailed-exitcode per root module that alerts on exit code 2, and a process where emergency console changes are followed by a pull request.

"We need to bring 40 existing S3 buckets under Terraform."

Use import blocks, not 40 terraform import commands, so the adoption is reviewed and repeatable. One import block with for_each over a map of bucket names covers them all. Run terraform plan -generate-config-out=generated.tf to draft the resource blocks (check that your version supports generation for for_each imports, and if not, write one resource block with for_each yourself), then clean the generated code: remove computed and default attributes, and replace hard-coded values with references. Iterate until the plan shows only imports and zero changes, then apply and delete the import blocks. Mention that a bucket in S3 is several resources in the AWS provider (versioning, encryption and lifecycle are separate resource types), so each needs importing too. For discovering what exists at larger scale, Terraform 1.14 added terraform query with list blocks.

"for_each fails with 'cannot be determined until apply'. Fix it."

The keys depend on an attribute that only exists after apply, such as an ID or a DNS name. Keys become resource addresses, so Terraform needs them during planning. Restructure so the keys come from values you control (variable map keys, static names, or another resource's own for_each keys) and keep the unknown attributes in the values. A two-step apply with -target gets you past it once, but a configuration that always needs one will eventually be applied wrongly by someone who did not know.

"The team wants to move this resource into a module without downtime."

Write the module, change the root to call it, and add moved { from = aws_x.y to = module.m.aws_x.y }. The plan should show has moved to and nothing else. If it also shows attribute changes, the module is not configuring the resource identically, and you fix that before applying. If the module is shared, put the moved block inside the module so every caller benefits, and keep it until at least the next major version.

"A pull request plan in CI shows a change nobody made."

Three candidates. The first is drift, which you confirm by comparing the planned change with the real resource. The second is provider behaviour: a provider upgrade in the lock file, or an attribute that the API normalises differently from how you wrote it, such as JSON policy formatting or case. The third is something else owning that attribute, like an autoscaler changing a desired count or a Kubernetes controller editing an object. The fix differs: correct the drift, normalise the value (for policies, jsonencode or the policy document data source), or use ignore_changes for attributes another system legitimately owns.

Trade-off questions

Here the interviewer wants to hear both sides, a default, and the condition that would change your mind.

CLI workspaces or separate root modules for environments? Workspaces share one backend and one set of permissions, and the selected workspace is invisible local state, so applying to the wrong environment is easy. Separate roots give each environment its own backend key, credentials and module version, which is why they are the default for dev, staging and prod. Workspaces are fine for near-identical, short-lived copies, such as one per feature branch.

terraform_remote_state or provider data sources for cross-state values? Remote state is simple and exact, but it grants the reader access to the whole state (secrets included) and couples the two configurations through output names. Data source lookups by tag or name, or a parameter store, need only API read access and survive the producer changing tools. Use remote state within one team; use lookups across teams.

count or for_each? for_each for anything with an identity, because instances are addressed by key and removing one does not shift the others. count for a zero-or-one toggle. Converting later costs a set of moved blocks, so choose for_each early.

Local modules or versioned modules? Local paths keep a module and its callers in one pull request, which is right while one team owns both. Versioned modules (Git tags or a registry) let callers upgrade on their own schedule and let production lag behind staging, which is what you need once other teams depend on it. The cost is a release process.

Terraform or Helm/Argo CD for what runs in Kubernetes? Terraform for the cluster, cloud resources, IAM and a small bootstrap set. GitOps for workloads, because they change daily, controllers constantly modify objects (which Terraform reads as drift), and the cluster's credentials are unknown in the same state that creates it.

Saved plan or plain apply in CI? Saved plan, so what is applied is what was reviewed, and a state change in between produces Saved plan is stale rather than a surprise. The cost is that the plan file contains sensitive values, so plan and apply in the same job, or protect the artifact like state.

ignore_changes or fix the drift? ignore_changes only for attributes another system is supposed to own. Using it to silence drift you do not understand hides real problems, and ignore_changes = all makes Terraform create-only for that resource.

Mock-provider tests or real-apply tests? Mocks are fast, free and need no credentials, so they run on every commit. But they prove only your logic, not that the cloud accepts the configuration. Keep a few real-apply tests per shared module, run on merge or nightly in a test account.

What a strong answer covers that a weak one misses

Strong candidates also state versions when a feature is recent, because the interviewer's team may be on an older one. Examples: use_lockfile for S3 locking is GA from 1.11, write-only arguments from 1.11, import in child modules from 1.16, and variables in module source from 1.15. Saying "on 1.16 you can do this; before that you would need that" shows you have upgraded a real estate.

Common interview questions

1. What is in a state file, and why does it matter?A mapping from each resource address to the real object's ID and every attribute the provider returned, plus outputs, a serial and a lineage. It matters because it is how Terraform knows what it manages, and because it holds secrets in plaintext. Store it remotely, encrypted, versioned and access-controlled.
2. What do serial and lineage do?Lineage identifies a state's history; serial increases with each write. Terraform refuses to push a state with a different lineage or a lower serial without force, which protects against overwriting newer state with a stale copy.
3. How does S3 state locking work today?Since 1.11, use_lockfile = true writes a .tflock object next to the state with a conditional write. The DynamoDB table arguments are deprecated. During migration you can run both so old and new clients honour the same lock.
4. Why can't a backend block use variables?The backend is initialised before variables are evaluated. Use partial configuration: leave values out of the block and pass -backend-config=file.hcl to init.
5. -migrate-state or -reconfigure?-migrate-state copies existing state to the new backend. -reconfigure ignores the old one. Use the second on an existing configuration and Terraform thinks it manages nothing.
6. How does Terraform decide order?From references between blocks, which form a dependency graph. File order is irrelevant. depends_on adds edges for dependencies no expression shows. Independent nodes run concurrently up to -parallelism, which defaults to 10.
7. What does (known after apply) mean and when does it hurt?An unknown value, such as a generated ID. It propagates to anything derived from it. It hurts in count and for_each, which must be known during planning, and in data sources, which get deferred to apply.
8. What does create_before_destroy change?Replacement order: create the new object, repoint dependants, destroy the old one. It avoids downtime but needs names that can coexist, hence name_prefix. It propagates to dependencies.
9. Precondition, postcondition, check block: differences?Preconditions and postconditions live in a resource's lifecycle and fail the run. A postcondition can use self. A check block is standalone, can have its own scoped data source, and only warns.
10. What does replace_triggered_by do?Replaces a resource when a referenced resource or attribute changes, even though its own arguments did not. A terraform_data resource holding a version string is the usual trigger.
11. How do you stop managing a resource without destroying it?A removed block with lifecycle { destroy = false }, applied through the normal plan. terraform state rm still works but leaves no reviewed record.
12. What does the lock file contain and should you commit it?The selected version and package checksums for each provider. Yes, commit it; otherwise each init may pick a different version. Add hashes for every platform with terraform providers lock -platform=….
13. How should a reusable module handle providers?Declare requirements with a minimum version, and no provider blocks. The caller configures providers and passes aliases through providers = { aws = aws.dr }. A module with its own provider block cannot use count or for_each and is hard to remove.
14. How do you test a module?fmt -check and validate on every commit, a linter and a security scanner, then terraform test with command = plan and a mock_provider for logic, and expect_failures for validation rules. A few real-apply tests in a test account for modules others depend on.
15. How do CI credentials work in a good setup?OIDC federation: the CI job gets a signed token and exchanges it for temporary cloud credentials. A read-only role plans pull requests; a write role, assumable only from the protected branch or environment, applies.
16. Why is a pull request plan a security concern?Plan executes provider code and data sources, including external, from the branch. Provisioners wait for apply, but the plan is enough. Anyone who can open a pull request can run code with the plan role, so keep it read-only and do not plan forks automatically.
17. What is the difference between sensitive and ephemeral?Sensitive redacts output but is stored in state and plans. Ephemeral values (1.10+) are never stored, and can only flow into provider configuration, locals, other ephemeral blocks, or write-only arguments (1.11+).
18. What is the modern replacement for terraform taint?terraform apply -replace=ADDR, which shows the replacement in the plan before doing it, rather than marking state and hoping someone reads the next plan.

Final self-test

• Name the three things a plan compares and the phase that costs most time.
• Say who decides whether a change is in place or a replacement, and where the plan tells you.
• Explain why sensitive = true does not protect state, and what does.
• Write the S3 backend block with native locking from memory, and say how to supply environment-specific values.
• Describe what you check before running force-unlock.
• Explain why for_each keys must be known at plan time and how to restructure when they are not.
• Rename a resource, move it into a module, and stop managing another, all without a destroy.
• Argue separate roots over workspaces for production, and name when workspaces are fine.
• Fix a lock-file checksum mismatch in CI and prevent it recurring.
• Describe a CI pipeline that applies only reviewed changes, with the credentials each stage uses.
• Say where Terraform should stop in a Kubernetes platform, and why.

Part two of three. At this level your configurations apply cleanly, so the problems change character. They are no longer "why won't init work" but "why does this plan want to replace the database", "why does CI fail when my laptop doesn't", and "who left the state locked". Start with the error table, then the practices in order of leverage, then the practice cards and the operational recipes you will reach for on a bad day.

Common errors at this level

Cumulative: Beginner's errors still apply, and these are the ones that appear once Terraform is shared by a team and a pipeline.

The practices that pay off most

In rough order of how much trouble each one prevents:

Practice cards

1. Read a plan in thirty secondsSave a plan with -out=tfplan, then run terraform show -json tfplan | jq -r '.resource_changes[] | select(.change.actions != ["no-op"]) | "\(.change.actions | join(",")) \(.address)"'. Do this on every plan for a week until you reach for it by habit.
2. See a secret in stateCreate an SSM parameter from a variable marked sensitive, apply, and grep terraform state pull for the value. Then you will never again say that sensitive values are protected.
3. Survive a lockStart an apply against a remote backend and kill the process mid-apply. Read the lock message from the next run, decide it is stale, and release it with force-unlock.
4. count to for_each, zero replacementsConvert a three-instance count resource to for_each with three moved blocks, and do not apply until the plan shows only moves.
5. Adopt a console resourceCreate a bucket by hand, then import it with an import block and -generate-config-out. Edit the generated code until the plan shows no changes.
6. Break the lock file on purposeGenerate a lock file on your laptop, run init -lockfile=readonly in a Linux container (docker run --rm -v "$PWD":/w -w /w hashicorp/terraform:1.16.4 init -lockfile=readonly), read the failure, then fix it with providers lock.
7. Write a test that needs no cloudAdd a tests/*.tftest.hcl with mock_provider, command = plan, one assert and one expect_failures, and run it with no credentials set.
8. Catch driftChange a tag in the console, run terraform plan -detailed-exitcode, and check the exit code is 2. Then decide between reverting it with apply and recording it with -refresh-only.
9. Watch a replacement orderingForce a replacement of something behind a dependency, once with and once without create_before_destroy, and compare the order of operations in the apply output.

Recipe: migrate S3 locking from DynamoDB to use_lockfile

Native S3 locking has been generally available since 1.11, and the DynamoDB arguments are deprecated. Migrate without a window where two clients could hold different locks:

1. Confirm everyone is on 1.11 or laterCheck required_version and the CI image. Older clients do not understand use_lockfile.
2. Grant the lock-object permissionsAdd s3:GetObject, s3:PutObject and s3:DeleteObject on <key>.tflock to every role that runs Terraform.
3. Run bothAdd use_lockfile = true alongside dynamodb_table and run terraform init -reconfigure. The state location has not changed, so there is nothing to migrate. Both locks are taken on every run.
4. Remove DynamoDBOnce every pipeline and person has picked up the change, remove dynamodb_table and re-run init. Delete the table later, when no configuration refers to it.

terraform {
  backend "s3" {
    bucket         = "acme-tfstate-me-central-1"
    key            = "platform/network/prod/terraform.tfstate"
    region         = "me-central-1"
    encrypt        = true
    use_lockfile   = true
    dynamodb_table = "tf-locks"      # transition only: remove in the next change
  }
}

Recipe: back up and restore state before surgery

Any time you are about to run a state subcommand, change a backend, or do a large refactor, take a copy first. It costs one command.

terraform state pull > "state-backup-$(date +%Y%m%d-%H%M%S).tfstate"
jq '{serial, lineage, terraform_version, resources: (.resources | length)}' state-backup-*.tfstate

If something goes wrong, your first recovery option is the backend's own history. With S3 versioning enabled, every previous state is an object version you can restore from the console or CLI. Pushing a local copy with terraform state push is the second option: Terraform refuses if the lineage differs or the serial is lower than the current state, and you should read that refusal as a warning, not an obstacle to force past. Store backups somewhere as protected as the state itself, because they contain the same secrets.

Recipe: recover from a half-finished apply

An apply fails partway: some resources were created, one failed, and the pipeline is red. Nothing is broken yet, as long as nobody panics.

1. Read the first error, not the lastLater errors are often consequences. The provider's message usually quotes the cloud API's error code, such as a quota, a name collision or a denied permission.
2. Trust that state is correctTerraform writes state as each resource completes. Whatever was created is recorded; a resource whose creation failed partway is recorded as tainted, so the next apply replaces it.
3. Fix the cause, re-plan, applyThe next plan shows only what is left, plus the replacement of anything tainted. Read it like any other plan.
4. If the lock is still heldA crashed runner may leave it. Check the lock info matches the failed job, then force-unlock.

If a resource was tainted but is actually fine, terraform untaint ADDR clears the mark. For the opposite case, a resource that is broken but not tainted, use terraform apply -replace=ADDR rather than the older taint command, so the replacement appears in a plan you review.

Recipe: move resources between two states

You are splitting a monolithic state and the compute resources need to move into a new root module without being recreated. Do it with configuration blocks so both halves are reviewed:

1. Take a state pull backup of both states.
2. In the new root, write the resource blocks and an import block for each resource, using the real IDs (terraform state show ADDR in the old root gives them).
3. In the old root, delete the resource blocks and add removed { from = … lifecycle { destroy = false } } for each.
4. Plan both. The old root should show only "will no longer be managed", and the new root only imports with zero changes.
5. Apply the new root first, then the old one. For a moment both states hold the resource, which is harmless. The reverse order leaves a moment where neither does.

import {
  for_each = {
    trainer = "i-0a1b2c3d4e5f60718"
    batch   = "i-0f1e2d3c4b5a69788"
  }
  to = aws_instance.worker[each.key]
  id = each.value
}

Recipe: upgrade a provider major version

Major provider releases remove arguments, change defaults, and occasionally change which attributes force replacement. Treat the upgrade as a change in its own right, not something folded into a feature pull request.

1. Read the upgrade guideEvery major provider release has one on the registry. Search your code for each removed or renamed argument it lists.
2. Upgrade in the lowest environment firstChange the constraint, run terraform init -upgrade, then providers lock for all platforms.
3. Plan and read every replacementAn upgrade that produces no changes is the goal. Any # forces replacement needs an explanation before apply.
4. Promote one environment at a timeSeparate roots per environment make this possible: production keeps the old constraint until staging has run the new one for a while.

Recipe: drift detection on a schedule

name: drift
on:
  schedule:
    - cron: "0 4 * * *"            # daily, 04:00 UTC
permissions:
  id-token: write
  contents: read
jobs:
  plan:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        root: [live/prod/network, live/prod/data, live/prod/compute]
    steps:
      - uses: actions/checkout@v7
      - uses: hashicorp/setup-terraform@v4
        with:
          terraform_version: "1.16.4"
          terraform_wrapper: false
      - uses: aws-actions/configure-aws-credentials@v6
        with:
          role-to-assume: ${{ secrets.TF_PLAN_ROLE }}
          aws-region: me-central-1
      - run: terraform -chdir=${{ matrix.root }} init -input=false -lockfile=readonly
      - run: terraform -chdir=${{ matrix.root }} plan -input=false -lock=false -detailed-exitcode

A job that exits 2 fails, and the failure names the root module that drifted. Use the read-only plan role: drift detection never needs write access, and -lock=false is acceptable here only because the plan writes nothing.

Recipe: move a state to a new bucket or key

Teams reorganise their state layout when they split a monolith or move state into an in-region bucket for data-residency reasons. The state has to move with the configuration. A copy made by hand is not enough, because serial and lineage must survive intact.

1. Freeze the rootPause the pipeline for this root module so nothing applies while the state moves. The move is quick, but a concurrent apply during it would write to the old location.
2. Back upRun terraform state pull into a timestamped file and record the serial.
3. Change the backend configurationEdit the backend block, or the partial backend/<env>.hcl file, to name the new bucket, key or region.
4. MigrateRun terraform init -migrate-state -backend-config=backend/prod.hcl. Terraform reads the old state, asks for confirmation, and writes it to the new location. Never use -reconfigure here: it would start from an empty state.
5. VerifyRun terraform state pull | jq '{serial, lineage}' and check that the lineage matches the backup and the serial is the same or slightly higher. Then run terraform plan, which must report no changes. A plan that proposes creating everything means the migration did not happen.
6. Retire the old objectLeave the old state in place, read-only, for a week before you delete it. Update any terraform_remote_state readers that pointed at the old key.

Recipe: rotate a secret held in a write-only argument

Write-only arguments (1.11+) keep a password out of state. That same property stops Terraform from ever seeing that the value changed. If you update the secret in AWS Secrets Manager and re-plan, you get "No changes", and the database keeps its old password. Rotation is therefore an explicit act:

1. Write the new value into the secret store, through its own rotation process or by hand.
2. Increase the companion version argument in the configuration, for example password_wo_version = 2, in a pull request.
3. Plan. The plan shows an in-place update of the version argument only. The password itself never appears, because it is ephemeral.
4. Apply through the pipeline. During apply, the provider reads the ephemeral secret fresh and sends the new password.

Keep the version number in the same pull request as a comment explaining the rotation, so the history shows when and why each rotation happened. If the application reads the same secret, rotate in an order it can survive, for example with a dual-user scheme on the database. Terraform only guarantees the order of its own calls.

Making plans readable for reviewers

A 900-line plan in a pull request is not reviewed; it is scrolled past. Three habits make plans short enough to read. Keep states small, so each pull request touches a small plan. Put a summary at the top of the pull request comment: the counts from the last line of the plan (Plan: 2 to add, 1 to change, 0 to destroy.) and the jq list of changing addresses, with any delete or replace called out in bold. And set default_tags on the AWS provider rather than tagging every resource by hand, so a tag change is one line in the code diff rather than a hundred edited resource blocks. The plan still lists a tags_all update on every tagged resource, so say in the pull request description that a default-tag change is the cause.

Habits worth adopting now

Run terraform fmt -recursive before every commit. Put it in a pre-commit hook. Formatting noise in diffs hides real changes.

Never apply from a laptop to production. Your laptop may have a different CLI, a stale lock file, and credentials broader than the pipeline's. Laptops plan; the pipeline applies.

Give every stateful resource prevent_destroy. It costs one line and turns a catastrophic plan into a failed one.

Write the moved block in the same commit as the rename. A rename without its moved block is a destroy waiting for someone to approve it.

Keep module inputs typed and validated. A validation message in the caller's file is worth ten minutes of reading provider errors.

Tag everything with the stack that owns it. When you find an unfamiliar resource in the console, a Stack = "prod/data" tag tells you which state to look in.

Keep your state in the region your data lives in. For teams in the Gulf and Egypt subject to data-residency rules, state is a copy of your infrastructure's configuration and sometimes its credentials, so the state bucket belongs in the same jurisdiction as the data.

terraform {
  required_version = "~> 1.16"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
  }
  backend "s3" {}              # bucket, key, region, use_lockfile from backend/<env>.hcl
}

provider "aws" {
  region = "me-central-1"
  default_tags {
    tags = {
      ManagedBy = "terraform"
      Stack     = "prod/data"
      Repo      = "acme/infra"
    }
  }
}

Senior tips go further: the trust model of providers and pipelines, policy as code on the plan JSON, importing hundreds of resources with terraform query, provider mirrors for restricted networks, upgrading a whole estate across CLI minors, and the incidents that only happen when fifty teams share one Terraform platform.

This is part three of three. It closes the series by taking every topic from Beginner and Mid one level further, where each one now has a security, scale or ownership side to it. It also covers the work that becomes yours when Terraform is something other teams depend on, rather than a tool you run for yourself.

The questions change at this level. Up to now they were about making a plan come out right. From here they are different: who can change production by merging a pull request, what an attacker learns from reading your state bucket, why a plan that took forty seconds last year now takes eleven minutes, what state looks like after an apply died halfway through, and how you let forty engineers ship infrastructure without all of them holding administrator credentials. A senior Terraform owner is judged less by the HCL they write than by the incidents that never happen, and by how fast the ones that do happen get closed.

Everything here is checked against Terraform 1.16, the current release line. Where a feature is newer, the section says which release brought it in. Where something is only in the 1.17 beta, it is marked as a preview and not taught as current.

Where this picks up

I start with the architecture, because nearly every failure mode in this track comes from forgetting which part of the system holds the truth at a given moment.

The architecture you are really running

At Beginner you learned that Terraform compares configuration, state and reality, then produces a plan. That is true, and it hides the fact that "Terraform" is at least five separate systems with different owners, different failure modes and different security properties. When something goes wrong in production, the first useful question is which of these five is involved.

Terraform core is the terraform binary. It parses HCL, builds a dependency graph from the references between blocks, and walks that graph with up to ten operations in flight by default (-parallelism=10). It holds no cloud knowledge at all.

Providers are separate executables that core starts as child processes and talks to over a gRPC plugin protocol (versions 5 and 6). The AWS provider, for example, is a large Go program that turns "create an aws_s3_bucket" into a series of AWS API calls. Two consequences follow. First, most bugs you meet in production are provider bugs, not core bugs, which is why the error text Provider produced inconsistent result after apply names the provider. Second, a provider is arbitrary code running with whatever credentials the process has, which makes it part of your supply chain. We come back to that later.

The backend stores state and, for most backends, holds the lock. The v1.x compatibility promise lists only the local and http backends as core-supported. The s3, azurerm, consul and kubernetes backends get only basic bug-fix maintenance, gcs, pg, cos and oss are community-maintained, and neither group is covered by the promise in the same way. This matters when you upgrade: backend arguments do get deprecated. The S3 backend's dynamodb_table argument is the current example.

State is a JSON document mapping every resource address to a real object ID and the full set of attributes the provider returned. It is the only thing that lets Terraform know that aws_db_instance.main is the database called prod-orders. Lose it and Terraform believes nothing exists. Corrupt it and Terraform believes the wrong things exist.

The APIs are outside your control entirely. They throttle you, they are eventually consistent (a role you just created may not be usable for several seconds), and they can fail after doing half of what you asked.

Failure modes, component by component

The value of this picture is that each component fails in a different, recognisable way.

The last row deserves a second look because it is the one that hurts most. An apply is not a transaction. Terraform walks the graph and records each resource as it completes. If the run dies on resource 37 of 60, the first 36 are real and recorded, resource 37 may or may not exist in the cloud, and the rest were never started. There is no rollback. The next plan is the recovery mechanism: it compares what is recorded with what exists and proposes the remainder. This is why you never "fix" a failed apply by hand in the console. Every hand change adds a new difference that the next plan has to explain.

When the backend itself cannot accept the final state write (the network dropped, the credentials expired mid-run), Terraform writes what it knows to a local file called errored.tfstate and tells you so. That file is the most up-to-date record of your infrastructure that exists anywhere. Do not delete it. Push it once the backend is reachable again, as covered in the incident playbooks below.

Scaling: what breaks first

Terraform does not have a hard ceiling on the number of resources in a state. It degrades, and it degrades in a predictable order. Knowing the order lets you act before the team starts to route around Terraform.

Refresh is the first thing to hurt

Every plan starts with a refresh by default: for each resource in state, the provider asks the real API what the object looks like now. That is at least one API call per resource, often several. A state with 3,000 resources means thousands of read calls before Terraform computes a single change. So plan time grows with the size of the state, not with the size of your change. The engineer changing one tag waits the same eleven minutes as the engineer rebuilding the network.

The API rate limit is the second wall and it arrives at the same time. Cloud APIs throttle per account and per region. When several large states refresh at once (a busy CI morning), they compete for the same quota, the providers back off and retry, and everything slows down together. In TF_LOG=DEBUG output you see the retries; in the pipeline you only see "slow".

Lock contention is the second

One state has one lock. When a state covers what several teams own, their pipelines queue behind each other. Someone discovers -lock=false. That flag is the moment a scaling problem becomes a correctness problem, because two concurrent applies against one state can each write a version that forgets the other's changes.

What people reach for, and what works

-refresh=false and -target both produce plans that do not reflect the whole truth. They have their place in an incident, and no place in a routine pipeline. If a team needs them every day, the state is too big.

The container CPU trap

Terraform 1.14 and later are built with Go 1.25, which respects the CPU bandwidth limit of the container it runs in. On a CI runner that gives the job one CPU, Terraform now schedules as if it has one CPU, where older builds saw every core on the host. The upgrade notes call this out. The symptom is a pipeline that got slower after a Terraform upgrade with no configuration change. Look at the job's CPU limit before you look at anything else.

Unknown values and two-phase applies

A different scaling problem shows up as an error rather than as slowness. count and for_each must be known at plan time. When a platform grows, people write modules whose for_each keys come from another resource's attributes, and the plan fails with:

Error: Invalid for_each argument

The "for_each" map includes keys derived from resource attributes that cannot
be determined until apply, and so Terraform cannot determine the full set of
keys that will identify the instances of this resource.

The fix is design, not a flag. Keys must come from things you know when you write the plan: variable values, static names, the keys of an input map. Unknown values belong in the map values, never in its keys. The escape hatch of a first -target apply followed by a full apply works once, but it turns every fresh environment into a two-step manual procedure. HCP Terraform Stacks can defer such changes automatically, and the CLI has an experimental deferral flag in the 1.18 alpha builds. Neither is a reason to design modules this way today.

Designing state boundaries

Because splitting state is the answer to both scaling and blast radius, where to draw the lines is the most important structural decision a Terraform owner makes. Get it right and every later problem is smaller. Get it wrong and the fix is a state migration, which is the riskiest routine operation in this whole guide.

Split by lifecycle and by owner

Two questions decide a boundary. How often does this change? And who changes it? Resources that change at very different rates should not share a state, because the slow, dangerous ones (the VPC, the database) get planned every time the fast ones (a feature flag, a DNS record) change. Resources that different teams own should not share a state, because each team then needs permission to change the other's.

Each box is a separate root module per environment, with its own backend key and its own lock. The arrows are data dependencies, and they go one way. A service reads the network's subnet IDs. The network never reads anything about a service.

Some rules of thumb that hold up:

• A state that takes longer than a couple of minutes to plan is a candidate for splitting.
• A state that more than one team applies to is a candidate for splitting.
• Stateful resources whose loss means data loss (databases, buckets holding models or training data, KMS keys) live in states that change rarely and carry prevent_destroy.
• Production and non-production never share a state, a backend key prefix or a credential.

Passing values between states

When one state needs another's outputs, you have three options, and they have different coupling.

terraform_remote_state reads another state's outputs directly. It is easy and it has a cost that people discover late: to read outputs, the reader needs read access to the whole state file, including every secret in it. A service team that can read the network state can read whatever the network state contains.

Provider data sources look the thing up in the cloud by name or tag. data "aws_vpc" "main" filtered by a tag, for example. The coupling is to the real infrastructure, not to someone else's state layout, and the permission needed is a normal read permission on the API.

A published parameter store (AWS SSM Parameter Store, Consul, or HCP Terraform's tfe_outputs data source) makes the interface explicit. The producer writes the values it intends to share; the consumer reads only those. This is the platform-grade option, because it gives the producer a contract to keep and lets you change the producing state's internals freely.

resource "aws_ssm_parameter" "private_subnets" {
  name  = "/platform/network/prod/private_subnet_ids"
  type  = "StringList"
  value = join(",", module.vpc.private_subnets)
}

data "aws_ssm_parameter" "private_subnets" {
  name = "/platform/network/prod/private_subnet_ids"
}

locals {
  private_subnet_ids = split(",", data.aws_ssm_parameter.private_subnets.value)
}

Stacks, briefly

HCP Terraform has a feature called Stacks that moves some of this design into the product. A stack declares components in *.tfcomponent.hcl files and deployments in *.tfdeploy.hcl files; each deployment has its own isolated state, and stacks pass values to each other with publish_output and upstream_input. It exists only in HCP Terraform, not in the open CLI workflow or on a self-managed backend. If you are on HCP Terraform and repeat the same set of root modules across many regions or accounts, it is worth evaluating. If you are not, the patterns above are what you build yourself.

The trust model

Security in Terraform starts with one fact that is easy to state and easy to forget: Terraform can do whatever its credentials can do. It has no permissions of its own. If the pipeline role is an administrator, then everyone who can get code into that pipeline is an administrator.

Who is really able to change production

Draw the path from a keyboard to a production change. An engineer opens a pull request. CI runs a plan. Someone approves. CI applies. At each step, ask what an attacker, or a careless colleague, could do.

The plan step is more dangerous than it looks. terraform plan runs providers, evaluates data sources and can call external programs through the external data source. A pull request from any branch that modifies configuration can therefore run code in your CI with whatever credentials the plan job has. If the plan job holds the apply role, a malicious pull request can change production without ever being merged. This is the most common serious flaw in Terraform pipelines.

Plan roles and apply roles

A read-only plan role needs permission to read every resource type in the state, and to read and write the state lock, but not to change resources. On AWS that is the service's Describe*, Get* and List* permissions plus the backend permissions. Two details trip people up. The state lock has to be writable even for a plan, because the plan takes the lock. And some data sources need permissions that feel like writes: reading a secret value is secretsmanager:GetSecretValue, which you may well not want the plan role to have.

The apply role is the powerful one. It should be assumable only from the protected branch or environment of your CI system, which OIDC federation lets you express directly in the cloud's trust policy. In GitHub Actions, the token's subject claim contains the repository and the ref or environment, so the AWS role's trust policy can say "only repo:acme/infra:environment:production".

name: terraform
on:
  pull_request:
  push:
    branches: [main]

permissions:
  id-token: write
  contents: read

jobs:
  plan:
    runs-on: ubuntu-latest
    env:
      TF_IN_AUTOMATION: "1"
      TF_INPUT: "0"
      TF_PLUGIN_CACHE_DIR: ${{ github.workspace }}/.plugin-cache
    steps:
      - uses: actions/checkout@v7
      - uses: aws-actions/configure-aws-credentials@v6
        with:
          role-to-assume: arn:aws:iam::123456789012:role/tf-plan-readonly
          aws-region: me-central-1
      - uses: hashicorp/setup-terraform@v4
        with:
          terraform_version: "1.16.x"
      - run: mkdir -p "$TF_PLUGIN_CACHE_DIR"
      - run: terraform -chdir=envs/prod init -lockfile=readonly
      - run: terraform -chdir=envs/prod plan -lock-timeout=5m -out=tfplan
      - run: terraform -chdir=envs/prod show -json tfplan > plan.json
      - uses: actions/upload-artifact@v7
        with:
          name: tfplan
          path: |
            envs/prod/tfplan
            plan.json
          retention-days: 1

  apply:
    if: github.ref == 'refs/heads/main'
    needs: plan
    runs-on: ubuntu-latest
    environment: production        # required reviewers are configured here
    env:
      TF_IN_AUTOMATION: "1"
      TF_INPUT: "0"
    steps:
      - uses: actions/checkout@v7
      - uses: actions/download-artifact@v8
        with:
          name: tfplan
      - uses: aws-actions/configure-aws-credentials@v6
        with:
          role-to-assume: arn:aws:iam::123456789012:role/tf-apply-prod
          aws-region: me-central-1
      - uses: hashicorp/setup-terraform@v4
        with:
          terraform_version: "1.16.x"
      - run: terraform -chdir=envs/prod init -lockfile=readonly
      - run: terraform -chdir=envs/prod apply tfplan

Read what this workflow commits to. The pull-request run gets only the read-only role. The apply job runs only on main, only after a human approves the production environment, and it applies the exact plan file the reviewers could inspect. -lockfile=readonly makes init fail rather than silently change which provider versions run. If state moved between plan and apply, Terraform refuses with Saved plan is stale and you re-plan, which is the behaviour you want.

Humans and break-glass

Engineers still need to run Terraform locally sometimes: to debug, to import, to recover. Give them the plan role by default, through single sign-on, and make the apply role something they assume through a short, logged, approved break-glass procedure. The rule to aim for is that nobody holds standing write credentials to production, and every use of the write path leaves a record.

Secrets in state, plans and pipelines

Mid-level taught you to mark variables sensitive = true. At this level you need to be precise about what that does, because a wrong belief here leads to real leaks.

sensitive = true redacts values in the command-line output. That is all. The value is still written to state in full, and it is still in any saved plan file. Anyone with read access to the state bucket can read every database password, every generated private key and every API token that any resource in that state ever handled. The state file is, for practical purposes, a secrets store with weaker access controls than your real secrets store.

For years that was simply how Terraform worked, and the advice was "protect the state". Protecting the state is still required. But since 1.10 and 1.11 you can also keep many secrets out of it entirely.

Ephemeral values and write-only arguments

An ephemeral resource is a block that reads something fresh during each phase and is never written to plan or state. A secret from AWS Secrets Manager, Azure Key Vault or HashiCorp Vault is the typical example. A write-only argument is a resource argument the provider accepts but never stores; by convention their names end in _wo. Together they let a secret flow from your secrets manager into a resource without ever landing on disk in Terraform's files.

ephemeral "aws_secretsmanager_secret_version" "db" {
  secret_id = "prod/orders/db"
}

resource "aws_db_instance" "orders" {
  identifier          = "orders-prod"
  engine              = "postgres"
  instance_class      = "db.r6g.large"
  allocated_storage   = 100
  username            = "orders_admin"
  password_wo         = ephemeral.aws_secretsmanager_secret_version.db.secret_string
  password_wo_version = 3          # bump to push a rotated password
  skip_final_snapshot = false

  lifecycle {
    prevent_destroy = true
  }
}

The password_wo_version argument exists because Terraform cannot compare a value it never stored. It has no way to know the password changed. So you tell it: bump the version when you rotate, and the provider sends the new value. Forgetting this is the most common mistake with write-only arguments. The rotation in Secrets Manager happens, the plan shows no change, and the database keeps the old password.

Ephemeral values can be referenced only in certain places: locals, other ephemeral blocks, ephemeral variables and outputs, write-only arguments, provider configuration, and provisioner and connection blocks. Try to put one into a normal resource argument and Terraform refuses, which is the point. A child module can declare an output with ephemeral = true to hand such a value to its caller, but a root module cannot persist one as an output. Variables can be declared ephemeral too, for tokens a pipeline passes in:

variable "registry_token" {
  type      = string
  ephemeral = true
  sensitive = true
}

Ephemeral values in provider configuration are one of the most useful patterns. A provider can authenticate with a short-lived token read by an ephemeral resource, for example a Kubernetes provider configured from a freshly issued cluster token, and the token never appears in state.

Terraform 1.16 adds a store block to the built-in terraform_data resource. Its input argument is write-only, so it accepts ephemeral values; sensitive routes the stored value to a sensitive output; and version controls when the output changes. It is for the narrow case where you need a value to survive between plan and apply without an external secrets manager. Reach for the secrets manager first.

Protecting the state you still have

Even with ephemeral values, state contains a great deal you would not publish: resource IDs, internal hostnames, IAM structures, and every secret for resources that do not yet support write-only arguments. The controls are the same for every backend:

• Encryption at rest with a key you control (SSE-KMS on S3, CMEK on GCS, customer-managed keys on Azure Storage), plus encrypt = true on the S3 backend.
• Least-privilege access. The state bucket is not readable by everyone who can read the account. Access is granted per key prefix.
• Versioning, so a corrupted or deleted state can be restored. Object lock, where you need tamper resistance.
• Access logging (CloudTrail data events on the bucket, or the equivalent), so you can answer "who read the production state last month".

The provider and module supply chain

Every terraform init downloads executable code and runs it with your cloud credentials. Providers are the obvious case. Modules matter too, because a module can declare resources, data sources and external programs that execute during plan. A senior owner treats both as dependencies with a supply chain, exactly as you would treat container images or Python packages.

What protects you by default

Providers from the public registry are signed, and terraform init checks the signature before installing. It then records hashes of what it installed in .terraform.lock.hcl. On every later init, Terraform checks that what it downloads matches those hashes. That makes the lock file your record of exactly which provider bytes are trusted, and why it belongs in Git and in code review.

provider "registry.terraform.io/hashicorp/aws" {
  version     = "6.14.0"
  constraints = "~> 6.0"
  hashes = [
    "h1:3mVb...",
    "zh:0a1b...",
    "zh:1c2d...",
  ]
}

The h1: hashes cover the unpacked package for the platforms Terraform has seen; the zh: hashes are the zip checksums the registry published. If developers use macOS on ARM and CI runs Linux on x86, the lock file generated on a laptop may lack the Linux h1: hash, and CI can fail with a checksum mismatch. Generate hashes for every platform you use:

terraform providers lock \
  -platform=linux_amd64 \
  -platform=linux_arm64 \
  -platform=darwin_arm64 \
  -platform=windows_amd64

A lock-file change in a pull request means someone upgraded a provider. It deserves the same review as a dependency bump anywhere else: which version, why, and has anyone read the changelog?

Modules are not locked

The lock file covers providers only. Modules are not hashed. A module referenced as git::https://github.com/acme/modules.git//vpc?ref=main changes whenever someone pushes to main, and your next plan runs the new code. Pin registry modules with an exact version, and pin Git modules to a tag or, for anything sensitive, a full commit SHA. Tags can be moved by anyone with push rights; a commit SHA cannot.

Mirrors and air-gapped installs

Regulated environments often cannot reach the public registry from CI at all, and even when they can, depending on it for every run is a reliability risk. The CLI configuration file (~/.terraformrc on Linux and macOS, terraform.rc in %APPDATA% on Windows, or wherever TF_CLI_CONFIG_FILE points) controls where providers come from.

provider_installation {
  network_mirror {
    url = "https://terraform-mirror.internal.example.com/providers/"
  }
}

plugin_cache_dir   = "/opt/terraform/plugin-cache"
disable_checkpoint = true

With this in place, every provider comes from your mirror and nothing else. You populate the mirror from a connected machine with terraform providers mirror, which downloads the packages your configuration needs for the platforms you name:

terraform providers mirror \
  -platform=linux_amd64 \
  -platform=darwin_arm64 \
  ./mirror

The lock file still applies. If your mirror serves a package whose hash differs from what the lock file recorded, init refuses. That protects you against a compromised mirror, as long as the lock file itself went through review.

Policy, guardrails and governance

Code review catches most mistakes. It does not scale to hundreds of pull requests a week, and reviewers miss the same things repeatedly: a bucket without encryption, an instance type nobody approved, a security group open to the internet. Policy as code moves those checks into the pipeline, where they run on every plan.

Where a guardrail can live

There are four layers, and each catches something the others cannot.

The most robust platforms use all four. The module enforces its own contract. The plan policy enforces organisation rules. Checks watch for health and drift. And the cloud's own policy stops the things that must never happen, regardless of whether they came through Terraform.

Policy on the JSON plan

terraform show -json tfplan produces a documented JSON representation of the plan. Its resource_changes array lists every resource with the actions planned for it (create, update, delete, or a delete and create pair for replacement) and the before and after values. Any policy engine can read it. Here is an Open Policy Agent rule, run through Conftest, that blocks the deletion or replacement of protected stateful resources unless someone approves it explicitly:

package main

protected_types := {"aws_db_instance", "aws_rds_cluster", "aws_s3_bucket"}

deny contains msg if {
  rc := input.resource_changes[_]
  protected_types[rc.type]
  "delete" in rc.change.actions
  msg := sprintf("%s would be deleted or replaced; this needs a change-window approval", [rc.address])
}

terraform show -json tfplan > plan.json
conftest test plan.json --policy policy/

Because replacement appears as a delete plus a create, this one rule catches the most dangerous class of plan: the harmless-looking change to an attribute that forces a stateful resource to be rebuilt.

HCP Terraform and Terraform Enterprise run policies natively, in either Sentinel (HashiCorp's policy language) or OPA, as policy sets attached to the whole organisation or to projects, with enforcement levels that range from advisory to mandatory. The CLI also has a native "Terraform Policy" feature, used through a -policies flag on plan, apply and query. In 1.16 it is experimental and needs an experimental build; the 1.17 beta makes it generally available. Plan for it, but do not build your governance on it yet.

Guardrails that live in the configuration

Two lifecycle settings are guardrails in their own right. prevent_destroy = true makes any plan that would destroy the resource fail with:

Error: Instance cannot be destroyed

Resource aws_db_instance.orders has lifecycle.prevent_destroy set, but the
plan calls for this resource to be destroyed.

It protects against the change that forces replacement, which is the dangerous case. It does not protect against someone deleting the whole block, because then the setting is gone too. That gap is what the plan policy above covers.

Terraform 1.16 adds destroy = false inside lifecycle. With it, a plan that would destroy the resource (a replacement, or a destroy run) drops it from state instead of deleting the real object. It is the right setting for resources whose ownership might move to another state. Be aware that lifecycle settings other than create_before_destroy are not stored in state, so the protection only applies while the resource block is in the configuration: delete the block and the object is destroyed. The removed block with lifecycle { destroy = false } remains the explicit way to hand off a resource.

check blocks run assertions after apply and report failures as warnings. They are for questions like "does the endpoint return 200?" or "is the certificate valid for at least thirty days?". HCP Terraform's health assessments re-run them on a schedule, alongside drift detection, so a failing check shows up between deploys.

check "orders_api_health" {
  data "http" "health" {
    url = "https://orders.internal.example.com/healthz"
  }

  assert {
    condition     = data.http.health.status_code == 200
    error_message = "orders API health endpoint returned ${data.http.health.status_code}"
  }
}

Governance is also people and paper

Policy engines enforce rules. Somebody has to decide what the rules are, own the exceptions and review them. The shape that works is a small written standard (naming, tagging, allowed regions, protected resource types, required modules), a policy set that implements it, an exception process with an expiry date, and a quarterly review of the exceptions. Without the expiry date, exceptions become the real policy.

Multi-tenancy: many teams, one platform

Once several teams use Terraform, the questions shift from "does my plan work?" to "can team A break team B?" and "can team A see team B's secrets?". Multi-tenancy is the discipline of making the answer to both "no" by construction, not by good behaviour.

What does not isolate tenants

CLI workspaces are the trap here. A CLI workspace (terraform workspace new staging) is another state instance of the same configuration, stored in the same backend, typically readable with the same credentials. HashiCorp's own documentation says they are not suitable for isolating environments that need different credentials or access controls. They are fine for near-identical copies owned by one team, such as short-lived review environments. They are not tenants.

Separate directories in one repository with one shared CI role are not isolation either. If every root module is applied by the same role, any team that can merge to any of them can, with a little creativity, change any of the others.

What does

Isolation comes from three things lining up: separate state locations, separate credentials, and separate approval paths.

On a self-managed backend, give each team a key prefix in the state bucket and an IAM policy that grants that team's roles access to that prefix only. Include the lock object: with S3-native locking, the lock is a file named after the state key with a .tflock suffix, and it needs s3:GetObject, s3:PutObject and s3:DeleteObject.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::acme-tfstate",
      "Condition": { "StringLike": { "s3:prefix": ["team-a/*"] } }
    },
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
      "Resource": [
        "arn:aws:s3:::acme-tfstate/team-a/*"
      ]
    }
  ]
}

The strongest isolation, and the one most mature cloud estates end up with, is a separate cloud account (AWS), subscription (Azure) or project (GCP) per team and environment. Then the team's apply role simply has no permissions in anyone else's account, and a mistake stays where it happened.

HCP Terraform and Terraform Enterprise model the same structure directly: an organisation contains projects, projects contain workspaces, and teams get permissions per project or per workspace. Variable sets attach credentials to projects, policy sets apply globally or per project, and agent pools run plans inside private networks the SaaS cannot reach. An HCP Terraform workspace is a very different thing from a CLI workspace: it has its own state, variables, runs and permissions.

Cost: of Terraform, and through Terraform

There are two cost questions, and senior owners get asked both.

The first is what running Terraform costs. The CLI is free to use under the Business Source License, which HashiCorp adopted in 1.6 (IBM completed its acquisition of HashiCorp in February 2025). HCP Terraform is priced on resources under management. Its legacy free plan ended on 31 March 2026, and organisations moved to an enhanced Free tier with up to 500 managed resources; the paid tiers are Essentials, Standard and Premium. Terraform Enterprise is self-hosted and licensed. Self-managed setups built on the open CLI with Atlantis, or on another automation product such as Spacelift, env0 or Scalr, trade licence cost for the engineering time to run them. Resource-count pricing has a practical consequence for design: splitting one module into twenty tiny resources may cost more on a per-resource plan.

The second question is what the infrastructure costs, and here Terraform is useful because every change is a reviewable diff. Put a cost estimate in the pull request. Infracost reads the plan and posts the monthly difference; HCP Terraform has built-in cost estimation. A reviewer who sees "+$4,200/month" next to a GPU node pool asks the right question before the invoice does. Enforce tags for cost allocation (team, service, environment) with provider default_tags and a policy rule, so the finance report can be traced back to a state and an owner.

provider "aws" {
  region = "me-central-1"

  default_tags {
    tags = {
      Team        = "ml-platform"
      Service     = "feature-store"
      Environment = "prod"
      ManagedBy   = "terraform"
    }
  }
}

Upgrades and migrations

Terraform estates rot quietly. The CLI moves forward a minor version every three or four months, providers ship major versions with breaking changes, backend arguments get deprecated, and one day a new region or a security fix forces an upgrade across a hundred states at once. Owning the upgrade path is part of owning the platform.

Upgrading the CLI

The v1.x compatibility promise covers the language, the core workflow commands and state compatibility going forward. It does not promise that you can go back: a state written by a newer version may not be readable by an older one. So in a shared estate the order is strict:

1. Read the upgrade notesEach minor release's CHANGELOG has an "UPGRADE NOTES" section. For 1.16 it says provisioners now apply bastion_host_key; for 1.14, that parallelism may drop in containers; for 1.15, that two S3 backend environment variables now accept only strict true or false.
2. Upgrade CI first, then peopleMove the pinned version in your CI images and the required_version constraint together. Engineers on older CLIs will then get Unsupported Terraform Core version instead of quietly writing state with the wrong version.
3. One minor at a timeSkipping from 1.12 to 1.16 means reading four sets of upgrade notes at once and not knowing which one broke you.
4. Canary on low-risk statesPlan every state with the new version, expect "No changes", and apply the empty plan to the least critical ones first.
5. Then the rest, in wavesWith a rollback plan that is "restore the previous state version", not "use the old CLI".

required_version = "~> 1.16" accepts any 1.x from 1.16 on. ~> 1.16.0 pins to 1.16 patch releases. For a platform, the second form in CI images and the first in shared modules is a good balance: modules stay usable across versions, and the pipeline changes only when you change it.

Upgrading providers

Provider major versions are where most breaking changes live: renamed arguments, split resources, changed defaults. The error you see after an unprepared bump is typically Unsupported argument: An argument named "x" is not expected here. The procedure is:

1. Read the provider's upgrade guide for the major version.
2. Raise the constraint in required_providers, run terraform init -upgrade, and commit the new lock file.
3. Plan every affected state. Any diff that is not explained by the upgrade guide is a bug to investigate before applying.

Automate the minor and patch bumps with Renovate or Dependabot, which update both the constraint and the lock file in a pull request. A steady trickle of small reviewed bumps is far safer than a yearly jump.

Migrating the backend: DynamoDB to S3-native locking

S3-native locking became generally available in 1.11, and the dynamodb_table argument is deprecated. You will see this warning until you move:

Warning: Deprecated parameter

The parameter "dynamodb_table" is deprecated. Use parameter "use_lockfile" instead.

You can enable both during the transition, so that old and new runs still respect each other's locks:

terraform {
  backend "s3" {
    bucket         = "acme-tfstate"
    key            = "network/prod/terraform.tfstate"
    region         = "me-central-1"
    encrypt        = true
    use_lockfile   = true
    dynamodb_table = "tf-locks"   # remove once every pipeline uses use_lockfile
  }
}

Roll it out, update the IAM policies to allow the .tflock object, confirm every pipeline and every engineer has run terraform init -reconfigure or -migrate-state as appropriate, then remove dynamodb_table in a second change.

Moving resources between states

Splitting a monolith state is the migration senior owners do most, and since 1.7 it can be done entirely in configuration, which means it is reviewable and repeatable. In the source state, stop managing the resource without destroying it:

removed {
  from = aws_s3_bucket.model_artifacts

  lifecycle {
    destroy = false
  }
}

In the destination state, adopt it:

import {
  to = aws_s3_bucket.model_artifacts
  id = "acme-model-artifacts-prod"
}

resource "aws_s3_bucket" "model_artifacts" {
  bucket = "acme-model-artifacts-prod"
}

Apply the destination first so the resource is never unmanaged, then the source. Both plans should show exactly what you expect: the destination plan says 1 to import, with no changes to the resource; the source plan says the resource will be removed from state, and nothing will be destroyed. If the destination plan shows updates, your new resource block does not match reality yet. Fix the configuration until it shows only the import.

Importing at scale

Bringing a hand-built estate under Terraform used to mean one terraform import command per resource. Now there are three tools that compose. An import block accepts for_each, so one block can adopt a whole map of resources. terraform plan -generate-config-out=generated.tf writes HCL for any import target that has no resource block yet. And since 1.14, terraform query finds the resources in the first place: you write list blocks in a *.tfquery.hcl file, and the command searches the cloud through the provider.

list "aws_instance" "all" {
  provider = aws
}

terraform query -generate-config-out=to-import/instances.tf

The output file contains import and resource blocks for what the query found. Treat generated configuration as a first draft: it contains every attribute, including computed ones and defaults, and it needs to be cut down to what you mean before anyone maintains it. Since 1.16, import blocks can live inside child modules, so you can adopt resources directly into the module that will own them.

Incident playbooks

Terraform incidents follow a small number of shapes. Knowing them in advance turns a frightening afternoon into a checklist. The first rule for all of them: before you touch anything, take a copy of the current state.

terraform state pull > "backup-$(date -u +%Y%m%dT%H%M%SZ).tfstate"

A lock that will not clear

The symptom is every pipeline failing with this:

Error: Error acquiring the state lock

Lock Info:
  ID:        6f1c2a9e-8d0b-4c4e-9a0f-2d7b3c1e5a44
  Path:      acme-tfstate/network/prod/terraform.tfstate
  Operation: OperationTypeApply
  Who:       runner@ci-7f9c
  Version:   1.16.4
  Created:   2026-09-29 07:12:41.18 +0000 UTC

Read the lock info. Who and Created tell you which run holds it and since when. Check that the run is truly dead: the CI job cancelled, the runner gone. Only then release it:

terraform force-unlock 6f1c2a9e-8d0b-4c4e-9a0f-2d7b3c1e5a44

Unlocking while the other run is still alive lets two applies race and corrupt state. If you cannot prove the other run is dead, wait for it.

A partial apply

The apply failed partway. Do not open the console to finish the job by hand. Re-run the plan. It will show the resources that were never created as creates, and any resource that failed during creation as tainted and due for replacement. If the failure was transient (throttling, an expired token), a new apply usually finishes the work. If it was a real error, fix the configuration and apply again. If Terraform left an errored.tfstate because it could not write to the backend, push that file once the backend works: terraform state push errored.tfstate.

State corrupted or wrongly edited

Someone ran terraform state rm on the wrong address, or pushed a bad file. With versioning enabled on the bucket, you can go back:

aws s3api list-object-versions \
  --bucket acme-tfstate \
  --prefix network/prod/terraform.tfstate \
  --query 'Versions[].{id:VersionId,modified:LastModified}' --output table

aws s3api get-object \
  --bucket acme-tfstate \
  --key network/prod/terraform.tfstate \
  --version-id 3HL4kqtJvjVBH40Nrjfkd \
  restored.tfstate

terraform state push checks two things before it overwrites remote state: the lineage (a unique ID each state gets when created) must match, and the serial (a counter that increases with each write) of what you push must not be older than what is there. An older version you have just restored will fail the serial check. terraform state push -force overrides it. That flag exists for exactly this case, and it is also the most dangerous flag in the CLI. Use it only with the lock held, a backup taken and a second person watching. Then plan immediately: the plan tells you what differs between the restored state and reality.

A secret leaked through state or a plan

The response order is the same as for any leaked credential: rotate first, then clean up. The state file has been copied by every run that pulled it, every backup and every artifact store. Removing the value from the current version does not remove it from history. Rotate the credential, then move the resource to an ephemeral value with a write-only argument, so the new secret never lands in state.

An accidental destroy

Someone applied a plan that destroyed a database or a bucket. Terraform cannot bring it back; only the cloud service can, from a snapshot or a versioned bucket, and only if that was configured. Restore from the service's own backup, then import the restored resource into state with an import block. Then do the post-mortem on the question that matters: why did the plan that did this get applied? The answers are usually a missing prevent_destroy, a missing delete policy on the plan, or a review that only looked at the summary line.

Drift nobody explained

A scheduled terraform plan -detailed-exitcode exits with 2 when there are changes to make. Run it nightly on every production state and alert on the 2. When drift appears, decide which side is right. If the manual change was correct, bring it into configuration. If it was not, apply to revert it. terraform apply -refresh-only updates state to match reality without changing any infrastructure, for the cases where you only want the record to be accurate.

Where Terraform stops

A senior engineer's credibility depends as much on saying "not Terraform" at the right time as on writing good Terraform. The tool is excellent at one thing: converging long-lived infrastructure towards a declared state, with a reviewable plan. It is poor at several things people keep asking it to do.

Kubernetes deserves its own paragraph

The Kubernetes and Helm providers work, and there is a clean line to draw. Terraform creates the cluster, the node pools (including GPU pools for training and serving), the networking, the IAM roles for workloads and the object storage for model artifacts. Then something else delivers what runs inside the cluster. Keep the cluster and the in-cluster workloads in separate states at the very least, because a provider configured from a cluster that does not exist yet is one of the classic unknown-value problems, and because Kubernetes controllers constantly change objects in ways that show up as drift. The natural next steps in this catalogue follow that line: Kubernetes, then Helm, then Argo CD.

Actions: day-two operations without state

Terraform 1.14 added a narrow answer to the "one-off operation" row. An action block declares a provider-defined operation, such as invoking a Lambda function, that does not change state. You trigger it from a resource's lifecycle.action_trigger (on events such as after_create or after_update, and since 1.16 before_destroy and after_destroy) or directly with terraform apply -invoke=action.aws_lambda_invoke.warm_cache. In 1.16, on_failure controls whether a failed action halts the apply, taints the resource or is ignored. Actions are useful for small, tightly scoped side effects of infrastructure changes: warming a cache, running a smoke test, triggering a model reload. They are not a workflow engine. If you are chaining four of them, you have written a runbook in the wrong tool.

The MLOps line

For an ML platform the division is usually clear. Terraform owns the durable foundation: accounts and networks, the Kubernetes clusters and GPU node pools that run training and inference, the buckets and registries that hold datasets and model artifacts, and the managed ML services such as SageMaker, Vertex AI or Azure ML workspaces. Pipelines, experiments, model versions and deployments of individual models belong to the ML tools and to GitOps. A model deploy that requires a Terraform apply is a sign the line is in the wrong place.

Running Terraform as a platform for other teams

Everything so far comes together here. When Terraform is a platform, your users are engineers on other teams, and your product is the paved road they use to get infrastructure without becoming Terraform experts. The goal is not that every team writes perfect Terraform. The goal is that the easy way to get infrastructure is also the safe way.

The module library is a product

Your modules are the interface. Treat them the way a library maintainer treats a public API:

• Opinionated defaults. The s3-bucket module encrypts, blocks public access, enables versioning and applies tags without being asked. Teams inherit good settings by using it, which is far more reliable than a checklist.
• A small surface. Expose the inputs teams really need, with validation blocks that explain themselves. Every extra input is a combination you have to test and support.
• Semantic versions and a changelog. A breaking change is a major version. Publish to a private registry or tag in Git, and never move a tag.
• Deprecation, not deletion. Since 1.15, variable and output blocks accept a deprecated = "message" attribute, so callers see a warning when they use an input you plan to remove. Give a release or two of warnings before the major version removes it.
• Tests. terraform test with .tftest.hcl files and mock_provider blocks lets you test a module's logic without creating real infrastructure, and -junit-xml feeds the results to CI. Keep one real-apply test per module against a sandbox account, on a schedule.

variable "name" {
  type        = string
  description = "Bucket name. Must include the team prefix."

  validation {
    condition     = can(regex("^[a-z0-9-]+-(dev|staging|prod)$", var.name))
    error_message = "Bucket names must be lowercase, hyphenated, and end in -dev, -staging or -prod."
  }
}

variable "force_destroy" {
  type        = bool
  default     = false
  description = "Deprecated: buckets holding data are never force-destroyed."
  deprecated  = "force_destroy will be removed in v4. Empty the bucket through the data-retention process instead."
}

mock_provider "aws" {}

variables {
  name = "ml-features-prod"
}

run "encryption_and_public_access_are_on" {
  command = plan

  assert {
    condition     = aws_s3_bucket_public_access_block.this.block_public_acls == true
    error_message = "public ACLs must be blocked by default"
  }
}

One pipeline, many teams

Give every team the same pipeline template: format check, validate, lint, plan with the read-only role, policy check on the JSON plan, cost estimate, a human approval, and an apply of the saved plan with the team's own write role. When a new control is needed, you change the template once. When a team needs something unusual, they ask you, and you find out what the platform is missing.

Whether that pipeline runs on GitHub Actions, GitLab CI, Atlantis or HCP Terraform matters less than the fact that there is one of it. Atlantis gives self-hosted pull-request automation. HCP Terraform and Terraform Enterprise give workspaces, policy sets, private registry, agents and audit trails as a product, at a per-resource price. Both are reasonable. Several hand-rolled pipelines per team are not.

Measure the platform

A platform without numbers is run on anecdotes. The measures that tell you the most:

terraform version -json reports the CLI and provider selections for a working directory, and terraform modules -json lists modules and versions, so a small script over your repositories can produce most of this table.

Support model

Write down what the platform team owns (modules, pipelines, policy, the state backend, upgrades) and what product teams own (their root modules, their resources, their approvals). Publish an upgrade calendar. Run office hours. Keep an issues channel. The most common failure of an infrastructure platform is not technical; it is a team that waited three weeks for a module change and wrote their own fork. Response time is a platform feature.

The review checklist

Use this list on any Terraform pull request or design document. It gathers what this series has covered into the questions that catch problems before they reach production.

The complete picture

Here is the final artefact of the series: a production root module for a feature store's data layer, with each earlier level's topics applied. Nothing in it is new, and you should be able to explain every line.

terraform {
  required_version = "~> 1.16.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
  }

  backend "s3" {
    bucket       = "acme-tfstate"
    key          = "ml-platform/feature-store/prod/terraform.tfstate"
    region       = "me-central-1"
    encrypt      = true
    use_lockfile = true
    assume_role {
      role_arn = "arn:aws:iam::123456789012:role/tf-state-ml-platform"
    }
  }
}

provider "aws" {
  region = "me-central-1"

  default_tags {
    tags = {
      Team        = "ml-platform"
      Service     = "feature-store"
      Environment = "prod"
      ManagedBy   = "terraform"
    }
  }
}

# Values another state publishes on purpose, not its whole state file
data "aws_ssm_parameter" "private_subnets" {
  name = "/platform/network/v1/prod/private_subnet_ids"
}

# Secrets are read fresh and never written to state or plan
ephemeral "aws_secretsmanager_secret_version" "db" {
  secret_id = "prod/feature-store/db"
}

module "offline_store" {
  source  = "app.terraform.io/acme/s3-bucket/aws"
  version = "3.2.1"
  name    = "ml-features-prod"
}

module "online_store" {
  source  = "app.terraform.io/acme/postgres/aws"
  version = "5.0.4"

  name                = "feature-store-prod"
  subnet_ids          = split(",", data.aws_ssm_parameter.private_subnets.value)
  master_password_wo  = ephemeral.aws_secretsmanager_secret_version.db.secret_string
  password_version    = 2
}

# A bucket adopted from a hand-built past, now owned here
import {
  to = module.offline_store.aws_s3_bucket.this
  id = "ml-features-prod"
}

check "online_store_reachable" {
  assert {
    condition     = module.online_store.status == "available"
    error_message = "feature store database is not available"
  }
}

# The pipeline, in order: each step comes from an earlier section
terraform fmt -check -recursive
terraform init -input=false -lockfile=readonly
terraform validate
terraform test -junit-xml=test-results.xml
terraform plan -input=false -lock-timeout=5m -out=tfplan      # read-only role
terraform show -json tfplan > plan.json
conftest test plan.json --policy policy/                      # protected deletes, tags, IAM
infracost breakdown --path plan.json                          # cost on the pull request
# ...human approval on the protected environment...
terraform apply -input=false tfplan                            # apply role, saved plan only

Read it as a set of decisions. The state has its own key, its own IAM role and in-region storage. The CLI is pinned to a minor line, and the provider to a major. The subnets come from a published parameter, not someone else's state. The database password comes from an ephemeral resource and goes into a write-only argument through the module, whose input variable must itself be declared ephemeral = true, with a version to bump on rotation. Modules come from a private registry at exact versions. A formerly hand-built bucket is imported into the module that owns it. A check reports health after apply. And the pipeline plans with one identity, applies the saved plan with another, and puts policy and price in front of a human before anything changes. The module input names in the example (master_password_wo, password_version, status) belong to a hypothetical internal module; your own modules define their own.

Where the series leaves you

You should now be able to look at a Terraform repository and see not only what it creates, but what it permits: who can change production and with which identity, what an attacker learns from the state bucket, which change would silently replace a database, which states are too large and why, which teams can reach each other's resources, what happens when an apply dies halfway, and what the next CLI or provider upgrade will cost.

Most Terraform incidents are prevented at review time, not fixed at run time. Carry the review checklist with you. If your next stop is what runs on the clusters you provision, continue with Kubernetes. If it is the secrets that feed your ephemeral resources, continue with Vault.

Sources

• Terraform documentation
• Terraform v1.16 CHANGELOG
• Terraform v1.15 CHANGELOG
• Terraform v1.14 CHANGELOG
• Terraform v1.11 CHANGELOG
• Terraform v1.10 CHANGELOG
• Terraform v1.17 CHANGELOG (pre-release)
• Upgrade guides
• Terraform v1.x compatibility promises
• CLI commands
• terraform plan
• terraform query
• terraform force-unlock
• The lifecycle meta-argument
• The for_each meta-argument
• The import block
• Invoking actions
• The terraform_data resource
• Manage sensitive data
• Dependency lock file
• S3 backend
• Tests
• Stacks
• HCP Terraform enhanced free tier

Part three of three. Senior Terraform interviews rarely ask you to write HCL. They give you an estate, an incident or a leadership question and watch how you reason: which parts of the system you name, which risks you raise without being asked, and whether you can explain a trade-off to someone who does not care about HCL. This file is organised the same way. System-design prompts first, then incident response, then trade-offs argued to leadership, then quick questions and a self-test. Allow about fifty minutes.

Part one: system design around Terraform

A good design answer follows the same shape each time. Start by asking about the constraints (how many teams, how many clouds, what regulation applies, what already exists), then describe the state layout, the identity model, the pipeline, the guardrails and the operations story, and finish with what you would measure. Interviewers are listening for whether you think about state and credentials first, because that is where the real risk sits.

Prompt: infrastructure as code for thirty product teams

"We have thirty product teams on AWS, one platform team of five, and everyone runs Terraform differently. Design what you would build."

A strong answer. I would start by asking what hurts today: slow plans, incidents, audit findings, or simply inconsistency. Then I would propose one account per team and environment under an organisation with service control policies, because account boundaries are the only isolation that does not depend on everyone writing correct IAM. State goes in one versioned, KMS-encrypted bucket in the home region with a key prefix per team and IAM scoped to each prefix, using S3-native locking with use_lockfile. Each team gets two roles in its own accounts, a read-only plan role and an apply role, both assumed through OIDC from CI, with the apply role trusted only from the protected production environment.

The platform team's product is a module registry with opinionated defaults and a single pipeline template: format, validate, test, plan with the read-only role, policy on the JSON plan, a cost estimate, approval, and apply of the saved plan. Guardrails live in four layers: module validations, plan policy, check blocks and drift detection, and service control policies for what must never happen. I would measure plan time per state, nightly drift, provider version spread and module forks, and publish an upgrade calendar.

What a weak answer misses. It jumps straight to folder structure and a tool name. It uses CLI workspaces for environments with one shared credential, which isolates nothing. It never mentions who can apply, or that a plan job holding write credentials can be abused by any pull request.

Prompt: a regulated ML platform in the Gulf

"A bank in Riyadh wants an ML platform: GPU training, model serving on Kubernetes, and a feature store. Regulators require data to stay in the Kingdom. How does Terraform fit?"

A strong answer. The residency requirement applies to Terraform's own files as much as to the data, because state and saved plans contain resource attributes, endpoints and, for resources without write-only arguments, secrets. So the state backend lives in an in-Kingdom region (for example Google Cloud's Dammam region with the gcs backend, or OCI's Jeddah or Riyadh regions with the oci backend available since 1.12), and I would check where any hosted automation platform stores state and plan files before routing regulated workloads through it. If that cannot be satisfied, a self-managed pipeline or Terraform Enterprise inside the bank's network is the answer. Providers come through an internal network mirror, because the build network probably cannot reach the public registry, and the lock file pins the hashes.

Terraform owns the foundation: networks, the Kubernetes cluster and GPU node pools, buckets for datasets and artifacts, and the managed services. It does not deploy models. Model serving goes to GitOps with Helm and Argo CD. Secrets come from the bank's secrets manager through ephemeral resources into write-only arguments, so database passwords never reach state. Cluster creation and in-cluster resources live in separate states.

What makes it senior. Saying unprompted that state files are regulated data, and drawing the line between infrastructure and model delivery.

Prompt: a platform you inherit

"You inherit one Terraform state with 4,000 resources. Plans take fourteen minutes, three teams apply to it, and there were two incidents last quarter. What do you do in your first ninety days?"

A strong answer. First, stop the bleeding without changing structure: take a state backup, enable versioning if it is off, add prevent_destroy to stateful resources and a plan policy that blocks protected deletes, and make sure the plan job cannot apply. Second, measure: time the plan with and without refresh, count resources by owner and by change frequency, and read the last quarter's applies. Third, design the split along owners and lifecycle, typically foundation, data and per-team service states. Fourth, execute the split one slice at a time with removed blocks using lifecycle { destroy = false } in the old state and import blocks in the new one, applying the destination first and checking that both plans show only the state moves. Every slice is a reviewed pull request, done in a quiet window, with the backup restored in rehearsal first.

What a weak answer misses. It starts with terraform state mv in a terminal, without backups or review, or it proposes rewriting everything from scratch.

Prompt: self-service without handing out admin

"Product teams want to create their own queues, buckets and databases without waiting for the platform team. Security does not want them holding admin rights. Design it."

A strong answer. Self-service and least privilege stop conflicting once the pipeline, not the person, holds the credentials. Teams write a root module that calls approved registry modules. Their pull request plans with a read-only role. A policy set checks that only approved modules and resource types are used, that tags exist, and that no IAM statement uses wildcards. A reviewer from the team approves, and a platform reviewer is required only when policy flags an exception. The apply role is scoped to the team's own account. Engineers never hold write credentials; break-glass is a separate, logged, time-limited path.

Follow-up you should expect. "What if a team needs something the modules do not support?" Answer: they raise it with the platform team, which has a published response time, and in the meantime an exception with an expiry date. Forks are measured, because each one means the platform was too slow.

Part two: incident response

For incident questions, interviewers want a calm order of operations. The shape: contain, preserve evidence, diagnose, fix, then prevent. In Terraform, "preserve evidence" nearly always means taking a copy of the state before you do anything else.

"Every pipeline fails with Error acquiring the state lock. What do you do?"

Read the lock information: the ID, Who, Operation and Created fields tell you which run holds it and since when. Check whether that run is still alive in CI. If it is, wait, or add -lock-timeout to pipelines so they queue instead of failing. If it is provably dead, release it with terraform force-unlock and the lock ID. Afterwards, find out why the run died without releasing the lock, usually a killed runner or an expired credential. Never suggest -lock=false in a shared environment, because two concurrent applies can each write a state that forgets the other's changes.

"An apply failed halfway. The on-call engineer wants to create the rest in the console. What do you say?"

No. An apply records each resource as it completes, so state already reflects what was done. Re-plan: the plan shows what remains, and any resource that failed during creation appears as tainted and due for replacement. If the failure was transient, re-apply. If Terraform could not write state at the end, it left an errored.tfstate file, which is the most current record and should be pushed once the backend is reachable. Console changes add drift that the next plan has to reconcile, and in a stateful resource they can cause a replacement.

"Someone ran terraform state rm on the production database by mistake."

The database itself is fine; only Terraform's record of it is gone. The next plan would try to create a new database, which would probably fail on a name conflict, or worse, succeed. Freeze applies on that state. Then there are two routes. Either re-adopt it with an import block, which is reviewable, or restore the previous state version from the versioned bucket and push it. The push will be refused on the serial check because the restored copy is older, and terraform state push -force overrides that, so do it with the lock held and a second person watching. Either way, plan immediately afterwards and expect no changes. Prevention: remove human write access to production state, and do refactors with moved and removed blocks in pull requests.

"A security scan found database passwords in the state bucket. What now?"

Rotate first. The state has been pulled by every run, copied into backups and possibly into CI artifacts, so the old password must be treated as public. Then find out who could read the bucket and check the access logs. Then fix the cause: move those resources to an ephemeral resource feeding a write-only argument, where the provider supports it, with a version argument to trigger the new value. Tighten bucket access to the prefix level. Finally, explain that sensitive = true was never protection, it only redacts output.

"Last night's plan shows a production database will be replaced. Nobody changed the configuration."

Something changed an input to a replace-forcing attribute. The usual suspects are a provider upgrade that changed a default or schema, drift from a manual change that Terraform now wants to revert, or a module version that floated because it was unpinned. Compare the lock file and module versions with the last good run. Read the plan line that says which attribute forces replacement. Do not apply. If the change came from an upgrade, pin back or adjust the configuration to match reality, and check the provider's upgrade guide. prevent_destroy would have turned this into an error rather than a risk, and a plan policy on deletes would have blocked it in CI.

"CI got much slower after a Terraform upgrade, with no config change."

Since 1.14, Terraform is built with Go 1.25, which respects the container's CPU quota. On a runner limited to one CPU, it now schedules as if it has one CPU. Check the job's CPU limit and raise it, or tune -parallelism. Then look at the usual suspects: provider cache missing (TF_PLUGIN_CACHE_DIR), refresh time growing with the state, API throttling in the debug log.

Part three: arguing trade-offs to leadership

Senior engineers get asked to recommend, not only to implement. Leadership questions test whether you can turn a technical position into cost, risk and time, and whether you can admit what your recommendation gives up. A useful pattern is three sentences: the decision, what it buys in the leader's terms, and what it costs or risks.

"Should we pay for HCP Terraform or keep running our own pipelines?"

Frame it as build versus buy with numbers. HCP Terraform is priced on resources under management; since the legacy free plan ended in March 2026, the Free tier covers up to 500 managed resources and paid tiers scale from there. It brings policy sets, a private registry, run agents, audit trails, drift detection and health assessments as a product. The self-managed path (open CLI with GitHub Actions or Atlantis, S3 state and OPA) costs no licence but costs engineering time to build and operate, and some of those features you will never build. Then add the constraints: where the hosted service keeps state and plans may matter for data residency, and resource-based pricing penalises very fine-grained modules. A sound recommendation might be: "Stay self-managed for the regulated estate, and trial the product on one non-regulated business unit for a quarter with a cost cap." Say what would change your mind.

"Should we move to OpenTofu because of the licence?"

Start with what the Business Source License restricts, and say you would ask legal to confirm it for your case: for most companies using Terraform internally to manage their own infrastructure, the licence changes little in practice. The case for moving is strategic (community governance, avoiding a single vendor) and sometimes legal, if you offer something that competes with HashiCorp's products. The cost is a migration of every pipeline, the risk of divergence between the two projects' features, and a smaller commercial ecosystem. Present it as reversible and testable: pin, canary a few low-risk states, compare features you rely on. A leader wants to hear that you would not switch the whole estate on principle in one quarter.

"Why should we spend a quarter splitting state instead of building features?"

Translate into their terms. "Our main state takes fourteen minutes to plan and three teams wait on the same lock, which costs roughly N engineer-hours a week. The same state means one mistake can affect every service we run; we had two incidents from it last quarter. Splitting it takes six weeks of one engineer and reduces both. The risk during the work is a state migration, which we manage with backups, rehearsals and reviewed pull requests." Always quantify the waiting, name the incidents and say how you control the migration risk.

"Can we let product teams provision their own infrastructure?"

Yes, if the credentials live in the pipeline and not with people. Explain it as a trade: the platform team gives up being a ticket queue, and in return must provide good modules, a fast support path and automated policy. Security gains a single auditable route to production instead of scattered admin keys. The risk is a team doing something expensive or unusual within the guardrails; cost estimates on every pull request and tag-based budgets address that.

Common interview questions

What is the difference between sensitive and ephemeral? Sensitive redacts a value in output but stores it in state and plan files. Ephemeral values are never written to plan or state, and can only be used in specific places: locals, ephemeral variables and outputs, write-only arguments, other ephemeral blocks, provider configuration and provisioners.

Why do write-only arguments need a version argument? Terraform never stores the value, so it cannot tell when it changes. You bump the version to tell the provider to send the new value.

What does the lock file protect, and what does it not? It records provider versions and package hashes so every run installs the same verified provider bytes. It does not cover modules; pin those by version, tag or commit SHA.

When would you use terraform_remote_state? Rarely across teams, because the reader needs access to the whole state, including secrets. Prefer provider data sources or a published parameter store.

How do you detect drift? A scheduled terraform plan -detailed-exitcode, which exits 2 when there are changes, alerting on 2. HCP Terraform health assessments do this as a product, plus re-running check blocks.

What does -refresh-only do? It updates state to match the real infrastructure without changing any infrastructure. It replaces the deprecated terraform refresh.

What is the modern replacement for terraform taint? terraform apply -replace=ADDR, which puts the replacement in a reviewable plan.

How do you bring existing resources under management at scale? terraform query with list blocks to discover them, -generate-config-out to draft configuration, import blocks with for_each, and since 1.16 import blocks inside child modules.

Why can't for_each keys depend on resource attributes? Terraform must know every instance address at plan time. Keys must come from known inputs; unknown values belong in map values.

Where does Terraform stop? Application delivery to Kubernetes, configuration inside servers, ordered runbooks, and logic that needs a real programming language.

What are Terraform actions? Provider-defined operations declared in action blocks (1.14 and later) that do not change state, triggered from lifecycle.action_trigger or terraform apply -invoke. They suit narrow side effects, not workflows.

What changed in S3 backend locking? Since 1.11, S3-native locking with use_lockfile = true is generally available and dynamodb_table is deprecated. You can run both during migration.

Final self-test

Answer each in a sentence before checking.

If you can answer these without looking, you are ready to walk into a senior platform interview and talk about Terraform the way the people who run it talk about it: in terms of state, credentials and blast radius.

Part three of three. At this level the problems are no longer failed plans. They are a state bucket that too many people can read, a plan that takes a quarter of an hour, a provider upgrade that wants to replace a database, and forty engineers who each learned Terraform differently. Start with the traps table, then the habits, the practice cards, the review checklists and the playbooks.

Traps that only appear at scale

Everything from Beginner and Mid still applies. These are the ones that cause incidents and audit findings rather than red pipelines.

Four traps worth a closer look

The table is for scanning. These four deserve more than a row, because each one looks harmless until the day it is not, and each one has a fix that is cheap only if you apply it early.

The plan job that could apply

This is the most serious flaw in Terraform pipelines, and it hides in plain sight. A plan is not a read-only operation in the sense security teams mean it. terraform plan starts every provider in the configuration, evaluates every data source, and can run arbitrary programs through the external data source. Anyone who can open a pull request that changes a .tf file can therefore run code inside your CI job, with whatever credentials that job holds. If the plan job and the apply job share one role, a pull request that is never reviewed or merged can still change production.

The fix is structural, not a matter of reviewing harder. Give the plan job a role that can read resources and write only the state lock. Give the apply job a separate role whose cloud trust policy accepts only the OIDC subject of your protected branch or environment. Then prove the separation: assume the plan role in a sandbox and try to create something. If it succeeds, you do not have a plan role yet.

# Run with the plan role's credentials. This should fail at the first write.
terraform -chdir=envs/sandbox apply -input=false -auto-approve

The secret that sensitive did not hide

Teams mark a variable sensitive = true, see (sensitive value) in the plan output, and conclude the secret is protected. It is not. The flag only redacts what the command line prints. The full value is still written to state, and still sits in cleartext in any saved plan file uploaded as a CI artifact. At scale this becomes an audit finding, because the state bucket is usually readable by more people than the secrets manager is.

The durable fix is to keep the value out of Terraform's files entirely: read it with an ephemeral resource and pass it to a write-only *_wo argument, where the provider supports one. The trap inside the fix is rotation. Terraform never stores a write-only value, so it cannot notice that the value changed. You must bump the matching version argument when you rotate, or the new secret never reaches the resource.

resource "aws_db_instance" "orders" {
  # ...
  password_wo         = ephemeral.aws_secretsmanager_secret_version.db.secret_string
  password_wo_version = 4   # was 3; bumped in the same PR as the rotation
}

The state everyone applies to

Every estate has one state that started small and became the place where everything goes. The symptoms arrive together: plans that take ten minutes for a one-line change because refresh is proportional to state size, pipelines that queue behind one lock, and incidents where one team's change breaks another team's service. When contention gets bad enough, someone reaches for -lock=false, and at that moment a performance problem becomes a correctness problem, because two concurrent applies can each write a state that forgets the other's work.

No flag fixes this. The fix is to split the state by owner and by how often things change, moving resources with a removed block (lifecycle { destroy = false }) in the old state and an import block in the new one, applying the destination first. Do it one slice at a time, with a state backup and a rehearsal before each slice.

The provider upgrade that wants a database

A Renovate pull request bumps a provider, and the plan now says a production database must be replaced. Nobody touched the database configuration. The cause is usually a changed default or a renamed attribute on a field that forces replacement. Reviewers who read only the summary line ("1 to add, 0 to change, 1 to destroy") may approve it as noise from a dependency bump.

Three layers stop this. prevent_destroy = true turns the replacement into a plan error. A policy on the JSON plan fails any delete action on protected resource types, which still works if someone deletes the whole resource block and the prevent_destroy setting with it. And a review habit of reading the provider's upgrade guide before approving any major bump catches the rest. Keep provider bumps small and frequent, so that when one of them does something surprising, you know which one it was.

Platform-ownership habits

A few habits deserve a paragraph each, because they are where experienced owners differ from everyone else.

Write things down before incidents, not after. A one-page runbook for a stuck lock, a partial apply, a state restore and a leaked secret takes an afternoon to write. During an incident at night it is the difference between a calm fix and a guess. Keep the runbook next to the code, and rehearse the state restore in a sandbox once a quarter so the -force push is familiar rather than frightening.

Keep a single source of truth for versions. The CLI version in CI images, required_version in root modules, provider constraints and module versions should each be set in one place and bumped by automation. When a version is copied into twenty places by hand, one of them is always wrong, and it is usually the one that runs production.

Treat modules as products with users. Publish a changelog. Use semantic versioning. Mark inputs with deprecated = "message" (available since 1.15) before removing them, so callers see warnings for a release or two. Answer requests quickly. The platform team's response time decides whether teams use the modules or fork them.

Say no to Terraform when it is the wrong tool. Application deploys, configuration inside servers and multi-step runbooks all belong elsewhere. Every one of them you let into Terraform makes plans slower, locks busier and drift noisier for everyone else.

Plan for data residency early. For teams in Saudi Arabia, the UAE, Egypt and elsewhere in the region, state and plan files are regulated data, because they hold endpoints, identities and often secrets. Decide where the backend lives, and where any hosted automation stores its files, before the first regulated workload arrives, not during the audit.

Practice cards

1. Prove the plan role cannot writeAssume the read-only plan role locally and try terraform apply on a sandbox state. It should fail at the first write call, not at the end.
2. Find the secrets in your stateRun terraform state pull | grep -iE 'password|secret|token|private_key' on a state you own. Move one of them to an ephemeral resource and a write-only argument, then run the grep again.
3. Rotate through a write-only argumentChange the secret in the secrets manager and plan: nothing changes. Bump the version argument and plan again. Remember the difference.
4. Break and fix the lock fileDelete the Linux hashes from a lock file, run init in a Linux container and read the error. Then run terraform providers lock for all platforms.
5. Release an orphaned lockKill an apply with kill -9 in a sandbox, read the lock info, and release it with terraform force-unlock.
6. Restore a state versionWith bucket versioning on, remove a resource from state, download the previous version and push it. Read the serial refusal, then push deliberately with -force and plan.
7. Move a resource between statesUse a removed block with destroy = false in one state and an import block in the other. Apply the destination first.
8. Write a delete policyExport a plan that replaces a database with show -json and write a Conftest rule that fails it.
9. Time your refreshTime a plan with and without -refresh=false on your largest state. Divide by the resource count. That number is your case for splitting.
10. Discover before you importWrite a list block in a .tfquery.hcl file for a hand-built resource type and run terraform query -generate-config-out. Trim the output until the plan shows only imports.

Review checklist for pull requests

Use this on every Terraform pull request you review. Automate the rows you can, and spend human attention on the rest.

Review checklist for design documents

Infrastructure design documents deserve their own review, because the expensive mistakes are made before any HCL exists.

Incident playbooks, condensed

Keep these next to the pipeline. Each starts with the same line.

terraform state pull > "backup-$(date -u +%Y%m%dT%H%M%SZ).tfstate"

Stuck lock. Read Who and Created in the lock info. Confirm the run is dead in CI. Then:

terraform force-unlock LOCK_ID

Partial apply. Do not finish it in the console. Re-plan, read what remains and what is tainted, and re-apply. If errored.tfstate exists, push it once the backend is reachable.

Lost or damaged state. Freeze applies. Restore from bucket versioning or re-adopt with import blocks. Plan and expect no changes before unfreezing.

Leaked secret. Rotate first. Check who read the bucket. Move the resource to ephemeral input and a write-only argument. Tighten access.

Unexpected replacement in a plan. Do not apply. Compare lock file and module versions with the last good run, find the attribute that forces replacement, and read the provider upgrade guide.

Drift alert. Decide which side is right. Bring a correct manual change into configuration, or apply to revert an incorrect one. Use terraform apply -refresh-only only when you want the record updated without touching infrastructure.

Before you call the platform done

A final list for the day you hand the platform over to the teams who will depend on it.

• Every state has one owning team, a documented purpose and a plan time you have measured.
• State is encrypted, versioned, access-logged, prefix-scoped and stored in the region your regulators expect.
• Plan and apply use separate identities from OIDC, and no human holds standing write access.
• Every pipeline applies the saved plan after policy, cost and human approval.
• The lock file has hashes for every platform, and providers come from a mirror where the network requires it.
• Modules are pinned, tested with terraform test, versioned, and deprecate before they remove.
• Protected resource types cannot be deleted without an exception, at the plan and in the cloud.
• A nightly drift plan runs on production states, and its alerts have an owner.
• The CLI and provider versions are tracked, and the next upgrade has a date.
• Runbooks exist for locks, partial applies, restores and leaked secrets, and the restore has been rehearsed.
• The boundary with Kubernetes, GitOps and the ML tooling is written down. The Kubernetes and Argo CD guides pick up from there.

If every line is true, you are running Terraform the way a platform should be run: teams ship infrastructure quickly and safely, and nobody needs to be a hero to do it.