Remote state is Terraform’s source of truth for what has been applied.

When it is designed well, state is findable, locked, protected, and recoverable. When it is an afterthought, a missing backend, a bad state key, or a stale lock turns a normal change into an incident.

This note is operational guidance for designing localStorage remote state, not another intro to backends.

Design State Around Blast Radius

Each root module should own a distinct state file scoped to a blast radius that a plan review can actually survive.

prod/network/terraform.tfstate
prod/compute/terraform.tfstate
nonprod/network/terraform.tfstate
shared/observability/terraform.tfstate

Ask per state file:

What infrastructure is in this state?
What is the smallest change this state must safely absorb?
Who can plan and apply it?
Who owns recovery if it is corrupted or deleted?

Big state files make plans slow, reviews impossible, and recovery risky. Small state files that mirror deployment units keep ownership readable.

Choose The Backend For Recovery

The minimum viable backend is one with locking and durable storage.

Common production choices:

  • AWS S3 with DynamoDB lock table.
  • Azure Blob with account-level lease and locking features.
  • GCS with object versioning.
  • HashiCorp HCP Terraform or a self-hosted TF state backend.

Whatever the choice, confirm the platform provides:

  • automatic locking during plan or apply.
  • history or versioning of state objects.
  • point-in-time recovery.
  • access that can be audited per operator.

If the backend cannot recover a bad write, it is not durable enough for shared infrastructure.

State Keys And Environments

State keys are part of the operations contract.

Use keys that include the environment and the root module:

terraform init -backend-config="key=prod/network/terraform.tfstate"

Never let two unrelated roots share one state file. A test that reuses the prod state key can tear down production with a clean plan.

Locking Is Not Optional

Confirm the backend enforces locking and that operators know the lock commands:

terraform plan     # acquires a lock
terraform apply    # acquires a lock
terraform force-unlock <lock-id>   # only after confirming no active run

A missing lock table is a race between two applies. The safest remediation is to build the lock resource before granting apply access to a second person.

Access Control

State contains sensitive values and identifiers. Protect it like the infrastructure it describes.

Minimum controls:

  • state access limited to the automation and operators that need it.
  • write access separated from read access where possible.
  • state read access not granted to every developer.
  • state buckets and containers private.
  • state access logged for investigation.

If a developer needs to see state for troubleshooting, grant read and document that state is sensitive. Do not hand out blanket access because it is convenient.

Backup And Recovery Points

Validate recovery before you need it:

# list backend state files for the environment
terraform state list

Some versions of a backend keep an up-to-date copy; some keep versions. Know which one you have.

Before risky changes or migrations, snapshot the state:

terraform state pull > before-change.tfstate

Store the snapshot outside the workspace. After a change, a comparison against the snapshot tells you what moved and whether it was intended.

Common Failure Modes

Stale Or Missing Backend

The root module runs against local state while the team believes state is remote. Two operators plan against different truths, and the first merge wins.

Lock Table Missing Or Broken

Two applies run at once. Both read the same state and one overwrites the other’s changes. Infrastructure and state disagree.

Stale Force-Unlock

An operator force-unlocks a lock that belongs to a still-running apply, then the two applies collide. Only force-unlock after confirming no active run.

Corrupt State File

A bad write or interrupted apply leaves a state object that fails to load. Recovery depends on a backend version or backup that nobody verified.

Overly Broad Access

Every operator and pipeline has write access to every state file. Anyone can plan a delete against any environment without an explicit ownership check.

Recovery Checklist

  • Is the backend’s locking mechanism actually configured?
  • Are state keys scoped by environment and root module?
  • Is state access limited and auditable?
  • Are backups or versions verified for the state files that matter?
  • Do operators know state pull, state push, and force-unlock behavior?
  • Is local state excluded from Git and pipelines?
  • Is recovery of the most business-critical state tested?

Practical Takeaway

Remote state is production data, and its design is a recovery decision.

Scope state to blast radius, enable locking, protect access, and verify recovery before the incident. A state file the team can find, lock, and restore is the difference between a bad apply and a long outage.

References