Terraform Change Review Field Notes

A Terraform plan can look like a small in-place update and still contain dangerous vSphere operations. When live VMs have been moved by incident response, storage maintenance, DRS, or a CSI controller, Terraform state may lag behind vCenter reality. A normal apply can try to move everything back, detach disks, or rewrite placement while you only meant to update a harmless metadata value. Situation The intended change was narrow: guestinfo.network-config: netmask /16 -> /22 The first plan was not narrow. It also included: ...

Module composition is where Terraform repos go from readable to unmaintainable. A module is a way to package decisions and raise the abstraction level. Done well, it reduces duplication and concentrates behavior. Done badly, it hides ownership, buries for_each keys, and turns every apply into a refactor. This note covers composing modules so the ownership boundaries stay visible. What A Module Should Own A module should own one operational concept and expose the minimum contract needed to use it. ...

Provider versions are pinned in required_providers and installed from a registry or mirror, but the operational work happens when the version moves. An upgraded provider can reformat state, change attributes, or force replacement. A versioned provider that nobody upgrades drifts until a forced move happens mid-incident or mid-release. This note is the operational discipline for keeping provider versions intentional. Pin Deliberately Every provider that matters should have an explicit constraint and a recorded version. ...

Remote state is Terraform’s source of truth for what has been applied. When it is designed well, state is findable, locked, protected, and recoverable. When it is an afterthought, a missing backend, a bad state key, or a stale lock turns a normal change into an incident. This note is operational guidance for designing localStorage remote state, not another intro to backends. Design State Around Blast Radius Each root module should own a distinct state file scoped to a blast radius that a plan review can actually survive. ...

A Packer template can contain a bootstrap framework without owning every piece of clone behavior. The boundary is sealing. A successful packer build produces a configured machine; it does not, by itself, prove that the resulting artifact is safe to clone. Sealing is the lifecycle stage where a configured build machine is deliberately converted into a reusable artifact, and ownership of state changes hands: configured machine -> sanitized machine -> sealed template -> clone -> uniquely identified machine That makes the ownership boundaries explicit: ...

Replacing Kubernetes nodes from a fresh OS template can be faster than repairing legacy VM drift in place, but speed only helps if the risky work is moved out of the maintenance window without creating identity conflicts. The useful rehearsal pattern is to pre-create replacement VMs from the current template, leave them powered off, and join them one at a time during the window. That turns the maintenance window into a controlled cluster-change sequence instead of a race to clone, customize, debug, and drain all at once. ...

A Packer template build can fail in three different places that look similar from the outside: Packer never reaches SSH, so file and shell provisioners never run. Packer places bootstrap files into the template, but does not execute runtime bootstrap. Terraform/cloud-init clones the VM but does not start the bootstrap entrypoint correctly. Do not diagnose all three as “bootstrap did not work.” Ask which layer failed. Placement Is Packer’s Job For a reusable vSphere template, Packer should place static resources only. For the follow-on sealing pattern after the bootstrap payload is placed and validated, see Packer Template Sealing After Clone-Time Bootstrap. ...

Attaching a second vSphere disk is not the same as using it. In one worker replacement, Terraform correctly attached a second disk to the VM, Packer correctly placed the static bootstrap entrypoint, and cloud-init completed. But the guest still showed the second disk as blank and unmounted: sda sda1 /boot/efi sda2 / sdb <blank> The result was subtle: bootstrap succeeded, but /var/lib/rancher stayed on the operating-system disk. For an RKE2 node, that means Rancher/RKE2 state and container log growth can still fill / even though Terraform created a data disk. ...

Copied Terraform environment roots are convenient until they drift. One root has a newer module call, another has an old variable name, a third has stale README instructions, and the next environment starts from whichever directory someone copied last. Use a small scaffolding script when a repository has a standard root-module shape. What To Generate For a vSphere environment root, generate the complete directory shape every time: environments/site-a/example/ main.tf variables.tf locals.tf outputs.tf terraform.tfvars README.md The script should create files that are immediately recognizable to operators: ...

Targeted Terraform applies are a sharp tool. They are not a normal workflow, but they are sometimes the safer option when a live cluster needs a narrow expansion and the full plan contains unrelated refactor drift. This note covers the pattern for adding a small set of new monitor nodes to an existing RKE2 cluster while avoiding changes to existing etcd, control-plane, worker, and load balancer VMs. Situation The environment already had Terraform-managed vSphere VMs: ...