Terraform Change Review Field Notes

Replacing Kubernetes nodes from a fresh OS template can be faster than repairing legacy VM drift in place, but speed only helps if the risky work is moved out of the maintenance window without creating identity conflicts. The useful rehearsal pattern is to pre-create replacement VMs from the current template, leave them powered off, and join them one at a time during the window. That turns the maintenance window into a controlled cluster-change sequence instead of a race to clone, customize, debug, and drain all at once. ...

A Packer template build can fail in three different places that look similar from the outside: Packer never reaches SSH, so file and shell provisioners never run. Packer places bootstrap files into the template, but does not execute runtime bootstrap. Terraform/cloud-init clones the VM but does not start the bootstrap entrypoint correctly. Do not diagnose all three as “bootstrap did not work.” Ask which layer failed. Placement Is Packer’s Job For a reusable vSphere template, Packer should place static resources only: ...

Attaching a second vSphere disk is not the same as using it. In one worker replacement, Terraform correctly attached a second disk to the VM, Packer correctly placed the static bootstrap entrypoint, and cloud-init completed. But the guest still showed the second disk as blank and unmounted: sda sda1 /boot/efi sda2 / sdb <blank> The result was subtle: bootstrap succeeded, but /var/lib/rancher stayed on the operating-system disk. For an RKE2 node, that means Rancher/RKE2 state and container log growth can still fill / even though Terraform created a data disk. ...

Copied Terraform environment roots are convenient until they drift. One root has a newer module call, another has an old variable name, a third has stale README instructions, and the next environment starts from whichever directory someone copied last. Use a small scaffolding script when a repository has a standard root-module shape. What To Generate For a vSphere environment root, generate the complete directory shape every time: environments/site-a/example/ main.tf variables.tf locals.tf outputs.tf terraform.tfvars README.md The script should create files that are immediately recognizable to operators: ...

Targeted Terraform applies are a sharp tool. They are not a normal workflow, but they are sometimes the safer option when a live cluster needs a narrow expansion and the full plan contains unrelated refactor drift. This note covers the pattern for adding a small set of new monitor nodes to an existing RKE2 cluster while avoiding changes to existing etcd, control-plane, worker, and load balancer VMs. Situation The environment already had Terraform-managed vSphere VMs: ...

Service Level Indicators (SLIs) are easier to understand when they are tied to a working service. Abstract definitions are useful, but a small lab makes the tradeoffs visible: what counts as success, what counts as failure, how latency should be measured, and how trends can reveal degradation before a full incident. This field note uses a companion lab in GitHub: https://github.com/trinidadgithub/IaC/tree/main/sli_app The lab runs a small Flask application, exposes Prometheus metrics, provisions Prometheus and Grafana with Terraform, and includes a basic SLI dashboard. It also introduces lightweight data science habits: percentiles, rolling windows, error-rate comparison, and avoiding misleading averages. ...

Terraform modules are easier to consume when engineers can understand their inputs without reading every line of source code. While reviewing an AWS Terraform Kinesis module, I created an input summary table with six columns: Input, Type, Default Value, Required, Notes, and Recommendation. The goal was simple: make the module safer and faster to use by turning variable definitions into an operator-friendly interface. The Problem This Solves Terraform modules often start clean and become harder to consume over time. Inputs are added for new capabilities, defaults change, conditional behavior grows, and security-sensitive options become mixed with ordinary configuration. The source code still contains the truth, but consuming the module requires reading variables.tf, resource blocks, locals, conditionals, and sometimes provider documentation. ...

A VM can have the correct FQDN intent and still receive the wrong resolver search suffix. The trap is treating these as the same setting: vm_domain -> identity/FQDN domain dns_search -> resolver search suffix list They are related, but they are not the same control. Symptom An environment sets DNS search suffixes to empty: dns_search = "[]" But new vSphere VMs still boot with a resolver search domain such as: search corp.example.com The node audit shows drift even though the Terraform input looked correct: ...

Importing existing vSphere VMs into Terraform can produce a clean source-of-truth checkpoint and still leave a plan that should not be applied. That is common when legacy Kubernetes nodes were built from an older template or outside the current module conventions. Audit Checkpoint A useful checkpoint looks like this: NetBox resources: no-op vSphere resources: update destroy actions: none This means NetBox ownership is reconciled, but Terraform still sees vSphere drift. ...

After existing vSphere VMs are imported into Terraform state, expect drift. The question is not whether drift exists. The question is whether applying that drift is safe. Generate The Audit Plan terraform plan -out=audit.tfplan terraform show -json audit.tfplan \ | jq -r '.resource_changes[]? | [.address, .type, (.change.actions | join(","))] | @tsv' Start with action types: no-op update create delete delete,create For imported production-like nodes, any delete, create, or delete,create action needs explicit review before apply. ...