vCenter Operations Field Notes

A Packer template can contain a bootstrap framework without owning every piece of clone behavior. The boundary is sealing. A successful packer build produces a configured machine; it does not, by itself, prove that the resulting artifact is safe to clone. Sealing is the lifecycle stage where a configured build machine is deliberately converted into a reusable artifact, and ownership of state changes hands: configured machine -> sanitized machine -> sealed template -> clone -> uniquely identified machine That makes the ownership boundaries explicit: ...

During a vSphere CNS backlog, stopping the source of new CSI work can be safer than continuing to submit attach, detach, resize, and update requests into a stuck backend queue. The trap is that GitOps may immediately undo the pause. Scaling a vSphere CSI controller Deployment to 0 is not a durable pause if Argo CD, an ApplicationSet, or a higher root Application owns it. The visible command succeeds, then self-heal restores the replicas and new controller pods start submitting tasks again. ...

A connected or host-backed virtual CD-ROM can become operational noise during VM maintenance. It can trigger device-lock prompts, confuse vMotion or storage work, and leave operators answering vCenter questions that have nothing to do with the actual change. If the VM is already locked by storage or vMotion tasks, clear the task contention first; see vSphere CSI CNS ExtendVolume Triage. For the incident-level journal on why CD-ROM cleanup should be treated as Kubernetes maintenance, see Journal: When Storage Triage Turns Into Platform Maintenance. If live VM reconfigure causes Kubernetes symptoms, use a power-cycle-first workflow with role gates instead; see vSphere CD-ROM Cleanup With Power-Cycle Gates. ...

A vSphere CSI controller log that says a CNS ExtendVolume task is pending does not immediately tell you whether the workload is blocked by resize, attachment, mount, or a stale backend task. The first job is to map the CNS volume ID back to Kubernetes state and avoid turning a storage delay into a destructive rollback. For the broader maintenance journal that ties this pattern to GitOps containment, vCenter task queues, VM device cleanup, and stop conditions, see Journal: When Storage Triage Turns Into Platform Maintenance. ...

When Kubernetes nodes reboot during a storage incident, the cause matters. An in-guest reboot, a Rancher/system-upgrade action, a human SSH session, and a vSphere HA reset all have different follow-up work. The useful pattern is to prove the reboot from multiple layers before assigning cause. Start With Kubernetes Check node readiness, boot ID, kernel, and recent events: kubectl get nodes -o wide kubectl describe node cp-1 | grep -A5 -E 'Conditions:|Events:' kubectl get events -A --sort-by=.lastTimestamp | grep -i 'reboot\|node' kubectl get node cp-1 -o jsonpath='{.status.nodeInfo.bootID}{"\n"}{.status.nodeInfo.kernelVersion}{"\n"}' Kubernetes can tell you that a node rebooted. It usually cannot tell you why. ...

Replacing Kubernetes nodes from a fresh OS template can be faster than repairing legacy VM drift in place, but speed only helps if the risky work is moved out of the maintenance window without creating identity conflicts. The useful rehearsal pattern is to pre-create replacement VMs from the current template, leave them powered off, and join them one at a time during the window. That turns the maintenance window into a controlled cluster-change sequence instead of a race to clone, customize, debug, and drain all at once. ...

A Packer template build can fail in three different places that look similar from the outside: Packer never reaches SSH, so file and shell provisioners never run. Packer places bootstrap files into the template, but does not execute runtime bootstrap. Terraform/cloud-init clones the VM but does not start the bootstrap entrypoint correctly. Do not diagnose all three as “bootstrap did not work.” Ask which layer failed. Placement Is Packer’s Job For a reusable vSphere template, Packer should place static resources only. For the follow-on sealing pattern after the bootstrap payload is placed and validated, see Packer Template Sealing After Clone-Time Bootstrap. ...

Attaching a second vSphere disk is not the same as using it. In one worker replacement, Terraform correctly attached a second disk to the VM, Packer correctly placed the static bootstrap entrypoint, and cloud-init completed. But the guest still showed the second disk as blank and unmounted: sda sda1 /boot/efi sda2 / sdb <blank> The result was subtle: bootstrap succeeded, but /var/lib/rancher stayed on the operating-system disk. For an RKE2 node, that means Rancher/RKE2 state and container log growth can still fill / even though Terraform created a data disk. ...

Copied Terraform environment roots are convenient until they drift. One root has a newer module call, another has an old variable name, a third has stale README instructions, and the next environment starts from whichever directory someone copied last. Use a small scaffolding script when a repository has a standard root-module shape. What To Generate For a vSphere environment root, generate the complete directory shape every time: environments/site-a/example/ main.tf variables.tf locals.tf outputs.tf terraform.tfvars README.md The script should create files that are immediately recognizable to operators: ...

Targeted Terraform applies are a sharp tool. They are not a normal workflow, but they are sometimes the safer option when a live cluster needs a narrow expansion and the full plan contains unrelated refactor drift. This note covers the pattern for adding a small set of new monitor nodes to an existing RKE2 cluster while avoiding changes to existing etcd, control-plane, worker, and load balancer VMs. Situation The environment already had Terraform-managed vSphere VMs: ...