vCenter Operations Field Notes

During a vSphere CNS backlog, stopping the source of new CSI work can be safer than continuing to submit attach, detach, resize, and update requests into a stuck backend queue. The trap is that GitOps may immediately undo the pause. Scaling a vSphere CSI controller Deployment to 0 is not a durable pause if Argo CD, an ApplicationSet, or a higher root Application owns it. The visible command succeeds, then self-heal restores the replicas and new controller pods start submitting tasks again. ...

A connected or host-backed virtual CD-ROM can become operational noise during VM maintenance. It can trigger device-lock prompts, confuse vMotion or storage work, and leave operators answering vCenter questions that have nothing to do with the actual change. If the VM is already locked by storage or vMotion tasks, clear the task contention first; see vSphere CSI CNS ExtendVolume Triage. For long-lived Kubernetes nodes and platform VMs, a CD-ROM is often unnecessary after provisioning. If one remains, it should be either disconnected as a client device or removed intentionally. ...

A vSphere CSI controller log that says a CNS ExtendVolume task is pending does not immediately tell you whether the workload is blocked by resize, attachment, mount, or a stale backend task. The first job is to map the CNS volume ID back to Kubernetes state and avoid turning a storage delay into a destructive rollback. The useful triage order is: CNS volume ID -> PV -> PVC -> pod -> VolumeAttachment -> CSI controller logs -> vCenter task state Map The CNS Volume ID Start by mapping the reported CNS volume ID to the Kubernetes PV and PVC: ...

When Kubernetes nodes reboot during a storage incident, the cause matters. An in-guest reboot, a Rancher/system-upgrade action, a human SSH session, and a vSphere HA reset all have different follow-up work. The useful pattern is to prove the reboot from multiple layers before assigning cause. Start With Kubernetes Check node readiness, boot ID, kernel, and recent events: kubectl get nodes -o wide kubectl describe node cp-1 | grep -A5 -E 'Conditions:|Events:' kubectl get events -A --sort-by=.lastTimestamp | grep -i 'reboot\|node' kubectl get node cp-1 -o jsonpath='{.status.nodeInfo.bootID}{"\n"}{.status.nodeInfo.kernelVersion}{"\n"}' Kubernetes can tell you that a node rebooted. It usually cannot tell you why. ...

Replacing Kubernetes nodes from a fresh OS template can be faster than repairing legacy VM drift in place, but speed only helps if the risky work is moved out of the maintenance window without creating identity conflicts. The useful rehearsal pattern is to pre-create replacement VMs from the current template, leave them powered off, and join them one at a time during the window. That turns the maintenance window into a controlled cluster-change sequence instead of a race to clone, customize, debug, and drain all at once. ...

A Packer template build can fail in three different places that look similar from the outside: Packer never reaches SSH, so file and shell provisioners never run. Packer places bootstrap files into the template, but does not execute runtime bootstrap. Terraform/cloud-init clones the VM but does not start the bootstrap entrypoint correctly. Do not diagnose all three as “bootstrap did not work.” Ask which layer failed. Placement Is Packer’s Job For a reusable vSphere template, Packer should place static resources only: ...

Attaching a second vSphere disk is not the same as using it. In one worker replacement, Terraform correctly attached a second disk to the VM, Packer correctly placed the static bootstrap entrypoint, and cloud-init completed. But the guest still showed the second disk as blank and unmounted: sda sda1 /boot/efi sda2 / sdb <blank> The result was subtle: bootstrap succeeded, but /var/lib/rancher stayed on the operating-system disk. For an RKE2 node, that means Rancher/RKE2 state and container log growth can still fill / even though Terraform created a data disk. ...

Copied Terraform environment roots are convenient until they drift. One root has a newer module call, another has an old variable name, a third has stale README instructions, and the next environment starts from whichever directory someone copied last. Use a small scaffolding script when a repository has a standard root-module shape. What To Generate For a vSphere environment root, generate the complete directory shape every time: environments/site-a/example/ main.tf variables.tf locals.tf outputs.tf terraform.tfvars README.md The script should create files that are immediately recognizable to operators: ...

Targeted Terraform applies are a sharp tool. They are not a normal workflow, but they are sometimes the safer option when a live cluster needs a narrow expansion and the full plan contains unrelated refactor drift. This note covers the pattern for adding a small set of new monitor nodes to an existing RKE2 cluster while avoiding changes to existing etcd, control-plane, worker, and load balancer VMs. Situation The environment already had Terraform-managed vSphere VMs: ...

A VM can have the correct FQDN intent and still receive the wrong resolver search suffix. The trap is treating these as the same setting: vm_domain -> identity/FQDN domain dns_search -> resolver search suffix list They are related, but they are not the same control. Symptom An environment sets DNS search suffixes to empty: dns_search = "[]" But new vSphere VMs still boot with a resolver search domain such as: search corp.example.com The node audit shows drift even though the Terraform input looked correct: ...