vCenter Operations Field Notes

Use this checklist when a Kubernetes workload has a PVC that should use vSphere CSI, but the pod is pending, stuck in init, or failing to mount the volume. For resize-specific CNS task triage, see vSphere CSI CNS ExtendVolume Triage. If the only visible symptom is ContainerCreating, first split storage failures from missing ConfigMap or Secret dependencies; see Kubernetes ContainerCreating: Split Storage Failures From Missing Manifests. If GitOps keeps restarting CSI controllers during a storage freeze, see GitOps-Owned vSphere CSI Maintenance Pauses. If pods are running but kubelet keeps reporting already mounted errors through /dev/mapper/mpath*, inspect host multipath policy before blaming the workload; see vSphere CSI Mount Loops Can Be Stale Multipath State. ...

govc is useful when Terraform says one thing and the VM behaves like something else. It gives a direct view into vSphere inventory, VM paths, NIC backing, and extra config. For virtual CD-ROM device cleanup, see vSphere CD-ROM Host Device Cleanup With govc. Load Environment Use a clearly named helper file, not vars.env if it is only for govc: source govc.env govc about Typical values: export GOVC_URL='vcenter.example.com' export GOVC_USERNAME='administrator@vsphere.local' export GOVC_PASSWORD='...' export GOVC_INSECURE=1 export GOVC_DATACENTER='DC-Site-A' Find A VM Path Search by VM name: ...

A Kubernetes node can be storage-ready before any new disk appears in lsblk. With CSI-backed storage, the disk appears only after the array maps a LUN to the node. Target State For a node with primary and iSCSI networks: ens192 -> service network, default route, DNS iscsi01 -> 169.253.0.x/24 iscsi02 -> 169.253.1.x/24 The node should have: deterministic netplan. reachable storage portals. open-iscsi installed and enabled. an initiator name. successful target discovery. persistent iSCSI node startup. Verify Network First ip -br addr ip route ping -c 3 169.253.0.11 ping -c 3 169.253.1.11 If portal pings fail, stop and fix networking before touching iSCSI. ...

VMware guest customization can write /etc/netplan/99-netcfg-vmware.yaml. If the template already has another active netplan file, the host can end up with multiple network authorities. That can break routing, DNS, bootstrap scripts, Kubernetes node communication, and storage initialization. Symptoms Common signs: Error: Conflicting default route declarations for IPv4 first declared in nic0 but also in ens192 or: Destination Host Unreachable or DNS search drift like: resolv_conf_search = . netplan_search = [] Inspect Netplan Ownership ls -l /etc/netplan sudo grep -R "routes:\|gateway\|search:" /etc/netplan -n If both a template file and VMware file are present, decide which one owns networking. ...

When an existing Terraform resource is moved into a module, Terraform sees a new resource address. If state is not moved first, the plan may try to create the module resource and destroy the old root resource. Symptom After converting a root resource to a module call, the plan shows addresses like this: # module.vm_group.vsphere_virtual_machine.vm["lb1"] will be created while the existing state still has: vsphere_virtual_machine.vm["lb1"] Check Current State Run from the environment root: ...

Terraform can successfully ask vSphere to clone a VM and still fail during guest customization. In that case, the problem is usually inside the guest/template, not Terraform syntax. Symptom Terraform reports an error like: Virtual machine customization failed An error occurred while customizing VM ... For details reference the log file /var/log/vmware-imc/toolsDeployPkg.log in the guest OS. Terraform may leave the VM in place to help troubleshooting. First Checks From the VM console or SSH if available: ...

Terraform can inject cloud-init data into vSphere clones through VMware guestinfo keys. That only works if the template is built to let cloud-init run on first boot. Symptom Terraform creates or replaces the VM, but the expected userdata.yaml behavior does not happen. Bootstrap does not run, SSH is not configured, or /var/log/platform-bootstrap.log is missing. Check Guestinfo From vSphere Use govc to inspect the VM extra config: source govc.env govc vm.info -e /DC-Site-A/vm/K8s-Cluster/OpsTools/cluster-a-api-lb-01 \ | grep 'guestinfo\|disk.enableUUID' Look for keys like: ...