Field Notes

“Vault is unreachable” was true, but it was not specific enough. The useful part of this incident was separating two failure domains that overlapped in time: Vault nodes that needed unseal after an infrastructure event, and client traffic that was still being reset after Vault was healthy internally. Incident Shape The incident context had two independent changes close together: 1. A rack power event affected the ESXi hosts running Vault VMs. 2. Firewall policy was tightened by removing a broad allow rule. After the VMs came back, Vault behaved like Vault should after restart when it uses Shamir sealing: nodes needed to be unsealed before they were usable. ...

Updating Kubernetes nodes is not just an apt-get dist-upgrade loop. The safe boundary is different for package download staging, package application, reboot, etcd quorum, CSI detach behavior, and Longhorn replica health. A node can be Ready and still be the wrong node to reboot next. Separate Staging From Applying Use download-only staging when the goal is to warm package caches before a maintenance window: sudo bash -lc ' set -euo pipefail export DEBIAN_FRONTEND=noninteractive apt-get -o DPkg::Lock::Timeout=300 update -qq apt-get -y \ -o DPkg::Lock::Timeout=300 \ --download-only \ -o Dpkg::Options::=--force-confdef \ -o Dpkg::Options::=--force-confold \ dist-upgrade echo "STAGED_DOWNLOAD_ONLY rc=0" echo "upgradable_count=$(apt list --upgradable 2>/dev/null | grep -c upgradable)" echo "cache_mb=$(du -sm /var/cache/apt/archives 2>/dev/null | awk '\''{print $1}'\'')" echo "reboot_required=$(test -f /var/run/reboot-required && echo YES || echo no)" ' After download-only staging, apt list --upgradable can still show the same updates. That is expected. The packages were downloaded, not installed. ...

The disk had space. Longhorn still could not place the replicas. That was the useful part of the incident. The first read looked like an attached volume that had not finished rebuilding. The evidence pointed somewhere else: physical free space was not the same as capacity available to the Longhorn scheduler. Incident Shape The affected Longhorn volume was attached and degraded: volume: pvc-35d1949c-4a61-448d-88b9-59b30c823c81 state: attached robustness: degraded current node: worker-4 size: 53687091200 Its scheduled condition was false: ...

expect_disconnect is useful, but it is not a broom. When a Packer vsphere-iso build fails with Script disconnected unexpectedly, the important question is whether the command already completed before SSH dropped. If the command was interrupted mid-package-install, telling Packer to expect the disconnect can turn a real partial install into a green build step. Symptom The build reaches a shell provisioner and fails after cloud-init finishes: Provisioning with shell script: /tmp/packer-shell123456789 Cloud-init finished... Synchronizing state of open-vm-tools.service with SysV service script... Executing: /usr/lib/systemd/systemd-sysv-install enable open-vm-tools Script disconnected unexpectedly. The template may build successfully in one vSphere environment and fail in another because network convergence and VMXNET3 behavior differ by host, port group, or timing. Do not assume the template is safe just because a sibling environment happened to tolerate the restart. ...

Log collection is workload traffic. When Grafana Alloy runs as a DaemonSet without a CPU limit, a log storm, destination retry loop, or backlog catch-up can consume most of a worker node. In virtualized clusters, that can also saturate the ESXi hosts underneath the cluster. Symptom The host-level signal may look like a vSphere capacity issue first: esxi-a CPU 95-105% esxi-b CPU 90-100% esxi-c CPU 5% Kubernetes then points at a smaller set of nodes: ...

A Terraform plan can look like a small in-place update and still contain dangerous vSphere operations. When live VMs have been moved by incident response, storage maintenance, DRS, or a CSI controller, Terraform state may lag behind vCenter reality. A normal apply can try to move everything back, detach disks, or rewrite placement while you only meant to update a harmless metadata value. Situation The intended change was narrow: guestinfo.network-config: netmask /16 -> /22 The first plan was not narrow. It also included: ...

A freshly downloaded kubeconfig can be valid and still fail when kubectl uses it. One failure pattern is duplicate user names across a merged KUBECONFIG. Rancher-generated kubeconfigs often use the same user name, such as rancher, across multiple clusters or downloaded files. When several kubeconfig files are merged, the first user with that name can win. A stale token from an older file can shadow the fresh token from the file you just downloaded. ...

An upgrade Plan can be syntactically correct and still create pods that can never schedule. One recurring pattern is a Plan that excludes control-plane and etcd nodes, but unintentionally includes monitor or infrastructure nodes with custom taints. The controller creates Jobs for those nodes. The scheduler rejects the pods. The Jobs age out with DeadlineExceeded. The cluster keeps generating warning events even though the nodes are healthy. The bug is not in the node. It is in the relationship between Plan selection and node taints. ...

Not every FailedMount event means a pod is down. In one cluster, several pods were Running and Ready, but kubelet kept emitting vSphere CSI mount warnings for weeks. The warnings had this shape: MountVolume.MountDevice failed for volume "pvc-<uid>": mount failed: /dev/mapper/mpathX already mounted on /var/lib/kubelet/plugins/kubernetes.io/csi/csi.vsphere.vmware.com/<volume-id>/globalmount The workload was not currently down. The node had stale storage reconciliation state. Split Current Impact From Event Noise Start with the workload, not the event text: ...

External Secrets Operator adds a useful boundary: applications can consume Kubernetes Secret objects while the source value lives in a dedicated secret manager. It also adds a new failure path. A backend token, SecretStore, controller, target Secret, and consuming workload all have to agree before rotation is actually safe. Ownership Model Write down the owner for each layer: backend secret manager -> source value and access policy SecretStore -> authentication and provider configuration ExternalSecret -> mapping from backend key to Kubernetes Secret target Secret -> Kubernetes object consumed by workloads application workload -> reload behavior and verification platform team -> controller health, RBAC, namespaces, and alerts If an ExternalSecret fails, do not assume it is an application bug or a secret-manager outage. Walk the chain. ...