Kubernetes And Rancher Field Notes

During a vSphere CNS backlog, stopping the source of new CSI work can be safer than continuing to submit attach, detach, resize, and update requests into a stuck backend queue. The trap is that GitOps may immediately undo the pause. Scaling a vSphere CSI controller Deployment to 0 is not a durable pause if Argo CD, an ApplicationSet, or a higher root Application owns it. The visible command succeeds, then self-heal restores the replicas and new controller pods start submitting tasks again. ...

A pod stuck in Pending because a PVC does not exist is not always a provisioning problem. Sometimes the data still exists in a retained PV and only the claim object is missing. The safe recovery is to prove the PV is the intended backing store, then recreate the PVC with an explicit volumeName so it binds to that retained PV instead of provisioning something new. Confirm The Symptom Start with the pod event: ...

A vSphere CSI controller log that says a CNS ExtendVolume task is pending does not immediately tell you whether the workload is blocked by resize, attachment, mount, or a stale backend task. The first job is to map the CNS volume ID back to Kubernetes state and avoid turning a storage delay into a destructive rollback. The useful triage order is: CNS volume ID -> PV -> PVC -> pod -> VolumeAttachment -> CSI controller logs -> vCenter task state Map The CNS Volume ID Start by mapping the reported CNS volume ID to the Kubernetes PV and PVC: ...

When Kubernetes nodes reboot during a storage incident, the cause matters. An in-guest reboot, a Rancher/system-upgrade action, a human SSH session, and a vSphere HA reset all have different follow-up work. The useful pattern is to prove the reboot from multiple layers before assigning cause. Start With Kubernetes Check node readiness, boot ID, kernel, and recent events: kubectl get nodes -o wide kubectl describe node cp-1 | grep -A5 -E 'Conditions:|Events:' kubectl get events -A --sort-by=.lastTimestamp | grep -i 'reboot\|node' kubectl get node cp-1 -o jsonpath='{.status.nodeInfo.bootID}{"\n"}{.status.nodeInfo.kernelVersion}{"\n"}' Kubernetes can tell you that a node rebooted. It usually cannot tell you why. ...

Replacing Kubernetes nodes from a fresh OS template can be faster than repairing legacy VM drift in place, but speed only helps if the risky work is moved out of the maintenance window without creating identity conflicts. The useful rehearsal pattern is to pre-create replacement VMs from the current template, leave them powered off, and join them one at a time during the window. That turns the maintenance window into a controlled cluster-change sequence instead of a race to clone, customize, debug, and drain all at once. ...

Longhorn has a CLI, longhornctl, but installing it should not change the operational source of truth during maintenance. For Rancher-managed Longhorn clusters, the most useful day-to-day state still usually comes from the Longhorn UI and Kubernetes CRDs such as volumes.longhorn.io, replicas.longhorn.io, nodes.longhorn.io, engines.longhorn.io, and settings.longhorn.io. The practical boundary is simple: use longhornctl as an operator tool, not as a reason to skip in-cluster evidence. Install Locally For an Ubuntu workstation, a user-local install avoids system-wide package changes. Confirm the workstation architecture first: ...

After an RKE2 upgrade or node maintenance event, Calico can fail readiness even when the node itself is Ready. One failure pattern is node-local and easy to miss: old Calico or Typha processes survive from a previous RKE2 runtime path and keep owning the ports that the current pods need. Kubernetes shows the new pods as unhealthy, but the real conflict is on the host. This is not a YAML problem first. It is a process ownership problem. ...

Certificate automation is not finished when the first certificate is issued. The operational work is the lifecycle: issuer health, renewal timing, DNS or HTTP challenge reliability, secret ownership, expiration alerts, and safe rotation. This field note focuses on cert-manager as a Kubernetes platform component, not just as an application dependency. Define Certificate Ownership Every production certificate should have a clear owner. Track: hostname. namespace. owning application or platform team. Certificate resource name. Issuer or ClusterIssuer. challenge type. backing secret. renewal policy. alert route. If a certificate expires and nobody knows who owns it, the platform has an ownership problem, not only a TLS problem. ...

Cluster autoscaler is capacity automation, not a replacement for capacity ownership. It can add nodes when pods cannot schedule and remove nodes when capacity is unused, but it only works inside the boundaries the platform team gives it: node groups, quotas, labels, taints, pod requests, disruption budgets, and cloud or virtualization capacity. This review is for platform teams that need to know whether autoscaling is safe, predictable, and observable. Start With The Capacity Contract Document the autoscaling boundary. ...

Ingress is where Kubernetes platform problems become visible to users. The application may be healthy, pods may be ready, and services may have endpoints, but a bad ingress change can still break the user path. Treat ingress as a production edge, not just another manifest. This checklist is for platform teams operating ingress controllers, DNS records, TLS certificates, and application routes across Kubernetes clusters. Define The Ingress Contract Before troubleshooting an ingress issue, write down the expected path. ...