Kubernetes And Rancher Field Notes

PostgreSQL operators make cluster lifecycle easier, but they do not remove the need to prove backup and failover behavior. A healthy operator can still manage a database whose backups are stale, replicas lag, clients cannot reconnect, or restores have never been tested. Backup Readiness For PostgreSQL, backup readiness has two parts: a base backup and the ability to recover to an accepted point using WAL or the operator’s equivalent archive mechanism. ...

A service mesh should solve a named operating problem. It should not appear because the platform has reached a maturity checklist. Meshes add a control plane, sidecars or node proxies, certificates, traffic policy, telemetry, and failure modes that did not exist before. The trade can be worth it. It needs an ownership model before the first namespace is enrolled. Name The Problem First Good mesh adoption starts with a concrete reason: ...

mTLS is often the reason teams adopt a service mesh. It is also where a mesh stops being invisible plumbing. The goal is not “turn on strict mode.” The goal is to prove that service identity, policy, and traffic behavior match the application’s real dependency graph. Inventory Service Calls Before enforcing policy, map the request path: source workload -> service -> destination workload -> external dependency Capture: namespace and service account for each workload. protocols and ports. internal and external dependencies. readiness and liveness probe paths. jobs, cronjobs, and maintenance callers. traffic that bypasses Kubernetes Service objects. Do not enforce mTLS from memory. Hidden callers become outage reports. ...

Service mesh incidents are request-path incidents. Debug them like one. The mesh adds useful telemetry, but it also adds another place where a request can fail. Start by proving whether the failure is application, ingress, Service/endpoints, NetworkPolicy, mesh policy, proxy health, or certificate state. Baseline The Path Write the expected path before changing anything: client -> ingress/load balancer -> source workload proxy -> destination service -> destination workload proxy -> application Then check the non-mesh objects first: ...

Kubernetes cost allocation fails when metadata is treated as a billing afterthought. The cluster already knows a lot about workload ownership: namespaces, labels, annotations, service accounts, node groups, storage classes, and ingress objects. FinOps starts when that metadata is consistent enough to map spend back to teams and services without a spreadsheet archaeology project. Start With Required Labels Define a small required set for namespaces first: owner service environment cost-center lifecycle Keep the list short. Labels that are never validated become decorative noise. ...

Kubernetes cost optimization usually starts with requests, not invoices. The scheduler reserves capacity based on CPU and memory requests. Cluster autoscaler responds to unschedulable pods based on those requests. If requests are inflated, nodes are added early and stay underused. If requests are too low, workloads run cheap until they become noisy, evicted, or throttled. Rightsizing is a reliability exercise with a cost outcome. Review Inputs Do not rightsize from a single dashboard screenshot. ...

Cross-region failover is not proven by a standby environment existing. It is proven by the standby environment accepting traffic with the expected identity, data, capacity, and observability. The hard part is not usually creating the second region. The hard part is proving it can become primary without discovering missing assumptions during the incident. Readiness Domains Review failover readiness by domain: routing and ingress. DNS ownership and TTL behavior. certificates and trust chains. identity provider reachability. secrets availability and rotation path. data replication freshness. compute and storage capacity. observability and alert routing. operator access to the recovery region. return-to-primary decision path. Each domain needs a current check, not only a design note. ...

A full-site disaster recovery plan is not a backup inventory. It is a decision model for operating when the normal platform boundary is gone. Backups answer whether data exists somewhere else. DR answers whether the organization can restore service in the right order, with the right authority, inside an agreed failure window. Start With Service Tiers Do not plan every system as if it has the same recovery objective. Define tiers first: ...

CSI node plugins often sit exactly where Kubernetes troubleshooting gets uncomfortable. The pod is a Kubernetes object, but the failure is on the host: iSCSI sessions, multipath devices, kernel routes, udev, or mounted filesystems. If SSH is not the approved path, operators may need to use the CSI node pod or a temporary privileged diagnostic pod to see the host. That is valid during an incident, but it needs a boundary. Host diagnostics should not quietly become host mutation. ...

Some Kubernetes storage failures are not attach failures. The volume is created. The attachment says it is attached. The pod still cannot start. In one cluster, several storage-backed pods were stuck in ContainerCreating or Init after node maintenance. Existing pods with storage kept running, but new pods could not stage volumes on a subset of workers. The important split was this: existing sessions keep working fresh CSI NodeStage requires discovery discovery fails only on legacy workers That shape points below Kubernetes and above the storage array. The CSI node plugin was asking the host to discover an iSCSI target, and the host-side open-iscsi configuration was changing how that discovery behaved. ...