Kubernetes Platform Operations

RKE2 clusters across multiple sites had inconsistent DNS search domain behavior. Some nodes appended internal domains to all lookups, some appended data center-specific domains, and a few were clean. In a Kubernetes cluster, uncontrolled search domain expansion causes: Pod DNS lookups that should resolve as-is getting unexpected suffix expansion. Inconsistent behavior across sites for the same service name. Debugging sessions that waste time on DNS before finding the real issue. Desired State Layer Target Netplan search: [] systemd-resolved No per-link or global domains Active resolver (/etc/resolv.conf) search . search . is systemd-resolved shorthand for “no search domains” and is the correct end state. ...

Kubernetes gives teams enough flexibility to create drift quickly. Platform conventions are the small set of decisions that keep clusters operable when many teams share them. The goal is not to standardize everything. The goal is to standardize the parts that affect troubleshooting, access, security, cost, and recovery. Namespace Ownership Every namespace should have an owner, purpose, and lifecycle expectation. Use consistent labels: metadata: labels: owner: team-platform environment: prod workload-tier: platform data-classification: internal Use annotations for human-facing metadata: ...

Kubernetes upgrades should be treated as controlled production changes, not package updates. The same applies to node operating-system patching when automatic updates can restart services, touch device handling, or trigger storage churn. See Ubuntu Unattended Upgrades Are Kubernetes Node Changes for the node patching failure pattern. The hard part is rarely clicking upgrade. The hard part is sequencing the change so operators know what can fail, what is safe to continue, and when to stop. ...

Rancher is useful because it gives operators a single control plane for many Kubernetes clusters. That centralization is also the risk: if access, workspace boundaries, cluster ownership, and GitOps conventions are loose, the management plane becomes another source of drift. The operating standard should make the boring parts explicit. Access Model Rancher should authenticate normal users through Active Directory. Roles should be assigned to AD groups, not individual users, wherever possible. ...