A Kubernetes Secret update is not the same thing as a completed rotation. The object can change in etcd while the application still uses an old environment variable, cached file, open connection, or credential loaded at process start. Rotation is complete only when the consumer uses the replacement and the old credential is revoked or made irrelevant. Identify The Consumer Path Start by finding how the workload consumes the secret: ...
Secret review should answer operational questions without exposing secret values. The useful evidence is ownership, age, source, consumers, expiry, rollout state, and whether stale credentials still work. Decoding secret data into a ticket or chat channel usually creates a second incident. Inventory Without Values Start with metadata: kubectl get secret -A \ -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,TYPE:.type,AGE:.metadata.creationTimestamp' Then inspect labels and annotations: kubectl -n app-ns get secret app-secret -o yaml Review metadata, not data values: ...
Egress control is where platform security, application dependencies, DNS, and network operations collide. The goal is not simply to block outbound traffic. The goal is to make allowed outbound paths reviewable, observable, and recoverable when a dependency changes. Define The Egress Path For each cluster, document the path: pod -> CNI policy -> node routing -> NAT gateway or firewall -> proxy, if used -> external service Track ownership for: ...
NetworkPolicy is production access control. Roll it out like a platform change, not a formatting cleanup. The risk is not only blocking traffic. The risk is blocking traffic without knowing which dependency the workload actually needed. Start With Namespace Ownership Before applying default deny, confirm: namespace owner. workload owner. expected ingress sources. expected egress destinations. DNS dependency. metrics and log scrape paths. admission webhook or sidecar dependencies. emergency rollback path. If the namespace owner cannot name the required network paths, observe first and enforce later. ...
GitHub Actions credentials are production access if the workflow can mutate production. The safe pattern is short-lived, scoped access tied to the repository, branch, environment, and workflow that needs it. Long-lived cloud keys in repository secrets should be the exception, not the default. Start With Permissions Set workflow permissions explicitly: permissions: contents: read Then add only what a job needs: permissions: contents: read id-token: write id-token: write enables OIDC token issuance. It does not by itself grant cloud access; the cloud trust policy still decides what the token can assume. ...
CSI node plugins often sit exactly where Kubernetes troubleshooting gets uncomfortable. The pod is a Kubernetes object, but the failure is on the host: iSCSI sessions, multipath devices, kernel routes, udev, or mounted filesystems. If SSH is not the approved path, operators may need to use the CSI node pod or a temporary privileged diagnostic pod to see the host. That is valid during an incident, but it needs a boundary. Host diagnostics should not quietly become host mutation. ...
A CIS benchmark review produces a long list of checks. The failure is not having fails; the failure is treating every check with equal weight. CIS profiles are dense and some controls conflict with how a platform is actually operated. This note is about reviewing CIS results in a way that produces evidence, priorities, and defensible decisions instead of a scary PDF. Scope First Before running any scanner, define the scope: ...
Kubernetes audit logging is the record of what happened to the cluster API. If it is not configured, enabled, and retained, a suspicious delete, a bad RBAC escalation, or a credential misuse becomes a debate instead of a line in a log. The API server generates the events; the platform team is responsible for making them useful. This note covers operating audit logs as working evidence, not just configuration. What Gets Recorded Audit log lines describe request-level events against the API server: ...
Pod Security Standards are the default answer for “make workloads run non-privileged” in Kubernetes. They replace the removed PodSecurityPolicy admission controller with three policies, Baseline, Restricted, and Privileged, that can be enforced or audited per namespace. The model is simple, but production mistakes come from exemptions, admission confusion, and treating the label as the control. This note covers operating Pod Security Standards on real clusters. Know The Three Policies Privileged: unrestricted, for system workloads that legitimately need it. Baseline: allows the default Kubernetes behavior while preventing known privilege escalation. Good starting point for most non-system namespaces. Restricted: the hardened target. No privileged containers, no host network, strict volume and capability limits. Baseline is the pragmatic default. Restricted is the goal where the workload can satisfy it. Privileged is a documented exception, not a default. ...
A STIG-oriented Linux baseline review checks a system against the hardening requirements that the security technical implementation guide defines. For platform teams, STIG work is usually about proving a defensible baseline exists: no root logins, sane permissions, locked-down services, audit enabled, and deviations that have an owner. This note is about running that review with automated content instead of a checklist walked by hand. What A STIG Baseline Actually Controls STIG content for Red Hat and Ubuntu distributions covers roughly the same layers: ...