Security And Compliance Field Notes

Secret rotation is not one operation. It is a lifecycle pattern that depends on the secret type, the consumer, the reload behavior, the rollback path, and the evidence the team needs afterward. Vault helps, but it does not remove the need to design rotation safely. Classify The Secret First Start by identifying what kind of secret is being rotated. Secret Type Rotation Pattern Static KV secret write new value, roll consumers, verify, remove old value if applicable Dynamic database credential reduce TTL, revoke leases, let Vault issue new credentials PKI certificate issue new certificate, reload consumer, verify live certificate Transit key rotate key version, rewrap or rewrite old ciphertext if needed API token create replacement, update consumers, revoke old token Kubernetes Secret update source, sync or rollout consumers, verify pod behavior Do not use one generic rotation runbook for every secret type. ...

Vault Kubernetes auth lets workloads authenticate to Vault using Kubernetes service account identity. That makes it a powerful bridge between platform identity and secret access. It also means mistakes in service account binding, namespace scoping, policy mapping, or token lifetime can become production secret exposure. This note focuses on operating the auth method safely. The Auth Contract For every workload using Kubernetes auth, document: cluster. namespace. service account. Vault auth mount. Vault role. attached policies. token TTL. secret paths allowed. owner. Example: ...

Vault PKI is useful when internal certificate issuance needs policy, auditability, and short-lived credentials instead of manual certificate handling. It is not just a place to mint certificates. It becomes part of the trust path for services, workloads, operators, and automation. This note focuses on operating the PKI secrets engine safely for internal certificates. Define The PKI Boundary Start by deciding what this PKI should and should not issue. Write down: ...

Vault transit gives applications cryptographic operations without handing them raw encryption keys. That is the main value: applications can encrypt, decrypt, sign, verify, or generate data keys through Vault while key material stays inside Vault’s trust boundary. Transit is not magic encryption. It is an operational contract between the application, Vault, policy, latency, audit logging, and recovery planning. Decide What Transit Owns Start with a clear use case. Good transit candidates: ...

Sometimes the maintenance path is blocked by host access, not Kubernetes health. In one RKE2 reboot window, SSH worked to the nodes, but noninteractive sudo did not. The maintenance automation needed to reboot hosts and verify node-level state. The workaround was a temporary privileged DaemonSet that wrote a short-lived sudoers rule onto each node, then stayed alive only long enough for the maintenance window. That pattern can be valid in a controlled emergency or tightly scoped window. It is also a host access change. Treat it with the same seriousness as adding an SSH key, changing a sudoers file, or granting a break-glass account. ...

Good maintenance windows produce evidence. Bad evidence bundles become a new secret store. During RKE2 reboot and upgrade work, the most useful artifacts were not complicated: preflight JSON, per-batch reboot logs, final cluster state captures, and error files. They proved which context was used, which nodes were touched, whether boot IDs changed, whether workers were drained, whether PodDisruptionBudgets blocked eviction, and whether the cluster returned to the expected baseline. That same evidence can expose internal hostnames, IP addresses, SSH usernames, kubeconfig paths, workload names, Vault paths, webhook URLs, and temporary access helpers. Treat maintenance output as operational evidence and sensitive data at the same time. For the access-helper side of this problem, see Temporary Privileged DaemonSets Are Host Access Changes. ...

An SRE agent can authenticate to Rancher and still fail every useful workload call. Seeing clusters in Rancher does not automatically mean the account can read pods, logs, events, or namespace-scoped workloads in downstream clusters. This pattern applies when an SRE automation identity reports symptoms like: namespaces is forbidden: User "u-example" cannot list resource "namespaces" in API group "" at the cluster scope pods is forbidden: User "u-example" cannot list resource "pods" in namespace "app-namespace" The important distinction is scope. The identity may have management-plane visibility and read-only node visibility, but no downstream project or namespace RBAC. ...

IoT device containers need TLS certificates to authenticate with AWS IoT Core. The certificate path is the critical configuration — if the container cannot find its credentials, it does not connect. For a runnable lab, see the terraform-docker-iot directory in the IaC repository. Certificate Injection Pattern Mount certificates from the host using Terraform volume mounts. Do not bake certs into the image: resource "docker_container" "iot_device" { image = docker_image.iot_device_image.name volumes { host_path = abspath("${path.module}/certs/root-CA.crt") container_path = "/certs/root-CA.crt" } volumes { host_path = abspath("${path.module}/certs/iot-thing.cert.pem") container_path = "/certs/iot-thing.cert.pem" } volumes { host_path = abspath("${path.module}/certs/iot-thing.private.key") container_path = "/certs/iot-thing.private.key" } } abspath(path.module) resolves the relative cert path to an absolute path relative to the Terraform module directory. This avoids path ambiguity when Terraform runs from different working directories. ...