Vault And Secrets Field Notes

External Secrets Operator adds a useful boundary: applications can consume Kubernetes Secret objects while the source value lives in a dedicated secret manager. It also adds a new failure path. A backend token, SecretStore, controller, target Secret, and consuming workload all have to agree before rotation is actually safe. Ownership Model Write down the owner for each layer: backend secret manager -> source value and access policy SecretStore -> authentication and provider configuration ExternalSecret -> mapping from backend key to Kubernetes Secret target Secret -> Kubernetes object consumed by workloads application workload -> reload behavior and verification platform team -> controller health, RBAC, namespaces, and alerts If an ExternalSecret fails, do not assume it is an application bug or a secret-manager outage. Walk the chain. ...

A Kubernetes Secret update is not the same thing as a completed rotation. The object can change in etcd while the application still uses an old environment variable, cached file, open connection, or credential loaded at process start. Rotation is complete only when the consumer uses the replacement and the old credential is revoked or made irrelevant. Identify The Consumer Path Start by finding how the workload consumes the secret: ...

Secret review should answer operational questions without exposing secret values. The useful evidence is ownership, age, source, consumers, expiry, rollout state, and whether stale credentials still work. Decoding secret data into a ticket or chat channel usually creates a second incident. Inventory Without Values Start with metadata: kubectl get secret -A \ -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,TYPE:.type,AGE:.metadata.creationTimestamp' Then inspect labels and annotations: kubectl -n app-ns get secret app-secret -o yaml Review metadata, not data values: ...

GitHub Actions credentials are production access if the workflow can mutate production. The safe pattern is short-lived, scoped access tied to the repository, branch, environment, and workflow that needs it. Long-lived cloud keys in repository secrets should be the exception, not the default. Start With Permissions Set workflow permissions explicitly: permissions: contents: read Then add only what a job needs: permissions: contents: read id-token: write id-token: write enables OIDC token issuance. It does not by itself grant cloud access; the cloud trust policy still decides what the token can assume. ...

Secret rotation is not one operation. It is a lifecycle pattern that depends on the secret type, the consumer, the reload behavior, the rollback path, and the evidence the team needs afterward. Vault helps, but it does not remove the need to design rotation safely. Classify The Secret First Start by identifying what kind of secret is being rotated. Secret Type Rotation Pattern Static KV secret write new value, roll consumers, verify, remove old value if applicable Dynamic database credential reduce TTL, revoke leases, let Vault issue new credentials PKI certificate issue new certificate, reload consumer, verify live certificate Transit key rotate key version, rewrap or rewrite old ciphertext if needed API token create replacement, update consumers, revoke old token Kubernetes Secret update source, sync or rollout consumers, verify pod behavior Do not use one generic rotation runbook for every secret type. ...

Vault Kubernetes auth lets workloads authenticate to Vault using Kubernetes service account identity. That makes it a powerful bridge between platform identity and secret access. It also means mistakes in service account binding, namespace scoping, policy mapping, or token lifetime can become production secret exposure. This note focuses on operating the auth method safely. The Auth Contract For every workload using Kubernetes auth, document: cluster. namespace. service account. Vault auth mount. Vault role. attached policies. token TTL. secret paths allowed. owner. Example: ...

Vault PKI is useful when internal certificate issuance needs policy, auditability, and short-lived credentials instead of manual certificate handling. It is not just a place to mint certificates. It becomes part of the trust path for services, workloads, operators, and automation. This note focuses on operating the PKI secrets engine safely for internal certificates. Define The PKI Boundary Start by deciding what this PKI should and should not issue. Write down: ...

Vault transit gives applications cryptographic operations without handing them raw encryption keys. That is the main value: applications can encrypt, decrypt, sign, verify, or generate data keys through Vault while key material stays inside Vault’s trust boundary. Transit is not magic encryption. It is an operational contract between the application, Vault, policy, latency, audit logging, and recovery planning. Decide What Transit Owns Start with a clear use case. Good transit candidates: ...

Local infrastructure labs often start with hardcoded passwords, localhost endpoints, and convenience tokens. That is normal for learning, but dangerous when the lab pattern becomes a production pattern without review. The useful distinction is not “lab bad, production good.” The useful distinction is knowing which shortcuts are temporary and what must change before the pattern is reused. Common Lab Shortcuts Terraform-managed Docker labs often include: Grafana admin credentials in container environment variables. Concourse local users such as admin:admin. Vault dev server tokens in shell environment files. database passwords pulled into Terraform state. generated private keys written to local files. privileged containers for CI workers or system exporters. localhost endpoints that assume a single operator workstation. Each shortcut may be acceptable in a disposable lab. None should cross into shared infrastructure by accident. ...

Terraform repositories often accumulate terraform.tfvars, *.auto.tfvars, shell env files, examples, and helper scripts. That works until nobody remembers which file is authoritative. Problem A single environment directory may contain: terraform.tfvars vars.auto.tfvars vars.env terraform.tfvars.example vars.auto.tfvars.example vars.env.example That creates three problems: Terraform may load values from multiple files. shell helper files look like Terraform configuration. secrets may drift into files that should be committed safely. How Terraform Loads Values Terraform automatically loads: ...