[{"categories":["field-notes"],"content":"During a vSphere CNS backlog, stopping the source of new CSI work can be safer than continuing to submit attach, detach, resize, and update requests into a stuck backend queue. The trap is that GitOps may immediately undo the pause.\nScaling a vSphere CSI controller Deployment to 0 is not a durable pause if Argo CD, an ApplicationSet, or a higher root Application owns it. The visible command succeeds, then self-heal restores the replicas and new controller pods start submitting tasks again.\nConfirm Ownership First Before scaling anything, inspect ownership and sync policy:\nkubectl -n vmware-system-csi get deploy vsphere-csi-controller -o yaml \\ | yq \u0026#39;.metadata.labels, .metadata.annotations\u0026#39; kubectl get applications.argoproj.io,applicationsets.argoproj.io -A \\ | grep -i \u0026#39;vsphere\\|csi\u0026#39; Then inspect the owning Application and ApplicationSet:\nkubectl -n argocd get application \u0026lt;app-name\u0026gt; -o yaml \\ | yq \u0026#39;.spec.syncPolicy\u0026#39; kubectl -n argocd get applicationset \u0026lt;applicationset-name\u0026gt; -o yaml \\ | yq \u0026#39;.spec.syncPolicy, .spec.template.spec.syncPolicy\u0026#39; Look for:\nautomated.prune=true automated.selfHeal=true applicationsSync: create-update Those settings mean live edits may not hold.\nPrefer A Narrow Pause Try the least broad pause first:\nkubectl -n argocd patch application \u0026lt;cluster-vsphere-csi-app\u0026gt; --type=json \\ -p=\u0026#39;[{\u0026#34;op\u0026#34;:\u0026#34;remove\u0026#34;,\u0026#34;path\u0026#34;:\u0026#34;/spec/syncPolicy/automated\u0026#34;}]\u0026#39; kubectl -n vmware-system-csi scale deploy/vsphere-csi-controller --replicas=0 If an ApplicationSet recreates the Application policy, pause the service-specific ApplicationSet rather than the entire GitOps platform when possible. If a parent Application self-heals that ApplicationSet, keep walking up the ownership chain until you understand which controller is restoring the change.\nDo not disable broad GitOps controllers unless the incident requires that blast radius and the rollback is explicit.\nScheduling Block When Self-Heal Will Not Stop When the GitOps chain cannot be paused cleanly during an active incident, a temporary scheduling block can stop CSI controller pods from running without deleting the desired state.\nFor a controller that only schedules on control-plane nodes, add a dedicated maintenance taint to those eligible nodes:\nkubectl taint nodes cp-1 cp-2 cp-3 \\ storage-maintenance/freeze-csi=true:NoSchedule kubectl -n vmware-system-csi delete pod -l app=vsphere-csi-controller Then verify the pause is real:\nkubectl -n vmware-system-csi get pods -l app=vsphere-csi-controller -o wide kubectl -n vmware-system-csi get deploy vsphere-csi-controller The desired replica count may still be 3, but the important operational signal is that replacement controller pods are Pending and have no assigned node. Pending pods are not submitting new CNS work.\nDo not taint worker nodes unless the CSI controller actually schedules there. A NoSchedule taint does not evict existing non-CSI pods, but it can still affect new scheduling, so keep the scope tight and documented.\nVerify The Backend Queue Stops Growing After the pause, watch vCenter queued/running tasks:\ngovc tasks -json -s queued -s running -n=300 \\ | jq -r \u0026#39;.Tasks[]? | [ .QueueTime, .State, .DescriptionId, (.EntityName // \u0026#34;\u0026#34;), .Key ] | @tsv\u0026#39; Compare the latest task timestamps before and after the pause. The goal is not that the old backlog disappears immediately. The goal is that new CSI-generated work stops appearing while vCenter drains or times out the existing tasks.\nRoll Back In Phases Do not unpause every cluster at once. If multiple clusters were paused to reduce CNS load, restore the lower-risk or actively needed cluster first and keep the problematic cluster paused until its VM-level locks are clear.\nRollback the scheduling block by removing the same taint key. The trailing - is the kubectl taint syntax for removal:\nkubectl taint nodes cp-1 cp-2 cp-3 \\ storage-maintenance/freeze-csi- Then verify:\nkubectl -n vmware-system-csi get pods -l app=vsphere-csi-controller -o wide kubectl get volumeattachments -o wide govc tasks -json -s queued -s running -n=300 Restore GitOps automation only after confirming the CSI controller is healthy and vCenter is not immediately accumulating new attach/detach work.\nOperating Rule A CSI pause is not successful when kubectl scale returns success. It is successful when no controller pod is running and the vCenter CNS task queue stops receiving new work from that cluster.\nIn GitOps-owned environments, verify the effective runtime state, not just the live object you patched.\n","permalink":"https://trinidadmarroquin.com/field-notes/gitops-owned-vsphere-csi-maintenance-pause/","section":"field-notes","summary":"During a vSphere CNS backlog, stopping the source of new CSI work can be safer than continuing to submit attach, detach, resize, and update requests into a stuck backend queue. The trap is that GitOps may immediately undo the pause.\nScaling a vSphere CSI controller Deployment to 0 is not a durable pause if Argo CD, an ApplicationSet, or a higher root Application owns it. The visible command succeeds, then self-heal restores the replicas and new controller pods start submitting tasks again.\n","tags":["kubernetes","vsphere","csi","argocd","gitops","storage","operations"],"title":"GitOps-Owned vSphere CSI Maintenance Pauses"},{"categories":["field-notes"],"content":"A pod stuck in Pending because a PVC does not exist is not always a provisioning problem. Sometimes the data still exists in a retained PV and only the claim object is missing.\nThe safe recovery is to prove the PV is the intended backing store, then recreate the PVC with an explicit volumeName so it binds to that retained PV instead of provisioning something new.\nConfirm The Symptom Start with the pod event:\nkubectl -n \u0026lt;namespace\u0026gt; describe pod \u0026lt;pod-name\u0026gt; kubectl -n \u0026lt;namespace\u0026gt; get events --sort-by=.lastTimestamp The key signal is:\npersistentvolumeclaim \u0026#34;\u0026lt;pvc-name\u0026gt;\u0026#34; not found Then inspect the workload volume reference:\nkubectl -n \u0026lt;namespace\u0026gt; get deploy \u0026lt;deployment-name\u0026gt; -o yaml \\ | yq \u0026#39;.spec.template.spec.volumes\u0026#39; Confirm the workload still references the missing claim name.\nFind The Retained PV List available PVs:\nkubectl get pv -o wide Inspect the candidate retained PV:\nkubectl get pv \u0026lt;pv-name\u0026gt; -o yaml Confirm:\nstatus.phase is Available. persistentVolumeReclaimPolicy is Retain. capacity matches the workload expectation. access mode matches the workload expectation. storage class matches the original claim. labels, path, CSI handle, or other metadata identify it as the intended PV. Do not bind a random available PV just because the size matches.\nRecreate The Claim Explicitly Create the PVC with spec.volumeName set to the retained PV:\napiVersion: v1 kind: PersistentVolumeClaim metadata: name: app-data namespace: app-namespace labels: app: app-api spec: accessModes: - ReadWriteOnce resources: requests: storage: 5Gi storageClassName: standard volumeMode: Filesystem volumeName: app-retained-pv Apply it:\nkubectl apply -f pvc-rebind.yaml Then verify binding:\nkubectl -n app-namespace get pvc app-data -o wide kubectl get pv app-retained-pv -o wide Expected:\nPVC: Bound PV: Bound Verify The Workload Watch the pod leave Pending or ContainerCreating:\nkubectl -n app-namespace get pod -o wide kubectl -n app-namespace rollout status deploy/app-api --timeout=180s kubectl -n app-namespace describe pod \u0026lt;pod-name\u0026gt; If the pod schedules but does not become ready, continue with mount, permissions, and application logs. The PVC rebind only fixes the missing-claim condition.\nGitOps Follow-Up If the workload is GitOps-managed, recreate the PVC in the source of truth or restore the missing manifest. A live PVC fix that is not represented in Git can disappear during a future prune or namespace rebuild.\nAlso check why the PVC disappeared while the PV remained. Common causes include manual deletion, prune behavior, a Helm chart split, or a previous migration that retained the PV intentionally but did not preserve the claim.\nOperating Rule A retained PV is recoverable evidence, not automatic permission to bind it.\nRebind only after confirming identity, capacity, access mode, storage class, and workload ownership. Then put the claim back into the declared state so the recovery survives reconciliation.\n","permalink":"https://trinidadmarroquin.com/field-notes/kubernetes-retained-pv-missing-pvc-rebind/","section":"field-notes","summary":"A pod stuck in Pending because a PVC does not exist is not always a provisioning problem. Sometimes the data still exists in a retained PV and only the claim object is missing.\nThe safe recovery is to prove the PV is the intended backing store, then recreate the PVC with an explicit volumeName so it binds to that retained PV instead of provisioning something new.\nConfirm The Symptom Start with the pod event:\n","tags":["kubernetes","storage","persistent-volumes","troubleshooting","operations"],"title":"Kubernetes Retained PV Missing PVC Rebind"},{"categories":["field-notes"],"content":"A connected or host-backed virtual CD-ROM can become operational noise during VM maintenance. It can trigger device-lock prompts, confuse vMotion or storage work, and leave operators answering vCenter questions that have nothing to do with the actual change. If the VM is already locked by storage or vMotion tasks, clear the task contention first; see vSphere CSI CNS ExtendVolume Triage.\nFor long-lived Kubernetes nodes and platform VMs, a CD-ROM is often unnecessary after provisioning. If one remains, it should be either disconnected as a client device or removed intentionally.\nAudit The Device Find the target VMs, then inspect CD-ROM state:\ngovc find /DC-Site-A/vm/K8s-Cluster/NonProd -type m | while read -r vm; do echo \u0026#34;VM: $vm\u0026#34; govc device.info -vm \u0026#34;$vm\u0026#34; \u0026#39;cdrom-*\u0026#39; 2\u0026gt;/dev/null \\ | awk \u0026#39;/^Name:|Label:|Summary:|Connected:|Start connected:/ { print \u0026#34; \u0026#34; $0 }\u0026#39; \\ || echo \u0026#34; No CD-ROM device\u0026#34; done Useful states:\nRemote ATAPI -\u0026gt; Client Device ATAPI ... -\u0026gt; Host Device or host-backed CD-ROM Connected -\u0026gt; whether the device is currently connected Start connected -\u0026gt; whether it reconnects on boot The goal for an unused CD-ROM is either:\nNo CD-ROM device or:\nSummary: Remote ATAPI Connected: false Start connected: false Try The Non-Destructive Path First For a VM that still has a CD-ROM, eject and disconnect the specific device:\nvm=\u0026#39;/DC-Site-A/vm/K8s-Cluster/NonProd/cluster-a-worker-01\u0026#39; cdrom=\u0026#34;$(govc device.info -vm \u0026#34;$vm\u0026#34; \u0026#39;cdrom-*\u0026#39; \\ | awk \u0026#39;/^Name:/ { name=$2 } /Label: *CD\\/DVD drive 1/ { print name; exit }\u0026#39;)\u0026#34; timeout 20s govc device.cdrom.eject -vm \u0026#34;$vm\u0026#34; -device \u0026#34;$cdrom\u0026#34; || true govc vm.question -vm \u0026#34;$vm\u0026#34; -answer=0 \u0026gt;/dev/null 2\u0026gt;\u0026amp;1 || true timeout 20s govc device.disconnect -vm \u0026#34;$vm\u0026#34; \u0026#34;$cdrom\u0026#34; || true govc vm.question -vm \u0026#34;$vm\u0026#34; -answer=0 \u0026gt;/dev/null 2\u0026gt;\u0026amp;1 || true govc device.info -vm \u0026#34;$vm\u0026#34; \u0026#34;$cdrom\u0026#34; \\ | awk \u0026#39;/^Name:|Label:|Summary:|Connected:|Start connected:/ { print }\u0026#39; govc vm.question is useful because vCenter may ask whether it should override a device lock. Answer only the expected device question; do not blindly answer unrelated VM prompts in a broad loop without inspecting the result.\nKnow The govc Limitation Some VMs have CD-ROM devices on an AHCI/SATA controller. In some govc builds, govc device.cdrom.add can only add a CD-ROM to an IDE controller. vSphere may also refuse to hot-add an IDE CD-ROM to a powered-on VM.\nThat means this sequence can be unsafe if run casually against powered-on VMs:\nremove AHCI CD-ROM attempt govc device.cdrom.add add fails because only IDE is supported or hot-add is blocked VM is left with no CD-ROM device That final state may be acceptable if CD-ROMs are not required. It is not acceptable if the runbook expected to replace the device immediately.\nChoose The Desired End State For server VMs that do not need virtual media, no CD-ROM is usually cleaner than a host-backed device. Verify and record that choice:\ngovc find /DC-Site-A/vm/K8s-Cluster/NonProd -type m | while read -r vm; do if govc device.info -vm \u0026#34;$vm\u0026#34; \u0026#39;cdrom-*\u0026#39; \u0026gt;/dev/null 2\u0026gt;\u0026amp;1; then echo \u0026#34;HAS CDROM: $vm\u0026#34; else echo \u0026#34;NO CDROM: $vm\u0026#34; fi done If the VM must keep a CD-ROM and govc cannot hot-add the desired client-device backing, repair it during a maintenance window while the VM is powered off, or use a vSphere UI/API path that can add the correct controller/device type safely.\nMaintenance Repair Pattern For VMs where a CD-ROM must exist and power-off is acceptable:\nvm=\u0026#39;/DC-Site-A/vm/K8s-Cluster/NonProd/cluster-a-worker-01\u0026#39; govc vm.power -off \u0026#34;$vm\u0026#34; govc device.cdrom.add -vm \u0026#34;$vm\u0026#34; cdrom=\u0026#34;$(govc device.info -vm \u0026#34;$vm\u0026#34; \u0026#39;cdrom-*\u0026#39; \\ | awk \u0026#39;/^Name:/ { print $2; exit }\u0026#39;)\u0026#34; govc device.disconnect -vm \u0026#34;$vm\u0026#34; \u0026#34;$cdrom\u0026#34; || true govc vm.power -on \u0026#34;$vm\u0026#34; govc device.info -vm \u0026#34;$vm\u0026#34; \u0026#39;cdrom-*\u0026#39; \\ | awk \u0026#39;/^Name:|Label:|Summary:|Connected:|Start connected:/ { print }\u0026#39; Do this only with normal VM maintenance controls: workload drain if the VM is a Kubernetes node, expected boot validation, and rollback criteria.\nOperating Rule Do not treat virtual CD-ROM cleanup as harmless inventory polish.\nAudit first, decide whether the desired state is “client device disconnected” or “no CD-ROM,” and understand whether your govc version can add the replacement device to a powered-on VM before removing anything.\n","permalink":"https://trinidadmarroquin.com/field-notes/vsphere-cdrom-host-device-cleanup-govc/","section":"field-notes","summary":"A connected or host-backed virtual CD-ROM can become operational noise during VM maintenance. It can trigger device-lock prompts, confuse vMotion or storage work, and leave operators answering vCenter questions that have nothing to do with the actual change. If the VM is already locked by storage or vMotion tasks, clear the task contention first; see vSphere CSI CNS ExtendVolume Triage.\nFor long-lived Kubernetes nodes and platform VMs, a CD-ROM is often unnecessary after provisioning. If one remains, it should be either disconnected as a client device or removed intentionally.\n","tags":["vsphere","vmware","govc","vcenter","operations"],"title":"vSphere CD-ROM Host Device Cleanup With govc"},{"categories":["field-notes"],"content":"A vSphere CSI controller log that says a CNS ExtendVolume task is pending does not immediately tell you whether the workload is blocked by resize, attachment, mount, or a stale backend task. The first job is to map the CNS volume ID back to Kubernetes state and avoid turning a storage delay into a destructive rollback.\nThe useful triage order is:\nCNS volume ID -\u0026gt; PV -\u0026gt; PVC -\u0026gt; pod -\u0026gt; VolumeAttachment -\u0026gt; CSI controller logs -\u0026gt; vCenter task state Map The CNS Volume ID Start by mapping the reported CNS volume ID to the Kubernetes PV and PVC:\nvolume_id=\u0026#39;\u0026lt;cns-volume-id\u0026gt;\u0026#39; kubectl get pv -o json \\ | jq -r --arg volume_id \u0026#34;$volume_id\u0026#34; \u0026#39; .items[] | select(.spec.csi.volumeHandle == $volume_id) | [ .metadata.name, .spec.claimRef.namespace, .spec.claimRef.name, .spec.capacity.storage, .status.phase ] | @tsv\u0026#39; Then inspect the PVC and PV directly:\nkubectl -n \u0026lt;namespace\u0026gt; get pvc \u0026lt;pvc-name\u0026gt; -o yaml kubectl get pv \u0026lt;pv-name\u0026gt; -o yaml Look for the requested size, current capacity, annotations, conditions, and storage class. A PVC that is already Bound at the expected size is a different problem than one stuck with expansion conditions.\nSeparate Resize From Attach Check whether Kubernetes still believes expansion is pending:\nkubectl -n \u0026lt;namespace\u0026gt; describe pvc \u0026lt;pvc-name\u0026gt; kubectl -n \u0026lt;namespace\u0026gt; get events --sort-by=.lastTimestamp Useful resize signals include:\nExternalExpanding Resizing FileSystemResizePending FileSystemResizeSuccessful Then check VolumeAttachments separately:\nkubectl get volumeattachments -o json \\ | jq -r --arg pv \u0026#39;\u0026lt;pv-name\u0026gt;\u0026#39; \u0026#39; .items[] | select(.spec.source.persistentVolumeName == $pv) | [ .metadata.name, .spec.nodeName, .status.attached, (.status.attachError.message // \u0026#34;\u0026#34;) ] | @tsv\u0026#39; An attach error such as DeadlineExceeded may be the active symptom even when the log line that started the investigation mentions ExtendVolume. Do not assume one log phrase is the whole incident.\nCheck The Workload Path Inspect the affected pod and node placement:\nkubectl -n \u0026lt;namespace\u0026gt; get pod \u0026lt;pod-name\u0026gt; -o wide kubectl -n \u0026lt;namespace\u0026gt; describe pod \u0026lt;pod-name\u0026gt; kubectl get node \u0026lt;node-name\u0026gt; -o wide This tells you whether the pod is blocked before scheduling, waiting on attach, waiting on mount, or already running after reconciliation caught up.\nFor vSphere CSI components, verify both controller and node pods:\nkubectl -n vmware-system-csi get pods -o wide kubectl -n vmware-system-csi logs deploy/vsphere-csi-controller --since=30m --all-containers=false If the CSI controller is intentionally paused during maintenance, record that explicitly. A paused controller can make unrelated storage symptoms look worse than they are.\nCheck CSI Operation And Leader State vSphere CSI also keeps Kubernetes-side operation records. When vCenter is slow or task visibility is inconsistent, these records can explain why CSI keeps retrying old work:\nkubectl -n vmware-system-csi get cnsvolumeoperationrequests.cns.vmware.com kubectl -n vmware-system-csi get cnsvolumeoperationrequest \u0026lt;operation-name\u0026gt; -o yaml Look for:\nfirstOperationDetails.taskStatus latestOperationDetails[].taskStatus latestOperationDetails[].error latestOperationDetails[].taskId volumeID Repeated TimeOut, VSLM task failed, or an operation that still says InProgress after vCenter no longer shows queued/running tasks is a signal to slow down. It may be stale CSI memory, vCenter task history lag, or a backend task that has not reconciled through every layer yet.\nAlso verify CSI sidecar leadership when attach status stops moving:\nkubectl -n vmware-system-csi get lease kubectl -n vmware-system-csi get lease external-attacher-leader-csi-vsphere-vmware-com -o yaml kubectl -n vmware-system-csi get pods -o wide If the external-attacher lease is held by a deleted pod, a live attacher may never update VolumeAttachment status. Deleting only the stale lease can be the smallest safe unblock, because a current controller pod should reacquire it:\nkubectl -n vmware-system-csi delete lease external-attacher-leader-csi-vsphere-vmware-com kubectl -n vmware-system-csi get lease external-attacher-leader-csi-vsphere-vmware-com -o yaml Do this only after confirming the holder pod is gone. Do not delete all leases or restart every CSI component blindly.\nBackend Attached, Kubernetes Still False A confusing failure mode is when vCenter and the guest show the disk attached, but Kubernetes still reports the VolumeAttachment as attached=false.\nCorrelate all three views:\nkubectl get volumeattachment \u0026lt;volumeattachment-name\u0026gt; -o yaml govc device.info -vm \u0026#39;\u0026lt;vm-path-or-name\u0026gt;\u0026#39; ssh operator@192.0.2.24 \u0026#39;lsblk; findmnt; sudo dmesg | tail -100\u0026#39; CSI logs may show a shape like:\nattachedStatus=false found=true That means the backend can see the disk, but the Kubernetes status path has not reconciled. At that point, patching VolumeAttachment.status.attached=true may look tempting. Treat that as a manual status intervention, not a normal fix. It bypasses the CSI controller\u0026rsquo;s failed update path and should require an explicit decision, current evidence that the correct disk is attached to the correct node, and a rollback plan.\nWhen in doubt, wait for vCenter/CNS to settle before patching storage status, deleting operation records, removing finalizers, or detaching disks.\nUse vCenter As Correlation, Not Guesswork If Kubernetes still shows pending attachment or resize, correlate with vCenter task state. Use read-only commands first:\ngovc tasks govc events -vm \u0026#39;\u0026lt;vm-path-or-name\u0026gt;\u0026#39; Capture whether CNS tasks are queued, running, succeeded, or no longer visible. If the Kubernetes PVC and pod recover while you are waiting, do not keep forcing remediation just because an earlier log line mentioned a pending task.\ngovc tasks can be misleading if you only run the default view. The default output is recent task history; it may not show the active backlog you care about. Query queued and running tasks explicitly:\ngovc tasks -json -s queued -s running -n=300 \\ | jq -r \u0026#39;.Tasks[]? | [ .StartTime, .QueueTime, .State, (.Progress|tostring), .DescriptionId, (.EntityName // \u0026#34;\u0026#34;), .Key ] | @tsv\u0026#39; For a suspected worker VM, scope the same query to the VM path:\nvm=\u0026#39;/DC-Site-A/vm/K8s-Cluster/cluster-a-worker-2\u0026#39; govc tasks -json -s queued -s running -n=300 \u0026#34;$vm\u0026#34; \\ | jq -r \u0026#39;.Tasks[]? | [ .StartTime, .QueueTime, .State, (.Progress|tostring), .DescriptionId, (.EntityName // \u0026#34;\u0026#34;), .Key ] | @tsv\u0026#39; A useful backlog summary is count by state for the affected VM:\ngovc tasks -json -n=300 \\ | jq -r --arg vm \u0026#39;cluster-a-worker-2\u0026#39; \u0026#39; [.Tasks[]? | select((.EntityName // \u0026#34;\u0026#34;) == $vm)] | group_by(.State)[] | [.[0].State, length] | @tsv\u0026#39; Look specifically for task types that can serialize or block storage progress:\ncom.vmware.cns.tasks.attachvolume com.vmware.cns.tasks.detachvolume com.vmware.cns.tasks.extendvolume com.vmware.cns.tasks.updatevolume vslm.vcenter.VStorageObjectManager.extendDisk VirtualMachine.attachDisk VirtualMachine.detachDisk Drm.ExecuteVMotionLRO The important distinction is whether CSI is still creating new work or whether vCenter/ESXi is still draining already-submitted work. If CSI is paused and no new controller pods are running, a growing queue probably points at the backend task backlog, not a fresh Kubernetes scheduling decision.\nWhen govc tasks -s queued -s running returns no rows but CSI operation CRs still mention an old task as InProgress, treat that as a split-brain signal between CSI\u0026rsquo;s remembered operation state and currently visible vCenter task state. Do not delete CSI operation records or patch storage status until the backend state, guest disk state, and Kubernetes object state have been reconciled deliberately.\nAvoid Destructive Rollback Storage rollback is not the same as rolling back an operational pause.\nSafe operational rollback examples:\nremove temporary CSI pause taints. restore GitOps sync policy after maintenance. restart or unpause CSI controllers when the runbook says to. Potentially destructive storage rollback examples:\ndeleting a replacement PVC. removing finalizers. detaching disks manually. reverting a database workload to an older volume. Do not perform storage rollback just because the workload was once pending. If the replacement PVC is now bound, attached, mounted, and serving the workload, treat it as live storage.\nFinal Health Gate Close the incident only after the storage and workload views agree:\nkubectl -n \u0026lt;namespace\u0026gt; get pod \u0026lt;pod-name\u0026gt; -o wide kubectl -n \u0026lt;namespace\u0026gt; get pvc \u0026lt;pvc-name\u0026gt; -o wide kubectl get volumeattachments -o wide kubectl -n vmware-system-csi get pods -o wide kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded Expected signals:\nPVC is Bound at the intended size. workload pod is Running and ready. matching VolumeAttachment is attached=true. CSI controller and node pods are healthy. no new storage-related non-running pods remain. no visible queued or running vCenter tasks explain a continuing symptom. Operating Rule Treat ExtendVolume task pending on CNS as a starting signal, not a diagnosis.\nMap the CNS ID to Kubernetes objects, separate resize from attach and mount, correlate with vCenter task state, and only then decide whether to wait, unpause controllers, restart CSI components, or plan a storage rollback.\n","permalink":"https://trinidadmarroquin.com/field-notes/vsphere-csi-cns-extendvolume-triage/","section":"field-notes","summary":"A vSphere CSI controller log that says a CNS ExtendVolume task is pending does not immediately tell you whether the workload is blocked by resize, attachment, mount, or a stale backend task. The first job is to map the CNS volume ID back to Kubernetes state and avoid turning a storage delay into a destructive rollback.\nThe useful triage order is:\nCNS volume ID -\u0026gt; PV -\u0026gt; PVC -\u0026gt; pod -\u0026gt; VolumeAttachment -\u0026gt; CSI controller logs -\u0026gt; vCenter task state Map The CNS Volume ID Start by mapping the reported CNS volume ID to the Kubernetes PV and PVC:\n","tags":["kubernetes","vsphere","vmware","csi","storage","troubleshooting","operations"],"title":"vSphere CSI CNS ExtendVolume Triage"},{"categories":["field-notes"],"content":"When Kubernetes nodes reboot during a storage incident, the cause matters. An in-guest reboot, a Rancher/system-upgrade action, a human SSH session, and a vSphere HA reset all have different follow-up work.\nThe useful pattern is to prove the reboot from multiple layers before assigning cause.\nStart With Kubernetes Check node readiness, boot ID, kernel, and recent events:\nkubectl get nodes -o wide kubectl describe node cp-1 | grep -A5 -E \u0026#39;Conditions:|Events:\u0026#39; kubectl get events -A --sort-by=.lastTimestamp | grep -i \u0026#39;reboot\\|node\u0026#39; kubectl get node cp-1 -o jsonpath=\u0026#39;{.status.nodeInfo.bootID}{\u0026#34;\\n\u0026#34;}{.status.nodeInfo.kernelVersion}{\u0026#34;\\n\u0026#34;}\u0026#39; Kubernetes can tell you that a node rebooted. It usually cannot tell you why.\nIf two control-plane nodes reboot at nearly the same timestamp, treat that as a high-signal clue. Simultaneous reboots are less likely to be a random operator SSH command on each node.\nCheck The Guest On each affected node, compare current and previous boots:\nuptime -s cat /proc/sys/kernel/random/boot_id journalctl --list-boots journalctl -b -1 -n 300 --no-pager Look for a clean shutdown path:\njournalctl -b -1 --no-pager \\ | grep -Ei \u0026#39;systemctl reboot|shutdown|poweroff|reboot:|Reached target.*Shutdown|Stopped target\u0026#39; Check whether a human session issued the reboot:\nlast -x | head -30 journalctl -b -1 _COMM=sudo --no-pager journalctl -b -1 --no-pager | grep -Ei \u0026#39;sudo|session opened|reboot|shutdown|poweroff\u0026#39; If the previous boot lacks a clean shutdown sequence, that points away from a normal in-guest reboot and toward reset, power loss, host failure, or HA action.\nCheck Package And Upgrade Paths A reboot into a newer kernel does not prove the kernel was installed at that moment. The kernel may have been installed earlier and only activated by the reset.\nCheck package logs and upgrade controllers:\ngrep -Ei \u0026#39;install|upgrade|linux-image|kernel\u0026#39; /var/log/apt/history.log /var/log/dpkg.log kubectl -n system-upgrade get plans,pods,jobs -o wide kubectl -n system-upgrade logs deploy/system-upgrade-controller --since=6h If there is no matching upgrade job and no package install near the reboot, keep looking.\nCheck vCenter Task And Event History Query visible power/reset tasks first:\ngovc tasks -json -b=6h -n=500 \\ | jq -r \u0026#39;.Tasks[]? | select((.DescriptionId // \u0026#34;\u0026#34;) | test(\u0026#34;Power|power|Reset|reset|Shutdown|Standby\u0026#34;)) | [ .StartTime, .CompleteTime, .State, .DescriptionId, (.EntityName // \u0026#34;\u0026#34;), (.Reason.UserName // \u0026#34;\u0026#34;), .Key ] | @tsv\u0026#39; Then inspect VM events:\ngovc events -vm \u0026#39;/DC-Site-A/vm/K8s-Cluster/cluster-a-cp-1\u0026#39; -n=100 Useful event phrases include:\nvSphere HA restarted this virtual machine VMware Tools heartbeat failure reset powered on migrating If vCenter events show an HA reset due to VMware Tools heartbeat failure, and guest logs lack a clean shutdown, classify it as an infrastructure reset, not a human or Kubernetes upgrade action.\nWatch The Side Effects A reset can undo an intentional maintenance shape. For example, a rebooted control-plane node may briefly allow paused controller pods to run again if the pause depended on node state, scheduling, or a just-deleted pod.\nAfter a HA reset, recheck:\nkubectl get nodes -o wide kubectl -n vmware-system-csi get pods -o wide kubectl describe nodes | grep -A3 \u0026#39;^Taints:\u0026#39; kubectl get events -A --sort-by=.lastTimestamp | tail -50 If a storage controller was intentionally paused, verify that it is still paused. If it briefly restarted, check whether it submitted new vCenter CNS tasks before re-pausing it.\nOperating Rule Do not stop at “the node rebooted.” Prove whether the reboot came from the guest, Kubernetes/Rancher automation, vCenter task history, or vSphere HA.\nThe classification changes the fix: upgrade cleanup, user-process review, host/HA investigation, or storage-controller containment.\n","permalink":"https://trinidadmarroquin.com/field-notes/vsphere-ha-reset-evidence-kubernetes-nodes/","section":"field-notes","summary":"When Kubernetes nodes reboot during a storage incident, the cause matters. An in-guest reboot, a Rancher/system-upgrade action, a human SSH session, and a vSphere HA reset all have different follow-up work.\nThe useful pattern is to prove the reboot from multiple layers before assigning cause.\nStart With Kubernetes Check node readiness, boot ID, kernel, and recent events:\nkubectl get nodes -o wide kubectl describe node cp-1 | grep -A5 -E \u0026#39;Conditions:|Events:\u0026#39; kubectl get events -A --sort-by=.lastTimestamp | grep -i \u0026#39;reboot\\|node\u0026#39; kubectl get node cp-1 -o jsonpath=\u0026#39;{.status.nodeInfo.bootID}{\u0026#34;\\n\u0026#34;}{.status.nodeInfo.kernelVersion}{\u0026#34;\\n\u0026#34;}\u0026#39; Kubernetes can tell you that a node rebooted. It usually cannot tell you why.\n","tags":["vsphere","vmware","kubernetes","rke2","incident-response","operations"],"title":"vSphere HA Reset Evidence For Kubernetes Nodes"},{"categories":["field-notes"],"content":"Replacing Kubernetes nodes from a fresh OS template can be faster than repairing legacy VM drift in place, but speed only helps if the risky work is moved out of the maintenance window without creating identity conflicts.\nThe useful rehearsal pattern is to pre-create replacement VMs from the current template, leave them powered off, and join them one at a time during the window. That turns the maintenance window into a controlled cluster-change sequence instead of a race to clone, customize, debug, and drain all at once.\nWhat To Move Out Of The Window Move clone and baseline configuration work earlier:\n1. Terraform creates replacement VMs from the current OS template. 2. Terraform applies final hostname, static IP, DNS, NTP, routes, SSH, users, sudo policy, CPU, memory, and disks. 3. The VM is validated without joining the live cluster. 4. Optional image-cache warming pulls known large workload images into the RKE2 containerd store. 5. The VM is shut down cleanly and left powered off. 6. Terraform outputs the VM path/name, intended role, IP, and maintenance notes. The acceptance check before the window is not just “Terraform applied.” It is that the replacement VM has the final identity it will use when it joins the cluster, and Terraform has no planned changes against the active old nodes unless those changes are explicitly part of the maintenance plan.\nKeep Version Skew Boring If replacement nodes are also moving the cluster forward by one Kubernetes minor, replace server-side nodes before allowing workers or monitoring nodes to run a newer kubelet minor than the API servers.\nFor a conservative RKE2 replacement rehearsal:\n1. Control-plane nodes, beginning with non-identity-takeover replacements. 2. Etcd nodes, one at a time with explicit etcd validation. 3. Monitoring or specialized worker pools. 4. General workers. If the rehearsal goal is only to prove VM power-on and RKE2 join mechanics, use a lower-risk worker-capacity proof at the current cluster version. That proves the provisioning path, but it does not prove the one-minor server-side upgrade path.\nDo Not Create Identity Conflicts The replacement VM should join with its intended final hostname and IP. Do not join a node, then rename or re-IP it into the old identity afterward.\nIf old hostname or IP reuse is required, the old identity must be removed from service before the replacement joins:\ncordon old node drain old node if disruption is acceptable stop old VM or otherwise guarantee the old IP is no longer active join replacement with the old hostname/IP from the start When possible, prefer new permanent names and IPs for replacement nodes. Save identity takeover for the nodes that truly need it, and do those last after the non-takeover path has been proven.\nPower On One Replacement At A Time During the maintenance window, power on only the current replacement VM:\ngovc vm.power -on \u0026#39;\u0026lt;vm-path-or-name\u0026gt;\u0026#39; govc vm.ip -wait=5m \u0026#39;\u0026lt;vm-path-or-name\u0026gt;\u0026#39; Then validate the host before running the RKE2 join step:\nssh operator@192.0.2.24 \\ \u0026#39;hostname; . /etc/os-release \u0026amp;\u0026amp; echo \u0026#34;$PRETTY_NAME\u0026#34;; uname -r; timedatectl status --no-pager; ip -brief addr show; ip route\u0026#39; Check that the VM booted with the expected hostname, IP, OS version, time sync, route, and access policy. If the VM has the wrong identity or reruns destructive initialization unexpectedly, power it off and stop before touching the cluster.\nWarm The Right Image Cache Image pre-pull can reduce maintenance-window variance, especially when large application images would otherwise cold-pull after the node joins. The important detail is where the image is pulled.\nFor RKE2, pull into the containerd store kubelet will actually use:\nsudo /var/lib/rancher/rke2/bin/ctr \\ --address /run/k3s/containerd/containerd.sock \\ -n k8s.io images pull registry.example.com/platform/app:1.2.3 Validate the cache from the same store:\nsudo /var/lib/rancher/rke2/bin/ctr \\ --address /run/k3s/containerd/containerd.sock \\ -n k8s.io images ls | grep \u0026#39;registry.example.com/platform/app\u0026#39; Do not confuse Docker, a separate containerd root, or a template-build cache with the RKE2 runtime cache. Also avoid baking short-lived join tokens or registry secrets into images, Terraform state, or durable logs.\nIf tags are mutable, refresh or verify them shortly before the window. Digest-pinned pulls are safer when the deployment process supports them.\nWorker Drain Shape When adding replacement worker capacity, avoid moving the same workload repeatedly across old nodes that will be drained later.\nA useful worker pattern is:\n1. Join one new replacement worker. 2. Confirm CNI, CSI, ingress, and representative workload scheduling. 3. Cordon old workers that should not receive newly evicted pods. 4. Drain one old worker only after reviewing PDB, RWO volume, and capacity risk. 5. Validate workloads and volume attachments. 6. Delete or decommission the old worker. 7. Repeat one worker at a time. Do not cordon the new replacement worker if it is supposed to absorb workloads. If new capacity is insufficient, stop and uncordon old workers rather than forcing drains into a bad placement state.\nValidation Gates For every replacement, capture a small before/after set:\nkubectl get nodes -o wide kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded kubectl get pvc -A kubectl get volumeattachments kubectl -n kube-system get pods -o wide kubectl get plans.upgrade.cattle.io -A For server and etcd nodes, add etcd health and membership checks after every add or removal:\nkubectl -n kube-system exec \u0026lt;etcd-pod\u0026gt; -- etcdctl \\ --cacert=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt \\ --cert=/var/lib/rancher/rke2/server/tls/etcd/server-client.crt \\ --key=/var/lib/rancher/rke2/server/tls/etcd/server-client.key \\ --endpoints=https://127.0.0.1:2379 endpoint health --cluster The per-node checklist should include node readiness, labels, taints, RKE2 service health, CNI pod health on the node, CSI node pod health, PVC mounts, VolumeAttachments, ingress behavior, and Rancher cluster visibility.\nStop And Roll Back Stop if any of these appear:\netcd health or membership is ambiguous. the replacement node does not become Ready within the agreed timeout. CSI node plugin or volume attachment behavior is unhealthy. critical pods remain Pending, CrashLoopBackOff, or ContainerCreating beyond the agreed timeout. Rancher marks the cluster disconnected and it does not recover quickly. ingress or load-balancer behavior breaks for representative services. duplicate node names, wrong node IPs, or stale provider IDs appear. Terraform or vCenter shows an unexpected change to active old VMs. Rollback is easiest when the old VM was only cordoned or powered off. If the old node still exists and is intact, uncordon or power it back on and let it return to Ready. If the replacement joined and failed post-join validation, cordon, drain, delete the replacement if safe, and keep the old node in service.\nFor etcd failures, stop replacement work immediately. Try to recover quorum by bringing a known-good old member back before considering restore from the latest off-node snapshot.\nOperating Rule Fast node replacement is not fast because operators skip checks. It is fast because VM cloning, baseline configuration, inventory collection, and optional image warming happen before the window.\nThe maintenance window should only do the work that must happen against the live cluster: power on one prepared VM, validate host identity, join it at the intended version and role, prove cluster health, decommission one old node, and decide whether to continue.\n","permalink":"https://trinidadmarroquin.com/field-notes/fast-os-template-node-replacement-rehearsal/","section":"field-notes","summary":"Replacing Kubernetes nodes from a fresh OS template can be faster than repairing legacy VM drift in place, but speed only helps if the risky work is moved out of the maintenance window without creating identity conflicts.\nThe useful rehearsal pattern is to pre-create replacement VMs from the current template, leave them powered off, and join them one at a time during the window. That turns the maintenance window into a controlled cluster-change sequence instead of a race to clone, customize, debug, and drain all at once.\n","tags":["rke2","kubernetes","rancher","vsphere","terraform","templates","upgrades","operations"],"title":"Fast OS Template Node Replacement Rehearsal"},{"categories":["field-notes"],"content":"Longhorn has a CLI, longhornctl, but installing it should not change the operational source of truth during maintenance. For Rancher-managed Longhorn clusters, the most useful day-to-day state still usually comes from the Longhorn UI and Kubernetes CRDs such as volumes.longhorn.io, replicas.longhorn.io, nodes.longhorn.io, engines.longhorn.io, and settings.longhorn.io.\nThe practical boundary is simple: use longhornctl as an operator tool, not as a reason to skip in-cluster evidence.\nInstall Locally For an Ubuntu workstation, a user-local install avoids system-wide package changes. Confirm the workstation architecture first:\nuname -s uname -m For a Linux x86_64 workstation, download the matching linux-amd64 release asset and checksum from the upstream Longhorn CLI releases, verify the checksum, then place the binary in ~/.local/bin:\nversion=\u0026#39;v1.12.0\u0026#39; base_url=\u0026#34;https://github.com/longhorn/cli/releases/download/${version}\u0026#34; curl -fL \u0026#34;${base_url}/longhornctl-linux-amd64\u0026#34; \\ -o /tmp/longhornctl-linux-amd64 curl -fL \u0026#34;${base_url}/longhornctl-linux-amd64.sha256\u0026#34; \\ -o /tmp/longhornctl-linux-amd64.sha256 cd /tmp sha256sum -c longhornctl-linux-amd64.sha256 chmod 0755 longhornctl-linux-amd64 mv longhornctl-linux-amd64 ~/.local/bin/longhornctl Verify the installed command:\nwhich longhornctl longhornctl version If ~/.local/bin is not already on PATH, add it through the workstation shell profile rather than installing the binary into a system directory by default.\nWhere It Fits longhornctl is useful for tasks such as install or upgrade checks, preflight checks, support bundle collection, and version-specific administrative workflows.\nDuring a maintenance window, keep the health gates tied to cluster state:\nkubectl -n longhorn-system get pods -o wide kubectl -n longhorn-system get volumes.longhorn.io kubectl -n longhorn-system get replicas.longhorn.io kubectl -n longhorn-system get nodes.longhorn.io Those checks show the current volume, replica, node, and engine state that matters before draining or rebooting storage-bearing nodes. longhornctl can complement that evidence, especially when collecting a support bundle, but it should not replace the explicit CRD checks in the runbook.\nMaintenance Rule Install tools before the window, verify their checksums, and record their versions. Then decide which command is authoritative for each gate.\nFor Longhorn maintenance, the safest split is:\nlonghornctl: workstation tool, preflight helper, support bundle helper kubectl + Longhorn CRDs: in-cluster operational state Longhorn UI: fast visual confirmation and guided admin workflows That boundary prevents a CLI install from becoming a process change. The tool is useful; the runbook still needs to prove storage health from the cluster itself.\n","permalink":"https://trinidadmarroquin.com/field-notes/longhornctl-workstation-install-operations-boundary/","section":"field-notes","summary":"Longhorn has a CLI, longhornctl, but installing it should not change the operational source of truth during maintenance. For Rancher-managed Longhorn clusters, the most useful day-to-day state still usually comes from the Longhorn UI and Kubernetes CRDs such as volumes.longhorn.io, replicas.longhorn.io, nodes.longhorn.io, engines.longhorn.io, and settings.longhorn.io.\nThe practical boundary is simple: use longhornctl as an operator tool, not as a reason to skip in-cluster evidence.\nInstall Locally For an Ubuntu workstation, a user-local install avoids system-wide package changes. Confirm the workstation architecture first:\n","tags":["longhorn","kubernetes","rke2","rancher","storage","cli","operations"],"title":"Longhornctl Workstation Install And Operations Boundary"},{"categories":["field-notes"],"content":"After an RKE2 upgrade or node maintenance event, Calico can fail readiness even when the node itself is Ready.\nOne failure pattern is node-local and easy to miss: old Calico or Typha processes survive from a previous RKE2 runtime path and keep owning the ports that the current pods need. Kubernetes shows the new pods as unhealthy, but the real conflict is on the host.\nThis is not a YAML problem first. It is a process ownership problem.\nSymptoms Start with the cluster view:\nkubectl -n calico-system get pods -o wide kubectl get nodes -o wide Typical signals:\none or more calico-node pods are Running but not ready. one or more calico-typha pods are CrashLoopBackOff or repeatedly failing readiness. the affected Kubernetes node is still Ready. the node may have been left cordoned by a previous upgrade or maintenance action. other nodes may be healthy, which makes this look node-specific rather than cluster-wide. Do not uncordon a node just because the upgrade Plan says complete. Check the CNI pods on that node first.\nCheck Host Port Owners On the affected host, check who owns the relevant ports:\nsudo ss -lntup | grep -E \u0026#39;9099|5473|179|4789\u0026#39; || true sudo ps -eo pid,ppid,cmd | grep -E \u0026#39;calico|typha|containerd-shim\u0026#39; | grep -v grep || true Useful ports to recognize:\n9099 Calico Felix health endpoint 5473 Typha 179 BGP, if used 4789 VXLAN Then compare the process path with the current RKE2 runtime path:\nreadlink -f /var/lib/rancher/rke2/bin sudo ls -ld /var/lib/rancher/rke2/data/*/bin 2\u0026gt;/dev/null Problem shape:\ncurrent RKE2 bin path: /var/lib/rancher/rke2/data/v1.32.x-rke2r1-.../bin port owner process: /var/lib/rancher/rke2/data/v1.31.x-rke2r1-.../bin/containerd-shim-runc-v2 That tells you an old runtime process may still be alive while the current Kubernetes pod is trying to start a replacement.\nAsk The Runtime First Before manually killing host processes, ask the current runtime whether it still owns them:\nsudo /var/lib/rancher/rke2/bin/crictl \\ --runtime-endpoint unix:///run/k3s/containerd/containerd.sock ps -a sudo /var/lib/rancher/rke2/bin/ctr \\ --address /run/k3s/containerd/containerd.sock \\ --namespace k8s.io tasks ls If the process is invisible to the current runtime but still owns a Calico or Typha port, it is likely orphaned. At that point, a controlled node reboot is usually safer than deleting old RKE2 data directories or killing process trees by hand.\nReboot One Node At A Time For a worker, the safe pattern depends on workload risk.\nIf you can drain:\nkubectl cordon worker-1 kubectl drain worker-1 --ignore-daemonsets --delete-emptydir-data ssh operator@worker-1.example.com \u0026#39;sudo reboot\u0026#39; kubectl wait node/worker-1 --for=condition=Ready --timeout=30m kubectl uncordon worker-1 If draining would cause unnecessary disruption and the platform owner accepts a no-drain reboot, still cordon first and verify after:\nkubectl cordon worker-1 ssh operator@worker-1.example.com \u0026#39;sudo reboot\u0026#39; kubectl wait node/worker-1 --for=condition=Ready --timeout=30m kubectl -n calico-system get pods -o wide | grep worker-1 kubectl uncordon worker-1 For control-plane nodes, use the existing control-plane maintenance process: one node at a time, cordon/no-drain unless your platform runbook says otherwise, and verify API, etcd, and Calico health before moving to the next node.\nVerify The Reboot Actually Happened Do not trust SSH reconnect alone. Record the boot ID before and after:\ncat /proc/sys/kernel/random/boot_id kubectl get node worker-1 -o jsonpath=\u0026#39;{.status.nodeInfo.bootID}{\u0026#34;\\n\u0026#34;}\u0026#39; The node is not remediated until the boot ID changes and the replacement Calico pods are healthy.\nPost-Reboot Checks After each node returns:\nkubectl get node worker-1 -o wide kubectl -n calico-system get pods -o wide | grep worker-1 kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded -o wide On the host, confirm the stale owners are gone:\nsudo ss -lntup | grep -E \u0026#39;9099|5473|179|4789\u0026#39; || true sudo ps -eo pid,ppid,cmd | grep -E \u0026#39;calico|typha|containerd-shim\u0026#39; | grep -v grep || true Then uncordon only when the node-local CNI state is healthy:\nkubectl uncordon worker-1 Watch For Stale Pod Objects Immediately after a reboot, Kubernetes can briefly show old pods as Unknown or ContainerStatusUnknown while kubelet and controllers reconcile. Do not confuse that transient cleanup with a persistent failure.\nUse a short settle window and check whether the replacement pods are actually ready:\nkubectl get pods -A -o wide --field-selector spec.nodeName=worker-1 kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded -o wide If the only remaining non-running pod is an old completed upgrade artifact or a long-standing unrelated issue, document it separately instead of blocking the CNI remediation.\nThe Pattern The useful triage order is:\nIdentify unhealthy Calico/Typha pods by node. Check host port owners for Calico and Typha ports. Compare process paths with the current RKE2 data directory. Ask crictl and ctr whether the current runtime owns the processes. Reboot one affected node at a time instead of deleting runtime data under live processes. Require boot-ID change, Kubernetes Ready, and Calico readiness before uncordoning. Separate stale pod objects from real post-reboot failures. The key lesson: a CNI readiness failure after RKE2 maintenance may be a host-runtime residue problem. Fix it like node maintenance, not like a manifest typo.\n","permalink":"https://trinidadmarroquin.com/field-notes/rke2-calico-stale-port-owners-after-upgrade/","section":"field-notes","summary":"After an RKE2 upgrade or node maintenance event, Calico can fail readiness even when the node itself is Ready.\nOne failure pattern is node-local and easy to miss: old Calico or Typha processes survive from a previous RKE2 runtime path and keep owning the ports that the current pods need. Kubernetes shows the new pods as unhealthy, but the real conflict is on the host.\nThis is not a YAML problem first. It is a process ownership problem.\n","tags":["rke2","kubernetes","calico","rancher","troubleshooting","operations"],"title":"RKE2 Calico Readiness Failures From Stale Port Owners"},{"categories":["field-notes"],"content":"Burn-rate alerting is easier to operate when the team understands the concepts before touching Prometheus rules.\nThe implementation details matter, but the operational idea is simple: alert when a service is consuming its allowed failure budget too quickly. That is different from alerting because a metric crossed a convenient threshold.\nThis note explains the mental model behind burn-rate alerts. For Prometheus examples and alert rule structure, see SLO Burn-Rate Alerting With Prometheus. For dashboard design after the alert fires, see What To Put On An SLO Dashboard.\nBurn-Rate Alerts A burn-rate alert tells you how fast a service is consuming its error budget.\nInstead of asking:\nIs the error rate above 5%? it asks:\nAre we using the allowed failure budget too quickly? That distinction matters. A 5% error rate may be catastrophic for one service and tolerable for another, depending on the SLO.\nFor example:\nSLO: 99% successful requests over 30 days Error budget: 1% failed eligible requests If the service is currently failing 1% of eligible requests, it is burning budget at the planned rate. If it is failing 2%, it is burning budget twice as fast as planned.\nBurn-rate alerts are useful because they connect paging to reliability policy. The alert is no longer just \u0026ldquo;red line crossed.\u0026rdquo; It is \u0026ldquo;the service is consuming reliability budget fast enough that an operator should act.\u0026rdquo;\nError-Budget Math An error budget is the failure allowance created by an SLO.\nerror budget = 100% - SLO target Examples:\nSLO Target Error Budget 99% 1% 99.9% 0.1% 99.99% 0.01% The burn-rate formula is:\nburn rate = observed failure rate / allowed failure rate For a 99% SLO, the allowed failure rate is 1%.\nObserved failure rate: 2% Allowed failure rate: 1% Burn rate: 2x For a 99.9% SLO, the allowed failure rate is 0.1%.\nObserved failure rate: 1% Allowed failure rate: 0.1% Burn rate: 10x That second example is the one that catches teams. A 1% error rate can sound small in a dashboard review, but for a 99.9% SLO it burns budget ten times faster than allowed.\nMulti-Window Alerting A single alert window is usually either too noisy or too slow.\nA short window catches sharp incidents quickly:\n5 minutes But short windows are noisy. One bad deploy, one dependency timeout, or one small traffic burst can create a scary ratio.\nA longer window confirms the problem is sustained:\n1 hour But long windows are slower to react if used alone.\nMulti-window alerting combines both ideas:\n5-minute burn rate is high AND 1-hour burn rate is high That means the problem is happening now and has lasted long enough to matter.\nA practical pattern is:\nAlert Type Windows Typical Action Fast burn 5m and 1h Page on-call Slow burn 30m and 6h Ticket or team-channel review Fast burn alerts are for active user-impacting failures. Slow burn alerts are for reliability drift that should not wait for a monthly review.\nCommon Failure Modes Burn-rate alerts can still be bad alerts if the underlying SLI or ownership model is weak.\nThe SLI Is Too Broad If all endpoints are aggregated together, a broken low-traffic path can disappear under a noisy high-traffic path.\nExample:\n/healthz receives 1,000,000 successful requests /checkout receives 500 failed requests The overall service may look healthy while the important user path is broken.\nUse service-level alerts for paging, but keep endpoint-level dashboards for diagnosis. For critical paths, define separate SLIs.\nExpected Client Errors Count Against The Budget Not every 4xx response is service unreliability. A 400 caused by invalid client input is different from a 500 caused by a broken database connection.\nThe team needs a written policy:\n5xx responses count against availability. Expected 4xx responses are excluded. Unexpected 4xx patterns are reviewed separately. Without that policy, every burn-rate discussion turns into a debate during the incident.\nLow Traffic Creates Noisy Ratios If one request arrives and fails, the failure rate is 100%.\nThat is mathematically true, but it may not be operationally meaningful.\nLow-traffic services usually need a minimum request-volume gate. The alert should only fire after enough requests have occurred for the ratio to be trustworthy.\nThe Alert Has No Owner A correct alert routed to a general channel is still weak operations.\nEvery burn-rate alert should have:\nservice owner. severity. routing destination. dashboard link. runbook or first-response notes. escalation path. If ownership is unclear, the alert creates noise instead of action.\nError Budget Does Not Change Behavior An error budget is not just a report. It should influence release risk.\nThe policy should answer:\nWhat happens when burn rate is high? What happens when the monthly budget is nearly exhausted? Do risky releases slow down? Who decides when reliability work takes priority? If nothing changes when the budget is exhausted, the SLO is decoration.\nIncident Review Checks After a burn-rate alert fires, review the alert as part of the incident review.\nAsk:\nDid the alert fire before customers reported the issue? Did it route to the right team? Did the dashboard answer the first questions? Was the SLI policy correct? Was the threshold too sensitive or too slow? Did the incident consume enough budget to change release risk? If customers reported the issue first, the alert was missing, too slow, or measuring the wrong thing.\nIf the alert routed to the wrong team, ownership metadata or Alertmanager routing needs work.\nIf the dashboard did not answer the first questions, the alert is not operationally complete. The on-call engineer should be able to see request rate, error rate, burn rate, latency, affected endpoint, recent deployments, and likely ownership from the first response view.\nDo not tune burn-rate alerts only to reduce noise. Tune them to improve decision quality.\nPractical Takeaway Burn-rate alerting works because it links alerting to a reliability promise.\nThe strongest implementation has four parts:\nA clear SLI policy. Error-budget math that matches the SLO. Multi-window alerts that separate fast incidents from slow drift. Incident reviews that improve the alert after it fires. If those pieces are missing, Prometheus can still evaluate the rule, but the organization may not know what decision the rule is supposed to support.\nReferences Google SRE Workbook - Alerting on SLOs Google SRE Workbook - Implementing SLOs SLO Burn-Rate Alerting With Prometheus What To Put On An SLO Dashboard Incident Review Template For SRE Teams On-Call Escalation Policy For Platform Teams ","permalink":"https://trinidadmarroquin.com/field-notes/burn-rate-alerting-concepts-for-operators/","section":"field-notes","summary":"Burn-rate alerting is easier to operate when the team understands the concepts before touching Prometheus rules.\nThe implementation details matter, but the operational idea is simple: alert when a service is consuming its allowed failure budget too quickly. That is different from alerting because a metric crossed a convenient threshold.\nThis note explains the mental model behind burn-rate alerts. For Prometheus examples and alert rule structure, see SLO Burn-Rate Alerting With Prometheus. For dashboard design after the alert fires, see What To Put On An SLO Dashboard.\n","tags":["sre","slo","sli","alerting","observability","incident-response","operations"],"title":"Burn-Rate Alerting Concepts For Operators"},{"categories":["field-notes"],"content":"Certificate automation is not finished when the first certificate is issued.\nThe operational work is the lifecycle: issuer health, renewal timing, DNS or HTTP challenge reliability, secret ownership, expiration alerts, and safe rotation.\nThis field note focuses on cert-manager as a Kubernetes platform component, not just as an application dependency.\nDefine Certificate Ownership Every production certificate should have a clear owner.\nTrack:\nhostname. namespace. owning application or platform team. Certificate resource name. Issuer or ClusterIssuer. challenge type. backing secret. renewal policy. alert route. If a certificate expires and nobody knows who owns it, the platform has an ownership problem, not only a TLS problem.\nResource Model cert-manager usually flows through these resources:\nCertificate -\u0026gt; CertificateRequest -\u0026gt; Order -\u0026gt; Challenge -\u0026gt; Secret Useful checks:\nkubectl get clusterissuer,issuer -A kubectl get certificates -A kubectl get certificaterequests -A kubectl get orders,challenges -A kubectl get secrets -A --field-selector type=kubernetes.io/tls When troubleshooting, avoid looking only at the final secret. The failed state is often visible in CertificateRequest, Order, or Challenge.\nIssuer Health Check issuer readiness before blaming an application ingress.\nkubectl describe clusterissuer letsencrypt-prod kubectl describe issuer -n app-namespace app-issuer Confirm:\nissuer is Ready=True. ACME account registration succeeded. referenced secrets exist. DNS provider credentials are valid for DNS-01. ingress class or solver path is correct for HTTP-01. staging and production issuers are not confused. Use Let\u0026rsquo;s Encrypt staging for testing. Production rate limits are not a validation tool.\nRenewal Windows Certificate renewal should be boring.\nReview:\ncertificate duration. renewBefore setting. alert threshold. owner notification path. rollback plan if renewal fails. Example certificate shape:\napiVersion: cert-manager.io/v1 kind: Certificate metadata: name: app-example-com namespace: app-namespace spec: secretName: app-example-com-tls dnsNames: - app.example.com issuerRef: name: letsencrypt-prod kind: ClusterIssuer duration: 2160h renewBefore: 360h Do not set renewal so close to expiration that one bad DNS provider outage creates an emergency.\nACME Challenge Checks For HTTP-01, confirm:\npublic DNS points to the ingress edge. port 80 is reachable from the internet. solver ingress uses the correct ingress class. redirects do not break the challenge path. NetworkPolicy does not block the solver pod. For DNS-01, confirm:\nDNS provider credentials are valid. credentials can write the required zone. delegated zones are understood. propagation time is accounted for. stale TXT records are not confusing validation. Useful commands:\nkubectl describe challenge -A kubectl describe order -A dig TXT _acme-challenge.app.example.com Secret Rotation Applications consume the generated TLS secret in different ways.\nIngress controllers usually reload automatically. Some workloads mount certificates directly and may need restart or reload behavior.\nReview:\nwhich pods consume the secret. whether the controller reloads on secret update. whether application pods need restart. whether old certificates remain cached by external load balancers. whether monitoring confirms the presented certificate changed. Check the live certificate, not only the Kubernetes secret:\nopenssl s_client -connect app.example.com:443 -servername app.example.com \u0026lt;/dev/null Expiration Alerting Expiration alerts should page early enough to fix the cause during business hours.\nAlert on:\ncertificate expiration within warning window. certificate expiration within critical window. Certificate not ready. failed ACME challenge. issuer not ready. cert-manager controller errors. The alert should include hostname, namespace, certificate name, issuer, and owner.\nAn alert that says \u0026ldquo;certificate expiring\u0026rdquo; without ownership is a scavenger hunt.\nCommon Failure Modes Wrong Issuer A certificate references staging instead of production, or a namespace issuer that no longer exists.\nHTTP-01 Solver Not Reachable The solver pod exists, but DNS, ingress class, redirects, firewall policy, or network policy prevents validation.\nDNS-01 Credentials Too Narrow The secret exists, but the DNS token cannot modify the required zone.\nSecret Name Collision Two certificates or workloads expect different certificates in the same secret name. Keep naming deliberate.\nWildcard Scope Drift Wildcard certificates are convenient but can hide ownership. Document who owns the wildcard and which services depend on it.\nRenewal Succeeds But Edge Still Presents Old Cert The Kubernetes secret updated, but the ingress controller, external load balancer, or application process did not reload.\nReview Checklist Use this checklist before considering certificate lifecycle healthy:\nEvery production certificate has an owner. Issuers are ready and monitored. ACME challenge type is documented. Renewal happens well before expiration. Expiration alerts include hostname, namespace, secret, and owner. Presented certificates are checked from outside the cluster. Secret consumers are known. Wildcard certificate scope is reviewed. Production issuers are not used for repeated testing. Practical Takeaway cert-manager automates certificate issuance, but the platform team still owns lifecycle reliability.\nMonitor the issuer, the challenge path, the renewal window, the secret, and the certificate actually presented to users. TLS is only healthy when the live edge presents the expected certificate and the team knows who owns the next renewal.\nReferences Kubernetes Ingress Operations Checklist Concourse Ingress DNS And TLS Cutover ","permalink":"https://trinidadmarroquin.com/field-notes/cert-manager-certificate-lifecycle/","section":"field-notes","summary":"Certificate automation is not finished when the first certificate is issued.\nThe operational work is the lifecycle: issuer health, renewal timing, DNS or HTTP challenge reliability, secret ownership, expiration alerts, and safe rotation.\nThis field note focuses on cert-manager as a Kubernetes platform component, not just as an application dependency.\nDefine Certificate Ownership Every production certificate should have a clear owner.\nTrack:\nhostname. namespace. owning application or platform team. Certificate resource name. Issuer or ClusterIssuer. challenge type. backing secret. renewal policy. alert route. If a certificate expires and nobody knows who owns it, the platform has an ownership problem, not only a TLS problem.\n","tags":["kubernetes","cert-manager","tls","certificates","acme","letsencrypt","platform-engineering","operations"],"title":"cert-manager Certificate Lifecycle Field Note"},{"categories":["field-notes"],"content":"Cluster autoscaler is capacity automation, not a replacement for capacity ownership.\nIt can add nodes when pods cannot schedule and remove nodes when capacity is unused, but it only works inside the boundaries the platform team gives it: node groups, quotas, labels, taints, pod requests, disruption budgets, and cloud or virtualization capacity.\nThis review is for platform teams that need to know whether autoscaling is safe, predictable, and observable.\nStart With The Capacity Contract Document the autoscaling boundary.\nCluster: Autoscaler owner: Node groups: Minimum nodes: Maximum nodes: Instance or VM types: Critical taints and labels: Scale-up expected time: Scale-down policy: Cloud, vSphere, or provider quota owner: If nobody owns the maximum size, quota, or image supply chain, autoscaler failures will be discovered during incidents.\nCheck Pending Pods First Autoscaler scale-up starts with unschedulable pods.\nkubectl get pods -A --field-selector=status.phase=Pending kubectl describe pod -n app-namespace pending-pod-name Look for scheduler reasons:\ninsufficient CPU. insufficient memory. node selector mismatch. taint not tolerated. topology spread constraints. persistent volume binding. pod affinity or anti-affinity rules. max node group size reached. Not every pending pod is solved by adding nodes. If labels, taints, or storage constraints prevent scheduling, autoscaler may not be able to help.\nReview Autoscaler Logs The autoscaler usually explains its decision.\nkubectl logs -n kube-system deploy/cluster-autoscaler Search for:\nscale-up decisions. node group max size reached. pods not triggering scale-up. failed cloud provider calls. node template mismatch. scale-down blocked by pods. insufficient quota or capacity. The useful question is:\nDid autoscaler choose not to scale, or did it try and fail? Those are different incidents.\nNode Group Design Autoscaler works best when node groups map to clear workload needs.\nReview each node group:\npurpose. min and max size. instance or VM shape. labels. taints. zones or failure domains. image or template version. storage and network assumptions. Avoid one giant generic node group if workloads have distinct needs. Also avoid too many special node groups that fragment capacity and make scheduling unpredictable.\nRequests Drive Scheduling Autoscaler responds to scheduler capacity, and scheduler capacity is based on requests.\nReview workload requests:\nkubectl top pods -A kubectl describe node node-name Look for:\npods with no CPU or memory requests. requests much higher than observed usage. requests too low for actual usage. namespace quotas that block scheduling. limit ranges that set surprising defaults. Bad requests create bad autoscaling. A cluster can add nodes and still have poor reliability if requests do not represent workload needs.\nScale-Up Blockers Common scale-up blockers:\nnode group is already at maximum size. provider quota is exhausted. requested instance or VM type is unavailable. node image or template is broken. bootstrap fails before the node joins. new node joins with wrong labels or taints. CNI fails and node remains NotReady. storage or CSI dependencies fail on new nodes. pod requires a zone with no scalable node group. For RKE2 or vSphere-style environments, node template readiness matters as much as the autoscaler configuration. A new node that cannot join the cluster is not capacity.\nScale-Down Safety Scale-down can be more dangerous than scale-up.\nReview:\nPodDisruptionBudgets. local storage usage. system and daemonset pods. critical workloads with anti-affinity. long-running jobs. stateful workloads. drain behavior. minimum node group sizes. If scale-down evicts the wrong workload at the wrong time, autoscaler becomes a reliability risk.\nUse PDBs to express disruption safety, but do not rely on them as the only control. Operators still need to understand which workloads should not be moved casually.\nObservability Panels Autoscaler dashboards should show:\npending pods by namespace and reason. node count by node group. desired versus current node count. scale-up events. scale-down events. autoscaler errors. node readiness after scale-up. time from pending pod to ready node. provider quota or capacity errors. unschedulable pod count. For incident response, pair autoscaler metrics with scheduler events and node readiness. A scale-up event is not successful until the node is ready and the pod schedules.\nCommon Failure Modes Autoscaler Cannot See The Right Node Group The pending pod requires labels, taints, or zone placement that no autoscaled group can provide.\nMax Size Is Too Low Autoscaler behaves correctly but stops at the configured maximum.\nQuota Blocks Provider Capacity The node group could scale, but cloud or virtualization quota prevents provisioning.\nNew Nodes Join Broken The provider creates the node, but bootstrap, CNI, kubelet, CSI, or certificates fail.\nScale-Down Fights Operations Autoscaler removes nodes during maintenance or while teams are debugging noisy workloads. Pause or constrain scale-down during risky windows when needed.\nRequests Are Wrong Pods either never trigger scale-up when they should, or trigger expensive scale-up because requests are inflated.\nReview Questions Use these during platform review:\nCan every critical workload schedule onto at least one autoscaled node group? Are min and max sizes documented and owned? Does the team know how long scale-up normally takes? Are pending pod reasons visible in dashboards? Are provider quota failures alerted? Do new nodes pass CNI, CSI, and node readiness checks? Are PDBs protecting critical workloads during scale-down? Is there a clear way to pause or limit autoscaler during maintenance? Practical Takeaway Cluster autoscaler should make capacity response predictable.\nIt is healthy when pending pods trigger understandable decisions, node groups match workload needs, scale-up produces ready nodes, scale-down respects disruption safety, and operators can explain why autoscaler did or did not act.\nIf autoscaler is a mystery during incidents, it is not an automation layer. It is another system to troubleshoot.\nReferences Packer Image Factory Workflow RKE2 Worker Join Failures From Calico Wrong Interface Selection Kubernetes Maintenance Evidence Bundles ","permalink":"https://trinidadmarroquin.com/field-notes/cluster-autoscaler-operational-review/","section":"field-notes","summary":"Cluster autoscaler is capacity automation, not a replacement for capacity ownership.\nIt can add nodes when pods cannot schedule and remove nodes when capacity is unused, but it only works inside the boundaries the platform team gives it: node groups, quotas, labels, taints, pod requests, disruption budgets, and cloud or virtualization capacity.\nThis review is for platform teams that need to know whether autoscaling is safe, predictable, and observable.\nStart With The Capacity Contract Document the autoscaling boundary.\n","tags":["kubernetes","cluster-autoscaler","autoscaling","capacity","platform-engineering","operations"],"title":"Cluster Autoscaler Operational Review"},{"categories":["field-notes"],"content":"An incident review should improve the system, not assign blame.\nThe useful output is a better service, a better alert, a better runbook, or a clearer ownership boundary. If the review only produces a timeline and a vague action item, it is documentation theater.\nThis template is designed for platform and SRE teams running production services, Kubernetes platforms, observability stacks, and shared infrastructure.\nRelated notes:\nSLO Burn-Rate Alerting With Prometheus What To Put On An SLO Dashboard On-Call Escalation Policy For Platform Teams Incident Summary Incident title: Incident date: Severity: Status: Services affected: Primary owner: Incident commander: Review date: Write a short summary in plain language.\nExample:\nCheckout API returned elevated 5xx responses for 42 minutes after a database connection pool change. The SLO burn-rate alert paged the on-call engineer, the change was rolled back, and error rates returned to baseline. Avoid starting with root cause if root cause was not known during the incident. Start with observed impact.\nCustomer Or User Impact Describe impact from the user\u0026rsquo;s perspective.\nWho was affected? What could they not do? How many requests, users, tenants, or clusters were affected? When did impact start? When did impact end? Was there data loss, degraded performance, or failed transactions? Good impact statements are specific:\nBetween 14:08 and 14:50 UTC, approximately 7.4% of checkout requests returned 5xx responses. Users could browse products but some checkout attempts failed before payment authorization. Weak impact statements hide the operational reality:\nThe service was degraded for some users. Detection Document how the incident was detected.\nDetected by: Detection time: First alert: First human acknowledgement: Customer report before alert: yes/no Review the detection path:\nDid the alert fire before customers reported the issue? Did it route to the right team? Did it include a useful dashboard link? Did the alert describe user impact or only metric symptoms? Was the alert too sensitive, too slow, or missing? If the incident was customer-reported before monitoring detected it, create a monitoring follow-up.\nTimeline Use objective timestamps.\n14:02 - Deployment started for checkout-api version 2026.07.28.1 14:08 - 5xx error rate increased above baseline 14:12 - Fast burn-rate alert fired 14:14 - On-call acknowledged page 14:18 - Incident channel opened 14:24 - Database connection pool change identified 14:32 - Rollback started 14:50 - Error rate returned to baseline 15:10 - Incident resolved Do not use the timeline to imply blame. The timeline is evidence for understanding detection, diagnosis, mitigation, and communication.\nWhat Happened Describe the technical sequence.\nUseful structure:\nTrigger: Immediate failure: Why users were affected: Why existing controls did not prevent it: Why detection happened when it did: Separate confirmed facts from reasonable hypotheses.\nExample:\nConfirmed: The new connection pool limit caused workers to queue requests under normal traffic. Confirmed: Checkout requests timed out and returned 5xx responses. Hypothesis: The staging load test did not reproduce production concurrency because it used synthetic traffic with lower fan-out. Contributing Factors Most incidents have multiple contributing factors.\nConsider:\nrecent deploys or configuration changes. missing canary or rollback guardrails. dependency behavior. resource exhaustion. unclear ownership. missing runbook steps. dashboards that did not answer first-response questions. alert thresholds that were not tied to SLO impact. Avoid stopping at the first technical cause. The cause may explain why the service failed, but not why the organization was vulnerable to that failure.\nMitigation And Recovery Document what restored service.\nMitigation used: Rollback required: yes/no Data repair required: yes/no Customer communication required: yes/no Verification used: Recovery is not complete just because a deployment was rolled back. Verify user-facing signals:\nrequest success ratio returned to baseline. burn rate returned below alert threshold. latency percentiles returned to baseline. dependency errors stopped. backlog or queue depth recovered. support/customer reports stopped. SLO And Error-Budget Impact If the service has an SLO, quantify the impact.\nSLO target: SLO window: Error budget consumed: Peak burn rate: Budget remaining after incident: Release risk changed: yes/no This is where SLOs become operational policy.\nIf an incident consumed a meaningful amount of budget, the team should decide whether to slow risky changes, prioritize reliability work, or adjust the SLO definition.\nCommunication Review Review internal and external communication.\nWas an incident channel opened? Was there a clear incident commander? Were updates sent on a predictable cadence? Were customer-facing teams informed? Was the status page updated, if applicable? Was resolution clearly communicated? Communication failures create second-order incidents. A technically mitigated incident can still be operationally messy if stakeholders do not know what happened or what to expect.\nWhat Went Well Capture strengths so they are repeated.\nExamples:\nburn-rate alert fired before customer reports. rollback completed quickly. dashboard showed the failing route. incident commander role was clear. dependency owner joined quickly. This section should be specific. \u0026ldquo;Team responded well\u0026rdquo; is too vague to repeat.\nWhat Could Be Improved Focus on system and process improvements.\nExamples:\nstaging load test did not represent production concurrency. dashboard lacked deployment annotations. runbook did not include rollback verification. escalation path for database ownership was unclear. alert description did not include the SLO failure policy. Avoid writing improvements as personal criticism.\nFollow-Up Actions Every action item needs an owner and a deadline.\nAction: Owner: Due date: Tracking link: Verification method: Good action item:\nAdd deployment annotations to the checkout SLO dashboard. Owner: platform-observability Due: 2026-08-07 Verification: next dashboard review confirms deploy markers appear within 60 seconds of rollout start. Weak action item:\nImprove monitoring. If an action cannot be verified, it is not ready.\nReview Checklist Use this checklist before closing the incident review:\nImpact is described from the user perspective. Timeline uses timestamps and facts. Detection path is reviewed. SLO and error-budget impact are quantified when possible. Contributing factors go beyond the immediate technical trigger. Follow-up actions have owners and due dates. Monitoring, dashboard, and runbook gaps are captured. The review avoids blame and focuses on system improvement. Template # Incident Review: \u0026lt;title\u0026gt; ## Summary ## Impact ## Detection ## Timeline ## What Happened ## Contributing Factors ## Mitigation And Recovery ## SLO And Error-Budget Impact ## Communication Review ## What Went Well ## What Could Be Improved ## Follow-Up Actions | Action | Owner | Due Date | Verification | |---|---|---|---| ## Review Checklist Practical Takeaway The incident review is part of the reliability system.\nUse it to improve alerts, dashboards, runbooks, escalation, and release safety. A useful review changes how the next incident is detected, understood, or mitigated.\nIf nothing changes after the review, the team wrote a report but did not improve operations.\n","permalink":"https://trinidadmarroquin.com/field-notes/incident-review-template-for-sre-teams/","section":"field-notes","summary":"An incident review should improve the system, not assign blame.\nThe useful output is a better service, a better alert, a better runbook, or a clearer ownership boundary. If the review only produces a timeline and a vague action item, it is documentation theater.\nThis template is designed for platform and SRE teams running production services, Kubernetes platforms, observability stacks, and shared infrastructure.\nRelated notes:\nSLO Burn-Rate Alerting With Prometheus What To Put On An SLO Dashboard On-Call Escalation Policy For Platform Teams Incident Summary Incident title: Incident date: Severity: Status: Services affected: Primary owner: Incident commander: Review date: Write a short summary in plain language.\n","tags":["sre","incident-response","postmortem","operations","observability","reliability"],"title":"Incident Review Template For SRE Teams"},{"categories":["field-notes"],"content":"Ingress is where Kubernetes platform problems become visible to users.\nThe application may be healthy, pods may be ready, and services may have endpoints, but a bad ingress change can still break the user path. Treat ingress as a production edge, not just another manifest.\nThis checklist is for platform teams operating ingress controllers, DNS records, TLS certificates, and application routes across Kubernetes clusters.\nDefine The Ingress Contract Before troubleshooting an ingress issue, write down the expected path.\nclient -\u0026gt; DNS -\u0026gt; load balancer or VIP -\u0026gt; ingress controller -\u0026gt; Kubernetes Service -\u0026gt; endpoint pod For each production hostname, track:\nDNS owner. load balancer or VIP owner. ingress class. namespace. Kubernetes Service. TLS secret or certificate source. application owner. rollback path. If the team cannot name the owner at each layer, ingress incidents will turn into routing debates.\nPre-Change Checklist Run this before changing ingress rules, certificates, controller configuration, or load balancer routing.\nkubectl get ingress -A kubectl get ingressclass kubectl get svc -A | grep -i ingress kubectl get pods -A -o wide | grep -i ingress kubectl get endpoints -A kubectl get certificaterequests,certificates,orders,challenges -A Confirm:\ntarget hostname resolves to the expected address. ingress object uses the intended ingressClassName. service selector matches running pods. endpoints exist for the service. TLS secret exists in the same namespace as the ingress. certificate is valid for the hostname. controller pods are healthy before the change. rollback manifest or previous release is available. Do not start an ingress change from an unknown baseline.\nDNS Checks DNS is the first dependency in the user path.\nCheck resolution from more than one place:\ndig app.example.com dig app.example.com @1.1.1.1 dig app.example.com @8.8.8.8 Confirm:\nrecord type is expected. returned address is the intended load balancer or VIP. TTL is appropriate for the change window. old records are not still returned by public resolvers. split-horizon DNS is understood if internal and external answers differ. For cutovers, lower TTL before the migration window. Raising TTL after validation is safer than discovering stale resolvers during rollback.\nTLS Checks TLS failures are often mistaken for application failures.\nCheck the certificate presented at the edge:\nopenssl s_client -connect app.example.com:443 -servername app.example.com \u0026lt;/dev/null Confirm:\ncertificate subject or SAN includes the hostname. certificate chain is complete. certificate is not expired. ingress uses the expected TLS secret. cert-manager or external certificate automation owns renewal. wildcard certificates are intentionally scoped. If cert-manager is used, also check the certificate resources:\nkubectl describe certificate -n app-namespace app-example-com kubectl get secret -n app-namespace app-example-com-tls Controller Health Ingress controller health matters before application debugging.\nCheck controller pods and events:\nkubectl get pods -n ingress-nginx -o wide kubectl describe pod -n ingress-nginx -l app.kubernetes.io/component=controller kubectl get events -n ingress-nginx --sort-by=.lastTimestamp Review:\npod restarts. failed readiness or liveness probes. config reload errors. admission webhook failures. resource pressure. node placement. recent controller upgrades. If the controller is unhealthy, do not spend the first thirty minutes debugging the application.\nRoute Validation Validate from outside and inside the cluster.\nExternal path:\ncurl -vk https://app.example.com/healthz Internal service path:\nkubectl run curl-test --rm -it --image=curlimages/curl --restart=Never -- \\ curl -sv http://app-service.app-namespace.svc.cluster.local:8080/healthz Endpoint path:\nkubectl get endpoints -n app-namespace app-service -o wide If internal service calls work but ingress fails, focus on ingress rules, TLS, controller logs, network policy, or load balancer behavior.\nIf service calls fail, ingress is only the messenger.\nObservability Panels Ingress dashboards should show:\nrequest rate by host. status code rate by host and path. 4xx and 5xx trends. latency percentiles. controller reload count. controller pod restarts. upstream response time. active connections. TLS certificate expiration. For incident response, keep the first screen focused on user impact and routing health. Detailed controller internals can live lower on the dashboard.\nCommon Failure Modes Wrong Ingress Class The ingress object may be valid but ignored by the intended controller.\nCheck:\nkubectl get ingress -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,CLASS:.spec.ingressClassName,HOSTS:.spec.rules[*].host Missing Endpoints The ingress routes to a service with no ready endpoints.\nCheck selectors and pod readiness before changing ingress again.\nTLS Secret In Wrong Namespace Ingress TLS secrets are namespace-scoped. A valid secret in another namespace does not help the current ingress.\nDNS Points To Old Edge The Kubernetes objects may be correct while DNS still points to the previous load balancer.\nController Reload Failed Ingress controllers can reject or fail to apply generated configuration. Check controller logs when symptoms do not match the manifest.\nNetworkPolicy Blocks The Controller If NetworkPolicy is enforced, ingress controller pods need allowed paths to application pods. A route can be correct and still blocked.\nRollback Checks A rollback is not complete until the user path is verified.\nAfter rollback, confirm:\nDNS resolves to the intended address. TLS certificate is valid. ingress controller accepted the configuration. service has ready endpoints. external curl succeeds. error rate returned to baseline. latency returned to baseline. customer-facing checks pass. Do not declare recovery from Kubernetes object state alone.\nReview Questions Use these during ingress design or post-incident review:\nCan the team identify the ingress owner in under one minute? Is DNS ownership documented? Are certificate renewal and expiration alerts in place? Does each production ingress specify an ingress class? Can the team validate external, service, and endpoint paths separately? Are ingress controller logs and metrics available during incidents? Is rollback tested for DNS and TLS, not only manifests? Practical Takeaway Ingress operations are edge operations.\nTreat DNS, TLS, controller health, service endpoints, and observability as one user path. A clean Kubernetes manifest is not enough. The only successful ingress change is one that preserves or restores the route users actually take.\nReferences Concourse Ingress DNS And TLS Cutover Network Saturation Evidence Checklist cert-manager Certificate Lifecycle Field Note ","permalink":"https://trinidadmarroquin.com/field-notes/kubernetes-ingress-operations-checklist/","section":"field-notes","summary":"Ingress is where Kubernetes platform problems become visible to users.\nThe application may be healthy, pods may be ready, and services may have endpoints, but a bad ingress change can still break the user path. Treat ingress as a production edge, not just another manifest.\nThis checklist is for platform teams operating ingress controllers, DNS records, TLS certificates, and application routes across Kubernetes clusters.\nDefine The Ingress Contract Before troubleshooting an ingress issue, write down the expected path.\n","tags":["kubernetes","ingress","rke2","nginx","tls","dns","platform-engineering","operations"],"title":"Kubernetes Ingress Operations Checklist"},{"categories":["field-notes"],"content":"An escalation policy should tell the on-call engineer what to do when ownership, severity, or impact is unclear.\nIt is not only a paging schedule. A schedule says who gets woken up. A policy says when to escalate, who should join, and what decisions are expected.\nFor platform teams, this matters because incidents often cross boundaries: Kubernetes, networking, storage, CI/CD, observability, identity, and application ownership.\nRelated notes:\nSLO Burn-Rate Alerting With Prometheus What To Put On An SLO Dashboard Incident Review Template For SRE Teams Policy Goals The policy should optimize for clear ownership and fast mitigation.\nGood escalation policies answer:\nWho owns first response? When should the incident be escalated? Who can make risk decisions? When should application teams be pulled in? When should leadership or customer-facing teams be notified? How is handoff handled? The goal is not to page more people. The goal is to page the right people earlier when the incident needs them.\nSeverity Levels Define severity by user impact and operational risk, not by how scary a metric looks.\nExample platform severity model:\nSeverity Meaning Expected Response SEV1 Widespread production outage, data loss risk, security incident, or critical customer impact Page immediately, incident commander assigned, frequent updates SEV2 Significant degradation, single critical service impaired, high burn rate, or major platform function degraded Page owning team, open incident channel, escalate if not mitigated quickly SEV3 Limited impact, slow burn, partial degradation, or non-critical platform issue Team-channel alert or ticket with owner and deadline SEV4 Informational, maintenance follow-up, or low-risk anomaly Backlog or routine review Tie severity to SLO signals where possible.\nExample:\nFast burn-rate alert for a customer-facing service defaults to SEV2. Fast burn-rate alert plus widespread customer reports defaults to SEV1. Slow burn-rate alert with no immediate customer report defaults to SEV3 unless budget impact is severe. Primary On-Call Responsibilities The primary on-call engineer owns first response.\nResponsibilities:\nacknowledge the page. assess user impact. open an incident channel when needed. start mitigation or rollback if the path is known. escalate when impact, ownership, or mitigation is unclear. document key timestamps. hand off cleanly if the incident exceeds their shift. The primary on-call does not need to solve every problem alone. Waiting too long to escalate is a policy failure, not a badge of ownership.\nSecondary On-Call Responsibilities The secondary on-call engineer provides depth and continuity.\nResponsibilities:\njoin SEV1 and SEV2 incidents when paged or requested. take investigation tasks from the primary. help validate dashboards, logs, traces, and recent changes. prepare handoff if the incident crosses shifts. support communication if no incident commander is assigned yet. The secondary should not silently shadow. If they join, they should take explicit tasks.\nIncident Commander Assign an incident commander for SEV1 and complex SEV2 incidents.\nResponsibilities:\nmaintain incident structure. assign owners for investigation, mitigation, and communication. keep updates on cadence. decide when to escalate further. protect responders from parallel requests. declare mitigation and resolution after evidence supports it. The incident commander does not have to be the deepest technical expert. Their job is coordination and decision flow.\nEscalation Triggers Escalate when any of these are true:\nuser impact is confirmed and mitigation is not obvious. the burn rate remains high after initial response. the service owner is unclear. the incident crosses platform and application boundaries. data integrity, security, or compliance may be involved. rollback is risky or blocked. the primary responder has been investigating for 15 minutes without a credible mitigation path. customer-facing teams need status updates. the incident may exceed the current on-call shift. Time-boxing matters. If the first responder spends too long alone, the incident loses time and context.\nOwnership Routing Platform teams often receive alerts for systems they enable but do not fully own.\nUse a routing model like this:\nSymptom Primary Owner Escalation Partner Kubernetes control plane unhealthy Platform Infrastructure/networking Node pressure or kubelet failures Platform Infrastructure/virtualization Storage attach or mount failures Platform Storage owner CI/CD deployment failure Platform or delivery engineering Application owner Application SLO burn-rate alert Application owner Platform if platform dependency suspected Observability pipeline outage Observability/platform Application owners if alerts are blind Certificate expiration risk Platform/security Application owner if app-specific Document ownership in the alert, dashboard, and service catalog. If ownership only exists in someone\u0026rsquo;s memory, it will fail during an incident.\nCommunication Cadence For SEV1 and SEV2 incidents, define update cadence.\nExample:\nSEV1: internal update every 15 minutes until mitigated SEV2: internal update every 30 minutes until mitigated Customer-facing update: coordinated through support, account, or status-page owner Each update should include:\ncurrent impact. current mitigation status. next action. next update time. Avoid optimistic guesses. Say what is known, what is being checked, and when the next update will arrive.\nHandoff Rules Handoff should preserve operational context.\nMinimum handoff:\nCurrent severity: Current impact: What changed: What was tried: Current hypothesis: Active mitigations: Open risks: Next action: Links to incident channel, dashboard, ticket, and logs: Do not hand off with only \u0026ldquo;see thread.\u0026rdquo; Threads are not operational summaries.\nFatigue And Safety On-call policy should account for responder fatigue.\nRules worth writing down:\npage secondary after a defined duration or severity. rotate incident commander during long incidents. require handoff after extended overnight response. avoid assigning the same person all follow-up actions after a major incident. review noisy alerts after every painful shift. Reliability depends on humans being able to make good decisions. Exhausted responders make worse decisions.\nEscalation Policy Template # On-Call Escalation Policy: \u0026lt;team/service\u0026gt; ## Scope Systems covered: Systems excluded: ## Severity Definitions | Severity | Definition | Response | |---|---|---| ## Primary On-Call Responsibilities: ## Secondary On-Call Responsibilities: ## Incident Commander Assignment rules: Responsibilities: ## Escalation Triggers ## Ownership Routing | Symptom | Primary Owner | Escalation Partner | |---|---|---| ## Communication Cadence ## Handoff Rules ## Fatigue And Safety Rules ## Review Cadence Review Cadence Review the escalation policy after:\nSEV1 incidents. painful SEV2 incidents. missed pages. pages routed to the wrong team. major ownership changes. team schedule changes. new critical services are added. An escalation policy is operational code. If the environment changes and the policy does not, the policy drifts.\nPractical Takeaway An on-call policy should reduce hesitation.\nThe primary on-call should know when to escalate. The secondary should know how to help. The incident commander should know when to coordinate instead of debug. Stakeholders should know when they will hear updates.\nClear escalation turns incident response from individual heroics into a repeatable operating model.\n","permalink":"https://trinidadmarroquin.com/field-notes/on-call-escalation-policy-platform-teams/","section":"field-notes","summary":"An escalation policy should tell the on-call engineer what to do when ownership, severity, or impact is unclear.\nIt is not only a paging schedule. A schedule says who gets woken up. A policy says when to escalate, who should join, and what decisions are expected.\nFor platform teams, this matters because incidents often cross boundaries: Kubernetes, networking, storage, CI/CD, observability, identity, and application ownership.\nRelated notes:\nSLO Burn-Rate Alerting With Prometheus What To Put On An SLO Dashboard Incident Review Template For SRE Teams Policy Goals The policy should optimize for clear ownership and fast mitigation.\n","tags":["sre","on-call","incident-response","operations","platform-engineering","observability"],"title":"On-Call Escalation Policy For Platform Teams"},{"categories":["field-notes"],"content":"Secret rotation is not one operation.\nIt is a lifecycle pattern that depends on the secret type, the consumer, the reload behavior, the rollback path, and the evidence the team needs afterward.\nVault helps, but it does not remove the need to design rotation safely.\nClassify The Secret First Start by identifying what kind of secret is being rotated.\nSecret Type Rotation Pattern Static KV secret write new value, roll consumers, verify, remove old value if applicable Dynamic database credential reduce TTL, revoke leases, let Vault issue new credentials PKI certificate issue new certificate, reload consumer, verify live certificate Transit key rotate key version, rewrap or rewrite old ciphertext if needed API token create replacement, update consumers, revoke old token Kubernetes Secret update source, sync or rollout consumers, verify pod behavior Do not use one generic rotation runbook for every secret type.\nDefine Rotation Ownership Every rotation needs:\nsecret owner. consumer owner. approving authority. change window if needed. rollback plan. verification command. evidence location. Example:\nSecret: kv/data/payments/api/database Owner: payments-platform Consumers: payment-api deployment Rotation type: static credential replacement Verification: application connects with new credential and old credential is rejected Rollback: restore previous credential only if old credential has not been revoked If the owner and consumer are different teams, coordinate before changing the value.\nStatic KV Rotation Static secrets are simple to store and easy to mishandle.\nGeneral flow:\n1. Create or obtain replacement secret. 2. Write replacement to Vault. 3. Roll or reload consumers. 4. Verify consumers use the replacement. 5. Revoke or disable old credential at the upstream system. 6. Confirm old credential no longer works. KV v2 write example:\nvault kv put kv/payments/api/database username=\u0026#34;payment_api\u0026#34; password=\u0026#34;replacement-value\u0026#34; Avoid storing old_password and new_password together unless a controlled dual-secret migration requires it.\nDynamic Credential Rotation Dynamic secrets should rely on leases.\nCheck leases:\nvault list sys/leases/lookup/database/creds/payment-api vault lease lookup \u0026lt;lease-id\u0026gt; Revoke a specific lease:\nvault lease revoke \u0026lt;lease-id\u0026gt; Revoke a prefix when intentionally forcing replacement credentials:\nvault lease revoke -prefix database/creds/payment-api Use prefix revocation carefully. It can break every consumer using that role at once.\nPKI Certificate Rotation Certificate rotation is successful only when the live endpoint presents the new certificate.\nFlow:\n1. Issue or renew certificate. 2. Deliver it to the consumer. 3. Reload or restart the service if required. 4. Check the presented certificate externally. 5. Revoke old certificate if needed. 6. Confirm expiration alerts are clear. Live check:\nopenssl s_client -connect app.example.com:443 -servername app.example.com \u0026lt;/dev/null Kubernetes object state is not enough. Verify the edge.\nTransit Key Rotation Transit rotation changes the key version used for new encryption.\nvault write -f transit/keys/customer-profile/rotate Existing ciphertext remains decryptable through older key versions unless policy prevents it.\nIf old ciphertext should move forward, plan a rewrap migration:\nvault write transit/rewrap/customer-profile ciphertext=\u0026#34;vault:v1:...\u0026#34; Do not raise min_decryption_version until the team proves old ciphertext no longer depends on that version.\nKubernetes Consumer Rotation Kubernetes adds another delivery layer.\nCheck how the secret reaches the pod:\nVault Agent file rendering. CSI secret mount. External Secrets syncing into Kubernetes Secrets. application direct Vault calls. Helm or pipeline injection. Each delivery model has different reload behavior.\nReview:\nDoes the pod need restart? Does the application reload files? Does the controller sync quickly enough? Is secret material copied into etcd? Are old pods still running with old values? For deployments, verify rollout:\nkubectl rollout status deployment/payment-api -n payments kubectl get pods -n payments -o wide Rotation Evidence Capture enough evidence to prove the rotation worked.\nUseful evidence:\nVault path or role, sanitized if needed. timestamp of new version or lease. consumer rollout timestamp. application health after rollout. old credential revocation result. live certificate check for TLS. audit event reference. incident or change ticket. Do not capture raw secret values in evidence bundles.\nCommon Failure Modes New Secret Written But Consumer Not Reloaded Vault has the new value, but the application still uses the old value from memory, file cache, or an old pod.\nOld Credential Not Revoked Rotation reduces little risk if the old credential still works.\nDynamic Lease Revocation Too Broad Prefix revocation breaks more workloads than intended.\nTransit Rotation Misunderstood The key rotates, but stored ciphertext remains on older versions. That may be fine, but it must be understood.\nEvidence Leaks Secrets Rotation logs, shell history, screenshots, and tickets accidentally include secret values.\nReview Checklist Secret type is classified before rotation. owner and consumer are known. delivery mechanism is documented. reload or rollout behavior is tested. old credential revocation is part of the plan. verification checks prove the consumer uses the new secret. evidence excludes raw secret values. audit logs can show who rotated or accessed the secret. rollback is possible or explicitly not allowed. Practical Takeaway Vault makes rotation easier to control, but the real work is consumer lifecycle.\nSuccessful rotation means the new secret is issued, delivered, consumed, verified, and the old secret is revoked or made irrelevant. Anything less is only a partial rotation.\nReferences Vault Token Lease Audit And Recovery Practices Vault PKI Secrets Engine For Internal Certificates Vault Transit Engine For Application Encryption Vault Kubernetes Auth Method Deep Dive ","permalink":"https://trinidadmarroquin.com/field-notes/secrets/secrets-rotation-patterns-with-vault/","section":"field-notes","summary":"Secret rotation is not one operation.\nIt is a lifecycle pattern that depends on the secret type, the consumer, the reload behavior, the rollback path, and the evidence the team needs afterward.\nVault helps, but it does not remove the need to design rotation safely.\nClassify The Secret First Start by identifying what kind of secret is being rotated.\nSecret Type Rotation Pattern Static KV secret write new value, roll consumers, verify, remove old value if applicable Dynamic database credential reduce TTL, revoke leases, let Vault issue new credentials PKI certificate issue new certificate, reload consumer, verify live certificate Transit key rotate key version, rewrap or rewrite old ciphertext if needed API token create replacement, update consumers, revoke old token Kubernetes Secret update source, sync or rollout consumers, verify pod behavior Do not use one generic rotation runbook for every secret type.\n","tags":["vault","secrets","rotation","security","operations","incident-response"],"title":"Secrets Rotation Patterns With Vault"},{"categories":["field-notes"],"content":"An SLO is not useful because it exists in a document. It becomes useful when it changes operational behavior.\nBurn-rate alerting is one way to make that happen. Instead of paging because an error rate crossed a random threshold, the alert asks a better question:\nHow quickly are we consuming the error budget? That question is easier to defend during an incident. It connects the alert to user impact, time window, and reliability policy instead of dashboard aesthetics.\nFor the conceptual model behind these rules, see Burn-Rate Alerting Concepts For Operators. For the first-response view operators need after the alert fires, see What To Put On An SLO Dashboard.\nStart With A Request-Based SLI For HTTP services, start with request success ratio before adding more complicated signals.\nExample policy:\nSuccessful requests: all non-5xx responses Failed requests: 5xx responses Excluded or reviewed separately: expected 4xx responses That policy has to be explicit. A 400 from invalid user input is not the same kind of failure as a 500 from a broken dependency. If the team cannot agree on what counts as failure, the burn-rate alert will only make the disagreement louder.\nWith Prometheus counters, the error-rate SLI usually looks like this:\nsum(rate(http_requests_total{status=~\u0026#34;5..\u0026#34;}[5m])) / sum(rate(http_requests_total[5m])) For the SLI lab used elsewhere on this site, the same shape is:\nsum(rate(sli_http_requests_total{status=~\u0026#34;5..\u0026#34;}[5m])) / sum(rate(sli_http_requests_total[5m])) Convert SLO Target To Error Budget If the SLO target is 99% successful requests over thirty days, the allowed error budget is 1%.\nSLO target: 99% Error budget: 1% Allowed failure: 0.01 of eligible requests Burn rate compares the current error rate against that allowed failure rate.\nburn rate = current error rate / allowed error rate If the service is failing 2% of requests and the budget allows 1%, the burn rate is 2x.\n0.02 / 0.01 = 2 At 2x, the service is consuming budget twice as fast as planned. If nothing changes, the thirty-day budget will be exhausted in about fifteen days.\nBasic Burn-Rate Query For a 99% SLO, the error budget is 0.01.\n( sum(rate(http_requests_total{status=~\u0026#34;5..\u0026#34;}[5m])) / sum(rate(http_requests_total[5m])) ) / 0.01 Using the SLI lab metric names:\n( sum(rate(sli_http_requests_total{status=~\u0026#34;5..\u0026#34;}[5m])) / sum(rate(sli_http_requests_total[5m])) ) / 0.01 This produces a multiplier:\nBurn Rate Meaning 0.5 Budget is being consumed at half the planned rate. 1 Budget is being consumed exactly at the planned rate. 2 Budget is being consumed twice as fast as planned. 14.4 Budget is being consumed fast enough to exhaust a 30-day budget in about two days. Do not page on every value above 1. A service can briefly run above budget without creating a paging-worthy incident. Burn-rate alerting works best with multiple windows.\nWhy Multi-Window Alerts Matter A short window catches fast-moving failures. A long window proves the problem is sustained.\nIf the short window fires alone, it may be a spike. If the long window fires alone, the incident may be moving too slowly for a page but still needs review. When both fire together, the signal is stronger.\nThe common pattern is:\nfast burn: short window + medium window slow burn: medium window + long window Fast burn alerts page. Slow burn alerts can usually route to a ticket or team channel unless the service is critical enough to page earlier.\nExample PrometheusRule This example assumes a 99% availability SLO and a thirty-day window. Adjust the metric names, labels, team routing, and thresholds for the service.\napiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: example-api-slo-burn-rate namespace: monitoring spec: groups: - name: example-api-slo.rules rules: - alert: ExampleApiFastBurnRate expr: | ( sum(rate(http_requests_total{service=\u0026#34;example-api\u0026#34;,status=~\u0026#34;5..\u0026#34;}[5m])) / sum(rate(http_requests_total{service=\u0026#34;example-api\u0026#34;}[5m])) ) / 0.01 \u0026gt; 14.4 and ( sum(rate(http_requests_total{service=\u0026#34;example-api\u0026#34;,status=~\u0026#34;5..\u0026#34;}[1h])) / sum(rate(http_requests_total{service=\u0026#34;example-api\u0026#34;}[1h])) ) / 0.01 \u0026gt; 14.4 for: 2m labels: severity: page service: example-api slo: availability annotations: summary: \u0026#34;example-api is burning availability error budget quickly\u0026#34; description: \u0026#34;The 5m and 1h burn rates are above 14.4x. Treat this as active user-impacting reliability loss.\u0026#34; - alert: ExampleApiSlowBurnRate expr: | ( sum(rate(http_requests_total{service=\u0026#34;example-api\u0026#34;,status=~\u0026#34;5..\u0026#34;}[30m])) / sum(rate(http_requests_total{service=\u0026#34;example-api\u0026#34;}[30m])) ) / 0.01 \u0026gt; 6 and ( sum(rate(http_requests_total{service=\u0026#34;example-api\u0026#34;,status=~\u0026#34;5..\u0026#34;}[6h])) / sum(rate(http_requests_total{service=\u0026#34;example-api\u0026#34;}[6h])) ) / 0.01 \u0026gt; 6 for: 15m labels: severity: ticket service: example-api slo: availability annotations: summary: \u0026#34;example-api is steadily burning availability error budget\u0026#34; description: \u0026#34;The 30m and 6h burn rates are above 6x. Review before the budget loss becomes an incident.\u0026#34; The exact threshold is not sacred. The behavior is the important part: page when burn is fast and sustained; create visible follow-up when burn is slower but still meaningful.\nAdd Guardrails For Low Traffic Low-volume services can produce noisy ratios. One failed request out of one request is 100% failure, but it may not deserve a page.\nAdd a minimum request-volume gate:\nsum(rate(http_requests_total{service=\u0026#34;example-api\u0026#34;}[5m])) \u0026gt; 1 Then combine it with the burn-rate expression:\n( ( sum(rate(http_requests_total{service=\u0026#34;example-api\u0026#34;,status=~\u0026#34;5..\u0026#34;}[5m])) / sum(rate(http_requests_total{service=\u0026#34;example-api\u0026#34;}[5m])) ) / 0.01 \u0026gt; 14.4 ) and sum(rate(http_requests_total{service=\u0026#34;example-api\u0026#34;}[5m])) \u0026gt; 1 For low-traffic internal services, request-count gates may be more useful than request-rate gates:\nsum(increase(http_requests_total{service=\u0026#34;example-api\u0026#34;}[5m])) \u0026gt; 50 The point is not to hide failures. The point is to avoid waking someone up for a mathematically correct but operationally weak signal.\nDashboard Before Pager Before enabling the alert, build a dashboard row that shows the same ingredients:\nrequest rate. 5xx error rate. burn rate by window. p95 and p99 latency. recent deploy markers, if available. top status codes by route or endpoint. The on-call engineer should be able to open the alert and answer:\nIs this real? Is this localized? Is this getting worse? What changed recently? Which service owner should act? If the alert does not lead to that dashboard context, the alert is not ready.\nFailure Modes Burn-rate alerts fail in predictable ways.\nThe SLI Is Too Broad Aggregating all routes can hide the broken path. A low-volume checkout endpoint can disappear under a high-volume health endpoint.\nPrefer service-level alerts for paging and route-level panels for diagnosis. For critical endpoints, define separate SLIs.\nThe SLI Includes Expected Client Errors If expected 4xx responses count against availability, a bad client or integration test can look like a service outage. Decide the policy before the alert exists.\nThe Alert Has No Ownership A good burn-rate alert still fails if it routes to a general channel where no one owns it. Every alert needs service ownership, escalation path, and a runbook or dashboard link.\nThe Team Treats Error Budget As Decoration If budget exhaustion never changes release behavior, the alert becomes another graph with a pager attached. The policy has to say what happens when budget is burning too quickly.\nIncident Review Checks After a burn-rate alert fires, review the alert itself:\nDid it fire before customers reported the issue? Did it route to the right team? Did the dashboard answer the first five questions? Was the SLI policy correct? Was the threshold too sensitive or too slow? Did the incident consume enough budget to change release risk? Do not tune the alert only to reduce noise. Tune it to improve decision quality.\nPractical Takeaway Burn-rate alerting is useful because it links paging to reliability policy. It gives the on-call engineer a reason to act beyond \u0026ldquo;a line crossed a threshold.\u0026rdquo;\nStart with one request-based SLO. Define the failure policy. Add multi-window burn-rate alerts. Gate low-traffic noise. Attach a dashboard. Review every page until the alert either earns trust or gets rewritten.\nReferences Google SRE Workbook — Alerting on SLOs Google SRE Workbook — Implementing SLOs Prometheus Alerting Rules Burn-Rate Alerting Concepts For Operators What To Put On An SLO Dashboard Incident Review Template For SRE Teams On-Call Escalation Policy For Platform Teams Building A Small SLI Lab With Flask, Prometheus, And Grafana ","permalink":"https://trinidadmarroquin.com/field-notes/slo-burn-rate-alerting-prometheus/","section":"field-notes","summary":"An SLO is not useful because it exists in a document. It becomes useful when it changes operational behavior.\nBurn-rate alerting is one way to make that happen. Instead of paging because an error rate crossed a random threshold, the alert asks a better question:\nHow quickly are we consuming the error budget? That question is easier to defend during an incident. It connects the alert to user impact, time window, and reliability policy instead of dashboard aesthetics.\n","tags":["sre","slo","sli","prometheus","alerting","observability","incident-response","operations"],"title":"SLO Burn-Rate Alerting With Prometheus"},{"categories":["field-notes"],"content":"Vault Kubernetes auth lets workloads authenticate to Vault using Kubernetes service account identity.\nThat makes it a powerful bridge between platform identity and secret access. It also means mistakes in service account binding, namespace scoping, policy mapping, or token lifetime can become production secret exposure.\nThis note focuses on operating the auth method safely.\nThe Auth Contract For every workload using Kubernetes auth, document:\ncluster. namespace. service account. Vault auth mount. Vault role. attached policies. token TTL. secret paths allowed. owner. Example:\nCluster: prod-rke2-a Namespace: payments Service account: payment-api Vault auth mount: auth/kubernetes-prod-a Vault role: payment-api Policies: payment-api-read TTL: 1h Owner: payments-platform If this mapping only exists in a pipeline variable or a Helm value, it will be difficult to review during an incident.\nConfigure The Auth Mount Enable a dedicated mount per trust boundary.\nvault auth enable -path=kubernetes-prod-a kubernetes Configure it with the Kubernetes API endpoint, CA certificate, and token reviewer JWT according to your platform model.\nvault write auth/kubernetes-prod-a/config \\ kubernetes_host=\u0026#34;https://kubernetes.default.svc\u0026#34; \\ kubernetes_ca_cert=@ca.crt \\ token_reviewer_jwt=\u0026#34;$TOKEN_REVIEWER_JWT\u0026#34; Production platforms often use explicit cluster-specific mounts instead of one generic auth/kubernetes mount for every cluster. That makes ownership, audit review, and decommissioning clearer.\nBind Roles Narrowly Vault roles should bind to specific service accounts and namespaces.\nvault write auth/kubernetes-prod-a/role/payment-api \\ bound_service_account_names=\u0026#34;payment-api\u0026#34; \\ bound_service_account_namespaces=\u0026#34;payments\u0026#34; \\ policies=\u0026#34;payment-api-read\u0026#34; \\ ttl=\u0026#34;1h\u0026#34; Avoid broad bindings like:\nbound_service_account_names=\u0026#34;*\u0026#34; bound_service_account_namespaces=\u0026#34;*\u0026#34; Wildcard bindings may be useful in tightly controlled automation patterns, but they should be rare, documented, and reviewed.\nPolicy Design The role authenticates the workload. The policy decides what it can read or do.\nExample KV read policy:\npath \u0026#34;kv/data/payments/payment-api/*\u0026#34; { capabilities = [\u0026#34;read\u0026#34;] } Example PKI issuance policy:\npath \u0026#34;pki_internal/issue/payments-services\u0026#34; { capabilities = [\u0026#34;update\u0026#34;] } Do not map many unrelated service accounts to one broad policy. Policy reuse should follow ownership and data boundaries, not convenience.\nToken Lifetimes Vault tokens issued through Kubernetes auth should be short-lived enough to reduce exposure but long-lived enough to avoid unnecessary churn.\nReview:\nrole ttl. role max_ttl. workload renewal behavior. Vault Agent or sidecar behavior. what happens during Vault outage. Short TTLs require reliable renewal. Long TTLs increase exposure when a workload identity is compromised.\nDelivery Patterns Common delivery models:\napplication calls Vault directly. Vault Agent renders secrets to files. CSI driver mounts secrets. external secret controller syncs secrets into Kubernetes Secrets. Each model has different risk.\nDirect application calls keep Kubernetes Secrets out of the path but add application complexity.\nVault Agent centralizes retrieval and renewal but introduces file permissions and reload behavior.\nSyncing into Kubernetes Secrets improves application compatibility but copies secret material into the Kubernetes API and etcd trust boundary.\nPick the model deliberately.\nOperational Checks List auth mounts:\nvault auth list Read a role:\nvault read auth/kubernetes-prod-a/role/payment-api Check policies:\nvault policy read payment-api-read Review Kubernetes service accounts:\nkubectl get serviceaccount -n payments payment-api -o yaml The Vault role, Kubernetes service account, and deployment manifest should agree.\nCommon Failure Modes Namespace Wildcards Leak Access A role allows the same service account name in every namespace. Another team creates a matching service account and receives unintended Vault policy.\nBroad Policy Defeats Narrow Auth The role is tightly bound, but the attached policy grants access to unrelated paths.\nToken Reviewer Breaks Kubernetes API, CA, token reviewer JWT, or RBAC changes prevent Vault from validating service account tokens.\nSecret Delivery Copies Data Too Widely External secret sync writes sensitive data into Kubernetes Secrets across namespaces without clear ownership or cleanup.\nWorkload Cannot Renew The workload receives a token but does not renew it, causing periodic failures that look like application bugs.\nAudit Review Vault audit logs should identify:\nauth mount. role. service account identity. namespace. policy. requested secret path. Use audit review to answer:\nWhich Kubernetes workload accessed this secret? Did access come from the expected namespace? Was the requested path allowed by a narrow policy? Did a decommissioned service account continue authenticating? Review Checklist Each cluster has an intentional auth mount strategy. roles bind to specific service accounts and namespaces. wildcard bindings are documented and rare. policies match workload ownership. token TTL and renewal behavior are tested. secret delivery model is documented. Kubernetes Secrets are avoided or justified for sensitive material. audit logs can tie access back to workload identity. decommissioning removes Vault roles and Kubernetes service accounts. Practical Takeaway Vault Kubernetes auth is strongest when service account identity maps to narrow Vault roles and policies.\nTreat the mapping as production access control. Review auth mounts, role bindings, policies, token lifetimes, delivery patterns, and audit logs together.\nReferences Vault PKI Secrets Engine For Internal Certificates Secrets Rotation Patterns With Vault Kubernetes Maintenance Evidence Bundles ","permalink":"https://trinidadmarroquin.com/field-notes/secrets/vault-kubernetes-auth-method-deep-dive/","section":"field-notes","summary":"Vault Kubernetes auth lets workloads authenticate to Vault using Kubernetes service account identity.\nThat makes it a powerful bridge between platform identity and secret access. It also means mistakes in service account binding, namespace scoping, policy mapping, or token lifetime can become production secret exposure.\nThis note focuses on operating the auth method safely.\nThe Auth Contract For every workload using Kubernetes auth, document:\ncluster. namespace. service account. Vault auth mount. Vault role. attached policies. token TTL. secret paths allowed. owner. Example:\n","tags":["vault","kubernetes","authentication","secrets","security","platform-engineering","operations"],"title":"Vault Kubernetes Auth Method Deep Dive"},{"categories":["field-notes"],"content":"Vault PKI is useful when internal certificate issuance needs policy, auditability, and short-lived credentials instead of manual certificate handling.\nIt is not just a place to mint certificates. It becomes part of the trust path for services, workloads, operators, and automation.\nThis note focuses on operating the PKI secrets engine safely for internal certificates.\nDefine The PKI Boundary Start by deciding what this PKI should and should not issue.\nWrite down:\nissuing use case. allowed domains. allowed common names. allowed subject alternative names. maximum TTL. renewal expectation. revocation process. certificate consumer. owner of the issuing role. Example boundary:\nPKI mount: pki_internal/ Purpose: internal service TLS Allowed domains: svc.internal.example, platform.internal.example Max TTL: 24h for workloads, 720h for platform services Owner: platform-security Excluded: public internet certificates, user certificates, unmanaged wildcard issuance If the boundary is unclear, Vault can become a convenient way to create unmanaged trust.\nRoot And Intermediate Design Avoid using a long-lived root directly for day-to-day issuance.\nA common pattern is:\noffline or tightly controlled root CA -\u0026gt; Vault intermediate CA -\u0026gt; short-lived service certificates Operational review questions:\nWhere is the root key stored? Who can generate or rotate the intermediate? How is the intermediate certificate backed up? How are CRL and issuing certificate URLs published? What happens if the intermediate is compromised? For lab environments, Vault can generate the root. For production, keep the root decision deliberate and documented.\nEnable And Configure A PKI Mount Example commands for an internal intermediate mount:\nvault secrets enable -path=pki_internal pki vault secrets tune -max-lease-ttl=8760h pki_internal Generate an intermediate CSR:\nvault write -format=json pki_internal/intermediate/generate/internal \\ common_name=\u0026#34;internal-platform-intermediate\u0026#34; \\ ttl=8760h \u0026gt; intermediate.csr.json Then sign that CSR with the chosen root process and import the signed intermediate:\nvault write pki_internal/intermediate/set-signed certificate=@intermediate.crt The exact signing process depends on whether the root is another Vault mount, an offline CA, or an enterprise CA workflow.\nConfigure Issuing URLs And CRL URLs Certificates should contain reachable URLs for issuer and revocation information.\nvault write pki_internal/config/urls \\ issuing_certificates=\u0026#34;https://vault.example.internal/v1/pki_internal/ca\u0026#34; \\ crl_distribution_points=\u0026#34;https://vault.example.internal/v1/pki_internal/crl\u0026#34; Use real internal URLs in production. The important part is that clients and operators know where issuer and revocation data live.\nCreate Narrow Roles Roles are the control point for issuance.\nExample service role:\nvault write pki_internal/roles/platform-services \\ allowed_domains=\u0026#34;svc.internal.example,platform.internal.example\u0026#34; \\ allow_subdomains=true \\ allow_bare_domains=false \\ allow_wildcard_certificates=false \\ max_ttl=24h \\ key_type=rsa \\ key_bits=2048 Review role settings carefully:\nallowed_domains should be narrow. wildcard issuance should be intentional. TTL should match consumer reload behavior. role names should map to owners or use cases. role policies should not be shared across unrelated teams. The role is where PKI policy becomes enforceable.\nIssue A Certificate Example:\nvault write pki_internal/issue/platform-services \\ common_name=\u0026#34;api.platform.internal.example\u0026#34; \\ alt_names=\u0026#34;api.svc.internal.example\u0026#34; \\ ttl=8h For automation, prefer machine identity and narrow policies over human tokens.\nThe policy should grant only the needed role path:\npath \u0026#34;pki_internal/issue/platform-services\u0026#34; { capabilities = [\u0026#34;update\u0026#34;] } Do not give broad pki_internal/* access to application deployment automation.\nRenewal And Reload Short-lived certificates reduce long-term exposure, but they require reliable reload behavior.\nFor each consumer, document:\nhow the certificate is requested. where it is stored. how the process reloads it. how renewal failure is detected. how much time remains before expiration when alerts fire. Vault can issue the certificate, but the platform still owns whether the workload uses the new certificate.\nFor Kubernetes workloads, consider whether cert-manager, Vault Agent, CSI drivers, or external secret controllers are responsible for delivery. Do not mix delivery mechanisms without a clear ownership model.\nRevocation And CRL Operations Revocation should be tested before it is needed.\nvault write pki_internal/revoke serial_number=\u0026#34;39:dd:2e:...\u0026#34; vault read pki_internal/crl Review:\nhow serial numbers are captured. who can revoke certificates. whether clients check CRLs. how CRL size is monitored. whether revoked certificates are still accepted by important clients. If clients do not check revocation data, revocation may be mostly administrative evidence. Know that before relying on it during an incident.\nAudit And Evidence Vault audit logs should show certificate issuance activity without exposing private key material.\nReview audit events for:\nissuing path. role name. token or entity identity. requested common name. requested TTL. source address. Do not paste raw audit logs into public notes or tickets without sanitizing hostnames, paths, tokens, and workload identifiers.\nCommon Failure Modes Role Allows Too Much A broad role can issue certificates for names outside the intended service boundary.\nTTL Exceeds Consumer Reality A short TTL is good only if renewal and reload are reliable. A one-hour certificate with no reload path creates outages.\nIntermediate Rotation Is Not Tested The team can issue certificates but has never rotated the intermediate or verified clients trust the new chain.\nRevocation Is Assumed But Not Enforced Certificates are revoked in Vault, but clients do not check revocation information.\nAutomation Uses Human Tokens Certificate issuance automation should use workload identity or tightly scoped machine auth, not copied operator tokens.\nReview Checklist PKI mount purpose is documented. Root and intermediate ownership is clear. issuing and CRL URLs are configured. roles are narrow and owner-mapped. wildcard issuance is disabled unless justified. automation policy grants only the required role path. renewal and reload behavior is tested. revocation workflow is tested. audit logs show expected issuance identity. Practical Takeaway Vault PKI is strongest when it issues short-lived certificates through narrow roles with clear ownership.\nDo not stop at successful issuance. Operate the full lifecycle: root and intermediate design, role policy, renewal, reload, revocation, audit, and incident evidence.\nReferences cert-manager Certificate Lifecycle Field Note Vault Policy Auth And Secrets Engines Vault Token Lease Audit And Recovery Practices ","permalink":"https://trinidadmarroquin.com/field-notes/secrets/vault-pki-secrets-engine-internal-certificates/","section":"field-notes","summary":"Vault PKI is useful when internal certificate issuance needs policy, auditability, and short-lived credentials instead of manual certificate handling.\nIt is not just a place to mint certificates. It becomes part of the trust path for services, workloads, operators, and automation.\nThis note focuses on operating the PKI secrets engine safely for internal certificates.\nDefine The PKI Boundary Start by deciding what this PKI should and should not issue.\nWrite down:\n","tags":["vault","pki","certificates","tls","secrets","security","operations"],"title":"Vault PKI Secrets Engine For Internal Certificates"},{"categories":["field-notes"],"content":"Vault transit gives applications cryptographic operations without handing them raw encryption keys.\nThat is the main value: applications can encrypt, decrypt, sign, verify, or generate data keys through Vault while key material stays inside Vault\u0026rsquo;s trust boundary.\nTransit is not magic encryption. It is an operational contract between the application, Vault, policy, latency, audit logging, and recovery planning.\nDecide What Transit Owns Start with a clear use case.\nGood transit candidates:\nencrypting sensitive fields before database storage. signing internal payloads. verifying signatures. centralizing encryption policy for a service. rotating encryption keys without distributing raw key material. Poor candidates:\nhigh-volume encryption where every request cannot tolerate a Vault call. secrets that should be generated dynamically instead of encrypted at rest. data where the application cannot handle Vault unavailability. workflows with no owner for key rotation or recovery. Transit protects keys. It does not remove the need for application design.\nEnable Transit vault secrets enable transit Create a key for one service or data boundary:\nvault write -f transit/keys/customer-profile Prefer key names that map to ownership and data purpose, not vague names like app-key or prod-key.\nExample boundary:\nKey: customer-profile Owner: identity-platform Purpose: encrypt selected profile fields before storage Consumers: profile-api production workload identity Rotation cadence: quarterly or after incident review Policy Boundaries Applications should receive the minimum transit capabilities they need.\nEncrypt-only policy:\npath \u0026#34;transit/encrypt/customer-profile\u0026#34; { capabilities = [\u0026#34;update\u0026#34;] } Encrypt and decrypt policy:\npath \u0026#34;transit/encrypt/customer-profile\u0026#34; { capabilities = [\u0026#34;update\u0026#34;] } path \u0026#34;transit/decrypt/customer-profile\u0026#34; { capabilities = [\u0026#34;update\u0026#34;] } Admin policy for key management should be separate:\npath \u0026#34;transit/keys/customer-profile\u0026#34; { capabilities = [\u0026#34;read\u0026#34;, \u0026#34;update\u0026#34;] } Do not give application workloads key management rights unless that is explicitly part of the design.\nEncrypt And Decrypt Flow Vault transit expects base64 plaintext for encryption.\nPLAINTEXT=$(printf \u0026#39;sensitive value\u0026#39; | base64) vault write transit/encrypt/customer-profile plaintext=\u0026#34;$PLAINTEXT\u0026#34; The result is a ciphertext value with Vault metadata:\nvault:v1:... Decrypt:\nvault write -field=plaintext transit/decrypt/customer-profile ciphertext=\u0026#34;vault:v1:...\u0026#34; | base64 --decode Applications should store the ciphertext, not the plaintext.\nKey Rotation Transit supports key rotation while preserving decrypt ability for older ciphertext versions.\nvault write -f transit/keys/customer-profile/rotate Review after rotation:\nvault read transit/keys/customer-profile Important settings:\ncurrent key version. minimum decryption version. minimum encryption version. deletion allowance. exportability. Do not raise min_decryption_version casually. Old ciphertext may become unreadable if the application still stores data encrypted with older versions.\nRewrapping Ciphertext Rewrap updates ciphertext to the latest key version without exposing plaintext to the caller.\nvault write transit/rewrap/customer-profile ciphertext=\u0026#34;vault:v1:...\u0026#34; Rewrap is useful when:\nthe key has rotated. stored ciphertext should be migrated forward. applications can process records safely over time. Plan rewrap like a data migration. It needs retry behavior, metrics, and a way to prove progress.\nAvailability And Latency Transit adds a runtime dependency on Vault.\nBefore adopting it, decide:\nCan the application fail closed if Vault is unavailable? Is local caching allowed? What operations need decrypt versus encrypt only? What is the acceptable latency budget? Does every request call Vault, or only specific write/read paths? For high-throughput services, measure transit latency under realistic load. Security architecture that breaks service reliability will eventually be bypassed.\nAudit Expectations Vault audit logs should show:\nauth identity. transit path. operation type. key name. source address. timestamp. They should not expose plaintext.\nUse audit logs to answer:\nWhich identity decrypted this data class? When did key rotation occur? Which workloads still use old policy paths? Did a non-application identity attempt decrypt operations? Common Failure Modes One Key For Everything A single transit key across unrelated services makes ownership, rotation, and incident response harder.\nApplication Has Admin Rights The app should usually encrypt and decrypt, not rotate, delete, export, or change key config.\nRotation Is Confused With Re-Encryption Rotating the key changes future encryption. Existing ciphertext remains on older versions until rewrapped or rewritten.\nVault Latency Is Ignored Every decrypt call is now part of the application path. Measure it.\nMinimum Decryption Version Breaks Old Data Raising minimum decryption version can make older ciphertext unreadable.\nReview Checklist Each transit key has one clear owner. Application policy excludes key administration. rotation and rewrap expectations are documented. Vault availability behavior is defined. latency impact is measured. audit logs are enabled and reviewed. minimum decryption version changes require explicit approval. ciphertext migration has retry and verification behavior. Practical Takeaway Vault transit is a strong pattern when applications need cryptographic operations without managing raw keys.\nOperate it like a production dependency: narrow policies, owner-mapped keys, tested rotation, understood rewrap behavior, latency monitoring, and audit review.\nReferences Vault Policy Auth And Secrets Engines Vault Token Lease Audit And Recovery Practices Secrets Rotation Patterns With Vault ","permalink":"https://trinidadmarroquin.com/field-notes/secrets/vault-transit-engine-application-encryption/","section":"field-notes","summary":"Vault transit gives applications cryptographic operations without handing them raw encryption keys.\nThat is the main value: applications can encrypt, decrypt, sign, verify, or generate data keys through Vault while key material stays inside Vault\u0026rsquo;s trust boundary.\nTransit is not magic encryption. It is an operational contract between the application, Vault, policy, latency, audit logging, and recovery planning.\nDecide What Transit Owns Start with a clear use case.\nGood transit candidates:\n","tags":["vault","transit","encryption","secrets","security","operations"],"title":"Vault Transit Engine For Application Encryption"},{"categories":["field-notes"],"content":"An SLO dashboard should help the on-call engineer make the first decision during an incident.\nIt is not a decoration layer for metrics. It is the operational view that answers:\nIs this user-impacting? How fast are we burning error budget? What changed? Who owns the response? If the dashboard cannot answer those questions quickly, the alert may be technically correct but operationally incomplete.\nThis note pairs with SLO Burn-Rate Alerting With Prometheus and Burn-Rate Alerting Concepts For Operators.\nStart With The Service Contract Put the SLO definition at the top of the dashboard.\nInclude:\nservice name. owning team. paging route. SLO target. SLO window. SLI query summary. what counts as failure. what is intentionally excluded. Example:\nService: checkout-api Owner: platform-payments SLO: 99.9% successful eligible requests over 30 days Failure policy: 5xx responses and dependency timeout responses count as failures Excluded: expected 4xx validation errors and health checks This prevents a common incident failure: operators arguing about the metric while users are already impacted.\nShow Request Volume First Every ratio needs context.\nShow request rate near the top of the dashboard:\nsum(rate(http_requests_total{job=\u0026#34;checkout-api\u0026#34;}[5m])) Request volume helps answer:\nIs the service receiving normal traffic? Did traffic drop because clients stopped calling it? Is a small number of requests creating a noisy failure ratio? Is the alert firing during low-traffic hours? For low-volume services, add a minimum-traffic panel or annotation. A 100% failure rate from one failed request is mathematically true, but it is not always page-worthy.\nShow Success Ratio And Error Ratio Show the user-facing success ratio in the same language as the SLO.\nExample success ratio:\nsum(rate(http_requests_total{job=\u0026#34;checkout-api\u0026#34;,code!~\u0026#34;5..\u0026#34;}[5m])) / sum(rate(http_requests_total{job=\u0026#34;checkout-api\u0026#34;}[5m])) Show the error ratio next to it:\nsum(rate(http_requests_total{job=\u0026#34;checkout-api\u0026#34;,code=~\u0026#34;5..\u0026#34;}[5m])) / sum(rate(http_requests_total{job=\u0026#34;checkout-api\u0026#34;}[5m])) Use the same failure policy as the alert rule. If the dashboard and alert use different definitions, incident response starts with confusion.\nShow Burn Rate Burn rate connects current failures to the reliability promise.\nFor a 99.9% SLO, the allowed failure rate is 0.001.\n( sum(rate(http_requests_total{job=\u0026#34;checkout-api\u0026#34;,code=~\u0026#34;5..\u0026#34;}[5m])) / sum(rate(http_requests_total{job=\u0026#34;checkout-api\u0026#34;}[5m])) ) / 0.001 Show at least two windows:\nPanel Purpose 5m burn rate Confirms the problem is happening now 1h burn rate Confirms the problem is sustained 6h burn rate Shows slow reliability drift 30d budget remaining Shows policy impact The on-call engineer should not have to calculate budget impact manually while triaging.\nShow Latency Percentiles Availability is not the whole user experience.\nShow latency percentiles for the SLO-critical path:\nhistogram_quantile( 0.95, sum by (le) (rate(http_request_duration_seconds_bucket{job=\u0026#34;checkout-api\u0026#34;}[5m])) ) Include:\np50 for baseline behavior. p95 for typical tail pain. p99 for severe tail behavior. Do not rely on averages. Averages hide the users who are having the worst experience.\nIf latency has its own SLO, show it as a separate SLO panel instead of burying it under availability.\nBreak Down The Failing Path After the top-level SLO panels, provide breakdowns for diagnosis.\nUseful breakdowns:\nendpoint or route. status code class. dependency. region or cluster. tenant or customer tier, if safe and appropriate. workload version. pod or instance. Example route error panel:\nsum by (route, code) ( rate(http_requests_total{job=\u0026#34;checkout-api\u0026#34;,code=~\u0026#34;5..\u0026#34;}[5m]) ) The service-level SLO tells you whether to respond. The breakdown panels help you decide where to respond.\nAdd Change Context Many incidents are change-related.\nAdd visible context for:\ndeployments. configuration changes. infrastructure changes. dependency maintenance. feature-flag changes. alert rule changes. In Grafana, use annotations or event panels. A simple deployment marker can reduce minutes of guessing.\nThe first useful incident question is often:\nWhat changed before the burn rate increased? The dashboard should make that question easy to answer.\nAdd Ownership And Response Links The dashboard should route the operator toward action.\nInclude links to:\nservice repository. runbook. Alertmanager route. escalation policy. recent deployments. logs view. trace search. Kubernetes namespace or workload view. If the service has no runbook, link to the incident review template and make creating a runbook a follow-up.\nAvoid Dashboard Failure Modes Common mistakes:\nToo many panels on the first screen. Alert queries and dashboard queries do not match. No traffic volume context. No owner or escalation link. Percentiles missing from latency panels. Endpoint breakdowns shown before the service-level SLO. Dashboards built for monthly review instead of incident response. The first screen should be boring and decisive. Put exploratory panels lower on the page.\nFirst-Response Layout A practical layout:\nRow Panels Service contract owner, SLO target, window, failure policy, runbook User impact request rate, success ratio, error ratio Budget impact burn rate 5m, burn rate 1h, budget remaining Latency p50, p95, p99 Failure breakdown route, status code, dependency, cluster Change context deploys, config changes, infra changes Response links logs, traces, runbook, escalation, repo That layout supports the first ten minutes of response. It does not try to replace deep debugging tools.\nReview Questions Use these questions during dashboard review:\nCan a new on-call engineer identify the owner in under 30 seconds? Does the dashboard use the same SLI definition as the alert? Can the dashboard show whether the alert is still active? Can it show whether the issue is getting better or worse? Can it show what changed before the alert fired? Can it separate user impact from internal noise? Can it guide the operator to the next system of record? If the answer is no, the dashboard is not finished.\nPractical Takeaway An SLO dashboard should support operational decisions, not just display metrics.\nStart with the service contract, show request volume, show SLO health, show burn rate, show latency percentiles, then provide breakdowns and response links.\nThe goal is simple: when a burn-rate alert fires, the dashboard should help the on-call engineer decide whether to page, mitigate, escalate, or watch.\nReferences SLO Burn-Rate Alerting With Prometheus Burn-Rate Alerting Concepts For Operators Building A Small SLI Lab With Flask, Prometheus, And Grafana ","permalink":"https://trinidadmarroquin.com/field-notes/slo-dashboard-first-response-view/","section":"field-notes","summary":"An SLO dashboard should help the on-call engineer make the first decision during an incident.\nIt is not a decoration layer for metrics. It is the operational view that answers:\nIs this user-impacting? How fast are we burning error budget? What changed? Who owns the response? If the dashboard cannot answer those questions quickly, the alert may be technically correct but operationally incomplete.\nThis note pairs with SLO Burn-Rate Alerting With Prometheus and Burn-Rate Alerting Concepts For Operators.\n","tags":["sre","slo","sli","observability","grafana","incident-response","operations"],"title":"What To Put On An SLO Dashboard"},{"categories":["field-notes"],"content":"Longhorn can report no scheduled replicas even when the Kubernetes PVC is still Bound and the data has not obviously disappeared. The message is easy to misread as a corrupt or missing volume. In practice, it often means Longhorn cannot place a replica on any eligible disk because scheduling rules, reservation, or nominal volume size have made the storage pool unschedulable.\nThe important distinction is this:\nactual filesystem usage != Longhorn scheduled capacity A Longhorn node can have free disk blocks while still refusing new replicas because the declared size of scheduled replicas exceeds the safe scheduling limit.\nFirst Checks Start with the affected volume in the Longhorn UI.\nCheck the volume detail page and the Replicas tab:\nIf there are no replicas, treat it as a restore or manual recovery case. If replicas exist but show Stopped, N/A, or Replica Scheduling Failure, continue with node and disk scheduling checks. If replicas are Failed or Unknown, consider Longhorn salvage only after confirming which replica data is trustworthy. Then check the Longhorn node and disk state:\nkubectl -n longhorn-system get nodes.longhorn.io kubectl -n longhorn-system get volumes.longhorn.io,replicas.longhorn.io -o wide For the affected Longhorn node:\nnode=\u0026#39;\u0026lt;longhorn-node-name\u0026gt;\u0026#39; kubectl -n longhorn-system get nodes.longhorn.io \u0026#34;$node\u0026#34; -o yaml Look for these fields under the relevant disk in status.diskStatus:\nstorageAvailable storageMaximum storageScheduled conditions[type=Schedulable] scheduledReplica If storageScheduled is greater than storageMaximum, or greater than the practical scheduling ceiling after reservation and minimum free space, deleting random stopped replicas will not fix the design problem.\nDo Not Delete Stopped Replicas Blindly A stopped replica is not automatically unused. It can be stopped because the workload is not running, the volume is detached, the node is under scheduling pressure, or the replica was replaced during a rebuild.\nDo not delete replicas from the Longhorn Node page just because their status is Stopped.\nBefore deleting anything, identify what the replica belongs to:\npvc_fragment=\u0026#39;\u0026lt;pvc-uuid-fragment\u0026gt;\u0026#39; kubectl get pv | grep \u0026#34;$pvc_fragment\u0026#34; || true kubectl -n longhorn-system get volumes.longhorn.io | grep \u0026#34;$pvc_fragment\u0026#34; || true kubectl -n longhorn-system get replicas.longhorn.io | grep \u0026#34;$pvc_fragment\u0026#34; || true If the PV, Longhorn volume, or Replica CR still exists, the data may still be managed and meaningful.\nScheduled Capacity Versus Used Capacity Longhorn schedules replicas against the declared volume size, not just current physical usage. A mostly empty 150 GiB PVC with three replicas can still consume 450 GiB of scheduled capacity across the cluster.\nInventory declared volume size and replica count:\nkubectl -n longhorn-system get volumes.longhorn.io \\ -o custom-columns=NAME:.metadata.name,REPLICAS:.spec.numberOfReplicas,SIZE:.spec.size,STATE:.status.state,ROBUSTNESS:.status.robustness For a node under pressure, inspect scheduled replicas:\nnode=\u0026#39;\u0026lt;longhorn-node-name\u0026gt;\u0026#39; disk=\u0026#39;\u0026lt;longhorn-disk-name\u0026gt;\u0026#39; kubectl -n longhorn-system get nodes.longhorn.io \u0026#34;$node\u0026#34; \\ -o json \\ | jq --arg disk \u0026#34;$disk\u0026#34; \u0026#39;.status.diskStatus[$disk].scheduledReplica\u0026#39; Large observability volumes such as Prometheus, Loki, MinIO, and log stores can dominate scheduled capacity. They may also have different recoverability requirements than customer databases. Do not assume every volume needs the same Longhorn replica count.\nCheck Real Disk Usage On the Longhorn storage node, compare filesystem usage with Longhorn\u0026rsquo;s scheduled view:\ndf -h /mnt/data sudo du -sh /mnt/data/* sudo du -sh /mnt/data/replicas/* | sort -hr | head -30 This separates two problems:\nHigh physical usage: the disk is actually full or close to full. High scheduled usage: Longhorn has committed more nominal capacity than the disk can safely schedule. Both matter, but they require different fixes.\nOrphaned Replica Data Longhorn creates Orphan CRs for replica data it sees on disk but no longer manages through current Longhorn objects.\nList orphans:\nkubectl -n longhorn-system get orphans.longhorn.io Capture evidence before cleanup:\nkubectl -n longhorn-system get orphans.longhorn.io -o yaml \\ \u0026gt; longhorn-orphans-before-cleanup.yaml Use the correct field for orphan type:\nkubectl -n longhorn-system get orphans.longhorn.io \\ -o jsonpath=\u0026#39;{range .items[*]}{\u0026#34;ORPHAN: \u0026#34;}{.metadata.name}{\u0026#34;\\nNODE: \u0026#34;}{.spec.nodeID}{\u0026#34;\\nTYPE: \u0026#34;}{.spec.orphanType}{\u0026#34;\\nDATA: \u0026#34;}{.spec.parameters.DataName}{\u0026#34;\\nCLEAN: \u0026#34;}{range .status.conditions[?(@.type==\u0026#34;DataCleanable\u0026#34;)]}{.status}{end}{\u0026#34;\\nERROR: \u0026#34;}{range .status.conditions[?(@.type==\u0026#34;Error\u0026#34;)]}{.status}{end}{\u0026#34;\\n\\n\u0026#34;}{end}\u0026#39; Only consider orphan cleanup when the evidence supports it:\norphanType is replica. DataCleanable is True. Error is False. The exact orphan DataName does not match a current replicas.longhorn.io object. Any parent volume that still exists is healthy before and after cleanup. Cross-check orphan data names against managed replicas:\nkubectl -n longhorn-system get replicas.longhorn.io \\ -o jsonpath=\u0026#39;{range .items[*]}{.metadata.name}{\u0026#34;\\n\u0026#34;}{end}\u0026#39; \\ | sort \u0026gt; /tmp/managed-longhorn-replicas.txt kubectl -n longhorn-system get orphans.longhorn.io \\ -o jsonpath=\u0026#39;{range .items[*]}{.spec.parameters.DataName}{\u0026#34;\\n\u0026#34;}{end}\u0026#39; \\ | sort \u0026gt; /tmp/orphan-longhorn-replicas.txt comm -12 /tmp/managed-longhorn-replicas.txt /tmp/orphan-longhorn-replicas.txt Expected output is empty.\nIf Orphan CR Deletion Does Not Free Space Deleting an Orphan CR should remove the associated orphaned replica data. If the CR disappears but disk usage does not change, verify whether the directories still exist on disk.\nSearch for known orphan suffixes:\nsudo find /mnt/data/replicas -maxdepth 1 -type d -name \u0026#39;\u0026lt;orphan-data-name\u0026gt;\u0026#39; sudo du -sh /mnt/data/replicas/\u0026lt;orphan-data-name\u0026gt; Before touching the filesystem manually, confirm all of the following:\nThe exact name returns NotFound from replicas.longhorn.io. A broad replica search finds no managed CR with that suffix. Parent volumes are still healthy. Longhorn manager logs do not show an active cleanup error that needs controller attention first. Example exact-name check:\nfor dir in \\ pvc-example-volume-aaaaaaaa \\ pvc-example-volume-bbbbbbbb do echo \u0026#34;=== $dir ===\u0026#34; kubectl -n longhorn-system get replicas.longhorn.io \u0026#34;$dir\u0026#34; 2\u0026gt;\u0026amp;1 done Check Longhorn manager logs:\nkubectl -n longhorn-system logs \\ -l app=longhorn-manager \\ --since=2h \\ --prefix \\ | grep -Ei \u0026#39;orphan|\u0026lt;replica-suffix-1\u0026gt;|\u0026lt;replica-suffix-2\u0026gt;\u0026#39; If manual cleanup is required, quarantine first. Move within the same filesystem so rollback is possible, then wait and recheck Longhorn health:\nsudo mkdir -p /mnt/data/orphan-quarantine-$(date +%Y%m%d) sudo mv /mnt/data/replicas/\u0026lt;orphan-data-name\u0026gt; \\ /mnt/data/orphan-quarantine-$(date +%Y%m%d)/ kubectl -n longhorn-system get volumes.longhorn.io \\ -o custom-columns=NAME:.metadata.name,STATE:.status.state,ROBUSTNESS:.status.robustness Moving a directory inside the same filesystem does not free space. It only creates a rollback window. After the cluster remains healthy, remove the quarantine:\nsudo rm -rf --one-file-system /mnt/data/orphan-quarantine-YYYYMMDD sync df -h /mnt/data sudo du -sh /mnt/data/replicas If du drops but df does not, check for deleted files still held open:\nsudo lsof +L1 /mnt/data Unused PVC Review When Longhorn remains unschedulable because storageScheduled is too high, reduce managed scheduled capacity. The safest path is to find PVCs that are genuinely unused and delete the PVC, not the PV first.\nBuild an inventory:\nkubectl get pvc -A \\ -o custom-columns=NAMESPACE:.metadata.namespace,PVC:.metadata.name,STATUS:.status.phase,VOLUME:.spec.volumeName,STORAGECLASS:.spec.storageClassName,CAPACITY:.status.capacity.storage,CREATED:.metadata.creationTimestamp kubectl get pv \\ -o custom-columns=PV:.metadata.name,STATUS:.status.phase,CLAIM_NAMESPACE:.spec.claimRef.namespace,CLAIM:.spec.claimRef.name,STORAGECLASS:.spec.storageClassName,RECLAIM:.spec.persistentVolumeReclaimPolicy,CAPACITY:.spec.capacity.storage,CSI_HANDLE:.spec.csi.volumeHandle Find current Pod references:\nkubectl get pods -A -o json \\ | jq -r \u0026#39;.items[] | .metadata.namespace as $ns | .metadata.name as $pod | .spec.volumes[]? | select(.persistentVolumeClaim != null) | [$ns, .persistentVolumeClaim.claimName, $pod] | @tsv\u0026#39; \\ | sort Also check StatefulSet volumeClaimTemplates. A StatefulSet PVC may not appear as a fixed claimName reference:\nkubectl get statefulsets -A -o json \\ | jq -r \u0026#39;.items[] | .metadata.namespace as $ns | .metadata.name as $sts | .spec.volumeClaimTemplates[]? | [$ns, .metadata.name, \u0026#34;statefulset-template\u0026#34;, $sts] | @tsv\u0026#39; \\ | sort Do not treat \u0026ldquo;not mounted by a running Pod\u0026rdquo; as proof that a PVC is disposable. It may belong to a scaled-down StatefulSet, retained customer data, suspended workload, or application recovery path.\nOwnership Search Before Deletion For large detached PVCs, search beyond normal workload kinds. Operators and custom resources may own storage indirectly.\nAt minimum, check:\nnamespace=\u0026#39;\u0026lt;namespace\u0026gt;\u0026#39; pvc=\u0026#39;\u0026lt;pvc-name\u0026gt;\u0026#39; pv=\u0026#39;\u0026lt;pv-name\u0026gt;\u0026#39; kubectl -n \u0026#34;$namespace\u0026#34; get all,pvc kubectl -n \u0026#34;$namespace\u0026#34; get secrets -l owner=helm kubectl api-resources --verbs=list --namespaced -o name \\ | sort -u \\ | while read -r resource; do kubectl -n \u0026#34;$namespace\u0026#34; get \u0026#34;$resource\u0026#34; -o yaml --request-timeout=10s 2\u0026gt;/dev/null \\ | grep -qE \u0026#34;$pvc|$pv\u0026#34; \u0026amp;\u0026amp; echo \u0026#34;MATCH: $resource\u0026#34; done Before deleting, export evidence:\nmkdir -p pvc-cleanup-backup/\u0026lt;namespace\u0026gt; kubectl -n \u0026#34;$namespace\u0026#34; get pvc \u0026#34;$pvc\u0026#34; -o yaml \\ \u0026gt; pvc-cleanup-backup/\u0026lt;namespace\u0026gt;/\u0026lt;pvc\u0026gt;-pvc.yaml kubectl get pv \u0026#34;$pv\u0026#34; -o yaml \\ \u0026gt; pvc-cleanup-backup/\u0026lt;namespace\u0026gt;/\u0026lt;pvc\u0026gt;-pv.yaml volume_handle=$(kubectl get pv \u0026#34;$pv\u0026#34; -o jsonpath=\u0026#39;{.spec.csi.volumeHandle}\u0026#39;) kubectl -n longhorn-system get volumes.longhorn.io \u0026#34;$volume_handle\u0026#34; -o yaml \\ \u0026gt; pvc-cleanup-backup/\u0026lt;namespace\u0026gt;/\u0026lt;pvc\u0026gt;-longhorn-volume.yaml This preserves object configuration, not application data. Get application-owner confirmation before deleting anything with possible customer or business value.\nDelete one PVC at a time:\nkubectl -n \u0026#34;$namespace\u0026#34; delete pvc \u0026#34;$pvc\u0026#34; With reclaim policy Delete, Kubernetes and Longhorn should remove the PVC, PV, and Longhorn volume. Verify all three disappear before deleting another large claim.\nMonitoring And Alerting Longhorn disk pressure should page the team before customers find failed PVC attaches.\nWatch effective disk usage including reservation:\n( longhorn_disk_usage_bytes + longhorn_disk_reservation_bytes ) / longhorn_disk_capacity_bytes Suggested alert shape:\napiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: longhorn-storage-capacity namespace: cattle-monitoring-system spec: groups: - name: longhorn-storage.rules rules: - alert: LonghornDiskUsageHigh expr: | ( longhorn_disk_usage_bytes + longhorn_disk_reservation_bytes ) / longhorn_disk_capacity_bytes \u0026gt; 0.80 for: 15m labels: severity: warning component: longhorn annotations: summary: \u0026#34;Longhorn disk {{ $labels.disk }} on node {{ $labels.node }} is above 80% used\u0026#34; description: \u0026#34;Longhorn disk usage including reservation is high. Replica scheduling failures may follow.\u0026#34; - alert: LonghornDiskUsageCritical expr: | ( longhorn_disk_usage_bytes + longhorn_disk_reservation_bytes ) / longhorn_disk_capacity_bytes \u0026gt; 0.90 for: 5m labels: severity: critical component: longhorn annotations: summary: \u0026#34;Longhorn disk {{ $labels.disk }} on node {{ $labels.node }} is above 90% used\u0026#34; description: \u0026#34;Longhorn replica scheduling and volume attaches may fail due to disk pressure.\u0026#34; Useful dashboard panels:\nLonghorn disk usage by node and disk. Longhorn node storage usage versus capacity. Top volumes by declared size and actual size. Volumes by state and robustness. Orphan count by node. For reactive detection, alert on Longhorn manager logs that contain Replica Scheduling Failure, Replica scheduling failed, or insufficient storage.\nPractical Recovery Order Use this order during an incident:\nConfirm whether replicas exist for the affected volume. Check Longhorn node and disk schedulability. Compare physical usage with storageScheduled. Do not delete stopped replicas unless they are proven orphaned and unmanaged. Clean reviewed Longhorn orphan resources first. If orphan CR cleanup fails to remove disk data, quarantine exact unmanaged directories before deletion. Identify genuinely unused PVCs through ownership search. Delete unused PVCs one at a time and verify PVC, PV, and Longhorn volume removal. Recheck storageScheduled, storageAvailable, and the disk Schedulable condition. Address the durable cause: add capacity, reduce replica count for appropriate workloads, move large observability data, or right-size retained PVCs through migration. Reusable Utilities The manual checks in this note have reusable versions in the public ops-toolbox repository:\nlonghorn-scheduler-pressure-report.sh reports Longhorn disk schedulability, scheduled capacity, available capacity, and scheduled replica counts. longhorn-orphan-report.sh reports Longhorn orphan resources and exact overlap with active Replica CRs. longhorn-pvc-ownership-audit.sh audits PVC ownership signals before cleanup review. Those tools are read-only by default. They are intended to preserve the audit workflow without carrying environment-specific names, customer data, or incident artifacts.\nReferences Longhorn Documentation Longhorn Orphaned Data Cleanup Longhorn Metrics ops-toolbox Longhorn utilities Longhorn To Enterprise Storage Migration Patterns ","permalink":"https://trinidadmarroquin.com/field-notes/longhorn-no-scheduled-replicas-disk-pressure/","section":"field-notes","summary":"Longhorn can report no scheduled replicas even when the Kubernetes PVC is still Bound and the data has not obviously disappeared. The message is easy to misread as a corrupt or missing volume. In practice, it often means Longhorn cannot place a replica on any eligible disk because scheduling rules, reservation, or nominal volume size have made the storage pool unschedulable.\nThe important distinction is this:\nactual filesystem usage != Longhorn scheduled capacity A Longhorn node can have free disk blocks while still refusing new replicas because the declared size of scheduled replicas exceeds the safe scheduling limit.\n","tags":["longhorn","kubernetes","storage","rke2","rancher","prometheus","troubleshooting","operations"],"title":"Longhorn No Scheduled Replicas Under Disk Pressure"},{"categories":["field-notes"],"content":"Sometimes the maintenance path is blocked by host access, not Kubernetes health.\nIn one RKE2 reboot window, SSH worked to the nodes, but noninteractive sudo did not. The maintenance automation needed to reboot hosts and verify node-level state. The workaround was a temporary privileged DaemonSet that wrote a short-lived sudoers rule onto each node, then stayed alive only long enough for the maintenance window.\nThat pattern can be valid in a controlled emergency or tightly scoped window. It is also a host access change. Treat it with the same seriousness as adding an SSH key, changing a sudoers file, or granting a break-glass account.\nWhat The Pattern Does The mechanism is simple:\nprivileged DaemonSet hostPath mount to the host filesystem container writes a temporary host access rule operator verifies noninteractive access maintenance runs operator removes the host rule operator deletes the DaemonSet operator verifies access is gone The Kubernetes object is only the delivery mechanism. The real change is on every host where the DaemonSet schedules.\nPre-Checks Before applying anything, prove the current state:\nkubectl get nodes -o wide kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded -o wide kubectl -n kube-system get daemonset Then check host access explicitly:\nssh ops-user@192.0.2.10 \u0026#39;true\u0026#39; ssh ops-user@192.0.2.10 \u0026#39;sudo -n whoami\u0026#39; Record the expected result. If sudo -n fails before the helper and succeeds after the helper, you can prove the helper changed access. If you skip the negative pre-check, you cannot prove what changed.\nScope The Blast Radius Do not apply a host-mutating DaemonSet casually to the entire cluster.\nDecide up front:\nwhich nodes need the access. whether storage nodes should be excluded. whether control-plane nodes and worker nodes need different windows. how long the helper is allowed to exist. who approved the access change. where the cleanup evidence will be stored. If the helper must run broadly, keep the window short and make cleanup a required step, not a follow-up ticket.\nAvoid Copy-Paste Blindness A helper like this usually needs powerful settings:\nsecurityContext: privileged: true volumes: - name: host-etc hostPath: path: /etc Those two lines mean the pod can modify host configuration. The review question is not “does the YAML apply?” The review question is “are we intentionally allowing a Kubernetes workload to mutate host access control?”\nUse a unique, temporary file name under /etc/sudoers.d/ or the equivalent access-control location for the operating system. Do not edit the main sudoers file in place. Set restrictive permissions. Make the cleanup command remove only the file the helper created.\nVerify Positive Access After rollout, verify both Kubernetes and host behavior:\nkubectl -n kube-system rollout status daemonset/temporary-host-access --timeout=5m kubectl -n kube-system get pods -l app=temporary-host-access -o wide ssh ops-user@192.0.2.10 \u0026#39;sudo -n whoami\u0026#39; The evidence should show:\nhelper scheduled on intended nodes noninteractive sudo works on intended nodes non-target nodes were not modified or were intentionally included If the helper does not schedule on a node, do not assume that node has access. If a node is tainted, cordoned, or unreachable, verify it separately.\nRun Maintenance, Then Clean Up Immediately The helper exists to support the maintenance action. It should not outlive that action.\nCloseout should remove both layers:\nremove host access file from every touched node delete the DaemonSet wait for DaemonSet pods to disappear verify noninteractive sudo fails again The negative verification matters:\nssh ops-user@192.0.2.10 \u0026#39;sudo -n whoami\u0026#39; Expected result after cleanup:\nsudo: a password is required Do not stop at kubectl delete daemonset. Deleting the DaemonSet removes the delivery pod. It does not prove the host file was removed.\nEvidence To Keep Retain a small evidence bundle:\npre-helper sudo -n failure. helper rollout status. post-helper sudo -n success on intended nodes. maintenance logs. host access file removal output. DaemonSet deletion or NotFound confirmation. post-cleanup sudo -n failure. Redact hostnames, IPs, usernames, file names, and cluster names before sharing outside the operations boundary.\nStop Criteria Stop and reassess if:\nthe helper schedules on unexpected nodes. the helper cannot be removed from a node. sudo remains passwordless after cleanup. a node is unreachable during cleanup. the DaemonSet uses broader host mounts than required. the access change was not explicitly approved for the window. The most dangerous failure mode is a successful maintenance window with forgotten elevated access left behind.\nThe Rule A temporary privileged DaemonSet is not just Kubernetes automation. It is a distributed host mutation.\nUse it only when the operational need is clear, keep it scoped to the maintenance window, and prove both sides of the change:\nbefore: access does not exist during: access exists only where intended after: access is gone That final negative check is what turns a risky workaround into a controlled operational procedure.\n","permalink":"https://trinidadmarroquin.com/field-notes/temporary-privileged-daemonset-maintenance-access/","section":"field-notes","summary":"Sometimes the maintenance path is blocked by host access, not Kubernetes health.\nIn one RKE2 reboot window, SSH worked to the nodes, but noninteractive sudo did not. The maintenance automation needed to reboot hosts and verify node-level state. The workaround was a temporary privileged DaemonSet that wrote a short-lived sudoers rule onto each node, then stayed alive only long enough for the maintenance window.\nThat pattern can be valid in a controlled emergency or tightly scoped window. It is also a host access change. Treat it with the same seriousness as adding an SSH key, changing a sudoers file, or granting a break-glass account.\n","tags":["kubernetes","rke2","security","maintenance","operations"],"title":"Temporary Privileged DaemonSets Are Host Access Changes"},{"categories":["field-notes"],"content":"An observability namespace can look like a platform outage even when customer workloads are healthy.\nIn one downstream-cluster investigation, the noisy symptoms were all in monitoring and alerting: logging resources disappearing, an agent operator stuck during deletion, alert delivery failures, external secret authentication errors, and exporter noise. The first useful move was to split the question in two:\nIs the cluster currently unhealthy? Is the observability stack being removed, broken, or reconciled by another owner? Those are different incidents.\nProve Cluster Health First Start with the generic health view before chasing monitoring components:\nkubectl get nodes -o wide kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded -o wide kubectl get pv,pvc -A kubectl get events -A --sort-by=.lastTimestamp Then check the customer or platform namespaces that matter for service availability:\nkubectl get deploy,statefulset,daemonset,pod -A -o wide kubectl get ingress,svc -A If nodes are Ready, critical pods are Running, and PVCs are bound, say that clearly. Do not let an observability failure become an assumed application outage.\nCheck The Ownership Plane Before repairing deleted observability resources, find out who owns them.\nCheck common local ownership surfaces:\nkubectl get applications.argoproj.io -A kubectl get helmreleases -A kubectl get helmcharts.helm.cattle.io -A helm list -A kubectl get bundles.fleet.cattle.io -A If Argo CD or Flux CRDs are absent locally, that does not prove there is no GitOps owner. A downstream cluster may only expose an agent locally while the upstream system owns the desired state elsewhere. In that case, local Helm state can tell you what is not managing the resource, but it may not identify the real owner.\nRead Deletion Evidence A namespace with a few leftovers may be in teardown, not partial failure.\nUseful checks:\nkubectl get all,cm,secret,pvc,sa,role,rolebinding -n logging kubectl get deploy -n telemetry -o yaml kubectl get events -n logging --sort-by=.lastTimestamp kubectl get events -n telemetry --sort-by=.lastTimestamp Look for:\nmany Killing events in a short window. deleted Deployments or DaemonSets with finalizers. only PVCs, TLS secrets, or root CA ConfigMaps left behind. an operator stuck at 0/1 while its Deployment has a deletion timestamp. no corresponding Helm release in the local cluster. That pattern points toward active removal or external pruning. The right next step is usually ownership confirmation, not manual recreation.\nSeparate Storage Topology From App Health Monitoring stacks often use persistent storage. Attach errors may be real, but still scoped to observability.\nCheck whether the storage driver is available on the nodes where the monitoring pods are trying to run:\nkubectl get nodes -L topology.kubernetes.io/zone -o wide kubectl get pods -n storage-system -o wide kubectl describe pod -n logging loki-example-0 kubectl describe pvc -n logging data-loki-example-0 The important question is whether the volume is trying to attach to a node that can actually run that CSI path. If only worker nodes advertise the needed CSI topology but monitoring pods are scheduled onto monitor-only nodes, the symptom is storage placement, not necessarily a broken cluster.\nAlert Delivery Is Its Own Failure Alertmanager failures should be triaged separately from workload health.\nCheck whether alerts are firing, whether Alertmanager is healthy, and whether notification delivery is failing:\nkubectl -n monitoring get pods -o wide kubectl -n monitoring logs deploy/alertmanager --since=2h kubectl -n monitoring get secret,configmap Common findings:\nwebhook returns 403 or another authorization failure. the alert route exists, but the receiver credential is expired or revoked. Alertmanager is correctly detecting issues but cannot notify the external system. That is an alert delivery incident. Treat it that way instead of assuming all firing alerts imply customer impact.\nExternal Secret Errors Need Consumers External secret controller errors can be urgent, but first check whether anything depends on the failing store:\nkubectl get clustersecretstore,secretstore -A kubectl get externalsecret -A kubectl -n external-secrets logs deploy/external-secrets --since=2h If a store cannot authenticate but no ExternalSecret objects consume it, that is still a configuration problem, but it may not explain current workload impact. If consumers exist and secrets are stale or missing, escalate it as an application dependency issue.\nExporter Noise Is Not Always Platform Failure Exporter errors can flood logs and alerts after database or application version drift.\nCheck the relationship between exporter version, target version, and the failing query:\nkubectl -n monitoring logs deploy/postgres-exporter --since=2h kubectl -n monitoring get cm,secret | grep exporter If an exporter repeatedly emits query-shape errors, the immediate operational value is to reduce false noise and fix the exporter/query compatibility. Do not let exporter noise mask higher-priority platform symptoms.\nThe Triage Pattern Use this order:\nProve node, pod, PVC, and critical workload health. Identify whether observability resources are locally managed or externally reconciled. Read deletion timestamps, finalizers, and events before recreating anything. Separate storage placement failures from application failures. Treat alert delivery failures as notification-path incidents. Check external secret consumers before declaring customer impact. Classify exporter errors as signal quality problems unless they block workloads. The conclusion should be explicit:\ncustomer workload impact: yes/no/unknown observability stack health: healthy/deleting/broken/externally owned alert delivery: healthy/failing/unknown secret dependency impact: yes/no/unknown next owner: platform/app/observability/security/GitOps That separation keeps the response factual. Observability can be broken at the same time the cluster is serving traffic. It can also be the only system that would have told you about a real outage. Triage both facts without merging them into one vague incident.\n","permalink":"https://trinidadmarroquin.com/field-notes/downstream-observability-teardown-triage/","section":"field-notes","summary":"An observability namespace can look like a platform outage even when customer workloads are healthy.\nIn one downstream-cluster investigation, the noisy symptoms were all in monitoring and alerting: logging resources disappearing, an agent operator stuck during deletion, alert delivery failures, external secret authentication errors, and exporter noise. The first useful move was to split the question in two:\nIs the cluster currently unhealthy? Is the observability stack being removed, broken, or reconciled by another owner? Those are different incidents.\n","tags":["kubernetes","observability","rancher","gitops","operations"],"title":"Downstream Observability Teardown Triage"},{"categories":["field-notes"],"content":"Good maintenance windows produce evidence. Bad evidence bundles become a new secret store.\nDuring RKE2 reboot and upgrade work, the most useful artifacts were not complicated: preflight JSON, per-batch reboot logs, final cluster state captures, and error files. They proved which context was used, which nodes were touched, whether boot IDs changed, whether workers were drained, whether PodDisruptionBudgets blocked eviction, and whether the cluster returned to the expected baseline.\nThat same evidence can expose internal hostnames, IP addresses, SSH usernames, kubeconfig paths, workload names, Vault paths, webhook URLs, and temporary access helpers. Treat maintenance output as operational evidence and sensitive data at the same time. For the access-helper side of this problem, see Temporary Privileged DaemonSets Are Host Access Changes.\nKeep The Bundle Shape Boring Use predictable file names that encode the maintenance phase without exposing unnecessary detail:\ncluster-a-preflight.json cluster-a-batch-1-control-plane.log cluster-a-batch-2-etcd.log cluster-a-workers.log cluster-a-final-state.json cluster-a-final-errors.err The useful pattern is phase-based:\npreflight batch execution post-batch health final validation errors and exceptions That lets another operator reconstruct what happened without reading a chat transcript or guessing which terminal command mattered.\nCapture Operator Intent Every execution log should start with the maintenance inputs that change behavior:\nKUBE_CONTEXT=cluster-a-prod DRAIN_WORKERS=true READY_TIMEOUT=30m REBOOT_SETTLE_SECONDS=90 POLL_SECONDS=15 WAIT_FOR_BOOT_ID=true VERIFY_HOST_REBOOT=true These values explain the run. If a worker was drained, the log should say so. If a control-plane node was cordoned but not drained, the log should say so. If the run required host boot-ID verification, the log should show that requirement before any node is touched.\nCapture Before And After State For node maintenance, retain enough state to prove the sequence:\nkubectl config current-context kubectl get nodes -o wide kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded -o wide kubectl get pdb -A kubectl get events -A --sort-by=.lastTimestamp For RKE2 and Longhorn, add role-specific checks:\nkubectl -n kube-system get pods -o wide kubectl -n longhorn-system get volumes.longhorn.io,replicas.longhorn.io,nodes.longhorn.io -o wide kubectl -n longhorn-system get pdb -o wide For reusable Longhorn evidence collection, the public ops-toolbox Longhorn utilities include read-only reports for scheduler pressure, orphan resources, and PVC ownership signals. If longhornctl is installed for preflight checks or support bundles, record its version and keep the authoritative health gates tied to Longhorn CRDs; see Longhornctl Workstation Install And Operations Boundary.\nThe important part is not the exact command list. The important part is that the same evidence exists before the first node and after the last node.\nDrain Output Is Evidence Do not discard drain output. It often reveals the real maintenance constraint.\nExamples of useful signals:\nwhich pods were evicted successfully. which DaemonSet pods were intentionally ignored. whether a PDB blocked eviction. whether a storage controller waited and then allowed eviction. whether the drain completed before reboot. For Longhorn, an instance-manager PDB can temporarily block eviction. That is useful information, not just noisy output. It tells the operator that storage components were in the disruption path and that future worker maintenance should account for Longhorn placement before assuming a normal drain will be quick.\nVerify Reboot With Two Views SSH availability only proves a host is reachable. It does not prove the host rebooted.\nCapture the pre-reboot and post-reboot boot IDs:\ncat /proc/sys/kernel/random/boot_id kubectl get node worker-1 -o jsonpath=\u0026#39;{.status.nodeInfo.bootID}{\u0026#34;\\n\u0026#34;}\u0026#39; The evidence bundle should show:\npre host boot ID pre Kubernetes boot ID post host boot ID changed post Kubernetes boot ID changed or reconciled node Ready after reboot This prevents a false success where SSH reconnects but the host never restarted, or Kubernetes still shows stale node information while the kubelet is catching up.\nIf a node reboots unexpectedly during a vSphere-backed incident, include vCenter task and event history in the evidence bundle. A vSphere HA reset from VMware Tools heartbeat failure looks different from an in-guest reboot or Rancher/system-upgrade action; see vSphere HA Reset Evidence For Kubernetes Nodes.\nRedact Before Sharing Raw maintenance artifacts are usually safe to keep in the controlled operations workspace. They are not safe to paste into tickets, public notes, blog posts, or vendor cases without review.\nRedact or generalize:\nreal cluster names. internal DNS names. RFC 1918 IP addresses. SSH usernames and key paths. kubeconfig paths. Vault paths and auth mount names. webhook URLs. customer workload names. temporary privileged helper names. organization-specific labels and namespaces. Prefer examples like:\ncluster-a-prod cp-1 etcd-1 worker-1 storage-1 192.0.2.10 vault.example.com Do not redact away the operational meaning. Keep the role, phase, and outcome.\nClose The Loop An evidence bundle should answer five questions:\nWhat context and cluster did the operator intend to touch? What was the health baseline before maintenance? Which nodes were touched, in what order, and with what drain behavior? What proved each host actually rebooted? What final checks proved the cluster returned to baseline? If the bundle answers those questions, it is useful during review. If it also has a redaction plan, it is safe to reuse for procedures, postmortems, and public writing.\nThe runbook is the plan. The evidence bundle is how you prove the plan actually happened.\n","permalink":"https://trinidadmarroquin.com/field-notes/kubernetes-maintenance-evidence-bundles/","section":"field-notes","summary":"Good maintenance windows produce evidence. Bad evidence bundles become a new secret store.\nDuring RKE2 reboot and upgrade work, the most useful artifacts were not complicated: preflight JSON, per-batch reboot logs, final cluster state captures, and error files. They proved which context was used, which nodes were touched, whether boot IDs changed, whether workers were drained, whether PodDisruptionBudgets blocked eviction, and whether the cluster returned to the expected baseline.\nThat same evidence can expose internal hostnames, IP addresses, SSH usernames, kubeconfig paths, workload names, Vault paths, webhook URLs, and temporary access helpers. Treat maintenance output as operational evidence and sensitive data at the same time. For the access-helper side of this problem, see Temporary Privileged DaemonSets Are Host Access Changes.\n","tags":["kubernetes","rke2","operations","runbooks","security"],"title":"Kubernetes Maintenance Evidence Bundles Need A Redaction Plan"},{"categories":["posts"],"content":"Node reboots are easy to underestimate in Kubernetes. They look smaller than upgrades because the target version does not change, but the risk profile can be just as high when storage is already unhealthy.\nIn one RKE2 maintenance window, the cluster needed operating-system reboots while Longhorn was not fully healthy. That changed the plan. The work could not be treated as a generic rolling reboot across every node. It needed role-based batches, explicit health gates, and a hard boundary around storage nodes until the Longhorn state was understood. It also needed retained evidence; see Kubernetes Maintenance Evidence Bundles Need A Redaction Plan for the artifact-handling side of this procedure.\nThe useful lesson was simple: when Longhorn is degraded, the reboot plan is a storage-risk plan first and a node-maintenance plan second.\nStart With The Storage State Before touching nodes, capture enough state to know whether a reboot will move the cluster closer to recovery or further away from it:\nkubectl get nodes -o wide kubectl get pods -A --field-selector=status.phase!=Running kubectl -n longhorn-system get pods -o wide kubectl -n longhorn-system get volumes.longhorn.io kubectl -n longhorn-system get engines.longhorn.io,replicas.longhorn.io For Longhorn, do not stop at kubectl get pods. A namespace full of Running pods does not prove volume health. Look for degraded volumes, detached volumes, failed replicas, rebuilding replicas, missing engines, and instance-manager churn.\nAlso identify which Kubernetes nodes are storage-bearing nodes. In a simple worker pool, that may be obvious. In a real cluster, confirm it from Longhorn placement and replica state instead of assuming every worker has the same blast radius.\nSplit Nodes By Failure Domain For RKE2 maintenance, avoid a single list of hosts named all_nodes. Split the work by role and risk:\ncontrol-plane / etcd nodes regular workers storage-bearing workers That split matters because each group has different stop criteria.\nControl-plane and etcd nodes should be rebooted in small batches, usually one at a time unless the cluster design and quorum math explicitly allow more. Regular workers can be drained and rebooted in controlled batches if workload disruption budgets allow it. Storage-bearing workers should be excluded when Longhorn is degraded unless the recovery plan specifically requires touching one.\nThe dangerous shortcut is to treat a successful first batch as proof that the next batch is safe. The first batch only proves that the first failure domain survived.\nGate Every Batch Use boring checks between every batch. The point is not to collect impressive output. The point is to decide whether the next reboot is allowed.\nControl-plane and etcd gates:\nkubectl get nodes -o wide kubectl -n kube-system get pods -l component=etcd -o wide sudo rke2 etcd-snapshot ls sudo ETCDCTL_API=3 etcdctl endpoint health \\ --cacert=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt \\ --cert=/var/lib/rancher/rke2/server/tls/etcd/client.crt \\ --key=/var/lib/rancher/rke2/server/tls/etcd/client.key \\ --endpoints=https://127.0.0.1:2379 Cluster gates:\nkubectl get nodes kubectl get pods -A --field-selector=status.phase!=Running kubectl get events -A --sort-by=.lastTimestamp Longhorn gates:\nkubectl -n longhorn-system get pods -o wide kubectl -n longhorn-system get volumes.longhorn.io kubectl -n longhorn-system get replicas.longhorn.io For a reusable version of the Longhorn storage checks, see the public ops-toolbox Longhorn utilities. The scheduler pressure report is especially useful before deciding whether a storage-bearing node is safe to reboot. If longhornctl is part of the operator workstation, keep its role explicit; see Longhornctl Workstation Install And Operations Boundary.\nContinue only when the previous batch has returned to the expected baseline. If the baseline already includes degraded storage, write down that accepted condition. Do not let a known pre-existing degraded state hide a new regression.\nVerify The Host Actually Rebooted Automation can report success while a host never restarted. SSH sessions can reconnect to the same boot. A reboot command can be skipped by sudo policy, shell behavior, or a wrapper script.\nCheck the boot ID before and after maintenance:\ncat /proc/sys/kernel/random/boot_id uptime -s Record the pre-reboot boot ID, issue the reboot, wait for SSH and Kubernetes readiness, then confirm the boot ID changed. This is a small check, but it prevents a false sense of completion during a long maintenance window.\nDrain Regular Workers, Not Storage Blindly For regular workers, use the normal Kubernetes maintenance shape:\nkubectl cordon worker-1 kubectl drain worker-1 --ignore-daemonsets --delete-emptydir-data sudo reboot kubectl uncordon worker-1 Then confirm the node and pods recovered:\nkubectl get node worker-1 -o wide kubectl get pods -A -o wide --field-selector spec.nodeName=worker-1 Do not apply that same pattern blindly to storage-bearing workers while Longhorn is degraded. Draining can move workload pods, but it does not magically make storage safe. A reboot can interrupt the only healthy replica path for a volume that is already degraded.\nStorage nodes need their own decision point:\nIs the affected volume healthy enough to lose this node temporarily? Are replicas placed on other healthy nodes? Is a rebuild in progress? Is the engine attached to this host? Is this node carrying the only remaining good replica for any volume? If those answers are unknown, the safe action is to skip the storage node and continue with lower-risk nodes only.\nKeep Temporary Access Temporary Maintenance windows often require short-lived access helpers: temporary sudo rules, one-time SSH keys, a jump-host allowance, or an emergency operator account. Those changes should be part of the runbook, not a forgotten side effect. If Kubernetes is used to deliver host access, treat that as a host mutation; see Temporary Privileged DaemonSets Are Host Access Changes.\nTrack them explicitly:\ncreate temporary access perform maintenance verify cluster health remove temporary access verify access was removed Cleanup is not administrative polish. It is part of returning the platform to its normal security baseline.\nStop Criteria For this kind of work, stop criteria should be written before the first reboot.\nStop if:\netcd health fails after a control-plane reboot. a control-plane node does not return Ready within the expected window. new Longhorn volumes become degraded or faulted. replica rebuilds begin unexpectedly on nodes still scheduled for reboot. workload disruption exceeds the approved maintenance impact. the next node is storage-bearing and Longhorn placement is not understood. The stop decision is easier when the plan has already said what evidence matters.\nThe Pattern The safest shape from this maintenance window was:\nCapture Kubernetes, etcd, and Longhorn state. Group nodes by role and storage risk. Reboot control-plane and etcd nodes in small batches with quorum checks. Reboot regular workers through cordon, drain, reboot, uncordon. Exclude storage-bearing workers while Longhorn is degraded unless a specific recovery plan requires them. Verify each host with boot IDs, not just SSH availability. Gate every batch on Kubernetes, etcd, and Longhorn health. Remove temporary maintenance access before closing the window. A reboot runbook is successful when it makes the next action obvious: continue, pause, or stop. Longhorn degradation removes the margin for vague sequencing. Treat every reboot as a storage-aware change, and the maintenance window becomes much easier to control.\n","permalink":"https://trinidadmarroquin.com/posts/rke2-node-reboots-longhorn-degraded-risk/","section":"posts","summary":"Node reboots are easy to underestimate in Kubernetes. They look smaller than upgrades because the target version does not change, but the risk profile can be just as high when storage is already unhealthy.\nIn one RKE2 maintenance window, the cluster needed operating-system reboots while Longhorn was not fully healthy. That changed the plan. The work could not be treated as a generic rolling reboot across every node. It needed role-based batches, explicit health gates, and a hard boundary around storage nodes until the Longhorn state was understood. It also needed retained evidence; see Kubernetes Maintenance Evidence Bundles Need A Redaction Plan for the artifact-handling side of this procedure.\n","tags":["rke2","kubernetes","longhorn","rancher","maintenance","operations"],"title":"RKE2 Node Reboots When Longhorn Is Already Degraded"},{"categories":["field-notes"],"content":"Ubuntu nodes can have the unattended-upgrades package installed without actively running upgrades. The package state alone is not enough. Check the config, service, timers, logs, and package history before deciding whether a node is safe.\nFor production Kubernetes nodes, the desired state is usually:\nunattended-upgrades package absent or inert unattended-upgrades.service inactive and disabled apt-daily.timer disabled or masked apt-daily-upgrade.timer disabled or masked /etc/apt/apt.conf.d/20auto-upgrades set to 0 patching handled through controlled maintenance windows Audit First Use multiple signals. An empty unattended-upgrades log is useful, but it does not prove the feature is disabled.\ndpkg -l unattended-upgrades cat /etc/apt/apt.conf.d/20auto-upgrades 2\u0026gt;/dev/null || true systemctl is-enabled unattended-upgrades.service apt-daily.timer apt-daily-upgrade.timer systemctl is-active unattended-upgrades.service apt-daily.timer apt-daily-upgrade.timer systemctl list-timers | grep apt || true sudo journalctl -u unattended-upgrades --since \u0026#34;7 days ago\u0026#34; --no-pager sudo ls -lh /var/log/unattended-upgrades/ 2\u0026gt;/dev/null || true Interpret the state carefully:\nSignal Meaning package installed The capability exists, but may be inactive 20auto-upgrades has Unattended-Upgrade \u0026quot;1\u0026quot; Automatic upgrades are enabled apt-daily-upgrade.timer enabled The system may trigger unattended upgrade activity empty unattended-upgrades log It may not have run recently, but verify timers and config journal entries during an incident window Correlate with package, service, storage, and kubelet events Disable Cleanly Prefer stopping and disabling systemd units before removing packages. Do not start with kill -9 unless a process is stuck and you have already tried a clean stop.\nsudo systemctl stop unattended-upgrades.service apt-daily.service apt-daily-upgrade.service 2\u0026gt;/dev/null || true sudo systemctl disable unattended-upgrades.service apt-daily.service apt-daily-upgrade.service 2\u0026gt;/dev/null || true sudo systemctl stop apt-daily.timer apt-daily-upgrade.timer 2\u0026gt;/dev/null || true sudo systemctl disable apt-daily.timer apt-daily-upgrade.timer 2\u0026gt;/dev/null || true sudo systemctl mask apt-daily.timer apt-daily-upgrade.timer 2\u0026gt;/dev/null || true Then make the config inert:\nsudo tee /etc/apt/apt.conf.d/20auto-upgrades \u0026gt;/dev/null \u0026lt;\u0026lt;\u0026#39;EOF\u0026#39; APT::Periodic::Update-Package-Lists \u0026#34;0\u0026#34;; APT::Periodic::Download-Upgradeable-Packages \u0026#34;0\u0026#34;; APT::Periodic::AutocleanInterval \u0026#34;0\u0026#34;; APT::Periodic::Unattended-Upgrade \u0026#34;0\u0026#34;; EOF If the operating model is to remove the package entirely:\nsudo apt-get remove --purge -y unattended-upgrades sudo rm -f /etc/apt/apt.conf.d/50unattended-upgrades Inject The Intent In Packer The durable place to express this policy is the node image, not only the remediation playbook. If Packer builds the Ubuntu template used for Kubernetes nodes, add a provisioner that makes the template\u0026rsquo;s intent explicit: automatic apt activity is off before any clone ever joins a cluster.\nPlace this after the base package installation stage and before final template cleanup. That lets the build install required packages first, then freeze the update policy into the image baseline.\nbuild { sources = [\u0026#34;source.vsphere-iso.ubuntu\u0026#34;] provisioner \u0026#34;shell\u0026#34; { inline = [ \u0026#34;while [ ! -f /var/lib/cloud/instance/boot-finished ]; do echo \u0026#39;Waiting for cloud-init...\u0026#39;; sleep 2; done\u0026#34;, \u0026#34;sudo apt-get update\u0026#34;, \u0026#34;sudo DEBIAN_FRONTEND=noninteractive apt-get install -y open-vm-tools openssh-server lvm2 xfsprogs\u0026#34;, ] } provisioner \u0026#34;shell\u0026#34; { inline = [ \u0026#34;set -eu\u0026#34;, \u0026#34;echo \u0026#39;Disabling unattended apt activity for Kubernetes node template...\u0026#39;\u0026#34;, \u0026#34;sudo systemctl stop unattended-upgrades.service apt-daily.service apt-daily-upgrade.service 2\u0026gt;/dev/null || true\u0026#34;, \u0026#34;sudo systemctl disable unattended-upgrades.service apt-daily.service apt-daily-upgrade.service 2\u0026gt;/dev/null || true\u0026#34;, \u0026#34;sudo systemctl stop apt-daily.timer apt-daily-upgrade.timer 2\u0026gt;/dev/null || true\u0026#34;, \u0026#34;sudo systemctl disable apt-daily.timer apt-daily-upgrade.timer 2\u0026gt;/dev/null || true\u0026#34;, \u0026#34;sudo systemctl mask apt-daily.timer apt-daily-upgrade.timer 2\u0026gt;/dev/null || true\u0026#34;, \u0026#34;sudo tee /etc/apt/apt.conf.d/20auto-upgrades \u0026gt;/dev/null \u0026lt;\u0026lt;\u0026#39;EOF\u0026#39;\\nAPT::Periodic::Update-Package-Lists \\\u0026#34;0\\\u0026#34;;\\nAPT::Periodic::Download-Upgradeable-Packages \\\u0026#34;0\\\u0026#34;;\\nAPT::Periodic::AutocleanInterval \\\u0026#34;0\\\u0026#34;;\\nAPT::Periodic::Unattended-Upgrade \\\u0026#34;0\\\u0026#34;;\\nEOF\u0026#34;, \u0026#34;sudo apt-get remove --purge -y unattended-upgrades || true\u0026#34;, \u0026#34;sudo rm -f /etc/apt/apt.conf.d/50unattended-upgrades\u0026#34;, \u0026#34;systemctl is-enabled apt-daily.timer apt-daily-upgrade.timer 2\u0026gt;/dev/null || true\u0026#34;, \u0026#34;test ! -e /etc/apt/apt.conf.d/50unattended-upgrades\u0026#34; ] } provisioner \u0026#34;shell\u0026#34; { inline = [ \u0026#34;sudo apt-get autoremove -y\u0026#34;, \u0026#34;sudo apt-get clean\u0026#34;, \u0026#34;sudo rm -rf /var/lib/apt/lists/*\u0026#34; ] } } If the organization prefers keeping the package installed but inert, omit the apt-get remove --purge line and keep the config plus masked timers. The important part is that the image declares the operational boundary: Kubernetes node patching is owned by maintenance automation, not by background apt timers.\nFor a more modular Packer layout, keep the policy in a dedicated script and call it from the template:\nprovisioner \u0026#34;file\u0026#34; { source = \u0026#34;scripts/disable-unattended-upgrades.sh\u0026#34; destination = \u0026#34;/tmp/disable-unattended-upgrades.sh\u0026#34; } provisioner \u0026#34;shell\u0026#34; { inline = [ \u0026#34;sudo install -m 0755 -o root -g root /tmp/disable-unattended-upgrades.sh /usr/local/sbin/disable-unattended-upgrades\u0026#34;, \u0026#34;sudo /usr/local/sbin/disable-unattended-upgrades\u0026#34;, \u0026#34;sudo rm -f /usr/local/sbin/disable-unattended-upgrades\u0026#34; ] } That keeps the Packer HCL readable while still making the image build fail if the policy script fails. Do not leave this as a wiki-only instruction. If the template is the source of truth for node operating-system behavior, the unattended-upgrades state belongs in the template build.\nAnsible Pattern In Ansible, avoid capturing newline-separated PIDs and passing them directly to kill. A task like this is unsafe when pgrep returns more than one process:\nshell: \u0026#34;kill -9 {{ apt_get_pid.stdout }}\u0026#34; Use services first, then verify, then process cleanup only as a last resort.\nvars: apt_timers: - apt-daily.timer - apt-daily-upgrade.timer apt_services: - apt-daily.service - apt-daily-upgrade.service - unattended-upgrades.service tasks: - name: Stop and mask automatic apt timers ansible.builtin.systemd: name: \u0026#34;{{ item }}\u0026#34; state: stopped enabled: false masked: true no_block: true loop: \u0026#34;{{ apt_timers }}\u0026#34; ignore_errors: true tags: [unattended, disable, timers] - name: Stop and disable unattended upgrade services ansible.builtin.systemd: name: \u0026#34;{{ item }}\u0026#34; state: stopped enabled: false no_block: true loop: \u0026#34;{{ apt_services }}\u0026#34; ignore_errors: true tags: [unattended, disable, services] - name: Disable apt periodic configuration ansible.builtin.copy: dest: /etc/apt/apt.conf.d/20auto-upgrades owner: root group: root mode: \u0026#34;0644\u0026#34; content: | APT::Periodic::Update-Package-Lists \u0026#34;0\u0026#34;; APT::Periodic::Download-Upgradeable-Packages \u0026#34;0\u0026#34;; APT::Periodic::AutocleanInterval \u0026#34;0\u0026#34;; APT::Periodic::Unattended-Upgrade \u0026#34;0\u0026#34;; tags: [unattended, disable, config] If process cleanup is still needed, separate discovery from action and handle an empty result safely:\n- name: Find lingering apt or unattended-upgrade processes ansible.builtin.shell: | pgrep -f \u0026#39;(apt-get.*update|/usr/bin/apt|apt.systemd.daily|unattended-upgrade|/usr/bin/dpkg)\u0026#39; || true register: apt_lingering changed_when: false failed_when: false tags: [unattended, cleanup] - name: Force kill lingering apt processes only as a last resort ansible.builtin.command: \u0026#34;kill -9 {{ item }}\u0026#34; loop: \u0026#34;{{ apt_lingering.stdout_lines | map(\u0026#39;trim\u0026#39;) | select(\u0026#39;match\u0026#39;, \u0026#39;^\\\\d+$\u0026#39;) | list }}\u0026#34; when: apt_lingering.stdout | length \u0026gt; 0 ignore_errors: true tags: [unattended, cleanup] That pattern works when zero, one, or many PIDs are returned. It also avoids failing when the discovery task is run and finds nothing.\nPurge With Lock Awareness If the package must be purged, wait briefly for apt and dpkg locks before invoking the apt module:\n- name: Wait for apt and dpkg locks to clear ansible.builtin.shell: | for i in $(seq 1 30); do if fuser /var/lib/dpkg/lock-frontend /var/lib/dpkg/lock /var/lib/apt/lists/lock /var/cache/apt/archives/lock \u0026gt;/dev/null 2\u0026gt;\u0026amp;1; then sleep 2 else exit 0 fi done exit 0 changed_when: false failed_when: false tags: [unattended, purge] - name: Purge unattended-upgrades ansible.builtin.apt: name: unattended-upgrades state: absent purge: true autoremove: true force_apt_get: true register: purge_unattended retries: 5 delay: 10 until: purge_unattended is succeeded tags: [unattended, purge] Verify The Desired State Verification should not depend on previous tasks having run in the same play. That matters when using --tags verify.\nsystemctl is-enabled apt-daily.timer apt-daily-upgrade.timer unattended-upgrades.service || true systemctl is-active apt-daily.timer apt-daily-upgrade.timer unattended-upgrades.service || true cat /etc/apt/apt.conf.d/20auto-upgrades 2\u0026gt;/dev/null || true dpkg -l unattended-upgrades 2\u0026gt;/dev/null || true Expected output for a disabled-but-installed model:\nAPT::Periodic::Unattended-Upgrade \u0026#34;0\u0026#34;; apt-daily.timer masked apt-daily-upgrade.timer masked unattended-upgrades.service inactive or disabled Expected output for a purged model:\npackage not installed timers disabled or masked no unattended-upgrades journal activity after the change Related article: Ubuntu Unattended Upgrades Are Kubernetes Node Changes.\n","permalink":"https://trinidadmarroquin.com/field-notes/ubuntu-unattended-upgrades-kubernetes-nodes/","section":"field-notes","summary":"Ubuntu nodes can have the unattended-upgrades package installed without actively running upgrades. The package state alone is not enough. Check the config, service, timers, logs, and package history before deciding whether a node is safe.\nFor production Kubernetes nodes, the desired state is usually:\nunattended-upgrades package absent or inert unattended-upgrades.service inactive and disabled apt-daily.timer disabled or masked apt-daily-upgrade.timer disabled or masked /etc/apt/apt.conf.d/20auto-upgrades set to 0 patching handled through controlled maintenance windows Audit First Use multiple signals. An empty unattended-upgrades log is useful, but it does not prove the feature is disabled.\n","tags":["ubuntu","ansible","kubernetes","updates","maintenance","operations"],"title":"Disabling Ubuntu Unattended Upgrades On Kubernetes Nodes"},{"categories":["notes"],"content":"Ubuntu unattended upgrades are useful on ordinary servers and workstations. On Kubernetes nodes, they are production changes.\nThat distinction matters because a Kubernetes worker is not just a Linux host. It is part of a larger control loop that includes kubelet, the container runtime, CNI, CSI, storage paths, systemd units, udev device handling, and sometimes iSCSI or multipath. A package update that is safe in isolation can become disruptive when it lands outside a maintenance window.\nThe Failure Pattern The incident pattern looks like this:\nunattended-upgrades starts during normal operations core OS packages change systemd, udev, or device-management triggers run storage or network services churn iSCSI / multipath / block-device handling stalls or resets CSI node plugin loses registration or stops responding kubelet volume operations fail on the node the node becomes degraded or NotReady The exact package list varies, but updates to components such as systemd, udev, libudev, libsystemd, PAM/NSS systemd libraries, or kernel-adjacent packages deserve special attention. They participate in process supervision, device events, service restart behavior, and block-device discovery.\nThat is the same control plane that Kubernetes storage relies on.\nWhy Storage Nodes Are Sensitive Kubernetes storage paths are layered:\napplication pod kubelet volume manager CSI node plugin filesystem mount block device multipath / iSCSI / storage network array target If device handling stalls under udev, if multipath commands block, or if iSCSI sessions churn, kubelet does not see that as a neat package-update event. It sees mount operations timing out, CSI calls failing, plugin sockets disappearing, or volumes that cannot detach cleanly.\nThe most important diagnostic line in this class of incident is usually not \u0026ldquo;package upgraded\u0026rdquo;. It is a kubelet or event message like:\ndriver name csi.example.com not found in the list of registered CSI drivers That means the failure crossed a boundary. It is no longer just storage noise. Kubelet has lost the node-local storage interface it needs to manage volumes.\nCorrelation Evidence To Collect When investigating a node that went NotReady after unattended upgrades, collect the timeline before rebooting if possible.\nStart with unattended-upgrades:\nsudo journalctl -u unattended-upgrades \\ --since \u0026#34;2026-07-20 06:00\u0026#34; \\ --until \u0026#34;2026-07-20 08:00\u0026#34; \\ --no-pager sudo grep -h \u0026#34;Packages that will be upgraded\u0026#34; \\ /var/log/unattended-upgrades/unattended-upgrades.log* sudo less /var/log/unattended-upgrades/unattended-upgrades-dpkg.log Then check the host services and kernel around the same window:\nsudo journalctl --since \u0026#34;2026-07-20 06:00\u0026#34; --until \u0026#34;2026-07-20 08:00\u0026#34; --no-pager | \\ grep -Ei \u0026#39;apt|unattended|systemd|udev|iscsi|multipath|containerd|kubelet\u0026#39; sudo journalctl -k --since \u0026#34;2026-07-20 06:00\u0026#34; --until \u0026#34;2026-07-20 08:00\u0026#34; --no-pager | \\ grep -Ei \u0026#39;scsi|reset|lun|multipath|iscsi|blk|i/o|timeout|hung|blocked\u0026#39; From Kubernetes, correlate node and volume symptoms:\nkubectl describe node worker-1 kubectl get events -A --sort-by=.lastTimestamp | \\ grep -Ei \u0026#39;worker-1|FailedMount|NodeNotReady|KubeletNotReady|CSI|VolumeFailed\u0026#39; The strongest case is a timeline where unattended upgrades begin, core device or service packages are installed, udev or systemd activity increases, storage paths churn, and kubelet then reports CSI or mount failures on the same node.\nDisable Automation, Not Patching The fix is not to stop patching Ubuntu nodes. The fix is to stop patching them silently.\nFor production Kubernetes nodes, prefer this posture:\nunattended-upgrades disabled apt timers disabled or masked package changes scheduled in maintenance windows nodes cordoned and drained before disruptive updates reboots handled intentionally post-update checks required before uncordon Disabling unattended upgrades does create a responsibility: the platform team must own patch cadence. That is still safer than letting a node restart services or touch device-management packages while stateful workloads are running.\nManual Node Patch Pattern Use the normal Kubernetes maintenance flow:\nkubectl cordon worker-1 kubectl drain worker-1 --ignore-daemonsets --delete-emptydir-data Patch the host:\nsudo apt update apt list --upgradable sudo NEEDRESTART_MODE=a apt upgrade -y if [ -f /var/run/reboot-required ]; then sudo reboot fi After the node returns:\nkubectl get node worker-1 -o wide kubectl describe node worker-1 kubectl uncordon worker-1 For storage-backed clusters, add checks for the node-local storage stack:\nsudo iscsiadm -m session sudo multipath -ll sudo journalctl -u kubelet -n 100 --no-pager The point is sequencing. Update one node or one controlled batch at a time, with a clear stop condition if storage, CSI, CNI, or kubelet health degrades.\nOperational Rule Treat Ubuntu package updates on Kubernetes nodes like cluster changes, not background hygiene.\nThat means they need:\ninventory scope. maintenance windows. drain and uncordon steps. pre-checks for CSI, CNI, and node readiness. post-checks for storage sessions, kubelet, and workload recovery. audit output showing which nodes were updated and which still need attention. For the companion audit and disable pattern, see Disabling Ubuntu Unattended Upgrades On Kubernetes Nodes.\n","permalink":"https://trinidadmarroquin.com/posts/ubuntu-unattended-upgrades-kubernetes-node-risk/","section":"posts","summary":"Ubuntu unattended upgrades are useful on ordinary servers and workstations. On Kubernetes nodes, they are production changes.\nThat distinction matters because a Kubernetes worker is not just a Linux host. It is part of a larger control loop that includes kubelet, the container runtime, CNI, CSI, storage paths, systemd units, udev device handling, and sometimes iSCSI or multipath. A package update that is safe in isolation can become disruptive when it lands outside a maintenance window.\n","tags":["ubuntu","kubernetes","rke2","updates","iscsi","multipath","csi","operations"],"title":"Ubuntu Unattended Upgrades Are Kubernetes Node Changes"},{"categories":["field-notes"],"content":"Rancher-managed RKE2 upgrades are Kubernetes workflows that intentionally mutate node-local filesystems.\nThat sounds odd until you inspect an upgrade pod. The pod is scheduled onto the node being upgraded, mounts host paths, compares the new RKE2 binary inside the upgrade image with the host binary, replaces the host binary, and restarts the node service.\nThat is the useful mental model:\nRancher desired version -\u0026gt; system-upgrade-controller Plan -\u0026gt; upgrade Job/Pod on each selected node -\u0026gt; host filesystem mounted under /host -\u0026gt; RKE2 binary/config/service update on that node -\u0026gt; node restarts components and rejoins The pod is not just a health check. It is the delivery mechanism for node-local changes.\nPre-Upgrade Evidence First Before letting a Rancher-managed plan move a cluster, capture state from both the target cluster and Rancher management cluster.\nTarget cluster captures:\nBACKUP_DIR=\u0026#34;$HOME/rancher-upgrade-backups/pre-v134-cluster-a-$(date +%Y%m%d-%H%M%S)\u0026#34; mkdir -p \u0026#34;$BACKUP_DIR\u0026#34; kubectl get nodes -o wide \u0026gt; \u0026#34;$BACKUP_DIR/kubectl-get-nodes-wide.txt\u0026#34; kubectl get pods -A -o wide \u0026gt; \u0026#34;$BACKUP_DIR/kubectl-get-pods-all-wide.txt\u0026#34; kubectl get events -A --sort-by=.lastTimestamp \u0026gt; \u0026#34;$BACKUP_DIR/kubectl-get-events-all.txt\u0026#34; kubectl get storageclasses -o yaml \u0026gt; \u0026#34;$BACKUP_DIR/storageclasses.yaml\u0026#34; kubectl get pv -o yaml \u0026gt; \u0026#34;$BACKUP_DIR/persistentvolumes.yaml\u0026#34; kubectl get pvc -A -o yaml \u0026gt; \u0026#34;$BACKUP_DIR/persistentvolumeclaims.yaml\u0026#34; kubectl get ingress -A -o yaml \u0026gt; \u0026#34;$BACKUP_DIR/ingresses.yaml\u0026#34; kubectl get services -A -o wide \u0026gt; \u0026#34;$BACKUP_DIR/services-wide.txt\u0026#34; kubectl get apiservices -o yaml \u0026gt; \u0026#34;$BACKUP_DIR/apiservices.yaml\u0026#34; kubectl get crds -o wide \u0026gt; \u0026#34;$BACKUP_DIR/crds-wide.txt\u0026#34; kubectl get namespaces -o yaml \u0026gt; \u0026#34;$BACKUP_DIR/namespaces.yaml\u0026#34; kubectl get plans -A -o yaml \u0026gt; \u0026#34;$BACKUP_DIR/system-upgrade-plans.yaml\u0026#34; kubectl get pods,jobs -A | grep -i upgrade \u0026gt; \u0026#34;$BACKUP_DIR/system-upgrade-pods-jobs.txt\u0026#34; || true Rancher management captures:\nkubectl --context rancher-mgmt get clusters.management.cattle.io -o yaml \\ \u0026gt; \u0026#34;$BACKUP_DIR/rancher-management-clusters.yaml\u0026#34; kubectl --context rancher-mgmt get clusters.provisioning.cattle.io -A -o yaml \\ \u0026gt; \u0026#34;$BACKUP_DIR/rancher-provisioning-clusters.yaml\u0026#34; kubectl --context rancher-mgmt get clusters.fleet.cattle.io -A -o yaml \\ \u0026gt; \u0026#34;$BACKUP_DIR/fleet-clusters.yaml\u0026#34; kubectl --context rancher-mgmt get bundles.fleet.cattle.io -A -o yaml \\ \u0026gt; \u0026#34;$BACKUP_DIR/fleet-bundles.yaml\u0026#34; kubectl --context rancher-mgmt get bundledeployments.fleet.cattle.io -A -o yaml \\ \u0026gt; \u0026#34;$BACKUP_DIR/fleet-bundledeployments.yaml\u0026#34; Also take an etcd snapshot and control-plane file archives according to the platform runbook. Verify the artifacts, not just the commands:\ntar tzf control-plane-backup.tgz \u0026gt;/dev/null ls -lh etcd-snapshot-name Non-empty .err files are not automatically failures. For example, Kubernetes v1.33+ can warn that core Endpoints is deprecated. Classify warnings separately from failed captures.\nKnow Which Plans Are Active Rancher can create managed Plans in cattle-system while older GitOps-managed Plans still exist in system-upgrade.\nCheck all Plans:\nkubectl get plans -A -o wide You may see both:\ncattle-system rke2-master-plan rancher/rke2-upgrade v1.34.9+rke2r1 cattle-system rke2-worker-plan rancher/rke2-upgrade v1.34.9+rke2r1 system-upgrade server-plan rancher/rke2-upgrade v1.32.8+rke2r1 system-upgrade agent-plan rancher/rke2-upgrade v1.32.8+rke2r1 Do not assume every Plan is active. Inspect labels and ownership:\nkubectl -n cattle-system get plan rke2-master-plan -o yaml kubectl -n cattle-system get plan rke2-worker-plan -o yaml Rancher-managed plans commonly show signals like:\nmetadata.labels.rancher-managed: \u0026#34;true\u0026#34; metadata.finalizers: systemcharts.cattle.io/rancher-managed-plan spec.concurrency: 1 spec.cordon: true The worker Plan should wait for the master Plan through prepare, rather than upgrading workers before control-plane completion.\nThe Upgrade Pod Writes Through /host Inspecting a Rancher-managed upgrade pod makes the mechanism visible:\nkubectl -n cattle-system get jobs,pods -o wide | grep -i rke2 kubectl -n cattle-system describe pod \u0026lt;upgrade-pod\u0026gt; kubectl -n cattle-system logs \u0026lt;upgrade-pod\u0026gt; --previous=false Log shape:\n[INFO] rke2 binary is running with pid 486375 RKE2_BIN_PATH=/usr/local/bin/rke2 FULL_BIN_PATH=/host/usr/local/bin/rke2 Comparing old and new binaries sha256sum /opt/rke2 /host/usr/local/bin/rke2 That tells you the upgrade container sees:\n/opt/rke2 new binary from the upgrade image /host/usr/local/bin/rke2 host binary on the node /host/proc/\u0026lt;pid\u0026gt;/cmdline host process metadata The pod is using Kubernetes scheduling to run node-local maintenance. This is why permissions, host mounts, and node selection matter.\nLabels Control The Blast Radius Rancher-managed Plans select nodes with labels such as:\nupgrade.cattle.io/kubernetes-upgrade=true Before the run, verify the selected nodes:\nkubectl get nodes -l upgrade.cattle.io/kubernetes-upgrade=true \\ -o custom-columns=NAME:.metadata.name,VERSION:.status.nodeInfo.kubeletVersion If the wrong nodes are selected, fix the labels before the controller starts new jobs:\nkubectl label node worker-6 worker-9 upgrade.cattle.io/kubernetes-upgrade- --overwrite kubectl label node cp-1 cp-2 cp-3 upgrade.cattle.io/kubernetes-upgrade=true --overwrite This is especially important when moving from one hop to the next. Stale labels can make a worker Plan start before the intended control-plane sequence.\nMonitor The Rollout By Plans And Nodes Poll instead of relying only on watches during control-plane restarts:\nwhile true; do date kubectl get nodes -o json \\ | jq -r \u0026#39;.items[] | [.metadata.name, (if (.spec.unschedulable // false) then \u0026#34;cordoned\u0026#34; else \u0026#34;schedulable\u0026#34; end), .status.nodeInfo.kubeletVersion, ([.status.conditions[] | select(.type==\u0026#34;Ready\u0026#34;)][0].status)] | @tsv\u0026#39; echo kubectl -n cattle-system get plans -o json \\ | jq -r \u0026#39;.items[] | [.metadata.name, ([.status.conditions[]? | select(.type==\u0026#34;Complete\u0026#34;)][0].status // \u0026#34;\u0026#34;), (.status.applying // [] | join(\u0026#34;,\u0026#34;)), .status.latestVersion] | @tsv\u0026#39; echo kubectl -n cattle-system get jobs,pods -o wide | grep -i rke2 || true sleep 30 done Completion shape:\nrke2-master-plan True v1.34.9-rke2r1 rke2-worker-plan True v1.34.9-rke2r1 Every node should be Ready, schedulable, and on the target version.\nPost-Upgrade Cleanup Signals Upgrade pods often become ContainerStatusUnknown or Failed because the node restarted underneath them. If the owning Jobs are complete and Plans are complete, these pods are usually stale artifacts.\nClean them after verification:\nkubectl -n cattle-system delete pod --field-selector=status.phase=Failed Then check the platform add-ons:\nkubectl get ds -A -o json \\ | jq -r \u0026#39;.items[] | select(.status.desiredNumberScheduled != .status.numberReady) | [.metadata.namespace, .metadata.name, .status.desiredNumberScheduled, .status.numberReady] | @tsv\u0026#39; kubectl get deploy -A -o json \\ | jq -r \u0026#39;.items[] | select((.status.readyReplicas // 0) \u0026lt; (.spec.replicas // 1)) | [.metadata.namespace, .metadata.name, (.status.readyReplicas // 0), (.spec.replicas // 1)] | @tsv\u0026#39; One post-upgrade symptom worth recognizing is a host-port stale bind:\nport 80 is already in use. Please check the flag --http-port For an ingress DaemonSet pod after a node restart, deleting the one bad pod can force a fresh sandbox and clear the stale bind:\nkubectl -n kube-system delete pod rke2-ingress-nginx-controller-abcde Do not delete the whole DaemonSet first. Confirm the failure is isolated to one pod/node.\nAcceptance Criteria The upgrade is complete when these are true:\nRancher desired version and reported actual version match. active Rancher-managed Plans are Complete=True. all nodes are Ready, schedulable, and on the target RKE2 version. stale failed upgrade pods are cleaned after Jobs complete. DaemonSets and deployments are at expected ready counts. event scans show no current port in use, sandbox, PDB, or image-pull blockers. pre-existing unrelated workload failures are documented separately. The operating rule: a Rancher RKE2 upgrade pod is a privileged node maintenance action packaged as a Kubernetes workload. Treat it with the same care you would give an SSH-based node patch, but use Kubernetes evidence to prove what it changed and when it is done.\n","permalink":"https://trinidadmarroquin.com/field-notes/rancher-rke2-upgrade-pods-host-filesystem/","section":"field-notes","summary":"Rancher-managed RKE2 upgrades are Kubernetes workflows that intentionally mutate node-local filesystems.\nThat sounds odd until you inspect an upgrade pod. The pod is scheduled onto the node being upgraded, mounts host paths, compares the new RKE2 binary inside the upgrade image with the host binary, replaces the host binary, and restarts the node service.\nThat is the useful mental model:\nRancher desired version -\u0026gt; system-upgrade-controller Plan -\u0026gt; upgrade Job/Pod on each selected node -\u0026gt; host filesystem mounted under /host -\u0026gt; RKE2 binary/config/service update on that node -\u0026gt; node restarts components and rejoins The pod is not just a health check. It is the delivery mechanism for node-local changes.\n","tags":["rancher","rke2","kubernetes","system-upgrade-controller","upgrades","operations"],"title":"Rancher RKE2 Upgrade Pods Mutate The Host Filesystem"},{"categories":["field-notes"],"content":"A Packer template build can fail in three different places that look similar from the outside:\nPacker never reaches SSH, so file and shell provisioners never run. Packer places bootstrap files into the template, but does not execute runtime bootstrap. Terraform/cloud-init clones the VM but does not start the bootstrap entrypoint correctly. Do not diagnose all three as “bootstrap did not work.” Ask which layer failed.\nPlacement Is Packer\u0026rsquo;s Job For a reusable vSphere template, Packer should place static resources only:\n/usr/local/bin/platform-bootstrap /etc/systemd/system/platform-bootstrap.service /var/lib/platform-bootstrap/ /var/log/platform-bootstrap.log The Packer build should verify the static entrypoint before sealing the image:\nbash -n /usr/local/bin/platform-bootstrap /usr/local/bin/platform-bootstrap --version /usr/local/bin/platform-bootstrap --check systemd-analyze verify /etc/systemd/system/platform-bootstrap.service It should not bake site-specific values into the template:\nno cluster token no server URL no static IP or gateway no DNS suffix no SSH private keys no environment secrets Terraform and cloud-init can provide runtime values later.\nIf /opt Or /usr/local/bin Is Empty If the final template is missing the bootstrap files entirely, check whether Packer ever reached the provisioner stage.\nTypical log shape when it did not:\nUsing SSH communicator to connect: 192.0.2.25 Waiting for SSH to become available... TCP connection to SSH ip/port failed: dial tcp 192.0.2.25:22: i/o timeout In that state, none of the later provisioners ran. The problem is not cloud-init or the bootstrap script. It is reachability from the Packer runner to the temporary build VM.\nFor Concourse-backed builds, test from the selected worker, not from your workstation:\nkubectl exec -n concourse concourse-worker-0 -- ip route get 192.0.2.25 kubectl exec -n concourse concourse-worker-0 -- nc -vz 192.0.2.25 22 If your workstation can reach the VM but the worker cannot, a local Packer build may succeed while the pipeline build never reaches SSH. Fix routing/firewall/worker placement before changing bootstrap code.\nProve The Pipeline Commit A pipeline that clones the repo fresh should print the cloned commit SHA. Otherwise, a successful build log cannot prove whether it used the commit that added the bootstrap stage.\nAdd a safe signal near the clone step:\ngit clone --depth 1 \u0026#34;$REPO_URL\u0026#34; repo git -C repo rev-parse --short HEAD Do not print clone URLs containing tokens. If the token is embedded in the URL, suppress command tracing around the clone.\nRuntime Execution Is Terraform And Cloud-Init\u0026rsquo;s Job After cloning from the template, verify the static resources first:\nls -l /usr/local/bin/platform-bootstrap systemctl cat platform-bootstrap Then verify cloud-init and bootstrap runtime state:\ncloud-init status --long sudo systemctl status platform-bootstrap --no-pager sudo journalctl -u platform-bootstrap --no-pager -n 100 sudo cat /var/lib/platform-bootstrap/status.json sudo test -f /var/lib/platform-bootstrap/complete \u0026amp;\u0026amp; echo complete For a Type=oneshot unit, inactive (dead) can be normal after success. Use the status file and completion marker to tell success from “never ran.”\nAcceptance Criteria The template/bootstrap path is healthy when these are true:\nPacker logs show SSH connected before file/shell provisioners. the final template contains the static bootstrap entrypoint and unit. the build validates script syntax, unit syntax, --version, and --check. the pipeline log records the safe Git commit SHA used for the build. cloned VMs receive runtime values through Terraform/cloud-init, not Packer. cloud-init starts bootstrap only after required mounts and configuration are present. The key distinction is placement versus execution. Packer places the mechanism. Terraform and cloud-init decide how that mechanism runs for each clone.\n","permalink":"https://trinidadmarroquin.com/field-notes/packer-bootstrap-placement-vs-runtime-execution/","section":"field-notes","summary":"A Packer template build can fail in three different places that look similar from the outside:\nPacker never reaches SSH, so file and shell provisioners never run. Packer places bootstrap files into the template, but does not execute runtime bootstrap. Terraform/cloud-init clones the VM but does not start the bootstrap entrypoint correctly. Do not diagnose all three as “bootstrap did not work.” Ask which layer failed.\nPlacement Is Packer\u0026rsquo;s Job For a reusable vSphere template, Packer should place static resources only:\n","tags":["packer","vsphere","cloud-init","terraform","bootstrap","concourse","operations"],"title":"Packer Bootstrap Placement Versus Runtime Execution"},{"categories":["posts"],"content":"The hard part of a Rancher-managed RKE2 upgrade is not always clicking the target version. The hard part is keeping every control plane that thinks it owns the upgrade aligned:\nKubernetes version-skew rules. Rancher UI version metadata. Rancher\u0026rsquo;s desired cluster version. system-upgrade-controller Plans. GitOps reconciliation for those Plans. node-level runtime cleanup after the upgrade. In one management-cluster upgrade, the safe Kubernetes path was clear:\nv1.31 -\u0026gt; v1.32 -\u0026gt; v1.33 But Rancher only advertised newer versions in the UI after the Rancher application itself was upgraded. The UI could offer v1.33 while the cluster still needed the intermediate v1.32 hop. That is where an otherwise normal upgrade becomes an ownership problem.\nDo Not Skip The Minor Hop If the current cluster is on v1.31, do not jump directly to v1.33 just because the UI dropdown offers it.\nUse the normal Kubernetes minor-hop model:\ncurrent: v1.31.x+rke2r1 next: v1.32.x+rke2r1 then: v1.33.x+rke2r1 Check the Rancher cluster object before changing anything:\nkubectl get clusters.management.cattle.io local \\ -o jsonpath=\u0026#39;{.spec.rke2Config.kubernetesVersion}{\u0026#34;\\t\u0026#34;}{.status.version.gitVersion}{\u0026#34;\\t\u0026#34;}{.status.conditions[?(@.type==\u0026#34;Ready\u0026#34;)].status}{\u0026#34;\\t\u0026#34;}{.status.conditions[?(@.type==\u0026#34;Connected\u0026#34;)].status}{\u0026#34;\\n\u0026#34;}\u0026#39; If desired and actual versions disagree before the upgrade, pause and understand why. Rancher warnings about version mismatch usually mean the cluster was changed outside the same Rancher control path.\nWhen The UI Cannot Offer The Intermediate Version Rancher version metadata can move forward faster than the cluster you are repairing. A newer Rancher release may advertise only a newer supported range, while your cluster still needs one intermediate minor.\nWhen that happens, do not use the UI to skip ahead. Inspect system-upgrade-controller instead:\nkubectl -n system-upgrade get deploy,pods,plans kubectl -n system-upgrade get plan server-plan -o yaml kubectl -n system-upgrade get plan agent-plan -o yaml Healthy plan shape for a conservative management-cluster upgrade:\nserver plan: concurrency 1 agent plan: concurrency 1 agent waits on server plan If the controller deployment is scaled to 0, treat that as a deliberate emergency brake until proven otherwise. First identify who owns that replica count.\nGitOps May Own The Upgrade Controller If Argo CD, Fleet, or another GitOps system manages the upgrade controller and Plans, live patches may not hold.\nSymptoms:\nkubectl patch succeeds minutes later the old value returns system-upgrade-controller replicas returns to 0 Plan version returns to the previous RKE2 version In that case, the real change belongs in Git, not directly in the cluster.\nBefore changing GitOps desired state, capture evidence:\nBACKUP_DIR=\u0026#34;$HOME/rancher-upgrade-backups/pre-v132-local-$(date +%Y%m%d-%H%M%S)\u0026#34; mkdir -p \u0026#34;$BACKUP_DIR\u0026#34; kubectl get nodes -o wide \u0026gt; \u0026#34;$BACKUP_DIR/kubectl-get-nodes-wide.txt\u0026#34; kubectl get pods -A -o wide \u0026gt; \u0026#34;$BACKUP_DIR/kubectl-get-pods-all-wide.txt\u0026#34; kubectl get events -A --sort-by=.lastTimestamp \u0026gt; \u0026#34;$BACKUP_DIR/kubectl-get-events-all.txt\u0026#34; kubectl -n system-upgrade get plans -o yaml \u0026gt; \u0026#34;$BACKUP_DIR/system-upgrade-plans.yaml\u0026#34; kubectl get clusters.management.cattle.io local -o yaml \u0026gt; \u0026#34;$BACKUP_DIR/rancher-local-management-cluster.yaml\u0026#34; helm -n cattle-system get values rancher \u0026gt; \u0026#34;$BACKUP_DIR/rancher-helm-values.yaml\u0026#34; helm -n cattle-system list \u0026gt; \u0026#34;$BACKUP_DIR/helm-list-cattle-system.txt\u0026#34; Also take an etcd snapshot from a control-plane node or an equivalent approved backup path:\nsudo rke2 etcd-snapshot save --name pre-v132-local-$(date +%Y%m%d-%H%M%S) Then make the smallest GitOps change required for the intermediate hop:\nsystem-upgrade-controller replicas: 1 server-plan version: v1.32.x+rke2r1 agent-plan version: v1.32.x+rke2r1 Render the overlay before publishing it:\nkustomize build path/to/system-upgrade/overlay \u0026gt;/tmp/system-upgrade.rendered.yaml Monitor With Polling During Control-Plane Restarts Watch streams often break during control-plane upgrades:\nunable to decode an event from the watch stream stream error: INTERNAL_ERROR That message is not automatically an upgrade failure. It can just mean the API server connection reset while control-plane components restarted.\nUse polling loops instead of -w during disruptive phases:\nwhile true; do clear date echo \u0026#34;== Nodes ==\u0026#34; kubectl get nodes -o wide echo echo \u0026#34;== Plans ==\u0026#34; kubectl -n system-upgrade get plans echo echo \u0026#34;== Upgrade Pods/Jobs ==\u0026#34; kubectl -n system-upgrade get pods,jobs sleep 15 done Expected sequence:\nserver-plan runs first control-plane nodes move one at a time server-plan reaches Complete agent-plan runs next workers move one at a time agent-plan reaches Complete Rancher desired version and actual version match Align GitOps After A UI-Driven Hop Once the cluster reaches the intermediate version and Rancher desired/actual versions match, the UI may become safe for the next minor hop.\nIf Rancher UI performs that upgrade, check the GitOps-managed Plans afterward:\nkubectl -n system-upgrade get plans kubectl get clusters.management.cattle.io local \\ -o jsonpath=\u0026#39;{.spec.rke2Config.kubernetesVersion}{\u0026#34;\\t\u0026#34;}{.status.version.gitVersion}{\u0026#34;\\n\u0026#34;}\u0026#39; Possible final state:\nRancher desired: v1.33.x+rke2r1 Rancher actual: v1.33.x+rke2r1 SUC Plans: v1.32.x+rke2r1 That means the cluster upgraded, but GitOps still describes the previous Plan version. Update the GitOps overlay to match the completed version. This should be a no-op for node upgrades if every node is already on the target, but it prevents the next reconciliation from fighting the real state.\nCleanup: Old Runtime Shims Can Survive The Upgrade After RKE2 upgrades, the active binary symlink should point at the new data directory:\nreadlink -f /var/lib/rancher/rke2/bin But orphaned containerd-shim-runc-v2 processes can keep executing from the old RKE2 data directory:\n/var/lib/rancher/rke2/data/v1.31.x-rke2r1-.../bin/containerd-shim-runc-v2 First ask containerd and CRI whether they still know about the containers:\nsudo /var/lib/rancher/rke2/bin/crictl \\ --runtime-endpoint unix:///run/k3s/containerd/containerd.sock ps -a sudo /var/lib/rancher/rke2/bin/ctr \\ --address /run/k3s/containerd/containerd.sock \\ --namespace k8s.io tasks ls If the old shim PIDs are invisible to the current runtime, they are likely orphaned. The cleanest remediation is a one-node-at-a-time reboot, not deleting old data directories while processes still execute from them.\nFor control-plane nodes:\nkubectl cordon cp-1 ssh -tt operator@cp-1.example.com \u0026#39;sudo reboot\u0026#39; kubectl wait node/cp-1 --for=condition=Ready --timeout=15m kubectl uncordon cp-1 Reboot Automation Needs A Real Reboot Gate A naive reboot script can finish too early because Kubernetes may still report stale Ready=True shortly after the reboot command is sent.\nBetter gates:\ncapture the node\u0026rsquo;s pre-reboot Kubernetes bootID. if passwordless SSH is available, capture the host\u0026rsquo;s /proc/sys/kernel/random/boot_id before reboot. after reboot, require SSH to return and the host boot ID to change. then require Kubernetes Ready=True and, when available, Kubernetes bootID to change. Kubernetes readiness alone is not enough immediately after sending the reboot command. It can be cached long enough to fool automation.\nAcceptance Criteria Call the minor hop complete only when these are true:\nRancher desired version matches Rancher reported actual version. every node reports the target RKE2 version. server-plan and agent-plan are complete or intentionally aligned to the completed version. GitOps desired state matches the live Plan versions and controller replica count. Rancher, Fleet, webhooks, ingress, CNI, and DNS are healthy. old RKE2 data directories are not held by live shim processes, or reboot cleanup is scheduled. the next minor target is planned, not guessed from the newest UI option. The operating rule is simple: Rancher can drive the upgrade, but GitOps and node runtime state still need reconciliation. The upgrade is not done when the UI says the version changed. It is done when every owner of that version agrees.\nRelated: Rancher RKE2 Upgrade Pods Mutate The Host Filesystem explains the node-local host filesystem update pattern behind Rancher-managed upgrade Jobs.\n","permalink":"https://trinidadmarroquin.com/posts/rancher-rke2-minor-hop-gitops-suc/","section":"posts","summary":"The hard part of a Rancher-managed RKE2 upgrade is not always clicking the target version. The hard part is keeping every control plane that thinks it owns the upgrade aligned:\nKubernetes version-skew rules. Rancher UI version metadata. Rancher\u0026rsquo;s desired cluster version. system-upgrade-controller Plans. GitOps reconciliation for those Plans. node-level runtime cleanup after the upgrade. In one management-cluster upgrade, the safe Kubernetes path was clear:\nv1.31 -\u0026gt; v1.32 -\u0026gt; v1.33 But Rancher only advertised newer versions in the UI after the Rancher application itself was upgraded. The UI could offer v1.33 while the cluster still needed the intermediate v1.32 hop. That is where an otherwise normal upgrade becomes an ownership problem.\n","tags":["rancher","rke2","kubernetes","gitops","argocd","upgrades","operations"],"title":"Rancher RKE2 Minor Hops When UI Metadata And GitOps Plans Disagree"},{"categories":["field-notes"],"content":"Attaching a second vSphere disk is not the same as using it.\nIn one worker replacement, Terraform correctly attached a second disk to the VM, Packer correctly placed the static bootstrap entrypoint, and cloud-init completed. But the guest still showed the second disk as blank and unmounted:\nsda sda1 /boot/efi sda2 / sdb \u0026lt;blank\u0026gt; The result was subtle: bootstrap succeeded, but /var/lib/rancher stayed on the operating-system disk. For an RKE2 node, that means Rancher/RKE2 state and container log growth can still fill / even though Terraform created a data disk.\nOwnership Boundary Use this split:\nPacker places static bootstrap resources in the template. Terraform attaches the intended VM disks. Terraform-rendered cloud-init owns guest disk layout and mount behavior. Bootstrap runs after the mounts exist. RKE2 uses the mounted paths normally. Do not expect the vSphere provider\u0026rsquo;s disk block to partition, format, or mount anything inside the guest. It only changes VM hardware.\nIntended Layout A clean RKE2 worker layout is:\n/dev/sda operating system disk /dev/sdb1 LABEL=rancher-data mounted at /var/lib/rancher Treat /dev/sdb1 as an example, not a contract. In vSphere clones, Linux disk enumeration can differ between otherwise similar nodes. One worker may see the data disk as /dev/sdb, while another may see the root disk on /dev/sdb and the data disk on /dev/sda. The durable contract should be the filesystem label, UUID, or a discovery step that excludes the current root disk.\nKeep OS logs on the OS disk. Put Rancher/Kubernetes-heavy logs on the Rancher data disk through bind mounts:\n/var/lib/rancher/log/pods -\u0026gt; /var/log/pods /var/lib/rancher/log/containers -\u0026gt; /var/log/containers That keeps noisy workload/container logs from consuming the root filesystem while avoiding a broad /var/log mount that can complicate OS debugging.\nCloud-Init Shape The Terraform-rendered userdata should describe the disk setup before it starts bootstrap. If disk naming is stable in your environment, cloud-init disk_setup can be enough:\n#cloud-config ssh_pwauth: true disable_root: true disk_setup: /dev/sdb: table_type: gpt layout: true overwrite: false fs_setup: - label: rancher-data filesystem: ext4 device: /dev/sdb1 overwrite: false mounts: - [ \u0026#34;LABEL=rancher-data\u0026#34;, \u0026#34;/var/lib/rancher\u0026#34;, \u0026#34;ext4\u0026#34;, \u0026#34;defaults,nofail\u0026#34;, \u0026#34;0\u0026#34;, \u0026#34;2\u0026#34; ] runcmd: - [ bash, -lc, \u0026#39;mkdir -p /run/sshd\u0026#39; ] - [ bash, -lc, \u0026#39;mkdir -p /var/lib/rancher/log/pods /var/lib/rancher/log/containers /var/log/pods /var/log/containers\u0026#39; ] - [ bash, -lc, \u0026#39;grep -qs \u0026#34; /var/log/pods \u0026#34; /etc/fstab || echo \u0026#34;/var/lib/rancher/log/pods /var/log/pods none bind,nofail 0 0\u0026#34; \u0026gt;\u0026gt; /etc/fstab\u0026#39; ] - [ bash, -lc, \u0026#39;grep -qs \u0026#34; /var/log/containers \u0026#34; /etc/fstab || echo \u0026#34;/var/lib/rancher/log/containers /var/log/containers none bind,nofail 0 0\u0026#34; \u0026gt;\u0026gt; /etc/fstab\u0026#39; ] - [ bash, -lc, \u0026#39;mountpoint -q /var/log/pods || mount /var/log/pods\u0026#39; ] - [ bash, -lc, \u0026#39;mountpoint -q /var/log/containers || mount /var/log/containers\u0026#39; ] - [ bash, -lc, \u0026#39;/usr/local/bin/platform-bootstrap\u0026#39; ] The important details are overwrite: false and running bootstrap after the mounts. The userdata is safe for fresh clones, but it is not a casual live-migration tool for existing nodes that already have data.\nWhen disk naming is not stable, do not hard-code /dev/sdb. Use a boot-time helper that discovers the non-root disk, labels it, and mounts by label:\nroot_source=\u0026#34;$(findmnt -n -o SOURCE /)\u0026#34; root_parent=\u0026#34;$(lsblk -no PKNAME \u0026#34;$root_source\u0026#34; | head -n1)\u0026#34; root_disk=\u0026#34;/dev/${root_parent}\u0026#34; data_disk=\u0026#34;$(lsblk -dn -o NAME,TYPE | awk \u0026#39;$2 == \u0026#34;disk\u0026#34; {print \u0026#34;/dev/\u0026#34; $1}\u0026#39; | grep -vx \u0026#34;$root_disk\u0026#34; | head -n1)\u0026#34; if [ -z \u0026#34;$root_parent\u0026#34; ] || [ -z \u0026#34;$data_disk\u0026#34; ]; then echo \u0026#34;no non-root data disk found\u0026#34; \u0026gt;\u0026amp;2 exit 1 fi if ! blkid -L rancher-data \u0026gt;/dev/null 2\u0026gt;\u0026amp;1; then parted -s \u0026#34;$data_disk\u0026#34; mklabel gpt mkpart primary ext4 0% 100% partprobe \u0026#34;$data_disk\u0026#34; mkfs.ext4 -L rancher-data \u0026#34;${data_disk}1\u0026#34; fi mkdir -p /var/lib/rancher grep -qs \u0026#39; /var/lib/rancher \u0026#39; /etc/fstab || \\ echo \u0026#39;LABEL=rancher-data /var/lib/rancher ext4 defaults,nofail 0 2\u0026#39; \u0026gt;\u0026gt; /etc/fstab mountpoint -q /var/lib/rancher || mount /var/lib/rancher The helper should be idempotent: if LABEL=rancher-data already exists, it should not repartition anything. Bootstrap should still run after the mount exists.\nValidate the cloud-init file before rebuilding nodes:\ncloud-init schema --config-file templates/userdata.yaml Replacement Flow For existing Kubernetes workers, prefer rolling replacement over in-place repartitioning:\nkubectl cordon worker-5 kubectl drain worker-5 --ignore-daemonsets --delete-emptydir-data kubectl delete node worker-5 Then replace only that VM through Terraform:\nterraform plan \\ -target=\u0026#39;module.vm_group.vsphere_virtual_machine.vm[\u0026#34;worker5\u0026#34;]\u0026#39; \\ -replace=\u0026#39;module.vm_group.vsphere_virtual_machine.vm[\u0026#34;worker5\u0026#34;]\u0026#39; \\ -out=tfplan-replace-worker5-disklayout terraform apply tfplan-replace-worker5-disklayout Using -target should be intentional and temporary. It is useful during a controlled one-node replacement when you specifically want to avoid pushing userdata or template changes to unrelated existing VMs in the same run.\nBefore applying, inspect the saved plan and confirm the replacement scope:\n1 to add, 0 to change, 1 to destroy only module.vm_group.vsphere_virtual_machine.vm[\u0026#34;worker5\u0026#34;] Verification After the VM boots, verify outside and inside the guest.\nTerraform/vSphere state:\nterraform state show \u0026#39;module.vm_group.vsphere_virtual_machine.vm[\u0026#34;worker5\u0026#34;]\u0026#39; \\ | grep -E \u0026#39;default_ip_address|power_state|vmware_tools_status|template_uuid\u0026#39; Expected external state:\npower_state = \u0026#34;on\u0026#34; vmware_tools_status = \u0026#34;guestToolsRunning\u0026#34; default_ip_address = \u0026#34;192.0.2.25\u0026#34; Guest checks:\ncloud-init status --long df -h lsblk -f findmnt /var/lib/rancher /var/log/pods /var/log/containers cat /etc/fstab sudo cat /var/lib/platform-bootstrap/status.json Also verify the root disk and data disk are not the same device:\nfindmnt -n -o SOURCE / findmnt -n -o SOURCE /var/lib/rancher lsblk -f blkid -L rancher-data Healthy shape:\n/dev/sda2 / /dev/sdb1 /var/lib/rancher /dev/sdb1 /var/log/pods /dev/sdb1 /var/log/containers findmnt should show /var/log/pods and /var/log/containers as bind mounts backed by the Rancher data disk.\nOperating Rule Do not stop at “Terraform attached the disk.”\nFor Rancher/RKE2 nodes, the acceptance test is that the guest mounted the data disk where RKE2 actually writes heavy state, cloud-init completed without errors, bootstrap ran after the mount existed, and container log paths cannot fill the OS disk during normal workload churn. Verify that on every worker; older nodes may still need live bind-mount correction even after the template is fixed for future clones.\n","permalink":"https://trinidadmarroquin.com/field-notes/terraform-cloud-init-rancher-data-disk-layout/","section":"field-notes","summary":"Attaching a second vSphere disk is not the same as using it.\nIn one worker replacement, Terraform correctly attached a second disk to the VM, Packer correctly placed the static bootstrap entrypoint, and cloud-init completed. But the guest still showed the second disk as blank and unmounted:\nsda sda1 /boot/efi sda2 / sdb \u0026lt;blank\u0026gt; The result was subtle: bootstrap succeeded, but /var/lib/rancher stayed on the operating-system disk. For an RKE2 node, that means Rancher/RKE2 state and container log growth can still fill / even though Terraform created a data disk.\n","tags":["terraform","vsphere","cloud-init","rke2","rancher","storage","operations"],"title":"Terraform Cloud-Init Ownership For Rancher Data Disks"},{"categories":["posts"],"content":"Rancher management-cluster upgrades are not just a Helm command. They are a sequence of readiness gates.\nThe version target matters, but it is not the first question. The first question is whether the management cluster is stable enough to survive the change.\nIn one upgrade run, the desired path was straightforward on paper:\nRancher first, then Kubernetes/RKE2. The actual work exposed the operational details that make or break the window: stale workers, version skew, node join scripts, root filesystem pressure, backup transfer permissions, Rancher pre-upgrade hooks, Fleet health, and downstream cluster impersonation.\nThe Seven-Step Shape For a Rancher-managed RKE2 environment, keep the sequence explicit:\nStabilize the management cluster. Take full backups and current-state captures. Upgrade Rancher before Kubernetes. Normalize old worker skew to the current management-cluster version. Upgrade control-plane and etcd nodes to the target Kubernetes/RKE2 minor. Upgrade workers to the target Kubernetes/RKE2 minor. Validate Rancher, Fleet, downstream clusters, and platform add-ons. The order is deliberate. Rancher must support the Kubernetes/RKE2 version it is about to manage. Worker capacity must exist before control-plane changes. Backups must be proven before any irreversible step.\nStabilize Before Upgrading Do not start by upgrading the broken cluster you wish you had. Start by stabilizing the cluster you actually have.\nBaseline checks:\nkubectl get nodes -o wide kubectl get pods -A --field-selector=status.phase!=Running kubectl get events -A --sort-by=.lastTimestamp kubectl get pdb -A kubectl get pods -A -o wide | egrep \u0026#39;rancher|cattle|fleet|ingress|metallb|calico\u0026#39; Stop and fix first if you find:\na long-term NotReady worker still carrying old DaemonSet pods. only one schedulable worker for management workloads. version skew where control-plane nodes are current but workers are several minors behind. FreeDiskSpaceFailed events on control-plane nodes. Rancher/Fleet/webhook pods already unhealthy. unresolved storage or ingress issues that would obscure upgrade failures. If a worker is permanently dead and no longer hosting useful state, remove it from Kubernetes before the upgrade:\nkubectl get pods -A --field-selector spec.nodeName=worker-3 -o wide kubectl delete node worker-3 Then add replacement workers before draining or upgrading the remaining old workers.\nWorker Replacement Has Its Own Traps When joining new RKE2 workers, retrieve the node token from a control-plane node:\nsudo cat /var/lib/rancher/rke2/server/node-token Be careful with how scripts write the token into /etc/rancher/rke2/config.yaml. A line break after token: changes the YAML shape and can leave the agent with an empty token or invalid join config.\nGood shape:\nserver: https://192.0.2.10:9345 token: K\u0026lt;redacted\u0026gt;::server:\u0026lt;redacted\u0026gt; node-label: - role=worker - os=Linux - cluster=cluster-a Avoid setting the Kubernetes reserved display-role label through kubelet --node-labels:\nnode-role.kubernetes.io/worker Modern kubelet validation rejects unknown labels in the kubernetes.io and k8s.io namespaces. If the agent repeatedly logs this, it is not waiting on the server. Kubelet is exiting:\nKubelet exited: exit status 1 failed to validate kubelet flags: unknown \u0026#39;kubernetes.io\u0026#39; or \u0026#39;k8s.io\u0026#39; labels specified with --node-labels Use a normal custom label during join, then set the display role after the node is registered:\nkubectl label node worker-4 node-role.kubernetes.io/worker= --overwrite The display role is for operator readability. It should not break node registration.\nFix Disk Pressure Before The Window RKE2 control-plane nodes can accumulate old runtime data, etcd files, containerd content, and logs. If events show disk pressure, check root filesystem usage and the main consumers:\ndf -h sudo du -xh /var/lib/rancher/rke2 | sort -h | tail -30 sudo du -xh /var/lib/kubelet | sort -h | tail -30 sudo du -xh /var/log | sort -h | tail -30 If vSphere has already grown the virtual disk but the guest still sees the old partition/LVM size, rescan and grow carefully:\nlsblk sudo pvs sudo vgs sudo lvs sudo growpart /dev/sda 3 sudo pvresize /dev/sda3 sudo lvextend -r -l +100%FREE /dev/mapper/ubuntu--vg-ubuntu--lv df -h Do this before the upgrade. Root filesystem pressure during a Rancher or RKE2 upgrade turns every symptom into a false lead.\nBackups Need Transferable Evidence Take an RKE2 etcd snapshot from a control-plane node:\nsudo rke2 etcd-snapshot save --name pre-upgrade-$(date +%Y%m%d-%H%M%S) Then back up control-plane state from every control-plane node:\n/etc/rancher/rke2/ /var/lib/rancher/rke2/server/db/ /var/lib/rancher/rke2/server/tls/ RKE2 service/config files used by the host Do not assume the snapshot is easy to copy as your user. RKE2 snapshots are often root-owned under the server DB path. If scp cannot read the directory, copy the snapshot to a temporary path with safe ownership first:\nsudo cp /var/lib/rancher/rke2/server/db/snapshots/pre-upgrade-* /tmp/ sudo chown operator:operator /tmp/pre-upgrade-* Capture Rancher and cluster state next:\nhelm list -A \u0026gt; helm-list-all-namespaces.txt helm list -n cattle-system \u0026gt; helm-list-cattle-system.txt helm get values rancher -n cattle-system -o yaml \u0026gt; rancher-helm-values.yaml helm status rancher -n cattle-system \u0026gt; rancher-helm-status.txt kubectl get nodes -o wide \u0026gt; kubectl-get-nodes-wide.txt kubectl get pods -A -o wide \u0026gt; kubectl-get-pods-all-wide.txt kubectl get crds \u0026gt; kubectl-get-crds.txt kubectl get events -A --sort-by=.lastTimestamp \u0026gt; kubectl-get-events-all.txt kubectl get secrets -n cattle-system \u0026gt; kubectl-get-secrets-cattle-system.txt kubectl get namespaces --show-labels \u0026gt; kubectl-get-namespaces-labels.txt Verify archives, do not just create them:\nfor file in *.tgz; do tar tzf \u0026#34;$file\u0026#34; \u0026gt;/dev/null \u0026amp;\u0026amp; echo \u0026#34;OK: $file\u0026#34; done Exit criteria for the backup step:\netcd snapshot exists and is copied off the control-plane node. config/tls/db archives exist for every control-plane node. archives pass tar tzf. Rancher Helm values and status are captured. current Kubernetes state captures are present. Upgrade Rancher Before Kubernetes Before changing Kubernetes/RKE2 minor versions, confirm the Rancher target supports the desired downstream and local cluster targets.\nPreserve current Rancher values:\nhelm get values rancher -n cattle-system -o yaml \u0026gt; rancher-helm-values.yaml Then upgrade with Helm:\nRANCHER_TARGET_VERSION=\u0026#34;2.14.3\u0026#34; helm upgrade rancher rancher-stable/rancher \\ --namespace cattle-system \\ --version \u0026#34;$RANCHER_TARGET_VERSION\u0026#34; \\ -f rancher-helm-values.yaml If the pre-upgrade hook fails with BackoffLimitExceeded, inspect the hook logs before retrying. One important failure mode is stale RKE1 provisioning artifacts blocking Rancher 2.12+ upgrades:\nRancher v2.12+ does not support RKE1. Detected RKE1-related resources. NodeTemplate: 2 Check whether stale node templates exist and whether any active node pools reference them:\nkubectl get nodetemplates.management.cattle.io -A -o wide kubectl get nodepools.management.cattle.io -A -o wide kubectl get clusters.provisioning.cattle.io -A If the NodeTemplate objects are stale and no NodePool objects reference them, remove the stale artifacts, delete the failed hook job, and retry:\nkubectl delete nodetemplate.management.cattle.io -n cattle-global-nt \u0026lt;template-name\u0026gt; kubectl delete job -n cattle-system rancher-pre-upgrade helm upgrade rancher rancher-stable/rancher \\ --namespace cattle-system \\ --version \u0026#34;$RANCHER_TARGET_VERSION\u0026#34; \\ -f rancher-helm-values.yaml Do not delete provisioning artifacts blindly. The safety check is that they are stale and not referenced by active node pools.\nValidate The Rancher Upgrade Validate from the inside out:\nkubectl rollout status deploy/rancher -n cattle-system --timeout=10m helm --namespace cattle-system list kubectl -n cattle-system get deploy rancher -o jsonpath=\u0026#39;{.spec.template.spec.containers[0].image}{\u0026#34;\\n\u0026#34;}\u0026#39; kubectl get pods -n cattle-system -o wide kubectl get pods -n cattle-fleet-system -o wide kubectl get pods -n cattle-fleet-local-system -o wide kubectl get bundles -A kubectl get clusters.provisioning.cattle.io -A Check the browser path and unauthenticated API behavior through the real hostname:\ncurl -k -I https://rancher.example.com/dashboard/ curl -k -I https://rancher.example.com/v3 Expected signals:\nRancher deployment rolled out. Helm shows the target Rancher chart and app version. rancher/rancher:\u0026lt;target\u0026gt; is running. Rancher pods are ready. webhook is running. Fleet controller and local Fleet agent are running. managed clusters are visible. /dashboard/ returns 200. /v3 may return 401 when unauthenticated, which still proves the endpoint is reachable. Watch For Post-Upgrade Rancher Symptoms After a major Rancher upgrade, downstream clusters may briefly report Fleet or agent churn. Do not assume every UI warning is fatal, but do not ignore these patterns:\nfleet-agent ... Pending termination 0/1 Bundles Ready unable to create impersonator account failed to get secret for service account: cattle-impersonation-system/... Use both management-cluster and downstream-cluster checks:\nkubectl get bundles -A kubectl get bundledeployments -A kubectl get clusters.provisioning.cattle.io -A kubectl get pods -A | egrep \u0026#39;fleet|cattle|rancher\u0026#39; If a downstream kubeconfig fails through Rancher impersonation, verify whether direct cluster access works before blaming Kubernetes itself. The failing layer may be Rancher-generated impersonation, not the downstream API server.\nRewrite The Runbook After The First Upgrade The first management cluster is where the runbook learns. Before touching the next dev or production Rancher cluster, add checks for what actually happened:\nremove or document stale RKE1 NodeTemplate artifacts before Rancher 2.12+. confirm no active NodePool references before deleting old templates. validate root filesystem capacity on every control-plane node. prove backup archives and etcd snapshots are readable off-node. verify replacement workers can join without reserved kubelet labels. label display roles after join instead of breaking kubelet startup. capture Rancher Helm values before the upgrade and after the upgrade. verify which system-upgrade-controller Plans are active before each hop. validate Fleet bundles and downstream impersonation after the UI loads. Operating Rule Do not treat Rancher upgrade readiness as “Rancher supports the target Kubernetes version.”\nUpgrade readiness means the management cluster is stable, worker capacity exists, disk pressure is gone, backups are verified, stale provisioning artifacts are understood, and Rancher/Fleet/downstream cluster health can be proven before moving on to Kubernetes itself.\nRelated:\nRancher RKE2 Minor Hops When UI Metadata And GitOps Plans Disagree Rancher RKE2 Upgrade Pods Mutate The Host Filesystem When A Latent Rancher Worker Upgrade Becomes An Outage Emergency Stop For Rancher System Upgrade Controller Kubernetes Upgrade Sequencing ","permalink":"https://trinidadmarroquin.com/posts/rancher-management-cluster-upgrade-preflight/","section":"posts","summary":"Rancher management-cluster upgrades are not just a Helm command. They are a sequence of readiness gates.\nThe version target matters, but it is not the first question. The first question is whether the management cluster is stable enough to survive the change.\nIn one upgrade run, the desired path was straightforward on paper:\nRancher first, then Kubernetes/RKE2. The actual work exposed the operational details that make or break the window: stale workers, version skew, node join scripts, root filesystem pressure, backup transfer permissions, Rancher pre-upgrade hooks, Fleet health, and downstream cluster impersonation.\n","tags":["rancher","rke2","kubernetes","upgrades","operations","backup","helm"],"title":"Rancher Management Cluster Upgrades Need More Than A Version Target"},{"categories":["field-notes"],"content":"Replacing an RKE2 worker can look like a token, hostname, or API availability problem when the real issue is lower in the node networking path.\nOne useful pattern is to separate three signals that often appear together:\nthe RKE2 agent cannot reach the supervisor through its local load balancer. the Kubernetes API reports duplicate node identity or stale node password state. Calico advertises an address from the wrong interface or subnet. Those symptoms are related, but they are not all fixed in the same place.\nSymptom The replacement worker starts rke2-agent, but the service never settles:\nrke2-agent: failed to get CA certs: Get \u0026#34;https://127.0.0.1:6444/cacerts\u0026#34;: context deadline exceeded rke2-agent: Waiting to retrieve kube-proxy configuration; server is not ready The 127.0.0.1:6444 address is not the Kubernetes API server itself. It is the node-local RKE2 agent load balancer. A timeout there means the local agent process could not establish a working upstream path to the configured server endpoint.\nDo not stop at the localhost address. Check what the agent is trying to reach.\nsudo systemctl status rke2-agent --no-pager sudo journalctl -u rke2-agent -n 200 --no-pager sudo cat /etc/rancher/rke2/config.yaml sudo ss -lntp | grep -E \u0026#39;6444|9345|6443\u0026#39; || true The expected agent config shape is small:\nserver: https://cluster-a-api.example.com:9345 token: REDACTED node-name: worker-1 If the token line is malformed, fix that first. If the server URL is wrong or DNS points at the wrong place, fix that first. If both are correct, move down the stack.\nSeparate Join Identity From Network Reachability RKE2 may also report node identity errors:\nNode password rejected, duplicate hostname or contents of /etc/rancher/node/password may not match server node-passwd entry That is a different class of problem than an API timeout.\nFor a true node replacement, make the lifecycle explicit:\nkubectl get nodes -o wide kubectl delete node worker-1 sudo systemctl stop rke2-agent sudo rm -rf /etc/rancher/node sudo rm -rf /var/lib/rancher/rke2/agent sudo systemctl start rke2-agent Only do this on the replacement node after confirming it is not a running production member you still need. The point is to avoid mixing stale node password state with the new machine\u0026rsquo;s join attempt.\nIf the agent still cannot reach the supervisor after identity cleanup, the remaining problem is likely connectivity, host routing, firewalling, certificate trust, proxy settings, or CNI/node IP behavior.\nCheck The Address Calico Selected When the worker finally appears, or when comparing against healthy nodes, check the Kubernetes node address next to the Calico annotation:\nkubectl get nodes -o json \\ | jq -r \u0026#39;.items[] | [ .metadata.name, (.status.addresses[]? | select(.type == \u0026#34;InternalIP\u0026#34;) | .address), (.metadata.annotations[\u0026#34;projectcalico.org/IPv4Address\u0026#34;] // \u0026#34;NONE\u0026#34;), (.metadata.annotations[\u0026#34;projectcalico.org/IPv4VXLANTunnelAddr\u0026#34;] // \u0026#34;NONE\u0026#34;) ] | @tsv\u0026#39; \\ | column -t Healthy shape:\nworker-1 192.0.2.10 192.0.2.10/24 198.51.100.10 worker-2 192.0.2.11 192.0.2.11/24 198.51.100.11 Problem shape:\nworker-1 192.0.2.10 169.254.10.25/24 198.51.100.10 That mismatch means Kubernetes and Calico disagree about the node address. In a multi-homed vSphere environment, Calico\u0026rsquo;s default autodetection can choose a storage, backup, migration, or otherwise non-primary interface. The node may look partially alive while pod networking and node-to-node paths fail in confusing ways.\nFix The Source Of Truth Do not hand-edit projectcalico.org/IPv4Address as the durable fix. In a Tigera operator-managed install, fix the Installation resource so every Calico node gets the same address selection policy.\nExample using an intended node CIDR:\nkubectl patch installation default --type merge -p \u0026#39;{ \u0026#34;spec\u0026#34;: { \u0026#34;calicoNetwork\u0026#34;: { \u0026#34;nodeAddressAutodetectionV4\u0026#34;: { \u0026#34;firstFound\u0026#34;: false, \u0026#34;cidrs\u0026#34;: [\u0026#34;192.0.2.0/24\u0026#34;] } } } }\u0026#39; Then roll or restart Calico components according to the platform runbook and re-check the node annotations.\nIf the cluster is not operator-managed, update the configured Calico autodetection method in the actual deployment source of truth. The principle is the same: make interface selection deterministic instead of relying on firstFound in a multi-network host.\nTriage Order Use this order to avoid chasing the wrong layer:\nConfirm /etc/rancher/rke2/config.yaml has a valid server, token, and node-name. Confirm DNS and direct TCP reachability to the RKE2 supervisor endpoint on 9345. Confirm local 127.0.0.1:6444 is created by the agent, not mistaken for a remote API address. Clear stale node identity only after deleting or intentionally replacing the old Kubernetes node object. Compare Kubernetes InternalIP with Calico projectcalico.org/IPv4Address. Fix Calico autodetection at the operator or manifest source of truth. Revalidate node readiness, CoreDNS, kube-proxy, and cross-node pod traffic. Acceptance Criteria The replacement is healthy when these are true:\nrke2-agent is active without repeated 127.0.0.1:6444 timeout loops. the node name is unique and has no stale password conflict. Kubernetes InternalIP uses the intended primary node network. Calico IPv4Address matches the intended node network, not a storage or auxiliary subnet. pods can communicate across nodes. the autodetection setting is versioned in the cluster\u0026rsquo;s source of truth. The key lesson is that an RKE2 worker join failure may expose multiple issues in sequence. Fix token and node identity problems when they are real, but still verify Calico node IP selection before declaring the replacement complete.\n","permalink":"https://trinidadmarroquin.com/field-notes/rke2-worker-join-calico-wrong-interface/","section":"field-notes","summary":"Replacing an RKE2 worker can look like a token, hostname, or API availability problem when the real issue is lower in the node networking path.\nOne useful pattern is to separate three signals that often appear together:\nthe RKE2 agent cannot reach the supervisor through its local load balancer. the Kubernetes API reports duplicate node identity or stale node password state. Calico advertises an address from the wrong interface or subnet. Those symptoms are related, but they are not all fixed in the same place.\n","tags":["rke2","kubernetes","calico","networking","troubleshooting","operations"],"title":"RKE2 Worker Join Failures From Calico Wrong Interface Selection"},{"categories":["field-notes"],"content":"Copied Terraform environment roots are convenient until they drift. One root has a newer module call, another has an old variable name, a third has stale README instructions, and the next environment starts from whichever directory someone copied last.\nUse a small scaffolding script when a repository has a standard root-module shape.\nWhat To Generate For a vSphere environment root, generate the complete directory shape every time:\nenvironments/site-a/example/ main.tf variables.tf locals.tf outputs.tf terraform.tfvars README.md The script should create files that are immediately recognizable to operators:\nmain.tf wires provider configuration and the shared module. variables.tf declares the environment inputs. locals.tf contains the VM list or per-environment object map. outputs.tf exposes useful VM and source-of-truth outputs. terraform.tfvars is blank or placeholder-only and ready for local population. README.md explains how to plan, preflight, apply, and destroy from that root. Do not generate a half-root that still requires copying files by hand. The helper exists to remove copy/paste decisions.\nPut The Script Where Operators Look If the repository already has a scripts/ directory, put the generator there:\nscripts/create-terraform-directory.sh environments/site-a/example That keeps scaffolding with the rest of the operational helpers, such as preflight checks and inventory lookups. Document both common invocation styles:\n# From repository root scripts/create-terraform-directory.sh environments/site-a/example # From an environment parent directory ../../scripts/create-terraform-directory.sh example The script should refuse to overwrite an existing directory unless an explicit force flag exists. Accidental regeneration over an active Terraform root is worse than a failed command.\nKeep Real Values Out Of Git A generated terraform.tfvars file is useful because it tells the operator what to populate, but it can become a secret or environment-specific data file quickly.\nUse one of these patterns deliberately:\nterraform.tfvars.example # committed sample values terraform.tfvars # ignored real values *.auto.tfvars # ignored or tightly controlled if environment-specific If the repository intentionally creates a blank terraform.tfvars, make sure .gitignore policy is clear and consistent. Do not rely on memory to keep credentials, vCenter endpoints, or private addressing out of commits.\nRelated: Terraform Variable File Hygiene covers the broader input-file ownership model.\nSmoke-Test The Generated Root The generator is infrastructure code. Test it before publishing:\nbash -n scripts/create-terraform-directory.sh scripts/create-terraform-directory.sh environments/site-a/.scaffold-test terraform fmt -check -recursive environments/site-a/.scaffold-test rm -r environments/site-a/.scaffold-test If the generated files include module references, also run a lightweight init in a disposable path when provider access allows it:\nterraform -chdir=environments/site-a/.scaffold-test init -backend=false Do not skip the smoke test. A broken generator spreads the same mistake to every future root.\nUpdate The README In The Same Change Scaffolding scripts often expose stale documentation. If the active repository layout is now:\nenvironments/ modules/ templates/ scripts/ archive/ then the README should not still describe legacy paths such as old Packer or Terraform directories. Update the sections operators actually use:\nrepository structure. common requirements. quick start. helper script catalog. preflight and guardrail workflow. Documentation drift is not cosmetic. If the quick start points at a deleted path, operators will copy an old root or bypass the guardrails.\nKeep Branches Clean Infrastructure repos often have unrelated local plans, generated JSON, or environment edits in the worktree. Stage only the scaffolding change:\ngit add README.md scripts/README.md scripts/create-terraform-directory.sh git diff --cached --stat git diff --cached If the branch requires ticket-prefixed commit messages, fix that before review. Rewriting a feature branch can be acceptable, but use --force-with-lease and only after confirming the branch is yours to rewrite.\nOperating Rule New Terraform roots should be generated, not copied from memory.\nThe generator, README, and smoke test together form the contract: every new environment starts with the same module shape, the same guardrail workflow, and the same warning about where real values belong.\n","permalink":"https://trinidadmarroquin.com/field-notes/terraform-environment-scaffolding-consistency/","section":"field-notes","summary":"Copied Terraform environment roots are convenient until they drift. One root has a newer module call, another has an old variable name, a third has stale README instructions, and the next environment starts from whichever directory someone copied last.\nUse a small scaffolding script when a repository has a standard root-module shape.\nWhat To Generate For a vSphere environment root, generate the complete directory shape every time:\nenvironments/site-a/example/ main.tf variables.tf locals.tf outputs.tf terraform.tfvars README.md The script should create files that are immediately recognizable to operators:\n","tags":["terraform","vsphere","automation","hcl","documentation","operations"],"title":"Terraform Environment Scaffolding For Consistent vSphere Roots"},{"categories":["posts"],"content":"Packer template builds are a good fit for CI/CD because they are repeatable, expensive enough to benefit from automation, and easy to forget after the first successful run. They also expose every hidden dependency in the platform: repository access, vCenter permissions, ISO availability, DNS, routing, worker placement, and secret handling.\nThe goal was not just to make one template build pass. The goal was to make template builds runnable per site from Concourse, with enough visibility to debug failures and enough guardrails to avoid leaking credentials.\nStart With The Smallest Useful Pipeline The first useful shape was simple:\nclone template repo -\u0026gt; packer init -\u0026gt; packer build for one site That proved the runtime container could reach the repository, install or use Packer plugins, read the site variable file, and talk to vCenter. Once a single site worked, the pipeline could fan out into site-specific jobs:\nbuild-site-a build-site-b build-site-c Parallelism mattered because template builds spend a lot of time waiting on remote infrastructure. If sites use separate vCenter managers and independent storage, running builds in parallel is reasonable. If several sites share one vCenter or datastore, serialize those jobs or use Concourse serial groups so the pipeline does not create artificial contention.\nThe important design choice is to model real infrastructure boundaries. Do not parallelize because Concourse can. Parallelize where the underlying platforms are independent enough to tolerate it.\nTreat Local Secrets As Temporary Plumbing During a proof of concept, it is common to start with a local vars.yaml file passed to fly set-pipeline:\nfly -t ci set-pipeline \\ -p packer-templates \\ -c concourse/pipeline.yaml \\ -l concourse/vars.yaml \\ -n That is acceptable only as temporary plumbing. The durable model should be Concourse variable interpolation backed by a credential manager:\nparams: GITLAB_TOKEN: ((gitlab-token)) VCENTER_USERNAME: ((site-a-vcenter-username)) VCENTER_PASSWORD: ((site-a-vcenter-password)) The pipeline should not require committing secrets.pkrvars.hcl. It should either pass sensitive values as -var arguments from Concourse variables or generate a temporary secrets file inside the task from environment variables.\nIf the long-term target is Vault, keep the variable names stable from the start. Concourse can read ((name)) from a local vars file during the POC and later resolve the same names through Vault. That makes the migration a credential-backend change instead of a pipeline rewrite.\nKeep Secrets Out Of Job Logs The fastest way to leak secrets is to run shell tasks with full trace output while echoing generated files:\nset -x echo \u0026#34;VCENTER_PASSWORD=${VCENTER_PASSWORD}\u0026#34; Avoid that pattern. Use trace output around safe commands only, or explicitly disable tracing before materializing secrets:\nset -euo pipefail apk add --no-cache git git clone --depth 1 \u0026#34;$REPO_URL\u0026#34; repo set +x cat \u0026gt; secrets.auto.pkrvars.hcl \u0026lt;\u0026lt;EOF vcenter_username = ${VCENTER_USERNAME@Q} vcenter_password = ${VCENTER_PASSWORD@Q} EOF set -x packer init templates/ubuntu-24-04.pkr.hcl packer build -force \\ -var-file=environments/site-a/variables.pkrvars.hcl \\ -var-file=secrets.auto.pkrvars.hcl \\ templates/ubuntu-24-04.pkr.hcl Also review clone URLs. An HTTPS clone command with a token embedded in the URL can appear in logs if the shell prints commands. Prefer Concourse resources or commands that avoid echoing credentials. If a token expires, rotate it and redeploy the pipeline, but also use the incident to remove any logging path that prints the token.\nMake Packer Logs Useful Without Drowning The Job Packer debug logging is useful when a build hangs during SSH or vSphere provisioning:\nPACKER_LOG=1 PACKER_LOG_PATH=packer.log packer build ... For CI, decide whether the log belongs in the console or as an artifact. Console logs are convenient during a POC, but verbose plugin output can bury the signal. A better production pattern is:\nnormal console output -\u0026gt; concise operator status packer.log -\u0026gt; retained artifact for deep debugging Useful console signals include:\nthe site being built. the selected Concourse worker. the vCenter endpoint name, sanitized if necessary. whether the build reached VM creation, SSH wait, provisioning, cleanup, and template conversion. a link or artifact name for the full Packer log. That gives operators enough context without making every successful build scroll for thousands of lines.\nWorker Placement Is Architecture, Not Scheduling Trivia Packer does not only talk to vCenter. The build process often needs to SSH from the Concourse task container to the temporary VM. That means the selected worker must have network reachability to the build VLAN.\nThis is where Concourse worker placement becomes an architectural decision:\ncentral cluster Concourse web Concourse database local workers for nearby sites remote clusters workers close to remote vCenter and build networks If central workers cannot reach a remote build network, the pipeline will fail even though vCenter can create the VM. Typical symptoms look like SSH timeouts:\nTCP connection to SSH ip/port failed: dial tcp 192.0.2.25:22: i/o timeout Before changing Packer code, prove the route from the worker:\nkubectl exec -n concourse concourse-worker-0 -- ip route get 192.0.2.25 kubectl exec -n concourse concourse-worker-0 -- nc -vz 192.0.2.25 22 If the worker subnet cannot reach the build subnet, the fix may be a firewall route, moving the build VM to a reachable port group, or placing a worker in the remote site. That is not a Packer problem.\nHost Networking Has A Cost Using hostNetwork: true for Concourse workers can be a practical bridge when task containers need the same routes as the Kubernetes nodes. It can also hide problems and expand the worker\u0026rsquo;s blast radius.\nThe decision should be explicit:\nUse host networking when the build workflow depends on node-level routes that pod networking does not expose. Document which subnets workers must reach and why. Keep the Concourse namespace and worker nodes tightly controlled. Revisit the decision when remote workers or proper pod routing are available. Host networking is not a generic performance tuning flag. In this workflow, it was about making SSH and vCenter-adjacent traffic follow the expected network path.\nISO Path And URL Fallback Need Clear Ownership vSphere ISO paths are fast when the ISO exists in the expected datastore, but brittle when names drift or datastores differ by site. URL-based ISO download is slower but more portable.\nA safe model is:\nprefer datastore ISO path when present fallback to ISO URL when missing cache downloads when possible fail with a clear error when neither source works When a VM drops into EFI Boot Manager, do not assume the answer is a bigger boot wait. Check whether the boot ISO was actually attached and whether the site-specific iso_path, datastore, and CD-ROM settings match the target vCenter.\nThe pipeline should make this visible in the job output:\nsite=site-a iso_source=datastore iso_path=[datastore-a] ISO/ubuntu.iso fallback_url_enabled=true That turns a boot screen mystery into a configuration check.\nProduction Readiness Checklist Before calling a Concourse-backed template build service production-ready, verify these items:\nRepository access uses a scoped credential and does not print tokens. Packer variables separate public site config from secrets. Secrets resolve through Concourse variables, with a path to Vault or another credential manager. Site jobs match the real vCenter and network ownership model. Worker placement is documented, including which source subnets need access to which vCenter and build networks. Packer logs are retained without dumping sensitive material to the console. ISO source behavior is consistent across sites. Build failures show whether the failure happened in clone, Packer init, VM creation, SSH wait, provisioning, or template conversion. Concourse web access has DNS, TLS, and authentication configured separately from build execution. The ingress side is its own checklist. See Concourse Ingress DNS And TLS Cutover for the browser-facing DNS, TLS secret, and externalUrl work.\nFor bootstrap-specific diagnosis, see Packer Bootstrap Placement Versus Runtime Execution. That note separates Packer file placement, SSH provisioner reachability, and Terraform/cloud-init runtime execution.\nOperating Rule Do not treat a successful template build as proof that the platform is ready.\nThe platform is ready when failed builds are diagnosable, secrets stay out of logs, workers run from networks that can reach their targets, and each site can be rebuilt without relying on someone\u0026rsquo;s workstation as the missing route.\n","permalink":"https://trinidadmarroquin.com/posts/concourse-packer-template-builds-network-secrets/","section":"posts","summary":"Packer template builds are a good fit for CI/CD because they are repeatable, expensive enough to benefit from automation, and easy to forget after the first successful run. They also expose every hidden dependency in the platform: repository access, vCenter permissions, ISO availability, DNS, routing, worker placement, and secret handling.\nThe goal was not just to make one template build pass. The goal was to make template builds runnable per site from Concourse, with enough visibility to debug failures and enough guardrails to avoid leaking credentials.\n","tags":["concourse","packer","vsphere","cicd","kubernetes","secrets","networking"],"title":"Turning Packer Template Builds Into A Concourse Workflow"},{"categories":["field-notes"],"content":"Containerizing a DNS drift detector is useful when the operator environment is inconsistent. The image can carry the shell runner, Ansible configuration, Python dependencies, and command defaults while the runtime provides the inventory, credentials, and report destination.\nThe important boundary is this: the image should run audits, not own remediation or secrets.\nImage Contents Keep the image small and boring:\nFROM python:3.12-slim ENV ANSIBLE_CONFIG=/app/ansible.cfg \\ PYTHONDONTWRITEBYTECODE=1 \\ PYTHONUNBUFFERED=1 RUN apt-get update \\ \u0026amp;\u0026amp; apt-get install -y --no-install-recommends \\ bash \\ ca-certificates \\ openssh-client \\ sshpass \\ bsdextrautils \\ \u0026amp;\u0026amp; rm -rf /var/lib/apt/lists/* WORKDIR /app COPY requirements.txt ./ RUN pip install --no-cache-dir -r requirements.txt COPY ansible.cfg ./ COPY bin/dns-drift-detector.sh /usr/local/bin/dns-drift-detector RUN chmod +x /usr/local/bin/dns-drift-detector ENTRYPOINT [\u0026#34;dns-drift-detector\u0026#34;] This gives the runner a predictable Bash and Ansible environment without baking in site inventory or operator credentials.\nRuntime Contract Mount inputs and outputs explicitly:\ndocker run --rm \\ -v \u0026#34;$PWD/inventory/site-a-rke2.yaml:/inventory/inventory.yaml:ro\u0026#34; \\ -v \u0026#34;$PWD/reports:/reports\u0026#34; \\ -v \u0026#34;$HOME/.ssh:/ssh:ro\u0026#34; \\ -e SSH_USER=operator \\ -e ANSIBLE_EXTRA_ARGS=\u0026#39;--private-key /ssh/id_rsa\u0026#39; \\ dns-drift-detector:local \\ --inventory /inventory/inventory.yaml \\ --group site_a_rke2 The container receives only what it needs for the run:\ninventory as read-only input. SSH material as read-only input. reports as writable output. target group as an explicit argument. Do not copy generated CSVs into the image, and do not commit them unless they are sanitized fixtures.\nAudit-Only Mode DNS search-domain remediation usually needs elevated host changes: netplan edits, systemd-resolved drop-ins, service restarts, and final audits. That workflow should remain separate from scheduled drift detection.\nThe detector should answer a narrower question:\nWhich targeted nodes differ from the expected resolver baseline right now? That keeps routine runs safe enough for CI, cron, or operator smoke tests. Remediation can consume the report later, but the audit image should not mutate hosts by default.\nSecrets Boundary There are two common local credential paths:\n# SSH key path -v \u0026#34;$HOME/.ssh:/ssh:ro\u0026#34; \\ -e ANSIBLE_EXTRA_ARGS=\u0026#39;--private-key /ssh/id_rsa\u0026#39; # Password path -e SSH_USER=operator \\ -e SSH_PASSWORD=\u0026#39;...\u0026#39; \\ -e BECOME_PASSWORD=\u0026#39;...\u0026#39; Keep both outside the image. The image should not contain .env, private keys, inventory secrets, Vault tokens, or cluster-specific group vars.\nUse .dockerignore defensively:\n.env reports/* __pycache__/ *.pyc .pytest_cache/ If the repo keeps reports/.gitkeep, ignore generated report contents while preserving the directory.\nCompose For Smoke Tests A local Compose file is useful, but it should remain a smoke-test wrapper around the same runtime contract:\nservices: dns-drift-detector: build: context: . image: dns-drift-detector:local env_file: - path: .env required: false volumes: - ./inventory.example.yaml:/inventory/inventory.yaml:ro - ./reports:/reports - ~/.ssh:/ssh:ro environment: ANSIBLE_EXTRA_ARGS: \u0026#34;--private-key /ssh/id_rsa\u0026#34; command: - --inventory - /inventory/inventory.yaml - --group - example_cluster_rke2 The example inventory should be harmless. Real inventories should be mounted at runtime.\nValidate Before Publishing Validate the shell entrypoint, image build, help output, and Compose rendering:\nbash -n bin/dns-drift-detector.sh docker build -t dns-drift-detector:local . docker run --rm dns-drift-detector:local --help docker compose config Then run one inventory in list-only mode before a full audit:\ndocker run --rm \\ -v \u0026#34;$PWD/inventory/site-a-rke2.yaml:/inventory/inventory.yaml:ro\u0026#34; \\ dns-drift-detector:local \\ --inventory /inventory/inventory.yaml \\ --group site_a_rke2 \\ --list-hosts Operating Rule Containerized audit tools should package execution consistency, not operational authority.\nKeep inventories, credentials, reports, and remediation outside the image so the detector can be rebuilt and shared without carrying site-specific risk.\nRelated: DNS Drift Detector Calico Overlay False Positives covers parser behavior after the containerized runner has produced reports.\n","permalink":"https://trinidadmarroquin.com/field-notes/containerized-dns-drift-detector-boundaries/","section":"field-notes","summary":"Containerizing a DNS drift detector is useful when the operator environment is inconsistent. The image can carry the shell runner, Ansible configuration, Python dependencies, and command defaults while the runtime provides the inventory, credentials, and report destination.\nThe important boundary is this: the image should run audits, not own remediation or secrets.\nImage Contents Keep the image small and boring:\nFROM python:3.12-slim ENV ANSIBLE_CONFIG=/app/ansible.cfg \\ PYTHONDONTWRITEBYTECODE=1 \\ PYTHONUNBUFFERED=1 RUN apt-get update \\ \u0026amp;\u0026amp; apt-get install -y --no-install-recommends \\ bash \\ ca-certificates \\ openssh-client \\ sshpass \\ bsdextrautils \\ \u0026amp;\u0026amp; rm -rf /var/lib/apt/lists/* WORKDIR /app COPY requirements.txt ./ RUN pip install --no-cache-dir -r requirements.txt COPY ansible.cfg ./ COPY bin/dns-drift-detector.sh /usr/local/bin/dns-drift-detector RUN chmod +x /usr/local/bin/dns-drift-detector ENTRYPOINT [\u0026#34;dns-drift-detector\u0026#34;] This gives the runner a predictable Bash and Ansible environment without baking in site inventory or operator credentials.\n","tags":["dns","ansible","docker","kubernetes","rke2","audit","operations"],"title":"Containerized DNS Drift Detector Boundaries"},{"categories":["Practical Data Science For DevOps And SRE"],"content":"Part 1 of the Practical Data Science For DevOps And SRE series.\nAverage latency is easy to explain and easy to graph. That is why it shows up everywhere. It is also why it can mislead reliability reviews.\nIf most requests are fast but a meaningful slice of users wait several seconds, the average may still look healthy. The users in the slow tail do not experience the average. They experience the request they are waiting on.\nPercentiles give DevOps and SRE teams a better way to talk about that tail.\nThe Problem With Averages An average compresses a distribution into one number. That can be useful for a quick summary, but it hides shape.\nConsider a small set of request durations in seconds:\nRequest Duration 1 0.12 2 0.13 3 0.14 4 0.15 5 0.16 6 0.18 7 0.21 8 0.25 9 1.80 10 2.40 The average is about 0.55 seconds. That sounds acceptable if the service target is vague.\nBut two users waited nearly two seconds or more. If this is a checkout flow, login path, search endpoint, or API used by another service, those slow requests matter.\nThe average did not lie mathematically. It just answered a weaker operational question.\nBetter Questions Instead of asking only:\nWhat is the average latency? Ask:\nHow fast is the service for a typical request? How slow is it for the slowest meaningful slice of users? Is the tail getting worse over time? Does the tail change by endpoint, method, or status code? Those questions map better to percentiles.\nMetric Practical Meaning p50 Half of requests are faster than this value. Useful for typical behavior. p95 95% of requests are faster than this value. Useful for most-user experience. p99 99% of requests are faster than this value. Useful for tail pain. Percentiles are not magic. They still need context. But they preserve more operational truth than an average alone.\nLab Context This article uses the SLI lab from the companion IaC repository:\nhttps://github.com/trinidadgithub/IaC/tree/main/sli_app The lab runs a Flask app, Prometheus, Grafana, and cAdvisor with Terraform-managed Docker containers. The Flask app intentionally creates variable latency and occasional failures so the graphs have something useful to show.\nThe application exposes a Prometheus histogram:\nsli_http_request_duration_seconds_bucket That histogram is what makes p95 and p99 latency analysis possible.\nRun The Lab From the lab directory:\ncd IaC/sli_app/terraform terraform init terraform apply Generate traffic from the sli_app directory:\ncd .. ITERATIONS=100 SLEEP_SECONDS=0 ./scripts/generate_traffic.sh Open Grafana:\nhttp://localhost:3000 Credentials:\nusername: admin password: admin01 Open the SLI dashboard:\nhttp://localhost:3000/d/sli-lab-flask/sli-lab-flask-service Query p95 Latency In Prometheus, p95 latency for the SLI lab can be queried with histogram_quantile:\nhistogram_quantile( 0.95, sum by (le, endpoint, method) ( rate(sli_http_request_duration_seconds_bucket[5m]) ) ) This reads as:\nFor each endpoint and method, estimate the latency value below which 95% of recent requests completed. That is more useful than asking whether the average looks acceptable across the whole service.\nQuery p99 Latency p99 uses the same pattern with a different quantile:\nhistogram_quantile( 0.99, sum by (le, endpoint, method) ( rate(sli_http_request_duration_seconds_bucket[5m]) ) ) p99 is more sensitive to rare slow requests. That can be valuable, but it can also be noisy when traffic volume is low. In a small lab, p99 may jump around. In production, p99 should be interpreted with request volume, endpoint criticality, and scrape window in mind.\nCompare Against Average Latency You can still calculate average latency from Prometheus histogram data:\nsum by (endpoint, method) ( rate(sli_http_request_duration_seconds_sum[5m]) ) / sum by (endpoint, method) ( rate(sli_http_request_duration_seconds_count[5m]) ) This query is not useless. It tells you the mean request duration over the window. The mistake is treating it as the whole user experience.\nIn the lab, compare the average latency query with the p95 and p99 queries. Watch for cases where the average looks calm while p95 or p99 shows a slower tail.\nWhat To Look For In Grafana On the SLI dashboard, focus on these panels:\nLatency Percentiles Request Rate By Endpoint Status Code Breakdown 5xx Error Rate Latency percentiles should not be reviewed alone. If p99 jumps during a tiny request window, it may represent one slow request. If p95 rises while request rate is healthy and error rate also increases, that is a stronger degradation signal.\nThe point is not to worship p95 or p99. The point is to see the shape of service behavior before turning it into a reliability claim.\nA Practical Review Pattern During a reliability review, use a small sequence of questions:\nWhat is the request rate for the endpoint? What are p50, p95, and p99 doing over the same window? Are slow requests concentrated on one endpoint or method? Did error rate rise at the same time? Did a deployment, infrastructure change, or dependency issue occur near the change in latency? This is practical data science. It is not advanced modeling. It is disciplined observation using the data already produced by the system.\nCommon Mistakes Percentiles are better than averages for tail behavior, but they can still be misused.\nDo not compare percentiles from different systems unless the measurement method is compatible. A p95 from load balancer logs, a p95 from application instrumentation, and a p95 from synthetic checks may describe different populations of requests.\nDo not review p99 without traffic volume. A p99 from ten requests is not the same kind of signal as a p99 from ten thousand requests.\nDo not aggregate unrelated endpoints too early. A slow write path can disappear inside a service-level aggregate if the read path has much higher traffic.\nDo not set thresholds before understanding normal behavior. A good SLO should come from user expectations and observed service behavior, not from a random round number on a dashboard.\nWhat This Supports Using percentiles supports better operational decisions:\nWhether an endpoint needs performance work. Whether an SLO should include latency. Whether a change degraded user experience. Whether a dashboard is hiding tail behavior. Whether alert thresholds are tied to meaningful user pain. Percentiles do not replace engineering judgment. They make the discussion harder to fake.\nField Note Takeaway Average latency is a useful supporting signal, but it is a weak primary reliability measure. Users experience individual requests, not averages.\nFor reliability reviews, start with request rate, p95, p99, error rate, and endpoint context. That combination gives a clearer picture of whether the service is healthy for most users and whether the slow tail is becoming operationally meaningful.\nWhen the lab is no longer needed, shut it down cleanly:\ncd IaC/sli_app/terraform terraform destroy References Prometheus Histograms And Summaries Prometheus histogram_quantile Google SRE Workbook: Implementing SLOs Building A Small SLI Lab With Flask, Prometheus, And Grafana ","permalink":"https://trinidadmarroquin.com/posts/practical-data-science-for-devops/using-percentiles-instead-of-averages/","section":"posts","summary":"Part 1 of the Practical Data Science For DevOps And SRE series.\nAverage latency is easy to explain and easy to graph. That is why it shows up everywhere. It is also why it can mislead reliability reviews.\nIf most requests are fast but a meaningful slice of users wait several seconds, the average may still look healthy. The users in the slow tail do not experience the average. They experience the request they are waiting on.\n","tags":["devops","sre","data-science","percentiles","latency","prometheus","grafana","sli","observability"],"title":"Using Percentiles Instead Of Averages In Reliability Reviews"},{"categories":["notes"],"content":"Calico is not just a YAML install. It becomes part of the cluster\u0026rsquo;s trust model, routing behavior, NetworkPolicy enforcement, and incident response surface.\nThe important decisions happen before the first manifest is applied:\nwho owns CNI installation. which node interface Calico should use. which pod CIDR the cluster will allocate from. whether encapsulation is VXLAN, VXLAN cross-subnet, IP-in-IP, or BGP. how operators will prove the install is healthy after upgrades and node replacement. This guide is written for RKE2 clusters where Calico is installed and managed intentionally, not discovered later as a side effect of another installer.\nPick One Ownership Model Do not mix CNI ownership models.\nThere are two common approaches:\nRKE2-managed CNI RKE2 config selects the CNI and RKE2 deploys/manages the manifests. Self-managed Calico RKE2 starts without a packaged CNI, then GitOps or an operator installs Calico. Both can work. The failure mode is combining them: RKE2 installs one CNI while GitOps installs another, or an operator tries to reconcile resources already owned by the distribution.\nFor a platform team, I prefer an explicit self-managed Calico path when the environment needs version pinning, custom autodetection, NetworkPolicy conventions, or repeatable GitOps promotion. That usually means:\n# /etc/rancher/rke2/config.yaml cni: none Then install Calico through a controlled path such as ArgoCD, Flux, or a reviewed bootstrap stage.\nIf you use the RKE2-managed Calico option instead, keep the same design discipline but place the configuration in the RKE2-supported location for your version. The operating rules below still apply: one owner, explicit node IP selection, and real validation.\nDefine The Network Inputs Before installing Calico, write down the network contract.\nExample:\ncluster: cluster-a-prod node network: 192.0.2.0/24 pod network: 198.51.100.0/24 service network: 203.0.113.0/24 node interface: ens192 encapsulation: VXLAN cross-subnet NetworkPolicy: enforced by Calico The exact CIDRs are environment-specific. The point is to separate three different address spaces:\nnode IPs: addresses used by Kubernetes nodes. pod IPs: addresses assigned to pods by Calico. service IPs: virtual service addresses assigned by Kubernetes. Do not let Calico infer the node interface in a multi-homed environment unless you are comfortable with the outcome. If a node has management, storage, backup, and workload networks, autodetection can pick the wrong address.\nInstall The Tigera Operator For a self-managed install, the Tigera operator owns Calico components and reconciles the Installation custom resource.\nPin the manifest version instead of applying a floating URL in production:\nkubectl apply -f tigera-operator-vX.Y.Z.yaml Then verify the operator is running:\nkubectl get pods -n tigera-operator kubectl get crd | grep -E \u0026#39;operator.tigera.io|projectcalico.org\u0026#39; Expected early signal:\ntigera-operator 1/1 Running At this point, the operator exists, but Calico is not fully configured until the Installation resource is applied.\nCreate The Installation Resource A minimal operator-managed install should be explicit about pod pools and node address autodetection.\nExample:\napiVersion: operator.tigera.io/v1 kind: Installation metadata: name: default spec: variant: Calico calicoNetwork: bgp: Disabled nodeAddressAutodetectionV4: cidrs: - 192.0.2.0/24 ipPools: - name: default-ipv4-ippool cidr: 198.51.100.0/24 encapsulation: VXLANCrossSubnet natOutgoing: Enabled nodeSelector: all() This example says:\nCalico should use node addresses from 192.0.2.0/24. Pods should receive addresses from 198.51.100.0/24. VXLAN cross-subnet is used instead of assuming pure L3 routing between all nodes. outbound pod traffic is NATed when leaving the pod network. BGP is disabled. For environments that intentionally use BGP, the design changes. You need route reflectors, peer policy, firewall rules, and failure-domain thinking. Do not enable BGP because it sounds more advanced. Use it because the network is designed to route pod CIDRs directly.\nApply the resource:\nkubectl apply -f calico-installation.yaml Then watch reconciliation:\nkubectl get installation default -o yaml kubectl get pods -n calico-system -o wide kubectl get pods -n tigera-operator -o wide Operate Autodetection Through The Tigera Operator In a Tigera operator-managed cluster, node IP autodetection is not a node-by-node setting to hand-edit first. The durable fix belongs in the Installation resource:\nspec.calicoNetwork.nodeAddressAutodetectionV4 When Calico selects the wrong interface or subnet, patch the operator source of truth so every calico-node pod receives the same intended policy.\nExample patch:\nkubectl --context cluster-a-prod patch installation.operator.tigera.io default \\ --type=merge \\ -p \u0026#39;{ \u0026#34;spec\u0026#34;: { \u0026#34;calicoNetwork\u0026#34;: { \u0026#34;nodeAddressAutodetectionV4\u0026#34;: { \u0026#34;firstFound\u0026#34;: false, \u0026#34;cidrs\u0026#34;: [\u0026#34;192.0.2.0/24\u0026#34;] } } } }\u0026#39; Then wait for the operator-managed DaemonSet to roll:\nkubectl --context cluster-a-prod -n tigera-operator get pods -o wide kubectl --context cluster-a-prod -n calico-system rollout status \\ ds/calico-node \\ --timeout=10m Confirm the operator accepted the patch:\nkubectl --context cluster-a-prod get installation.operator.tigera.io default -o json \\ | jq -r \u0026#39;.spec.calicoNetwork.nodeAddressAutodetectionV4\u0026#39; Expected shape:\n{ \u0026#34;firstFound\u0026#34;: false, \u0026#34;cidrs\u0026#34;: [ \u0026#34;192.0.2.0/24\u0026#34; ] } Also inspect what the rendered calico-node pod receives:\nkubectl --context cluster-a-prod -n calico-system get ds calico-node -o json \\ | jq -r \u0026#39;.spec.template.spec.containers[] | select(.name == \u0026#34;calico-node\u0026#34;) | .env[]? | select(.name == \u0026#34;IP_AUTODETECTION_METHOD\u0026#34; or .name == \u0026#34;IP\u0026#34; or .name == \u0026#34;FELIX_IPAUTODETECTIONMETHOD\u0026#34;) | \u0026#34;\\(.name)=\\(.value)\u0026#34;\u0026#39; Finally, verify at least one node annotation against its Kubernetes InternalIP:\nkubectl --context cluster-a-prod get node worker-1 -o jsonpath=\u0026#39;{.status.addresses[?(@.type==\u0026#34;InternalIP\u0026#34;)].address}{\u0026#34;\\t\u0026#34;}{.metadata.annotations.projectcalico\\.org/IPv4Address}{\u0026#34;\\n\u0026#34;}\u0026#39; The host portion should match:\n192.0.2.10 192.0.2.10/24 The safe remediation order is:\naudit mismatched nodes. patch the Tigera Installation resource. wait for calico-node rollout. validate node annotations. only then use node-level remediation for stale nodes that did not refresh. Node-level remediation usually means cordoning the affected node, optionally draining workers, deleting the calico-node pod on that node, waiting for the node to become Ready, and confirming projectcalico.org/IPv4Address now matches InternalIP. If the annotation still does not match, leave the node cordoned and investigate the operator config instead of repeatedly deleting pods.\nVerify Node Address Selection The first serious verification is whether Calico chose the same node IP that Kubernetes reports as InternalIP.\nkubectl get nodes -o json \\ | jq -r \u0026#39;.items[] | [ .metadata.name, (.status.addresses[]? | select(.type == \u0026#34;InternalIP\u0026#34;) | .address), (.metadata.annotations[\u0026#34;projectcalico.org/IPv4Address\u0026#34;] // \u0026#34;NONE\u0026#34;), (.metadata.annotations[\u0026#34;projectcalico.org/IPv4VXLANTunnelAddr\u0026#34;] // \u0026#34;NONE\u0026#34;) ] | @tsv\u0026#39; \\ | column -t Healthy shape:\nworker-1 192.0.2.10 192.0.2.10/24 198.51.100.10 worker-2 192.0.2.11 192.0.2.11/24 198.51.100.11 Problem shape:\nworker-1 192.0.2.10 203.0.113.10/24 198.51.100.10 That means Calico selected an address that does not match the node\u0026rsquo;s Kubernetes InternalIP. In a multi-network vSphere or bare-metal environment, that is usually an autodetection problem.\nFix the autodetection policy first. Avoid hand-patching node annotations unless you are following a short-lived emergency runbook.\nVerify Calico Components Check the Calico workloads:\nkubectl get pods -n calico-system -o wide kubectl get daemonset -n calico-system kubectl get deployment -n calico-system Look for:\ncalico-node running on every schedulable node. Typha replicas running if your cluster uses Typha. no repeated restarts on calico-node or calico-kube-controllers. pods scheduled across expected failure domains. Then inspect node-level readiness:\nkubectl get nodes -o wide kubectl describe node worker-1 | grep -A8 Conditions If nodes are NotReady, do not assume Calico is the root cause. A bad CNI install can cause NotReady symptoms, but so can kubelet failure, compute pressure, disk pressure, certificate problems, or host firewall drift.\nVerify Pod Networking Deploy a temporary test workload:\nkubectl create namespace net-test kubectl -n net-test run client \\ --image=curlimages/curl:latest \\ --restart=Never \\ --command -- sleep 3600 kubectl -n net-test run server \\ --image=nginx:stable \\ --restart=Never Wait for both pods:\nkubectl -n net-test get pods -o wide Test basic service discovery and pod connectivity:\nkubectl -n net-test expose pod server --port 80 kubectl -n net-test exec client -- \\ curl -sS -I http://server.net-test.svc.cluster.local Then test cross-node scheduling by checking -o wide. If both test pods land on the same node, force one to another node or repeat until you test node-to-node pod traffic.\nCleanup:\nkubectl delete namespace net-test Verify NetworkPolicy Enforcement Calico is often installed because the platform wants real NetworkPolicy enforcement. Prove that policy works.\nCreate a namespace and a default-deny policy:\nkubectl create namespace policy-test kubectl -n policy-test apply -f - \u0026lt;\u0026lt;\u0026#39;EOF\u0026#39; apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny spec: podSelector: {} policyTypes: - Ingress - Egress EOF Then deploy a client/server pair and verify traffic is blocked until explicit allow rules are added. If NetworkPolicy objects apply but do not change traffic behavior, the cluster does not have enforcement working even if the CNI pods are running.\nCleanup:\nkubectl delete namespace policy-test Host Firewall And Port Checks Calico depends on node-to-node traffic. The exact ports depend on your encapsulation and routing mode.\nFor VXLAN, pay attention to UDP 4789. For Typha, pay attention to TCP 5473. For BGP designs, TCP 179 becomes part of the routing contract.\nUseful checks:\nsudo ss -lntup | grep -E \u0026#39;4789|5473|179\u0026#39; || true sudo ip link show | grep -E \u0026#39;vxlan.calico|cali\u0026#39; sudo ip route | grep -E \u0026#39;bird|cali|tunl|vxlan\u0026#39; || true If the host firewall is managed separately, test it intentionally. Do not let unmanaged ufw or ad-hoc firewall rules coexist with Kubernetes networking unless the platform has a clear policy for that.\nUpgrade And Change Control Treat Calico upgrades like platform changes, not application deploys.\nBefore changing Calico:\ncapture current Installation, IPPools, BGPPeers, and FelixConfiguration. confirm node readiness. confirm pod networking with a test namespace. identify maintenance or rollback path. confirm the GitOps controller will not revert or race the change. Useful snapshots:\nkubectl get installation default -o yaml \u0026gt; calico-installation.before.yaml kubectl get ippool -o yaml \u0026gt; calico-ippools.before.yaml kubectl get felixconfiguration -o yaml \u0026gt; calico-felix.before.yaml kubectl get nodes -o wide \u0026gt; nodes.before.txt After the change, repeat the same captures and compare.\nCommon Failure Modes Wrong node IP selected:\nCalico IPv4Address host portion does not match Kubernetes InternalIP Usually fixed by changing nodeAddressAutodetectionV4, then rolling Calico components according to the runbook.\nPods cannot reach pods on other nodes:\nsame-node traffic works, cross-node traffic fails Check encapsulation mode, host firewalls, MTU, and routing.\nNetworkPolicy has no effect:\npolicies exist, traffic still flows Confirm Calico is the active CNI and policy engine, not merely installed alongside another CNI.\nClean audit shows zero targets:\n0 mismatch targets That can mean every node is clean, not that the audit failed. Use all-node audit evidence before declaring success.\nAcceptance Criteria Calico is ready when these are true:\none system owns CNI installation. calico-node is running on expected nodes. Kubernetes node InternalIP matches Calico IPv4Address host portion. pod-to-pod traffic works across nodes. service DNS and ClusterIP access work. NetworkPolicy enforcement is proven with a deny/allow test. host firewall rules are compatible with the chosen Calico mode. Calico configuration is versioned and reviewed. operators know how to audit and remediate node IP autodetection drift. Related Field Notes Calico IP Audit Zero Targets Does Not Mean Zero Nodes — how to interpret mismatch-only Calico audit scripts. DNS Drift Detector Calico Overlay False Positives — why Calico overlay links should not be parsed as DNS search domains. Remediation Audits With NotReady Nodes And Calico Checks — how to keep NotReady nodes visible during remediation and Calico validation. RKE2 Calico Readiness Failures From Stale Port Owners — how to diagnose old node-local Calico or Typha processes holding readiness ports after RKE2 changes. RKE2 kube-proxy CrashLoopBackOff After Upgrade Due To UFW — reminder that host firewall drift can look like a Kubernetes dataplane failure. The installation is only the beginning. The durable operating model is explicit CNI ownership, clear node IP selection, repeatable validation, and audit output that tells operators what was actually checked.\n","permalink":"https://trinidadmarroquin.com/posts/installing-configuring-calico-rke2-ground-up/","section":"posts","summary":"Calico is not just a YAML install. It becomes part of the cluster\u0026rsquo;s trust model, routing behavior, NetworkPolicy enforcement, and incident response surface.\nThe important decisions happen before the first manifest is applied:\nwho owns CNI installation. which node interface Calico should use. which pod CIDR the cluster will allocate from. whether encapsulation is VXLAN, VXLAN cross-subnet, IP-in-IP, or BGP. how operators will prove the install is healthy after upgrades and node replacement. This guide is written for RKE2 clusters where Calico is installed and managed intentionally, not discovered later as a side effect of another installer.\n","tags":["calico","rke2","kubernetes","networking","cni","tigera"],"title":"Installing And Configuring Calico On RKE2 From The Ground Up"},{"categories":["field-notes"],"content":"Targeted Terraform applies are a sharp tool. They are not a normal workflow, but they are sometimes the safer option when a live cluster needs a narrow expansion and the full plan contains unrelated refactor drift.\nThis note covers the pattern for adding a small set of new monitor nodes to an existing RKE2 cluster while avoiding changes to existing etcd, control-plane, worker, and load balancer VMs.\nSituation The environment already had Terraform-managed vSphere VMs:\nlb1, lb2 etc1, etc2, etc3 mstr1, mstr2, mstr3 wrkr1, wrkr2, wrkr3 The intended change was only:\nmntr1, mntr2, mntr3 The module refactor also introduced NetBox resources for VMs, interfaces, IP addresses, primary IP assignment, and tags. That meant the first full plan was not safe to apply directly.\nDo Not Trust A Failed Or Partial Plan First issue: the NetBox provider prompted for missing credentials:\nprovider.netbox.api_token provider.netbox.server_url That is not a valid reviewed plan. The correct response is to stop, set credentials through the expected environment variables, and rerun the plan.\nexport NETBOX_SERVER_URL=\u0026#39;https://netbox.example.com\u0026#39; export NETBOX_API_TOKEN=\u0026#39;\u0026lt;redacted\u0026gt;\u0026#39; Do not paste long-lived tokens into shared logs or tickets. If a token lands in a transcript, rotate it.\nWhy The Full Plan Was Unsafe After credentials were available, the full plan still was not safe:\nterraform plan -detailed-exitcode The full plan wanted to create NetBox records for existing VMs and also detected unrelated drift on existing data disks. In this case, existing nodes had larger data disks than the refactored defaults, so Terraform interpreted the config as a disk shrink.\nThat is a stop sign.\nThe full plan was answering a different question:\nWhat would happen if this whole refactor were applied now? The operational question was narrower:\nCan we create only mntr1, mntr2, mntr3 and their required NetBox records? Build A Targeted Saved Plan Use -target only for the intended resource addresses and save the reviewed plan:\nterraform plan -out=targeted.tfplan \\ -target=\u0026#39;netbox_tag.cluster\u0026#39; \\ -target=\u0026#39;module.vm_group.netbox_virtual_machine.vm[\u0026#34;mntr1\u0026#34;]\u0026#39; \\ -target=\u0026#39;module.vm_group.netbox_interface.vm[\u0026#34;mntr1\u0026#34;]\u0026#39; \\ -target=\u0026#39;module.vm_group.netbox_ip_address.vm[\u0026#34;mntr1\u0026#34;]\u0026#39; \\ -target=\u0026#39;module.vm_group.netbox_primary_ip.vm[\u0026#34;mntr1\u0026#34;]\u0026#39; \\ -target=\u0026#39;module.vm_group.vsphere_virtual_machine.vm[\u0026#34;mntr1\u0026#34;]\u0026#39; \\ -target=\u0026#39;module.vm_group.netbox_virtual_machine.vm[\u0026#34;mntr2\u0026#34;]\u0026#39; \\ -target=\u0026#39;module.vm_group.netbox_interface.vm[\u0026#34;mntr2\u0026#34;]\u0026#39; \\ -target=\u0026#39;module.vm_group.netbox_ip_address.vm[\u0026#34;mntr2\u0026#34;]\u0026#39; \\ -target=\u0026#39;module.vm_group.netbox_primary_ip.vm[\u0026#34;mntr2\u0026#34;]\u0026#39; \\ -target=\u0026#39;module.vm_group.vsphere_virtual_machine.vm[\u0026#34;mntr2\u0026#34;]\u0026#39; \\ -target=\u0026#39;module.vm_group.netbox_virtual_machine.vm[\u0026#34;mntr3\u0026#34;]\u0026#39; \\ -target=\u0026#39;module.vm_group.netbox_interface.vm[\u0026#34;mntr3\u0026#34;]\u0026#39; \\ -target=\u0026#39;module.vm_group.netbox_ip_address.vm[\u0026#34;mntr3\u0026#34;]\u0026#39; \\ -target=\u0026#39;module.vm_group.netbox_primary_ip.vm[\u0026#34;mntr3\u0026#34;]\u0026#39; \\ -target=\u0026#39;module.vm_group.vsphere_virtual_machine.vm[\u0026#34;mntr3\u0026#34;]\u0026#39; The Terraform warning about -target is expected. The warning is useful because it reminds you that the result is incomplete by design. That is acceptable only when the operational goal is intentionally narrow and the plan is reviewed as such.\nProve The Saved Plan Scope Human-readable review:\nterraform show -no-color targeted.tfplan Machine-readable review:\nterraform show -json targeted.tfplan \\ | jq -r \u0026#39;.resource_changes[] | [(.change.actions | join(\u0026#34;,\u0026#34;)), .address] | @tsv\u0026#39; Expected shape:\ncreate netbox_tag.cluster create module.vm_group.netbox_virtual_machine.vm[\u0026#34;mntr1\u0026#34;] create module.vm_group.netbox_interface.vm[\u0026#34;mntr1\u0026#34;] create module.vm_group.netbox_ip_address.vm[\u0026#34;mntr1\u0026#34;] create module.vm_group.netbox_primary_ip.vm[\u0026#34;mntr1\u0026#34;] create module.vm_group.vsphere_virtual_machine.vm[\u0026#34;mntr1\u0026#34;] create module.vm_group.netbox_virtual_machine.vm[\u0026#34;mntr2\u0026#34;] create module.vm_group.netbox_interface.vm[\u0026#34;mntr2\u0026#34;] create module.vm_group.netbox_ip_address.vm[\u0026#34;mntr2\u0026#34;] create module.vm_group.netbox_primary_ip.vm[\u0026#34;mntr2\u0026#34;] create module.vm_group.vsphere_virtual_machine.vm[\u0026#34;mntr2\u0026#34;] create module.vm_group.netbox_virtual_machine.vm[\u0026#34;mntr3\u0026#34;] create module.vm_group.netbox_interface.vm[\u0026#34;mntr3\u0026#34;] create module.vm_group.netbox_ip_address.vm[\u0026#34;mntr3\u0026#34;] create module.vm_group.netbox_primary_ip.vm[\u0026#34;mntr3\u0026#34;] create module.vm_group.vsphere_virtual_machine.vm[\u0026#34;mntr3\u0026#34;] The important checks:\nevery action is create. there are no update, delete, or replace actions. no existing lb, etc, mstr, or wrkr addresses appear. the saved plan includes required dependency resources such as netbox_tag.cluster. If the JSON output contains any existing node address, stop and rebuild the plan.\nInclude Missing NetBox Tags As Managed Dependencies The monitor nodes used a cluster tag:\ncluster:cluster-a The apply failed when that tag did not exist in NetBox:\ncould not locate referenced tag \u0026#34;cluster:cluster-a\u0026#34; in netbox, no results The fix was to make the tag a first-class Terraform resource:\nresource \u0026#34;netbox_tag\u0026#34; \u0026#34;cluster\u0026#34; { name = \u0026#34;cluster:cluster-a\u0026#34; slug = \u0026#34;cluster-cluster-a\u0026#34; color_hex = \u0026#34;2196f3\u0026#34; description = \u0026#34;RKE2 Kubernetes cluster cluster-a - identifies all VMs that are members of this cluster\u0026#34; } Then reference it from VM tag lists:\nlocals { cluster_netbox_tag_name = netbox_tag.cluster.name } netbox_tags = [\u0026#34;monitor\u0026#34;, \u0026#34;rke2-node\u0026#34;, local.cluster_netbox_tag_name] That creates an explicit dependency so Terraform creates the tag before NetBox VM records need it.\nOne caveat: if someone creates the tag manually before Terraform applies, import it instead of creating a duplicate:\nterraform import netbox_tag.cluster \u0026lt;tag-id\u0026gt; Verify Hot Add Before Apply For live cluster nodes, CPU and memory hot-add settings matter. Check the saved plan, not just the source code:\nterraform show -json targeted.tfplan \\ | jq -r \u0026#39;.resource_changes[] | select(.type == \u0026#34;vsphere_virtual_machine\u0026#34;) | [.name, .index, (.change.after.cpu_hot_add_enabled|tostring), (.change.after.memory_hot_add_enabled|tostring)] | @tsv\u0026#39; Expected shape:\nvm mntr1 true true vm mntr2 true true vm mntr3 true true This is another reason to inspect the saved plan JSON. It proves what Terraform will send to vSphere for the exact resources being applied.\nApply Only The Reviewed Plan Apply the saved plan file, not a fresh plan:\nterraform apply \u0026#34;targeted.tfplan\u0026#34; Expected summary for this pattern:\n16 added, 0 changed, 0 destroyed That includes:\none NetBox cluster tag. NetBox VM/interface/IP/primary IP records for three monitor nodes. three vSphere virtual machines. Afterward, do not run a broad terraform apply until the full-plan drift is resolved. In this case, the full refactor still needed data disk sizing reconciliation for existing nodes.\nGuardrails terraform validate only proves configuration syntax and provider schema compatibility. terraform plan -detailed-exitcode is not useful if the provider prompts or errors. -target is acceptable for narrowly scoped live operations, but only with a saved plan and explicit address review. never use a filtering variable that removes existing keys from for_each unless you have proven it will not plan destroys. NetBox dependencies such as tags should be managed or imported before dependent VM records are created. apply the reviewed .tfplan, not a new plan. rotate credentials if they were pasted into a transcript. The operating rule is simple: when a live cluster needs three new nodes, the plan should prove exactly three new VMs and their dependencies. Anything else belongs in a separate refactor window.\n","permalink":"https://trinidadmarroquin.com/field-notes/terraform-targeted-plan-live-cluster-nodes/","section":"field-notes","summary":"Targeted Terraform applies are a sharp tool. They are not a normal workflow, but they are sometimes the safer option when a live cluster needs a narrow expansion and the full plan contains unrelated refactor drift.\nThis note covers the pattern for adding a small set of new monitor nodes to an existing RKE2 cluster while avoiding changes to existing etcd, control-plane, worker, and load balancer VMs.\nSituation The environment already had Terraform-managed vSphere VMs:\n","tags":["terraform","vsphere","netbox","kubernetes","rke2","operations","guardrails"],"title":"Terraform Targeted Plans For Live Cluster Node Expansion"},{"categories":["field-notes"],"content":"Service Level Indicators (SLIs) are easier to understand when they are tied to a working service. Abstract definitions are useful, but a small lab makes the tradeoffs visible: what counts as success, what counts as failure, how latency should be measured, and how trends can reveal degradation before a full incident.\nThis field note uses a companion lab in GitHub:\nhttps://github.com/trinidadgithub/IaC/tree/main/sli_app The lab runs a small Flask application, exposes Prometheus metrics, provisions Prometheus and Grafana with Terraform, and includes a basic SLI dashboard. It also introduces lightweight data science habits: percentiles, rolling windows, error-rate comparison, and avoiding misleading averages.\nWhat This Lab Demonstrates The application intentionally creates variable latency and occasional failures. A perfectly reliable demo app is not useful for learning reliability measurement.\nThe lab helps answer practical questions:\nWhat percentage of requests are successful? How slow is the service for typical users and tail users? Is the error rate stable, improving, or getting worse? Are 4xx responses client errors, service errors, or excluded from the SLI? Which metrics belong on a Grafana dashboard? Which measurements could support an SLO later? This is not a production architecture. It is a small observability lab for making SLI design concrete.\nRepository Layout The lab lives under sli_app in the IaC repository.\nsli_app/ ├── app.py ├── Dockerfile ├── requirements.txt ├── prometheus.yml ├── scripts/ │ └── generate_traffic.sh └── terraform/ ├── main.tf ├── outputs.tf ├── prometheus.yml └── grafana/ ├── dashboards/ │ ├── system-docker-monitoring.json │ └── sli-lab-dashboard.json └── provisioning/ └── datasources/ └── datasources.yml Terraform creates a local Docker network and runs:\nComponent Purpose Local URL Flask app Example service http://localhost:5000 App metrics Prometheus metrics endpoint http://localhost:8000 Prometheus Metrics storage/querying http://localhost:9090 Grafana Dashboards http://localhost:3000 cAdvisor Container metrics http://localhost:8080 Prometheus scrapes the Flask app through the shared Docker network at sli_app:8000.\nService Shape The Flask application exposes a few endpoints:\nEndpoint Purpose Behavior / Basic landing route Returns 200 /healthz Local health check Returns 200 /api/data Simulated read path Adds random latency and occasional 500 errors /api/submit Simulated write path Requires JSON input, adds random latency, occasional 500 errors The distinction between /api/data and /api/submit is useful because read and write paths often have different latency and reliability expectations.\nMetrics Exposed The application uses prometheus_client and exposes metrics from a separate port, 8000.\nMetric Type Labels Purpose sli_http_requests_total Counter endpoint, method, status Request volume, availability, and error-rate analysis sli_http_request_duration_seconds_bucket Histogram endpoint, method, status, le p95/p99 latency analysis The explicit labels make it possible to separate endpoint behavior and status classes. That matters when distinguishing expected 400 responses from service-side 500 failures.\nRunning The Lab Clone the repository and apply Terraform:\ngit clone https://github.com/trinidadgithub/IaC.git cd IaC/sli_app/terraform terraform init terraform validate terraform apply Grafana credentials:\nusername: admin password: admin01 Generate traffic from the sli_app directory:\nchmod +x scripts/generate_traffic.sh ./scripts/generate_traffic.sh For a longer run:\nITERATIONS=100 SLEEP_SECONDS=0 ./scripts/generate_traffic.sh The traffic script sends valid read/write requests and occasional invalid submit requests to produce expected 400 responses.\nDefining The SLIs SLIs should represent user-visible service behavior, not just container health.\nSLI Practical Definition Why It Matters Availability Ratio of successful requests to total eligible requests Measures whether users can complete requests Latency p95 or p99 request duration by endpoint Captures tail user experience better than averages Error Rate Ratio of server-side failures to total eligible requests Shows service-side reliability degradation Throughput Request rate by endpoint Helps interpret latency and error changes under load Container uptime is not the same as availability. A container can be running while every request fails.\nPromQL Examples Request Rate sum by (endpoint, method) ( rate(sli_http_requests_total[5m]) ) Availability This version treats 5xx responses as service failures:\n1 - ( sum(rate(sli_http_requests_total{status=~\u0026#34;5..\u0026#34;}[5m])) / sum(rate(sli_http_requests_total[5m])) ) If the SLI should exclude expected client errors from the denominator:\n1 - ( sum(rate(sli_http_requests_total{status=~\u0026#34;5..\u0026#34;}[5m])) / sum(rate(sli_http_requests_total{status!~\u0026#34;4..\u0026#34;}[5m])) ) The policy decision matters. A bad client request may not indicate service unreliability, but a broken API contract might.\nError Rate sum(rate(sli_http_requests_total{status=~\u0026#34;5..\u0026#34;}[5m])) / sum(rate(sli_http_requests_total[5m])) p95 Latency histogram_quantile( 0.95, sum by (le, endpoint, method) ( rate(sli_http_request_duration_seconds_bucket[5m]) ) ) p99 Latency histogram_quantile( 0.99, sum by (le, endpoint, method) ( rate(sli_http_request_duration_seconds_bucket[5m]) ) ) Practical Data Science Layer This lab becomes more valuable when metrics are treated as time-series data rather than isolated values.\nPercentiles Instead Of Averages Average latency hides tail pain. If most users receive a response quickly but a meaningful minority wait several seconds, the average may look acceptable while real users suffer.\nUse:\np50 for typical behavior p95 for most-user experience p99 for tail pain Average latency is still useful as a supporting signal, but it should not be the primary user-experience SLI.\nError-Rate Analysis Error rate should be reviewed by endpoint, method, and status class:\nsum by (endpoint, method, status) ( rate(sli_http_requests_total[5m]) ) This helps separate:\nexpected 400 responses from bad input service-side 500 responses endpoint-specific failure patterns read-path versus write-path behavior Define what counts against the SLI before reviewing the graph. Otherwise, teams are tempted to redefine reliability after the fact.\nTrend Awareness A single five-minute window can be noisy. Compare short and longer windows to detect direction.\nShort-window error rate:\nsum(rate(sli_http_requests_total{status=~\u0026#34;5..\u0026#34;}[5m])) / sum(rate(sli_http_requests_total[5m])) Longer-window error rate:\nsum(rate(sli_http_requests_total{status=~\u0026#34;5..\u0026#34;}[30m])) / sum(rate(sli_http_requests_total[30m])) If the five-minute rate is much higher than the thirty-minute rate, the service may be entering a failure window. If both are rising, the issue may be sustained.\nSimple Baseline Comparison PromQL offset can compare current behavior with a prior window:\nhistogram_quantile( 0.95, sum by (le) (rate(sli_http_request_duration_seconds_bucket[5m])) ) histogram_quantile( 0.95, sum by (le) (rate(sli_http_request_duration_seconds_bucket[5m] offset 1h)) ) This is not advanced forecasting. It is operational awareness: is the service behaving materially differently from a recent stable period?\nGrafana Dashboard The lab provisions an SLI Lab - Flask Service dashboard with panels for:\nRequest rate by endpoint Availability 5xx error rate p95 and p99 latency Status code breakdown Short-window versus long-window error-rate comparison The dashboard is intentionally small. It focuses on operational questions rather than every metric available.\nWhat This Teaches This lab reinforces several SRE lessons:\nSLIs must be user-centered. Availability is not container uptime. Averages hide tail latency. 4xx and 5xx responses should not be blindly grouped together. Short windows detect fast changes but are noisy. Longer windows show sustained behavior but can hide spikes. Dashboards should support decisions. SLOs should be based on carefully chosen SLIs, not whatever metric is easiest to graph. Gaps Addressed From The Original Notes The original notes had the right general direction but needed cleanup before becoming a repeatable lab.\nKey improvements:\nUpdated Python and Flask dependency guidance. Removed unrelated outbound network traffic from the app. Used explicit Prometheus counters and histograms. Added labels for endpoint, method, and status. Kept the existing Terraform-managed Docker lab pattern. Pinned Prometheus and Grafana image versions. Added a repeatable traffic generation script. Added an SLI-specific Grafana dashboard. Defined availability from request success ratio, not uptime. Added explicit 4xx versus 5xx discussion. Used histogram percentiles for latency instead of averages. Added trend and baseline comparison examples. Removed unrelated GitHub branch-protection and C-programming notes. Where To Take This Next This lab is intentionally small, but it creates enough signal to support more realistic reliability conversations. The next useful layer would be to turn the observed SLIs into a simple SLO, such as 99% successful eligible requests over 30 days, and then calculate how quickly different failure rates consume that error budget.\nAlerting can also be added once the SLO is clear. A high 5xx rate or sustained p95 latency spike is more meaningful when it is tied to a user-facing objective instead of an arbitrary threshold.\nThe same data can support a follow-up field note on percentiles. Comparing average latency with p95 and p99 latency is a practical way to show why averages often hide the user experience that reliability reviews are supposed to protect.\nReferences Google SRE Book — Service Level Objectives Google SRE Workbook — Implementing SLOs Prometheus Querying Basics Prometheus Histograms And Summaries Grafana Prometheus Data Source Flask Documentation ","permalink":"https://trinidadmarroquin.com/field-notes/sli-lab-flask-prometheus-grafana/","section":"field-notes","summary":"Service Level Indicators (SLIs) are easier to understand when they are tied to a working service. Abstract definitions are useful, but a small lab makes the tradeoffs visible: what counts as success, what counts as failure, how latency should be measured, and how trends can reveal degradation before a full incident.\nThis field note uses a companion lab in GitHub:\nhttps://github.com/trinidadgithub/IaC/tree/main/sli_app The lab runs a small Flask application, exposes Prometheus metrics, provisions Prometheus and Grafana with Terraform, and includes a basic SLI dashboard. It also introduces lightweight data science habits: percentiles, rolling windows, error-rate comparison, and avoiding misleading averages.\n","tags":["sre","sli","slo","prometheus","grafana","flask","docker","terraform","observability","data-science"],"title":"Building A Small SLI Lab With Flask, Prometheus, And Grafana"},{"categories":["DevOps Dirty Dozen"],"content":"Part 12 of the DevOps Dirty Dozen Series: Nomen est omen — the name is a sign.\nInsight: Reminds us that mere labeling without substance is meaningless.\nDevOps became popular because it named a real problem: development and operations were too often separated by incentives, handoffs, tools, and blame. The promise was not a new department, a new title, or a new toolchain. The promise was a better way of building and operating software together.\nThen the word started to drift.\nDevOps became a job title, a team name, a tool category, a transformation slide, a vendor label, and sometimes a substitute for the harder work it was supposed to represent. Organizations adopted the vocabulary without changing the system. They talked about DevOps while preserving the same silos, approvals, bottlenecks, and fear.\nThat is the DevOps-as-a-buzzword anti-pattern: adopting the name while avoiding the change.\nA new label does not transform an old system.\nThe Anatomy Of DevOps In Name Only DevOps as a buzzword is not always obvious. It often looks like progress from a distance.\nRenamed Teams: Operations becomes DevOps, but the work remains ticket intake, environment gatekeeping, and after-hours firefighting. The name changes. The operating model does not.\nTool-First Transformation: The organization buys CI/CD, observability, secrets management, or platform tooling and declares DevOps achieved. Collaboration, ownership, and feedback remain unchanged.\nOld Handoffs In New Language: Teams still throw work over the wall, but now the wall has a pipeline attached to it. Automation accelerates the handoff without improving shared responsibility.\nTransformation Theater: Leaders present roadmaps, maturity models, and slogans, but teams do not receive time, authority, or incentives to change how work actually flows.\nCargo-Cult Practices: Standups, pipelines, postmortems, dashboards, and platform teams are copied from successful organizations without understanding the constraints they were designed to solve.\nTransformation theater is easy to present and hard to operate.\nThe Cost Of The Buzzword The damage is not merely semantic. When DevOps becomes branding instead of practice, it creates real organizational risk.\nFalse Progress: Leaders believe transformation is underway because language and tooling changed. The deeper constraints remain hidden.\nCynicism Increases: Engineers can tell when vocabulary is disconnected from reality. When teams hear DevOps used to describe unchanged behavior, trust erodes.\nTooling Gets Blamed For Cultural Failure: A CI/CD platform cannot fix approval bottlenecks. An observability tool cannot create psychological safety. When tools fail to transform the system alone, the tool is blamed instead of the operating model.\nOld Incentives Persist: If teams are still rewarded for local optimization, avoiding risk, protecting turf, or closing tickets instead of improving flow, DevOps language will not change behavior.\nReal Improvement Gets Harder: Once the organization has already \u0026ldquo;done DevOps,\u0026rdquo; it becomes harder to argue for the work DevOps actually requires.\nBuzzwords become debt when they hide the work still unfinished.\nA Real-World Example: The DevOps Team That Became A Queue I have seen organizations create a DevOps team to accelerate delivery, only to turn that team into another centralized queue. Developers still opened tickets for environments, pipelines, secrets, deployments, and troubleshooting. Operations still owned production pain. Security still arrived late. The new DevOps team sat in the middle trying to satisfy everyone.\nAt first, it looked like progress. There was a team with the right name. There were pipelines. There were dashboards. There was a backlog full of platform work.\nBut the delivery system had not changed. The DevOps team became the new bottleneck because the organization had not distributed ownership or reduced handoffs. Developers were still not empowered to operate their services. Operations knowledge was still centralized. Incidents still escalated to specialists instead of service-owning teams.\nThe fix was not to rename the team again. The fix was to change the interaction model: platform capabilities instead of ticket fulfillment, paved roads instead of bespoke requests, service ownership instead of handoff, and shared operational standards instead of one team absorbing everyone else\u0026rsquo;s complexity.\nThe word DevOps was not wrong. The implementation was incomplete.\nWhat Real DevOps Requires DevOps is not a department or a product SKU. It is a set of operating principles that must be visible in daily work.\nShared Ownership: Teams that build services need meaningful responsibility for operating them. Operations expertise should be embedded, shared, and amplified — not isolated.\nFast Feedback: Monitoring, testing, user feedback, incident reviews, and deployment signals must flow back into engineering decisions.\nReliable Automation: Automation should reduce toil, enforce standards, and make safe behavior easier. It should not automate confusion or hide ownership gaps.\nPsychological Safety: Teams need to surface risk, failure, and uncertainty without blame. Without safety, feedback loops become reporting theater.\nSmall, Reversible Change: Delivery improves when changes are understandable, observable, and recoverable.\nContinuous Improvement: DevOps is never complete. The system must be reviewed, adapted, and improved as constraints change.\nThe label only matters if the pillars underneath are real.\nHow To Tell If DevOps Is Real Ask practical questions. Avoid slogans.\nQuestion Healthy Signal Warning Signal Who owns production behavior? Service teams share operational responsibility. Production is owned by a separate group after handoff. How does feedback reach engineering? Incidents, metrics, and user reports change priorities. Feedback is observed but rarely changes work. What happens after failure? Teams improve the system without blame. People are blamed or action items disappear. How are platforms consumed? Paved roads enable teams to self-serve safely. Every request becomes a ticket to a central team. How are tools evaluated? Tools are tied to outcomes and ownership. Tools are adopted because they look like DevOps. How does change happen? Small, observable, reversible steps. Large releases with unclear rollback. The answers matter more than the vocabulary.\nWhy The Buzzword Persists DevOps-as-a-buzzword persists because labels are easier than systems change.\nLabels Are Visible: A new team name, tool purchase, or transformation program is easy to announce. Changing incentives and ownership is harder to show.\nVendors Sell The Word: The market attaches DevOps to products because buyers recognize it. That does not mean the product creates the practice.\nLeaders Want A Finish Line: \u0026ldquo;We adopted DevOps\u0026rdquo; is more comfortable than \u0026ldquo;we are continuously improving how work flows through a complex sociotechnical system.\u0026rdquo;\nTeams Need Language For Pain: Sometimes teams use the word DevOps because they know the old model hurts, even if they do not yet know how to change it.\nPartial Improvements Are Mistaken For Transformation: A new pipeline or dashboard can be useful. It is just not the whole system.\nA slogan can point at a problem. It cannot solve the system.\nApplying The Scientific Method The cure for buzzword DevOps is evidence.\nAsk: What specific constraint are we trying to improve — lead time, reliability, toil, handoffs, recovery, quality, security, or ownership?\nHypothesize: \u0026ldquo;If we introduce self-service deployment with guardrails, lead time will decrease without increasing change failure rate.\u0026rdquo;\nTest: Apply the change to one service or team. Keep the scope small enough to learn.\nMeasure: Track outcomes, not adoption theater. Did flow improve? Did reliability hold? Did toil decrease? Did teams gain autonomy?\nIterate: Keep what works, revise what does not, and avoid declaring victory because the label is present.\nCarl Sagan\u0026rsquo;s Baloney Detection Kit When DevOps language appears, challenge it with grounded questions:\n\u0026ldquo;What changed in daily work?\u0026rdquo; — If the answer is only terminology, transformation has not happened.\n\u0026ldquo;Which handoff disappeared?\u0026rdquo; — DevOps should reduce harmful handoffs, not rename them.\n\u0026ldquo;What can teams now do safely without waiting?\u0026rdquo; — Real improvement increases safe autonomy.\n\u0026ldquo;Which metric improved without creating a worse tradeoff?\u0026rdquo; — Look for balanced outcomes, not vanity metrics.\n\u0026ldquo;What did we stop doing?\u0026rdquo; — If nothing was retired, the organization may have added DevOps on top of the old system.\nThe question is not whether we say DevOps. The question is what changed.\nMoving Forward Together This final anti-pattern brings the series full circle. Every item in the DevOps Dirty Dozen can hide behind the word DevOps: silos, tool overload, automating chaos, blame, over-reliance on tools, ignored feedback, hero culture, big bang deployments, metrics misuse, resistance to change, and neglected systems.\nThat is why the label is not enough.\nDevOps is useful only when it changes how work flows, how teams learn, how systems are operated, and how responsibility is shared. It is not a badge. It is not a maturity certificate. It is not a team name. It is a practice of continuously improving the sociotechnical system that delivers and operates software.\nNomen est omen — the name is a sign. But the sign is not the destination.\nIf your organization says it is doing DevOps, ask what became safer, faster, clearer, or more reliable because of it. The answer should be visible in the work, not just in the slide deck.\nReferences The DevOps Handbook by Gene Kim, Patrick Debois, John Willis, and Jez Humble Accelerate: The Science of Lean Software and DevOps by Nicole Forsgren, Jez Humble, and Gene Kim The Three Ways: Principles Underpinning DevOps Team Topologies by Matthew Skelton and Manuel Pais DORA: Generative organizational culture Google SRE Book — Introduction ","permalink":"https://trinidadmarroquin.com/posts/devops-dirty-dozen/devops-as-a-buzzword/","section":"posts","summary":"Part 12 of the DevOps Dirty Dozen Series: Nomen est omen — the name is a sign.\nInsight: Reminds us that mere labeling without substance is meaningless.\nDevOps became popular because it named a real problem: development and operations were too often separated by incentives, handoffs, tools, and blame. The promise was not a new department, a new title, or a new toolchain. The promise was a better way of building and operating software together.\n","tags":["devops","sre","culture","platform-engineering","continuous-improvement","systems-thinking"],"title":"Beyond The Label: The DevOps-As-A-Buzzword Anti-Pattern"},{"categories":["field-notes"],"content":"An SRE agent can authenticate to Rancher and still fail every useful workload call. Seeing clusters in Rancher does not automatically mean the account can read pods, logs, events, or namespace-scoped workloads in downstream clusters.\nThis pattern applies when an SRE automation identity reports symptoms like:\nnamespaces is forbidden: User \u0026#34;u-example\u0026#34; cannot list resource \u0026#34;namespaces\u0026#34; in API group \u0026#34;\u0026#34; at the cluster scope pods is forbidden: User \u0026#34;u-example\u0026#34; cannot list resource \u0026#34;pods\u0026#34; in namespace \u0026#34;app-namespace\u0026#34; The important distinction is scope. The identity may have management-plane visibility and read-only node visibility, but no downstream project or namespace RBAC.\nStart With The Actual Request Do not translate \u0026ldquo;can\u0026rsquo;t get logs\u0026rdquo; into cluster-wide view access by default.\nBreak the request down into required Kubernetes permissions:\nget/list/watch pods get pods/log get/list events get/list deployments get/list replicasets get/list services get/list endpoints Those can be granted with namespaced RoleBindings that reference a read-only ClusterRole.\nThe permission that changes the risk profile is namespace discovery:\nlist namespaces namespaces is cluster-scoped. Granting it requires cluster-scoped RBAC, usually a ClusterRoleBinding. If the agent can be configured with explicit namespaces, skip namespace listing and keep access namespaced.\nPrefer Group Bindings Over User IDs Bind an AD group or identity-provider group instead of a Rancher user ID when possible:\nsubjects: - kind: Group name: \u0026#34;activedirectory_group://CN=sre-investigate,OU=groups,DC=example,DC=com\u0026#34; Group bindings are easier to audit, rotate, and reuse across clusters. Direct user IDs such as u-example are harder to reason about later and couple the manifest to Rancher internals.\nUse An Existing Readonly ClusterRole Define the capability package once as a ClusterRole:\napiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: platform:readonly rules: - apiGroups: [\u0026#34;\u0026#34;] resources: [\u0026#34;pods\u0026#34;] verbs: [\u0026#34;get\u0026#34;, \u0026#34;list\u0026#34;, \u0026#34;watch\u0026#34;] - apiGroups: [\u0026#34;\u0026#34;] resources: [\u0026#34;pods/log\u0026#34;] verbs: [\u0026#34;get\u0026#34;] - apiGroups: [\u0026#34;\u0026#34;] resources: [\u0026#34;events\u0026#34;, \u0026#34;services\u0026#34;, \u0026#34;endpoints\u0026#34;] verbs: [\u0026#34;get\u0026#34;, \u0026#34;list\u0026#34;] - apiGroups: [\u0026#34;apps\u0026#34;] resources: [\u0026#34;deployments\u0026#34;, \u0026#34;replicasets\u0026#34;] verbs: [\u0026#34;get\u0026#34;, \u0026#34;list\u0026#34;] Then bind that role only inside the namespaces the SRE agent needs:\napiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: platform:sre-investigate-readonly namespace: shared-infra subjects: - kind: Group name: \u0026#34;activedirectory_group://CN=sre-investigate,OU=groups,DC=example,DC=com\u0026#34; roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: platform:readonly A namespaced RoleBinding can reference a ClusterRole. That does not make the binding cluster-wide. The binding scope remains the namespace where the RoleBinding lives.\nKeep Namespace Lists Honest Before generating RBAC for requested namespaces, verify they exist in the target clusters:\nkubectl --context cluster-a-prod get namespaces kubectl --context cluster-a-uat get namespaces If requested namespaces do not exist, remove them from the change instead of shipping dead RoleBindings. Dead bindings create noise and make reviewers wonder whether the access request was understood.\nExample decision:\nrequested: app-namespace, shared-infra, argocd-app-namespace exists: shared-infra ship: shared-infra only Render Through GitOps For a GitOps-managed RBAC repository, update the generator inventory rather than hand-editing generated overlays:\nbindings: - group: name: sre_investigate dn: \u0026#34;CN=sre-investigate,OU=groups,DC=example,DC=com\u0026#34; role: readonly scope: environment: prod clusters: all namespaces: - shared-infra Then regenerate:\npython3 rbac/hack/generate-overlays/generate.py The generated output should create namespaced RoleBindings in each intended overlay and should not introduce ClusterRoleBinding.\nValidate The Rendered Access Build the changed overlays:\nkubectl kustomize rbac/overlays/prod/cluster-a \u0026gt;/tmp/prod-cluster-a-rbac.yaml kubectl kustomize rbac/overlays/uat/cluster-a \u0026gt;/tmp/uat-cluster-a-rbac.yaml Confirm the expected subject and namespace appear:\ngrep -n \u0026#39;sre-investigate\\|namespace: shared-infra\u0026#39; /tmp/prod-cluster-a-rbac.yaml Confirm broad access was not introduced:\ngrep -R \u0026#39;kind: ClusterRoleBinding\u0026#39; rbac/overlays/prod rbac/overlays/uat || true grep -R \u0026#39;secrets\\|pods/exec\\|pods/portforward\u0026#39; rbac/overlays/prod rbac/overlays/uat || true After ArgoCD syncs, validate with impersonation or the target identity where possible:\nkubectl --context cluster-a-prod auth can-i get pods \\ --as=\u0026#39;u-example\u0026#39; \\ -n shared-infra kubectl --context cluster-a-prod auth can-i get pods/log \\ --as=\u0026#39;u-example\u0026#39; \\ -n shared-infra kubectl --context cluster-a-prod auth can-i list namespaces \\ --as=\u0026#39;u-example\u0026#39; Expected minimum-access result:\nget pods in shared-infra yes get pods/log in shared-infra yes list namespaces no That no is intentional if the SRE gateway is configured with explicit namespaces.\nRisk Rating For a customer-facing cluster, this change is usually moderate risk, not trivial:\n2/5 when limited to specific namespaces and read-only resources 3/5 or higher if expanded to all namespaces or cluster-scoped discovery 4/5+ if secrets, exec, port-forward, or write verbs are added The difference between 2/5 and 3/5 is usually scope creep. A few namespaced RoleBindings for logs and events are very different from fleet-wide view or namespace discovery across every cluster.\nReview Checklist The identity is an AD/group subject, not a brittle Rancher user ID. The rendered objects are RoleBinding, not ClusterRoleBinding. The bound role does not include secrets, exec, port-forward, impersonation, or write verbs. The namespaces exist in the target clusters. Generated files were produced by the generator, not hand-edited. kubectl kustomize succeeds for all affected overlays. The SRE agent operator understands whether namespaces must be configured explicitly. The safest fix is not \u0026ldquo;give the agent view everywhere.\u0026rdquo; It is to grant the smallest namespaced read path that lets the agent collect logs and events, then expand only when the next concrete requirement proves it is necessary.\n","permalink":"https://trinidadmarroquin.com/field-notes/sre-agent-kubernetes-log-access-rbac/","section":"field-notes","summary":"An SRE agent can authenticate to Rancher and still fail every useful workload call. Seeing clusters in Rancher does not automatically mean the account can read pods, logs, events, or namespace-scoped workloads in downstream clusters.\nThis pattern applies when an SRE automation identity reports symptoms like:\nnamespaces is forbidden: User \u0026#34;u-example\u0026#34; cannot list resource \u0026#34;namespaces\u0026#34; in API group \u0026#34;\u0026#34; at the cluster scope pods is forbidden: User \u0026#34;u-example\u0026#34; cannot list resource \u0026#34;pods\u0026#34; in namespace \u0026#34;app-namespace\u0026#34; The important distinction is scope. The identity may have management-plane visibility and read-only node visibility, but no downstream project or namespace RBAC.\n","tags":["kubernetes","rancher","rbac","argocd","kustomize","sre","security"],"title":"SRE Agent Kubernetes Log Access With Namespaced RBAC"},{"categories":["DevOps Dirty Dozen"],"content":"Part 11 of the DevOps Dirty Dozen Series: Semper vigilans — always vigilant.\nInsight: Encourages ongoing attention and adaptation.\nAutomation is not finished when it runs successfully once. Monitoring is not complete when the dashboard loads. A platform is not healthy because the last deployment worked. Systems change. Dependencies move. Credentials expire. Traffic shifts. Teams reorganize. What was safe last quarter can become fragile without anyone touching it directly.\nThe \u0026ldquo;set it and forget it\u0026rdquo; mentality treats operational systems as static. DevOps and SRE work assumes the opposite: systems are alive, and living systems require care.\nThis anti-pattern is dangerous because it rarely fails loudly at first. It decays quietly. The pipeline still runs, but with old assumptions. The dashboard still renders, but nobody knows whether the queries are meaningful. The alert still exists, but the owning team changed names two reorganizations ago. The automation is present, but the confidence is gone.\nA dashboard nobody reviews is not observability. It is decoration with a refresh interval.\nThe Anatomy Of Set It And Forget It This anti-pattern shows up anywhere a team deploys a mechanism and stops revisiting whether it still does the job.\nForgotten Automation: A pipeline, cron job, or script keeps running for years. Nobody remembers who owns it, what assumptions it makes, or what would break if it stopped.\nStale Monitoring: Dashboards remain online but no longer reflect current architecture. Metrics point at retired services, renamed namespaces, old labels, or incomplete data sources.\nUnreviewed Alerts: Alerts fire into channels nobody watches, page teams that no longer own the service, or trigger for conditions that stopped being actionable months ago.\nAging Credentials: Tokens, certificates, IAM roles, kubeconfigs, and service accounts continue to exist because removing them feels risky. Eventually they become security debt.\nStatic Runbooks: A runbook captures the system as it existed during one incident. The next time it is needed, commands fail because paths, names, APIs, or permissions changed.\nPolicy Drift: Tagging, encryption, backup, and retention rules are documented once but never verified continuously. Compliance becomes a snapshot instead of a practice.\nAutomation does not stay correct by existing. It stays correct by being maintained.\nThe Cost Of Neglect The cost of set-it-and-forget-it operations is usually paid during incidents, audits, migrations, and upgrades.\nFalse Confidence: Teams believe a control exists because the artifact exists. The backup job exists. The alert exists. The runbook exists. Nobody has tested whether it works.\nSlow Incident Response: During an outage, stale documentation and broken automation waste precious minutes. The team discovers decay at the worst possible time.\nSecurity Exposure: Old credentials and forgotten access paths accumulate. If nobody owns review and cleanup, the environment gets more permissive over time.\nUpgrade Fragility: Systems that have not been revisited become harder to upgrade. Unknown dependencies turn routine maintenance into archaeology.\nOperational Learning Stops: A system that is never reviewed cannot improve. Teams repeat old patterns because the operating model never gets challenged.\nOperational decay is quiet until the moment it becomes urgent.\nA Real-World Example: The Backup Nobody Restored A team had a nightly backup job for a critical internal application. The job ran successfully for years. Green status. Logs uploaded. Retention configured. Everyone assumed the system was protected.\nDuring a storage failure, the team attempted a restore and discovered three problems at once: the backup included application data but not a required configuration directory, the restore command referenced an old namespace, and the service account used for recovery no longer had the needed permissions.\nThe backup had not failed. The operational assumption had failed.\nThe job was created when the application was simpler. Over time, the architecture changed, the namespace changed, permissions changed, and the restore path was never rehearsed. The backup pipeline preserved the appearance of resilience while the actual recovery capability decayed.\nThe fix was not merely to repair the backup. The fix was to schedule restore tests, assign ownership, document recovery objectives, and track the backup system as production infrastructure.\nWhy Teams Fall Into The Trap Set-it-and-forget-it behavior usually comes from pressure, not laziness.\nNew Work Is More Visible Than Maintenance: Shipping a new pipeline gets attention. Reviewing an old one rarely does.\nSuccess Creates Complacency: If something has worked for a long time, teams assume it will keep working. Longevity becomes mistaken for reliability.\nOwnership Gets Blurry: People move teams. Services change hands. Vendors change APIs. The artifact remains, but the ownership model does not.\nMaintenance Has No Calendar: If review is not scheduled, it depends on memory. Memory is not an operational control.\nNobody Wants To Touch The Old Thing: Older automation often lacks tests, documentation, and clear rollback. Teams avoid it because changing it feels riskier than ignoring it.\nWhen ownership fades, maintenance becomes optional by accident.\nBuilding A Practice Of Vigilance The antidote is not constant anxiety. It is scheduled, boring, repeatable review.\nAssign Owners To Operational Artifacts: Pipelines, dashboards, alerts, secrets, runbooks, backup jobs, and policies need owners just like services do.\nReview On A Cadence: Quarterly is often enough for stable systems. High-risk systems may need monthly review. The key is that review is scheduled, not remembered.\nTest Recovery Paths: Backups, restores, failovers, rollback steps, and incident runbooks should be rehearsed before they are needed.\nExpire What Should Not Live Forever: Credentials, exceptions, temporary firewall rules, feature flags, and elevated permissions should have expiration dates.\nMeasure Artifact Usefulness: Which dashboards drove decisions? Which alerts resulted in action? Which runbooks were used successfully? Retire or repair the rest.\nTreat Automation As Code: Automation should have version control, review, tests where feasible, clear ownership, and deprecation paths.\nVigilance becomes sustainable when it is scheduled.\nApplying The Scientific Method Operational vigilance can be treated as a testable practice.\nAsk: Which operational artifacts do we rely on but rarely verify?\nHypothesize: \u0026ldquo;If we run quarterly restore tests, we will find recovery gaps before incidents.\u0026rdquo;\nTest: Pick one backup, one runbook, one dashboard, or one alert group and review it end to end.\nObserve: Did it work? Was ownership clear? Were permissions current? Did the artifact still match the system?\nIterate: Update, assign ownership, schedule the next review, or retire the artifact.\nCarl Sagan\u0026rsquo;s Baloney Detection Kit Set-it-and-forget-it thinking survives on assumptions. Challenge them:\n\u0026ldquo;The backup is green.\u0026rdquo; — Has anyone restored from it recently?\n\u0026ldquo;The dashboard exists.\u0026rdquo; — Who uses it, and what decision does it support?\n\u0026ldquo;The alert will page us.\u0026rdquo; — Does it route to the right team, and is the condition still actionable?\n\u0026ldquo;That credential is probably still needed.\u0026rdquo; — By whom? For what system? When was it last used?\n\u0026ldquo;The runbook worked last time.\u0026rdquo; — Has the system changed since then?\nThe loop is not complete until the system is verified again.\nMoving Forward Together The strongest operational systems are not the ones that never change. They are the ones that are revisited often enough to stay true.\nDevOps is full of useful artifacts: automation, dashboards, runbooks, policies, tests, alerts, backups, templates, modules, and deployment pipelines. None of them remain valuable by default. They remain valuable because teams keep them aligned with reality.\nSemper vigilans is not a call to panic. It is a call to responsible stewardship. Build the system, automate the work, document the path — then come back and prove it still works.\nWhat operational artifact is your team trusting because it existed last quarter? When was the last time someone verified it end to end?\nReferences The DevOps Handbook by Gene Kim, Patrick Debois, John Willis, and Jez Humble Google SRE Book — Monitoring Distributed Systems Google SRE Workbook — Practical Alerting Google SRE Workbook — Postmortem Culture NIST SP 800-57: Recommendation for Key Management AWS Well-Architected Framework — Operational Excellence Pillar ","permalink":"https://trinidadmarroquin.com/posts/devops-dirty-dozen/set-it-and-forget-it/","section":"posts","summary":"Part 11 of the DevOps Dirty Dozen Series: Semper vigilans — always vigilant.\nInsight: Encourages ongoing attention and adaptation.\nAutomation is not finished when it runs successfully once. Monitoring is not complete when the dashboard loads. A platform is not healthy because the last deployment worked. Systems change. Dependencies move. Credentials expire. Traffic shifts. Teams reorganize. What was safe last quarter can become fragile without anyone touching it directly.\nThe \u0026ldquo;set it and forget it\u0026rdquo; mentality treats operational systems as static. DevOps and SRE work assumes the opposite: systems are alive, and living systems require care.\n","tags":["devops","sre","automation","monitoring","maintenance","technical-debt","operations"],"title":"Always Vigilant: The DevOps Set It And Forget It Anti-Pattern"},{"categories":["field-notes"],"content":"Terraform modules are easier to consume when engineers can understand their inputs without reading every line of source code. While reviewing an AWS Terraform Kinesis module, I created an input summary table with six columns: Input, Type, Default Value, Required, Notes, and Recommendation. The goal was simple: make the module safer and faster to use by turning variable definitions into an operator-friendly interface.\nThe Problem This Solves Terraform modules often start clean and become harder to consume over time. Inputs are added for new capabilities, defaults change, conditional behavior grows, and security-sensitive options become mixed with ordinary configuration. The source code still contains the truth, but consuming the module requires reading variables.tf, resource blocks, locals, conditionals, and sometimes provider documentation.\nThat creates friction for several audiences:\nApplication engineers need to know which inputs are required and what safe defaults look like. Platform engineers need to explain intended usage without becoming the help desk for every module invocation. Security and governance reviewers need to understand encryption, IAM, tagging, alarms, and data retention behavior quickly. Technical managers need a readable view of what the module allows and where the risk is concentrated. The input summary table acts as a thin translation layer between Terraform implementation detail and practical module consumption.\nWhy Module Input Documentation Matters Terraform module inputs are the module\u0026rsquo;s public interface. If that interface is unclear, users make assumptions. Some assumptions are harmless. Others affect encryption, IAM access, retention, observability, and cost.\nGood input documentation helps answer questions before the module is used:\nWhich values are truly required? Which defaults are safe for production? Which inputs are conditionally required? Which settings affect security posture? Which settings affect cost? Which values should normally be left alone? Which values require coordination with another team? Without that context, engineers may copy a minimal example, accept defaults blindly, or cargo-cult values from another workspace. The module may still deploy successfully, but success at terraform apply time does not mean the configuration is operationally sound.\nThe Input Summary Format The table format I used was intentionally simple:\nColumn Purpose Input Variable name exposed by the module Type Terraform type, such as string, bool, list(string), map(string), or object Default Value Default from variables.tf, or - when no default exists Required Whether the caller must provide a value, including conditional requirements Notes Plain-language behavior and constraints Recommendation Practical guidance for normal usage A small excerpt from the Kinesis module summary looked like this:\nInput Type Default Value Required Notes Recommendation name string - Yes Unique stream name. Final resource name is built from {name_prefix}{name}. Use a descriptive name that identifies workload and purpose. stream_mode string \u0026quot;ON_DEMAND\u0026quot; No Valid values are PROVISIONED or ON_DEMAND. PROVISIONED requires shard_count. Use ON_DEMAND unless throughput is predictable and intentionally provisioned. shard_count number null Conditional Required only when stream_mode is PROVISIONED. Size according to expected throughput and AWS Kinesis guidance. encryption_type string \u0026quot;KMS\u0026quot; No Valid values are NONE or KMS. Keep KMS; avoid NONE unless there is an approved exception. firehose_encryption_enabled bool true No Controls server-side encryption for the Firehose S3 target. Keep enabled and clarify whether AWS-managed or customer-managed keys are required. This format is not complicated, which is the point. A useful module interface summary should be easy to scan, easy to review, and easy to paste into a design document or onboarding guide.\nHow The Format Improves Usability The table makes the module easier to consume because it separates caller-facing decisions from implementation details.\nInstead of asking engineers to infer behavior from Terraform expressions like this:\nkms_key_id = var.encryption_type == \u0026#34;NONE\u0026#34; ? null : var.kms_key_id The table states the operational behavior directly:\nkms_key_id is used only when encryption_type is KMS. Prefer a customer-managed key for production workloads. That shift matters. Most module consumers do not need to understand every internal expression on first pass. They need to know what to provide, what the default does, and what decision they are making by overriding it.\nSecurity And Governance Value The Kinesis module exposed several inputs with security or governance impact:\nencryption_type kms_key_id firehose_encryption_enabled policy_write_roles policy_read_roles tags retention_period alarm_sns_topics alarm_metric_thresholds Documenting these inputs in a summary table made review easier because the risky decisions were visible in one place.\nFor example, the module defaulted encryption_type to KMS, which is a good baseline. But kms_key_id defaulted to alias/aws/kinesis, while the description recommended a customer-managed key. That distinction matters for governance. The module is encrypted by default, but the default may not satisfy stricter key ownership, rotation, audit, or separation-of-duties requirements.\nLikewise, firehose_encryption_enabled defaulted to true, but the implementation used S3-managed encryption (AES256) when enabled. The input description said the default was encrypted with an AWS-owned CMK. That wording is worth tightening because AWS-owned, AWS-managed, customer-managed, and S3-managed encryption are not interchangeable governance terms.\nThe table made those gaps easier to discuss without turning the review into a line-by-line Terraform walkthrough.\nRisks And Gaps Discovered Documenting the inputs surfaced several areas worth improving.\nEncryption Wording The module differentiated Kinesis stream encryption and Firehose/S3 output encryption, but the descriptions could be clearer.\nFor Kinesis:\nencryption_type = \u0026quot;KMS\u0026quot; enables KMS encryption. kms_key_id = \u0026quot;alias/aws/kinesis\u0026quot; uses the AWS-managed Kinesis key by default. Production guidance may require a customer-managed key instead. For Firehose output:\nfirehose_encryption_enabled = true enabled S3 server-side encryption. The implementation used AES256, which is S3-managed encryption. If customer-managed KMS is required for S3, the module would need additional inputs for KMS key selection. Conditional Requirements shard_count is not always required. It becomes required when stream_mode is PROVISIONED. This is exactly the kind of condition that should be explicit in the Required column.\nWithout that note, callers may either provide unnecessary values for ON_DEMAND streams or omit required values for provisioned streams.\nDefaults Need Interpretation Defaults are not automatically recommendations. Some defaults are safe. Some are placeholders. Some are low-friction starting points but not production standards.\nExamples:\nretention_period = 24 is valid but may be too short for replay or incident recovery needs. tags = {} is technically valid but may violate tagging policy if required tags are not supplied elsewhere. alarm_metric_thresholds = {} means no alarms are provisioned unless thresholds are configured. The Recommendation column is useful because it explains whether the default should be accepted, reviewed, or overridden.\nAlarms Are Opt-In The Kinesis module included alarm_sns_topics and alarm_metric_thresholds, but alarms were not enabled unless thresholds were configured. That is a reasonable module pattern, but it needs to be obvious.\nOtherwise, a caller may provide SNS topics and assume alerting exists, when no alarm resources are created because thresholds are empty.\nTagging Depends On Merge Behavior The module merged caller-provided tags with tags from a shared variable module:\ntags = merge(var.tags, module.eits_vars.tags) That behavior should be documented because tag precedence matters. If shared tags override caller tags, that is one governance outcome. If caller tags override shared tags, that is another. The input summary should identify the merge pattern or link to the module\u0026rsquo;s tagging standard.\nValidation Is Uneven encryption_type had validation for NONE or KMS, which is good. Other inputs could benefit from similar validation or stronger typing guidance.\nPotential improvements:\nValidate stream_mode as ON_DEMAND or PROVISIONED. Validate retention_period between 24 and 8760. Validate stream_consumer_names count at or below the Kinesis limit. Validate IAM role ARN format for read/write role lists. Add validation or preconditions for shard_count when stream_mode = \u0026quot;PROVISIONED\u0026quot;. Reusable Pattern Across AWS Modules This documentation pattern can be reused across other AWS Terraform modules with very little change.\nGood candidates include:\nS3 bucket modules Lambda modules SQS and SNS modules DynamoDB modules IAM role modules VPC and subnet modules RDS modules CloudWatch alarm modules EKS add-on modules The same six columns work because most module consumption questions are consistent across AWS services:\nWhat do I have to provide? What happens if I provide nothing? Which defaults are safe? Which values affect security? Which values affect cost? Which settings are conditional? What does the platform team recommend? For security-sensitive modules, the summary can be extended with additional columns such as Security Impact, Cost Impact, or Policy Requirement. I would avoid adding those by default unless they are consistently maintained. A small table that stays current is more valuable than a large table nobody trusts.\nPractical Recommendations For future Terraform module documentation, I would standardize on this approach:\nGenerate the first pass from variables.tf. Capture variable name, type, default, and description.\nManually add operational context. The Notes and Recommendation columns should be written by someone who understands how the module is used.\nCall out conditional requirements explicitly. Avoid hiding important behavior in prose.\nSeparate default from recommendation. A default tells users what Terraform does. A recommendation tells users what they should normally do.\nReview security-sensitive wording carefully. Encryption, IAM, logging, retention, and tagging language should match governance terminology.\nAdd validation where documentation reveals ambiguity. If the table needs a long warning for an input, the module may need validation or preconditions.\nKeep the table close to the module. Store it in README.md or generated documentation so it changes with the module.\nReuse the pattern across modules. A consistent input summary format lowers cognitive load for every engineer consuming platform modules.\nNext Steps The Kinesis module input table was useful as documentation, but it also acted as a review tool. It exposed unclear encryption wording, conditional requirements, observability defaults, and validation opportunities.\nThe next step is to make this repeatable:\nAdd the input summary table to the module README. Review encryption language with security/governance stakeholders. Add validation for stream_mode, retention_period, and conditional shard_count behavior. Clarify alarm behavior when SNS topics are set but thresholds are empty. Document tag merge precedence. Apply the same summary format to the next AWS Terraform module. The broader lesson is simple: module documentation is not just user assistance. It is part of the module interface, and it is one of the cheapest ways to improve platform usability, security review, and operational consistency.\n","permalink":"https://trinidadmarroquin.com/field-notes/terraform-module-input-summary-pattern/","section":"field-notes","summary":"Terraform modules are easier to consume when engineers can understand their inputs without reading every line of source code. While reviewing an AWS Terraform Kinesis module, I created an input summary table with six columns: Input, Type, Default Value, Required, Notes, and Recommendation. The goal was simple: make the module safer and faster to use by turning variable definitions into an operator-friendly interface.\nThe Problem This Solves Terraform modules often start clean and become harder to consume over time. Inputs are added for new capabilities, defaults change, conditional behavior grows, and security-sensitive options become mixed with ordinary configuration. The source code still contains the truth, but consuming the module requires reading variables.tf, resource blocks, locals, conditionals, and sometimes provider documentation.\n","tags":["terraform","aws","documentation","platform-engineering","governance","modules"],"title":"Terraform Module Input Summary Pattern"},{"categories":["DevOps Dirty Dozen"],"content":"Part 10 of the DevOps Dirty Dozen Series: Cui resistitur, crescere videtur — what is resisted appears to grow.\nInsight: Suggests that resisting change only amplifies the challenges.\nChange resistance is easy to misread as stubbornness. Sometimes it is. More often, it is a signal that the organization has not made change safe enough, clear enough, or valuable enough for people to trust it.\nIn DevOps, resistance to change shows up as old deployment processes nobody wants to touch, manual approvals that outlived their original risk, legacy tools kept alive by fear, and infrastructure patterns everyone complains about but nobody replaces. The team knows the system is aging. The work to improve it keeps getting deferred.\nThe irony is that avoiding change does not preserve stability. It usually preserves fragility.\nStability is not the same as immobility. Some anchors look like process.\nThe Anatomy Of Resistance To Change Resistance to change appears in many forms. Some are cultural. Some are technical. Most are both.\nThe Sacred Manual Step: A manual approval, spreadsheet, or checklist remains mandatory because it once prevented a real problem. Nobody can explain whether it still reduces risk, but removing it feels dangerous.\nThe Untouchable Tool: A legacy system remains central because everyone is afraid to migrate away from it. It is brittle, poorly understood, and increasingly expensive, but it is familiar.\nThe Frozen Architecture: Teams work around old design decisions instead of revisiting them. New services inherit old constraints because changing the foundation would require coordination.\nThe Fear-Based No: New practices are rejected before they are evaluated. Infrastructure as code, GitOps, automated testing, progressive delivery, or better observability are dismissed because the first imagined failure ends the conversation.\nThe Permanent Pilot: A new approach is tested forever but never adopted. The pilot succeeds technically, but the organization never commits to changing the default path.\nA pilot that never changes the default path is just a sandbox with better press.\nThe Cost Of Standing Still Staying with familiar tools and processes may feel safe, but the cost compounds.\nOperational Risk Grows Quietly: Unsupported versions, manual runbooks, undocumented exceptions, and fragile integrations accumulate. The system becomes riskier while appearing unchanged.\nDelivery Slows Down: Teams spend more time navigating old process than delivering value. The friction becomes normal, so nobody counts it.\nTalent Burns Out Or Leaves: Engineers who repeatedly propose improvements and see them deferred eventually stop trying. Some disengage. Some leave. The organization loses the people most capable of helping it evolve.\nSecurity Posture Degrades: Old tooling may lack modern auditability, encryption defaults, policy enforcement, or identity integration. Avoiding migration becomes a security decision, even when nobody labels it that way.\nChange Eventually Arrives As Crisis: The migration that could have been planned over months becomes a forced upgrade after end-of-life, audit finding, outage, or vendor deprecation.\nDeferred change does not disappear. It compounds.\nA Real-World Example: The Approval That Became Theater I have seen release processes where every production deployment required a manual approval meeting. The process began after a serious incident, and at the time it made sense. The organization needed more visibility and control.\nYears later, the meeting remained. The systems had changed. The deployment tooling had improved. Automated tests existed. Rollback paths were clearer. But the approval meeting persisted because nobody wanted to be the person who removed a control associated with safety.\nThe meeting no longer caught meaningful risk. Engineers summarized changes already reviewed in pull requests. Approvers nodded through systems they did not operate. Emergency fixes bypassed the meeting anyway. The process created delay without real control.\nThe breakthrough came when the team stopped arguing about whether approval was good or bad and started asking what risk the approval was supposed to reduce. Some changes still needed review: database migrations, network changes, permission expansions, irreversible operations. Many routine deployments did not.\nThe new process did not remove control. It moved control closer to the risk: automated checks for low-risk changes, explicit review for high-risk changes, and post-deployment verification for everything.\nResistance softened when the change was framed as better risk management, not less governance.\nWhy Teams Resist Change Teams resist change for reasons that often make sense locally.\nPast Change Hurt: If previous migrations caused outages, people remember. Skepticism is not irrational when experience taught the team that change is painful.\nThe Current System Is Understood: A flawed system with known failure modes can feel safer than an improved system with unknown ones.\nIncentives Favor Avoidance: If teams are punished for failed change but not rewarded for reducing future risk, the safest career move is to leave things alone.\nMigration Work Is Invisible: Leadership sees feature delivery. It may not see the operational debt paid down by modernization work until the avoided incident never happens.\nNo One Owns The Transition: Everyone agrees the future state is better, but nobody owns the bridge from current state to future state. Without ownership, change remains an idea.\nPeople rarely resist the destination. They resist the unsafe path to get there.\nMaking Change Safer The answer is not to force change harder. The answer is to make change smaller, clearer, and more reversible.\nDefine The Risk Being Reduced: Every change should explain what risk, toil, cost, or constraint it addresses. Change for its own sake creates fatigue.\nStart With Reversible Moves: Prefer changes that can be rolled back, toggled off, or run in parallel. Reversibility lowers fear.\nUse Incremental Migration Paths: Replace big migrations with expand-and-contract patterns, compatibility windows, and side-by-side operation where possible.\nCreate A Retirement Plan: New tools and processes should include what they replace. Otherwise the organization adds change without removing complexity.\nMeasure Friction: Track lead time, approval wait time, manual handoffs, failed changes, and repeated incidents. Data helps show when the old way is no longer safe.\nProtect Time For Modernization: If improvement work only happens after feature work, it never happens. Reserve explicit capacity for reducing operational debt.\nThe safest change is the one with checkpoints, evidence, and a way back.\nApplying The Scientific Method Change resistance decreases when improvement is treated as an experiment instead of a mandate.\nAsk: What problem are we trying to solve, and what evidence shows it matters?\nHypothesize: \u0026ldquo;If we replace the manual deployment meeting with automated policy checks for low-risk changes, lead time will decrease without increasing change failure rate.\u0026rdquo;\nTest: Apply the new approach to one service, one team, or one class of low-risk change.\nObserve: Measure lead time, failed changes, rollback frequency, and team confidence.\nIterate: Expand, adjust, or stop based on evidence. A failed experiment is still progress if it teaches the team what risk remains.\nCarl Sagan\u0026rsquo;s Baloney Detection Kit When resistance appears, challenge both sides of the argument.\n\u0026ldquo;We have always done it this way.\u0026rdquo; — Why was the process created? Does that reason still exist?\n\u0026ldquo;The new way is better.\u0026rdquo; — Better by what evidence? Faster? Safer? Cheaper? Easier to audit?\n\u0026ldquo;Changing it is too risky.\u0026rdquo; — Compared to what? What is the risk of leaving it unchanged for another year?\n\u0026ldquo;We will migrate later.\u0026rdquo; — When exactly? Who owns it? What trigger moves it from intention to work?\n\u0026ldquo;The pilot worked.\u0026rdquo; — Did it change the default path, or did it remain an exception?\nThe debate should not be old versus new. It should be evidence versus assumption.\nMoving Forward Together Resistance to change is not always the enemy. Sometimes it is the organization asking for proof, safety, and a better migration path. Listen to that signal. Then do the work to make change trustworthy.\nThe real anti-pattern is not caution. Caution is healthy. The anti-pattern is allowing caution to harden into permanent inertia while risk grows underneath.\nHealthy DevOps culture does not chase every new tool or trend. It also does not cling to yesterday because yesterday is familiar. It creates a disciplined path for evaluating change, reducing risk, and evolving before crisis forces the issue.\nWhat process, tool, or architecture is your team defending because it is safe — and what evidence would prove whether it actually still is?\nReferences The DevOps Handbook by Gene Kim, Patrick Debois, John Willis, and Jez Humble Accelerate: The Science of Lean Software and DevOps by Nicole Forsgren, Jez Humble, and Gene Kim Team Topologies by Matthew Skelton and Manuel Pais Google SRE Workbook — Non-Abstract Large System Design DORA: Generative organizational culture Martin Fowler: Strangler Fig Application ","permalink":"https://trinidadmarroquin.com/posts/devops-dirty-dozen/resistance-to-change/","section":"posts","summary":"Part 10 of the DevOps Dirty Dozen Series: Cui resistitur, crescere videtur — what is resisted appears to grow.\nInsight: Suggests that resisting change only amplifies the challenges.\nChange resistance is easy to misread as stubbornness. Sometimes it is. More often, it is a signal that the organization has not made change safe enough, clear enough, or valuable enough for people to trust it.\nIn DevOps, resistance to change shows up as old deployment processes nobody wants to touch, manual approvals that outlived their original risk, legacy tools kept alive by fear, and infrastructure patterns everyone complains about but nobody replaces. The team knows the system is aging. The work to improve it keeps getting deferred.\n","tags":["devops","sre","change-management","legacy-systems","platform-engineering","culture"],"title":"The Drag Of Yesterday: The DevOps Resistance To Change Anti-Pattern"},{"categories":["field-notes"],"content":"Concourse can be healthy inside the cluster while still failing from the operator\u0026rsquo;s browser. The common gap is not the web pod. It is the handoff between DNS, ingress controller placement, certificate material, and Concourse\u0026rsquo;s own external URL.\nUse this checklist when moving Concourse from a temporary URL or NodePort to a real hostname such as concourse.example.com.\nConfirm The Ingress Target Start by finding where ingress actually lands. Do not assume every worker should be in DNS until the ingress controller placement confirms it.\nkubectl --context cluster-a-prod get daemonset -A | grep ingress kubectl --context cluster-a-prod get pods -n kube-system \\ -l app.kubernetes.io/name=rke2-ingress-nginx -o wide kubectl --context cluster-a-prod get ingress -n concourse The important outputs are:\nthe ingress class used by the Concourse ingress. the node names running ingress controller pods. the IP addresses published in ingress status. whether control-plane or etcd nodes are unintentionally serving ingress. On RKE2, the default ingress controller can run as a DaemonSet on every Linux node unless restricted. That may include control-plane or etcd nodes. Decide whether that is acceptable before placing all published addresses into DNS.\nRegister DNS To Ingress Nodes For a simple internal round-robin setup, create multiple A records for the same hostname:\nconcourse.example.com. 300 IN A 192.0.2.10 concourse.example.com. 300 IN A 192.0.2.11 concourse.example.com. 300 IN A 192.0.2.12 If using dynamic DNS updates, build the transaction explicitly so the intended record is replaced as one unit:\nnsupdate \u0026lt;\u0026lt;EOF server 192.0.2.53 zone example.com update delete concourse.example.com A update add concourse.example.com 300 A 192.0.2.10 update add concourse.example.com 300 A 192.0.2.11 update add concourse.example.com 300 A 192.0.2.12 send EOF Verify against the authoritative resolver, not only the workstation cache:\nnslookup concourse.example.com 192.0.2.53 dig @192.0.2.53 concourse.example.com A +short Create The TLS Secret Kubernetes ingress expects a TLS secret with the server certificate and private key. If the certificate was issued by an internal CA, put the server certificate first, then intermediates, then the root if your environment requires it.\ncat concourse.example.com.crt intermediate-ca.crt root-ca.crt \u0026gt; fullchain.pem openssl verify \\ -CAfile root-ca.crt \\ -untrusted intermediate-ca.crt \\ concourse.example.com.crt kubectl --context cluster-a-prod -n concourse create secret tls concourse-web-tls \\ --cert=fullchain.pem \\ --key=concourse.example.com.key Do not commit private keys, generated full chains, or downloaded PEM bundles unless the repository is explicitly designed to hold public certificate material. The durable configuration should reference the secret name, not contain the secret contents.\nUpdate Helm Values There are two separate changes:\ningress TLS tells Kubernetes which secret to serve for the hostname. Concourse externalUrl tells Concourse what URL to advertise and use during login flows. Example values:\nconcourse: web: externalUrl: \u0026#34;https://concourse.example.com\u0026#34; web: ingress: enabled: true ingressClassName: nginx hosts: - concourse.example.com tls: - hosts: - concourse.example.com secretName: concourse-web-tls Apply the values through the normal Helm path:\nhelm --kube-context cluster-a-prod upgrade --install concourse concourse/concourse \\ --namespace concourse \\ --values concourse-values.yaml \\ --version 20.2.4 \\ --wait \\ --timeout 10m Verify The Cutover Check Kubernetes state first:\nkubectl --context cluster-a-prod get ingress -n concourse -o yaml \\ | grep -A6 \u0026#39;host:\\|tls:\\|secretName:\u0026#39; kubectl --context cluster-a-prod exec -n concourse deployment/concourse-web -- \\ env | grep CONCOURSE_EXTERNAL_URL Then test each ingress node directly with curl --resolve. This separates DNS round-robin from backend health:\nfor ip in 192.0.2.10 192.0.2.11 192.0.2.12; do curl -sS -o /dev/null \\ --resolve concourse.example.com:443:${ip} \\ -w \u0026#34;${ip} http=%{http_code} connect=%{time_connect} tls=%{time_appconnect} total=%{time_total}\\n\u0026#34; \\ https://concourse.example.com done Finally, check the Concourse API through the public hostname:\ncurl -sS https://concourse.example.com/api/v1/info Expected signals:\nHTTPS returns 200 for the web UI. /api/v1/info returns Concourse version metadata. CONCOURSE_EXTERNAL_URL is https://concourse.example.com. each DNS target completes TCP connect and TLS handshake. Failure Patterns DNS resolves but browser gets a timeout. DNS may point to nodes that are not running ingress or cannot receive traffic on 443. TLS works on one IP but not another. Ingress controller placement or host firewall state differs by node. Login redirects to HTTP. Ingress TLS was added, but Concourse externalUrl still uses http://. Certificate warning remains. The secret may contain only the leaf certificate instead of the full chain, or the wrong secret name is referenced by ingress. Helm notes still show HTTP. Do not trust chart notes alone. Verify the rendered ingress and the running CONCOURSE_EXTERNAL_URL environment variable. The clean cutover is four independent confirmations: DNS points to ingress nodes, ingress references the TLS secret, Concourse advertises the HTTPS URL, and every published IP completes an HTTPS request.\n","permalink":"https://trinidadmarroquin.com/field-notes/concourse-ingress-dns-tls-cutover/","section":"field-notes","summary":"Concourse can be healthy inside the cluster while still failing from the operator\u0026rsquo;s browser. The common gap is not the web pod. It is the handoff between DNS, ingress controller placement, certificate material, and Concourse\u0026rsquo;s own external URL.\nUse this checklist when moving Concourse from a temporary URL or NodePort to a real hostname such as concourse.example.com.\nConfirm The Ingress Target Start by finding where ingress actually lands. Do not assume every worker should be in DNS until the ingress controller placement confirms it.\n","tags":["concourse","kubernetes","ingress","tls","dns","helm"],"title":"Concourse Ingress DNS And TLS Cutover"},{"categories":["DevOps Dirty Dozen"],"content":"Part 9 of the DevOps Dirty Dozen Series: Falsus in uno, falsus in omnibus — false in one thing, false in all.\nInsight: Warns against misleading metrics that distort the truth.\nMetrics are supposed to help teams see reality more clearly. Used well, they reveal bottlenecks, validate improvement, and help engineering organizations make better decisions. Used poorly, they become theater.\nThe danger is not measurement itself. The danger is measuring what is easy, rewarding what is visible, and mistaking activity for progress. A dashboard can be full and still tell the wrong story. A team can improve every reported metric while the system becomes harder to operate.\nMetrics misuse is what happens when numbers stop informing judgment and start replacing it.\nA dashboard can be green and still be lying to you.\nThe Anatomy Of Metrics Misuse Bad metrics usually do not look bad at first. They are tidy, countable, and easy to report upward. That is what makes them dangerous.\nCommon patterns include:\nActivity Metrics Disguised As Outcome Metrics: Counting commits, tickets closed, story points completed, or pipelines executed can describe motion. It does not prove customer value, reliability, or delivery improvement.\nVanity Dashboards: Charts are built because the data is available, not because the team knows what decision the chart should support. The dashboard becomes decoration.\nMetric Gaming: Once a metric becomes a performance target, people adapt to the metric. If teams are rewarded for closing tickets, tickets get smaller. If they are rewarded for deployment count, deployments may increase without improving outcomes.\nAverages That Hide Pain: Average latency, average resolution time, and average failure rate often hide the worst user experience. Averages are comfortable. Tail behavior is where users suffer.\nSingle-Metric Management: Leadership picks one metric and optimizes aggressively. Deployment frequency goes up while change failure rate worsens. MTTR improves because teams roll forward without fixing root causes. The system optimizes locally and degrades globally.\nWhen the measure becomes the target, behavior bends around the number.\nThe Cost Of Bad Metrics Misused metrics do more than waste reporting time. They actively reshape behavior.\nTeams Optimize For Appearance: If the metric rewards visible activity, teams produce visible activity. The organization gets more motion and less learning.\nReal Risks Stay Hidden: A team can hit sprint targets while reliability declines. A platform can show high deployment frequency while rollback paths are broken. Bad metrics create false confidence.\nPsychological Safety Erodes: When metrics are used as weapons, engineers hide bad news. Incidents are softened. Estimates are padded. The data becomes less truthful because the organization made truth unsafe.\nDecision Quality Declines: Leaders make resource decisions based on distorted signals. The team with the best reporting looks healthiest, even if the most important work is happening elsewhere.\nContinuous Improvement Stalls: Improvement requires honest feedback. If the measurement system rewards performance theater, the feedback loop is corrupted.\nBad metrics do not just report distortion. They create it.\nA Real-World Example: The Ticket Closure Trap One operations team I worked with was measured heavily on weekly ticket closure count. On paper, the numbers improved. Backlog volume went down. Reports looked better. Leadership saw momentum.\nBut the team had changed its behavior to satisfy the metric. Large recurring problems were split into small tickets. Tickets were closed when a workaround was applied, not when the underlying issue was resolved. Follow-up work moved into chat threads where it no longer counted against backlog. The metric improved while the operational reality got worse.\nThe signal eventually surfaced during incident review. Several outages traced back to the same unresolved automation failure. The tickets had all been closed. The problem had never been fixed.\nThe metric was not useless. Ticket closure can be a helpful operational signal. The misuse was treating closure count as a proxy for reliability. The better question was not \u0026ldquo;How many tickets did we close?\u0026rdquo; It was \u0026ldquo;How many recurring causes did we eliminate?\u0026rdquo;\nMeaningful Metrics Versus Vanity Metrics Useful metrics connect to outcomes and decisions. Vanity metrics make teams feel productive without forcing a useful choice.\nVanity Metric Better Question Number of commits Did the change improve customer or operational outcomes? Story points completed Did lead time improve without increasing failure rate? Tickets closed Did recurring causes decrease? Number of dashboards Which dashboard changed a decision? Alert count Which alerts are actionable and reduce time to detect? Test count What failure modes are covered and what escaped? Deployment count alone Are deployments safe, reversible, and low-risk? This does not mean activity metrics have no value. They can be useful diagnostic signals. The mistake is promoting them into success measures without context.\nThe question is not whether the team is busy. The question is whether the system is improving.\nChoosing Metrics That Improve Behavior Metrics should help teams improve the system, not perform for the dashboard.\nStart With A Decision: Before adding a metric, ask: \u0026ldquo;What decision will this help us make?\u0026rdquo; If there is no decision, there may not be a reason to measure it.\nPair Speed With Safety: Deployment frequency without change failure rate is incomplete. Lead time without quality is incomplete. MTTR without recurrence is incomplete. Metrics need balancing pairs.\nMeasure Trends, Not Just Targets: A single number can be gamed or misunderstood. Trends reveal direction. Direction matters more than the snapshot.\nUse Percentiles For User Experience: Averages hide pain. For latency and reliability, look at p95, p99, error budgets, and user-impacting failure modes.\nMake Metrics Team-Owned: Metrics should be tools for the team closest to the work. When metrics are imposed only for executive reporting, they drift toward performance theater.\nReview Metrics For Harm: Ask whether a metric encourages bad behavior. If it does, change it. A metric that damages judgment is worse than no metric.\nGood metrics balance speed with safety, activity with outcome, and delivery with learning.\nApplying The Scientific Method Metrics are hypotheses about what matters. Treat them that way.\nAsk: What outcome are we trying to improve?\nHypothesize: \u0026ldquo;If we reduce lead time while keeping change failure rate stable, users will receive value sooner without reliability loss.\u0026rdquo;\nMeasure: Select metrics that test the hypothesis, not metrics that merely describe activity.\nObserve: Watch for intended and unintended behavior changes. Are teams improving the system or gaming the score?\nRevise: Retire metrics that no longer help. A metric can be useful for a season and harmful later.\nCarl Sagan\u0026rsquo;s Baloney Detection Kit When reviewing a metric, challenge it directly:\n\u0026ldquo;What does this actually prove?\u0026rdquo; — Does the metric connect to user value, reliability, flow, or learning?\n\u0026ldquo;What behavior does this reward?\u0026rdquo; — If people optimize for the number, will the system improve or merely look better?\n\u0026ldquo;What does this hide?\u0026rdquo; — Are averages hiding outliers? Are success rates hiding degraded users? Are closed tickets hiding unresolved causes?\n\u0026ldquo;Can this be gamed?\u0026rdquo; — If yes, assume it eventually will be, even unintentionally.\n\u0026ldquo;What metric balances this one?\u0026rdquo; — Speed needs safety. Output needs quality. Recovery time needs recurrence rate.\nA metric should reflect reality, not flatter the organization.\nMoving Forward Together Metrics are powerful because they focus attention. That is also why they are dangerous. The wrong metric does not simply mislead leadership; it teaches teams what the organization actually values.\nIf the organization values closed tickets, it will get closed tickets. If it values deployment count, it will get deployments. If it values learning, safety, and customer outcomes, the metrics should make those things visible.\nThe goal is not to measure everything. The goal is to measure honestly enough that teams can improve without fear and leaders can make decisions without illusion.\nWhat number does your organization celebrate that might be hiding the real problem? What would happen if that metric disappeared tomorrow?\nReferences Accelerate: The Science of Lean Software and DevOps by Nicole Forsgren, Jez Humble, and Gene Kim The DevOps Handbook by Gene Kim, Patrick Debois, John Willis, and Jez Humble Google SRE Book — Service Level Objectives Google SRE Workbook — Implementing SLOs Goodhart\u0026rsquo;s Law DORA Research Program DORA: Generative organizational culture ","permalink":"https://trinidadmarroquin.com/posts/devops-dirty-dozen/metrics-misuse/","section":"posts","summary":"Part 9 of the DevOps Dirty Dozen Series: Falsus in uno, falsus in omnibus — false in one thing, false in all.\nInsight: Warns against misleading metrics that distort the truth.\nMetrics are supposed to help teams see reality more clearly. Used well, they reveal bottlenecks, validate improvement, and help engineering organizations make better decisions. Used poorly, they become theater.\nThe danger is not measurement itself. The danger is measuring what is easy, rewarding what is visible, and mistaking activity for progress. A dashboard can be full and still tell the wrong story. A team can improve every reported metric while the system becomes harder to operate.\n","tags":["devops","sre","metrics","observability","dora","goodharts-law","reliability"],"title":"Measuring The Wrong Things: The DevOps Metrics Misuse Anti-Pattern"},{"categories":["field-notes"],"content":"Node remediation scripts often fail in boring ways: wrong inventory group, hidden Vault dependency, or one unreachable node that makes the final audit look stuck. Those failures matter because they can turn a clean remediation into a false incident, or worse, hide the one node that still needs attention.\nUse this workflow when cleaning Kubernetes node drift across a whole RKE2 cluster and validating Calico state at the same time.\nStart With A Per-Cluster Inventory The remediation target should be the cluster group, not a hand-written list of hosts. Generate inventory from the kubeconfig so the source of truth is Kubernetes node state:\n./generate-inventory.sh \\ --kubeconfig ~/.kube/config \\ --environment cluster-a-prod \\ --output inventory/cluster-a-prod.yaml The generated inventory should preserve enough runtime facts to help operators interpret remediation output:\nall: children: cluster_a_prod: rke2_servers: children: etcd: mstr: rke2_agents: children: wrkr: cluster_a_prod: children: wrkr: hosts: worker-1: ansible_host: 192.0.2.10 rke2_node_ready: true worker-2: ansible_host: 192.0.2.11 rke2_node_ready: false rke2_node_ready is not a fix by itself. It is evidence. It tells the person reading Ansible output that a missing or slow host may already be unhealthy from Kubernetes\u0026rsquo; point of view.\nVerify The Target Group A common failure is targeting the Kubernetes context name when Ansible inventory normalized the group name.\nBad signal:\n[WARNING]: Could not match supplied host pattern, ignoring: cluster-a-prod [WARNING]: No hosts matched, nothing to do Fix the wrapper or command so it resolves the inventory group name before running Ansible:\nansible-inventory -i inventory/cluster-a-prod.yaml --graph Then run the remediation against the real group:\nansible cluster_a_prod \\ -i inventory/cluster-a-prod.yaml \\ -u ubuntu \\ -b \\ -m ping Do not proceed until the target count matches expectations.\nAvoid Local Vault Lookup Failures If ad-hoc Ansible uses group_vars/all.yaml, local credential lookups can fail before any remote host is touched:\nThe lookup plugin \u0026#39;community.hashi_vault.vault_kv2_get\u0026#39; failed to load Failed to import the required Python library (hvac) For emergency audits or remediation, pass explicit operator credentials or an override file rather than letting local workstation Vault dependencies decide whether the cluster can be checked:\nansible cluster_a_prod \\ -i inventory/cluster-a-prod.yaml \\ -u ubuntu \\ -b \\ -e ansible_user=ubuntu \\ -m shell \\ -a \u0026#39;hostname -s\u0026#39; This does not replace proper secret management. It keeps a local tooling gap from being misread as node remediation failure.\nInterpret A Hung Final Audit After DNS or resolver cleanup, a final audit that appears stuck may not mean the script is broken. It may mean one host is unreachable while the others completed.\nUseful checks:\nps -ef | grep \u0026#39;[a]nsible\u0026#39; kubectl --context cluster-a-prod get nodes -o wide If interrupted output says something like this, treat it as partial success:\nWARNING: audited 10 of 11 hosts. Check warnings/failures above. Then compare with Kubernetes node state:\nNAME STATUS ROLES INTERNAL-IP worker-1 Ready worker 192.0.2.10 worker-2 NotReady worker 192.0.2.11 worker-3 Ready worker 192.0.2.12 At that point the remediation result is not simply pass or fail. It is:\n10 nodes remediated and audited. 1 node needs separate NodeNotReady triage. the final cluster declaration must wait until the missing node is recovered or explicitly excluded. Separate NodeNotReady From Remediation Drift For the NotReady node, switch from cluster-wide remediation to node triage:\nkubectl --context cluster-a-prod describe node worker-2 kubectl --context cluster-a-prod get events -A \\ --field-selector involvedObject.kind=Node,involvedObject.name=worker-2 \\ --sort-by=.lastTimestamp On the node or through out-of-band access, check the local services:\nsudo systemctl status rke2-agent --no-pager sudo journalctl -u rke2-agent --since \u0026#39;1 hour ago\u0026#39; sudo crictl ps Common causes include kubelet/RKE2 agent failure, expired node certificates, host networking drift, disk pressure, compute pressure, or a node that is reachable by Kubernetes API metadata but not reachable by SSH. In one incident, workloads consumed the available CPU and memory on a node until rke2-agent and kubelet had too little capacity left to report and recover normally.\nDo not rerun broad remediation repeatedly until the NotReady node is understood. Repeated retries can hide which changes already succeeded.\nRun Calico Audits With Human Evidence Calico IP audit scripts often emit only mismatches into remediation target files. That is good for automation, but weak for human confidence.\nUse two outputs:\nmismatch-only files for remediation. all-node files for audit evidence. Example command:\n./audit-calico-ip.sh \\ --dc site-a \\ --ctx-regex \u0026#39;^(cluster-a-prod|cluster-a-uat|cluster-a-qa)$\u0026#39; \\ --show-all The all-node report should include a status column:\nCONTEXT NODE NODE_INTERNAL_IP CALICO_IPV4ADDRESS STATUS cluster-a-prod worker-1 192.0.2.10 192.0.2.10/24 MATCH cluster-a-prod worker-2 192.0.2.11 198.51.100.11/24 MISMATCH cluster-a-prod worker-3 192.0.2.12 NONE MISSING_ANNOTATION cluster-a-qa - - - UNREACHABLE_CONTEXT Keep existing automation files mismatch-only:\ncalico-ip-audit/site-a/mismatches.tsv calico-ip-audit/site-a/targets.tsv calico-ip-audit/site-a/targets-prod.tsv Add human evidence separately:\ncalico-ip-audit/site-a/all-nodes.tsv This prevents a clean audit from looking empty while preserving safe remediation inputs.\nDecision Rules No hosts matched means inventory targeting is wrong, not that drift is clean. Vault lookup errors are local execution failures until proven otherwise. audited 10 of 11 hosts is partial success and requires explicit accounting. NotReady nodes should stay visible in inventory and reports. Calico 0 targets means no mismatches only if all-node evidence proves nodes were checked. Remediation should not patch Calico annotations one by one unless the platform runbook explicitly calls for that. Prefer fixing autodetection policy and rolling Calico components safely. The operational goal is not just to change settings. It is to prove which nodes were targeted, which nodes changed, which nodes were skipped, and why.\n","permalink":"https://trinidadmarroquin.com/field-notes/remediation-audits-notready-calico-inventory/","section":"field-notes","summary":"Node remediation scripts often fail in boring ways: wrong inventory group, hidden Vault dependency, or one unreachable node that makes the final audit look stuck. Those failures matter because they can turn a clean remediation into a false incident, or worse, hide the one node that still needs attention.\nUse this workflow when cleaning Kubernetes node drift across a whole RKE2 cluster and validating Calico state at the same time.\n","tags":["kubernetes","rke2","calico","ansible","inventory","notready","operations"],"title":"Remediation Audits With NotReady Nodes And Calico Checks"},{"categories":["DevOps Dirty Dozen"],"content":"Part 8 of the DevOps Dirty Dozen Series: Gutta cavat lapidem — a drop hollows out the stone.\nInsight: Advocates for small, consistent efforts over overwhelming changes.\nBig bang deployments are seductive because they feel efficient. One release window. One coordination call. One massive bundle of work moved into production at once. Weeks or months of effort finally land, and the organization gets to say the project shipped.\nThen something breaks.\nThe problem with big bang deployments is not that large changes always fail. The problem is that when they fail, they fail with too much context, too many variables, and too much pressure. The blast radius is large, the rollback is unclear, and every team on the bridge has a different theory about which part of the release caused the damage.\nDevOps works best when change is small enough to understand, validate, and reverse. Big bang deployments invert that principle. They delay learning until the most expensive possible moment: production cutover.\nThe larger the release, the larger the uncertainty cloud around failure.\nThe Anatomy Of Big Bang Deployments Big bang deployments happen when many independent changes are bundled into one release event. They often appear under names that sound responsible: release train, quarterly launch, migration weekend, coordinated cutover, platform refresh.\nThe pattern is not defined by size alone. It is defined by coupling and irreversibility.\nCommon signs include:\nMany Changes, One Window: Application changes, database migrations, infrastructure updates, configuration changes, and dependency upgrades all land together.\nRollback Is Theoretical: The rollback plan exists as a section in the change ticket, but nobody has rehearsed it end to end. Some parts can roll back. Others cannot.\nTesting Happens Too Late: Integration testing occurs near the end, after all pieces are merged. Failures discovered late are treated as release blockers instead of design feedback.\nRelease Night Becomes A War Room: Dozens of people join a bridge call, each responsible for one slice of the system. Coordination becomes the control plane.\nSuccess Criteria Are Vague: The release is considered successful if nothing obvious is on fire after the window closes. Latent failure, degraded user experience, and operational debt are discovered later.\nWhen the deployment needs a war room, the system is telling you the change is too large.\nThe Cost Of Big Bang Deployments Big bang deployments concentrate risk in ways that make systems harder to operate.\nRoot Cause Analysis Becomes Guesswork: When fifty changes ship together, the first question during an incident is not \u0026ldquo;what changed?\u0026rdquo; It is \u0026ldquo;which part of everything changed?\u0026rdquo;\nRollback Becomes Dangerous: Rolling back a large deployment may undo unrelated fixes, conflict with database state, or leave dependent systems in incompatible versions. Teams hesitate, and hesitation extends outages.\nFeedback Arrives Too Late: Problems that could have been caught with incremental rollout are discovered after the entire change set is live. The cost of learning increases.\nRelease Anxiety Increases: The larger the deployment, the more fear surrounds it. Teams delay releases further to avoid risk, which makes the next release even larger. The cycle feeds itself.\nOwnership Blurs: With many teams involved in one release, accountability becomes diffuse. Everyone owns a piece, but nobody owns the system behavior created by the pieces together.\nRollback is easy only when the change was designed to be reversible.\nA Real-World Example: The Migration Weekend That Became A Month A team planned a platform migration over a long weekend. The change included a Kubernetes version upgrade, ingress controller replacement, database parameter changes, new DNS records, and application configuration updates. Each change had been tested in isolation. The combined release was scheduled for a single maintenance window.\nThe first few hours went well. Nodes upgraded. Pods rescheduled. DNS propagated. Then intermittent failures appeared in one customer workflow. The logs pointed at the application. The application team pointed at ingress. The ingress team pointed at DNS. The database team pointed at connection pooling. Every theory was plausible because everything had changed.\nRollback was not clean. DNS had propagated. Database settings had changed. Some workloads were already running against new assumptions. Returning to the previous state would have been another big bang deployment under worse conditions.\nThe maintenance window closed with the platform technically online but operationally fragile. For the next month, teams chased edge cases created by the combined change. None of the individual changes were reckless. Bundling them together made the release unrecoverable.\nThe lesson was blunt: integration risk is real risk. Testing pieces separately does not prove the combined release is safe.\nWhy Big Bang Deployments Persist Teams usually know large releases are risky. They still happen because the organization rewards batching.\nChange Approval Favors Fewer Events: If every release requires heavy process, teams batch work to reduce administrative overhead. The change process accidentally creates larger, riskier changes.\nStakeholders Want A Launch Date: Project plans often converge on a single visible date. The business gets certainty. Engineering inherits concentrated risk.\nArchitecture Couples The Release: If services, schemas, clients, and infrastructure cannot evolve independently, the organization has no choice but to deploy them together.\nTesting Environments Are Not Trusted: When staging is unreliable or incomplete, teams defer real validation until production. Production becomes the first honest integration environment.\nRollback Is Not Part Of Design: Features are built to go forward, not backward. Without feature flags, compatibility windows, and migration strategy, small deployment becomes difficult.\nHeavy process often creates the very risk it was meant to control.\nMoving From Big Bang To Incremental Delivery Escaping big bang deployments requires designing for smaller change. It is not only a release-management decision. It is an architecture, testing, and culture decision.\nSeparate Deploy From Release: Deploy code dark, then release behavior through feature flags or configuration. This lets teams validate production deployment mechanics before exposing users to the change.\nUse Expand-And-Contract Migrations: Database changes should support old and new application versions during a compatibility window. Add first, migrate usage, remove later.\nLimit Change Per Release: One application change and one infrastructure change in the same window may be manageable. Ten of each is not. If a release cannot be explained clearly in a few sentences, it is probably too large.\nCanary Before Full Rollout: Start with one node, one tenant, one region, or one small percentage of traffic. Observe before expanding.\nMake Rollback Real: A rollback plan is only real if it has been tested. If rollback is impossible, document that explicitly and design a forward-fix path before deployment begins.\nReduce Change Approval Friction: If the process makes small changes expensive, people will batch. Lightweight approvals for low-risk incremental changes reduce the incentive to create big releases.\nSmall changes create more learning opportunities and fewer catastrophic surprises.\nApplying The Scientific Method Incremental delivery is experimentation applied to release engineering.\nAsk: What is the smallest change that can validate the assumption?\nHypothesize: \u0026ldquo;If we deploy this change to one tenant first, we expect no increase in error rate or latency over thirty minutes.\u0026rdquo;\nTest: Deploy to a narrow slice of production with clear monitoring and rollback criteria.\nObserve: Watch technical metrics and user-facing signals. Do not rely only on deployment success.\nIterate: Expand, pause, roll back, or adjust based on evidence. The release plan should respond to reality, not the calendar.\nCarl Sagan\u0026rsquo;s Baloney Detection Kit Big bang deployments often survive because teams repeat comfortable assumptions. Challenge them:\n\u0026ldquo;It is safer to do it all at once.\u0026rdquo; — Safer for coordination, or safer for production? Those are not the same.\n\u0026ldquo;We tested everything in staging.\u0026rdquo; — Is staging production-like enough to prove the claim? Does it have the same data shape, traffic, dependencies, and failure modes?\n\u0026ldquo;Rollback is documented.\u0026rdquo; — Has it been rehearsed? Does it include database state, DNS, queues, caches, and downstream consumers?\n\u0026ldquo;This has to ship together.\u0026rdquo; — Is that a real technical constraint, or a planning habit? Can compatibility be introduced to decouple the pieces?\n\u0026ldquo;We only deploy quarterly because releases are risky.\u0026rdquo; — Are releases risky because they are rare and large?\nA canary is not ceremony. It is a controlled question asked of production.\nMoving Forward Together Big bang deployments are not a badge of discipline. They are often evidence that the organization has allowed change to become too hard, too coupled, or too frightening.\nThe goal is not reckless speed. The goal is recoverability. Smaller releases do not eliminate failure, but they make failure easier to understand and contain. They turn deployment from an event into a habit.\nThe proverb says a drop hollows out the stone. It does not happen through force. It happens through repetition. The same is true of reliable delivery: small changes, repeated safely, reshape systems more effectively than occasional acts of release-day heroism.\nWhat is the largest release your team still treats as normal? What would it take to split it into ten smaller, safer changes?\nReferences Continuous Delivery by Jez Humble and David Farley Accelerate: The Science of Lean Software and DevOps by Nicole Forsgren, Jez Humble, and Gene Kim The DevOps Handbook by Gene Kim, Patrick Debois, John Willis, and Jez Humble Martin Fowler: Feature Toggles Google SRE Book — Managing Critical State The Twelve-Factor App: Disposability ","permalink":"https://trinidadmarroquin.com/posts/devops-dirty-dozen/big-bang-deployments/","section":"posts","summary":"Part 8 of the DevOps Dirty Dozen Series: Gutta cavat lapidem — a drop hollows out the stone.\nInsight: Advocates for small, consistent efforts over overwhelming changes.\nBig bang deployments are seductive because they feel efficient. One release window. One coordination call. One massive bundle of work moved into production at once. Weeks or months of effort finally land, and the organization gets to say the project shipped.\nThen something breaks.\n","tags":["devops","sre","deployments","release-engineering","change-management","rollback","continuous-delivery"],"title":"Small Changes, Safer Systems: The DevOps Big Bang Deployments Anti-Pattern"},{"categories":["field-notes"],"content":"DNS drift detectors need to distinguish resolver search domains from interface names.\nOn Kubernetes nodes using Calico, resolvectl domain can include link names such as:\nLink 3 (caliabc123): Link 5 (vxlan.calico): Those strings can look domain-like to a naive regex. If the detector treats vxlan.calico as an active DNS search domain, clean nodes appear drifted.\nSymptom The DNS audit shows expected resolver state:\nresolv_conf_search = . netplan_search = [] resolv_conf_type = symlink:/run/systemd/resolve/stub-resolv.conf But the detector still marks nodes as drift because resolved_domains includes Calico overlay links.\nFilter Overlay Links Before Drift Classification Filter Calico link lines from resolvectl domain before extracting domains:\nRESOLVED_DOMAINS=$(resolvectl domain 2\u0026gt;/dev/null \\ | grep -Ev \u0026#39;^Link [0-9]+ \\((cali[^)]*|vxlan\\.calico)\\):\u0026#39; \\ | sed -E \u0026#39;s/^Global:[[:space:]]*//; s/^Link [0-9]+ \\([^)]*\\):[[:space:]]*//\u0026#39; \\ | grep -v \u0026#39;^[[:space:]]*$\u0026#39; \\ | tr \u0026#39;\\n\u0026#39; \u0026#39; \u0026#39; \\ | sed \u0026#39;s/[[:space:]]\\+/ /g; s/^ //; s/ $//\u0026#39;) [ -z \u0026#34;$RESOLVED_DOMAINS\u0026#34; ] \u0026amp;\u0026amp; RESOLVED_DOMAINS=\u0026#34;NONE\u0026#34; The key is filtering by link name before evaluating whether a remaining value is a real search domain.\nDo Not Parse CSV With awk -F, If the report stores fields like resolved_domains, commas or quoted values can break naive parsing.\nAvoid this:\nawk -F, \u0026#39;$7 == \u0026#34;drift\u0026#34;\u0026#39; report.csv Use a CSV parser and emit tab-separated output for display:\npython3 - \u0026#34;$OUTPUT_FILE\u0026#34; \u0026lt;\u0026lt;\u0026#39;PY\u0026#39; | column -t -s $\u0026#39;\\t\u0026#39; import csv import sys with open(sys.argv[1], newline=\u0026#39;\u0026#39;, encoding=\u0026#39;utf-8\u0026#39;) as report: reader = csv.reader(report) header = next(reader, None) if header: print(\u0026#39;\\t\u0026#39;.join(header)) for row in reader: if len(row) \u0026gt;= 7 and row[6] == \u0026#39;drift\u0026#39;: print(\u0026#39;\\t\u0026#39;.join(row)) PY Suppress Successful Ansible Noise If the detector captures raw Ansible output, do not print normal success lines as warnings:\ngrep -v \u0026#39;DNS_DRIFT|\u0026#39; \u0026#34;$AUDIT_RAW\u0026#34; \\ | grep -vE \u0026#39;^[^|]+ \\| (CHANGED|SUCCESS) \\| rc=0 \u0026gt;\u0026gt;$\u0026#39; \\ \u0026gt; \u0026#34;$WARNINGS\u0026#34; || true Warnings should mean something operators need to read.\nValidate The Fix Run syntax checks:\nbash -n dns-drift-detector.sh If the detector runs in a container, rebuild the local image:\ndocker build -t dns-drift-detector:local dns-drift-detector Then run the same inventory audit again:\ndocker run --rm \\ -v \u0026#34;$PWD/ansible/inventory/site-a-ops-rke2.yaml:/inventory/inventory.yaml:ro\u0026#34; \\ -v \u0026#34;$PWD/dns-drift-detector/reports:/reports\u0026#34; \\ -v \u0026#34;$HOME/.ssh:/ssh:ro\u0026#34; \\ -e SSH_USER=operator \\ -e ANSIBLE_EXTRA_ARGS=\u0026#39;--private-key /ssh/id_rsa\u0026#39; \\ dns-drift-detector:local \\ --inventory /inventory/inventory.yaml \\ --group site_a_ops_rke2 Expected result:\nresolved_domains = NONE status = clean Drift remaining = header only Operating Rule Audit tools should ignore infrastructure interface names before classifying DNS drift.\nCalico overlay links are evidence about networking, not resolver search domains.\n","permalink":"https://trinidadmarroquin.com/field-notes/dns-drift-detector-calico-overlay-filter/","section":"field-notes","summary":"DNS drift detectors need to distinguish resolver search domains from interface names.\nOn Kubernetes nodes using Calico, resolvectl domain can include link names such as:\nLink 3 (caliabc123): Link 5 (vxlan.calico): Those strings can look domain-like to a naive regex. If the detector treats vxlan.calico as an active DNS search domain, clean nodes appear drifted.\nSymptom The DNS audit shows expected resolver state:\nresolv_conf_search = . netplan_search = [] resolv_conf_type = symlink:/run/systemd/resolve/stub-resolv.conf But the detector still marks nodes as drift because resolved_domains includes Calico overlay links.\n","tags":["dns","calico","ansible","kubernetes","rke2","audit","operations"],"title":"DNS Drift Detector Calico Overlay False Positives"},{"categories":["field-notes"],"content":"Use this when a Rancher system-upgrade-controller worker Plan is actively cordoning or draining nodes and a normal live-object patch does not stick because GitOps restores it.\nConfirm The Controller Is Active kubectl get plan -n system-upgrade kubectl get jobs,pods -n system-upgrade -o wide kubectl get events -n system-upgrade --sort-by=.lastTimestamp Look for jobs like:\napply-agent-plan-on-worker-1-... Repeated jobs can mean the upgrade job is timing out while trying to stop rke2-agent or rke2-server. In that case the controller may start over and attempt the shutdown again, keeping the selected worker in the disruption path.\nCheck worker schedulability:\nkubectl get nodes kubectl describe node worker-1 | grep -E \u0026#39;Unschedulable|Taints|Ready\u0026#39; Check GitOps Ownership Before assuming a patch will hold, check ownership annotations and Argo/ApplicationSet resources:\nkubectl get plan agent-plan -n system-upgrade -o yaml kubectl get application -n argocd system-upgrade -o yaml kubectl get applicationset -n argocd system-upgrade -o yaml If Argo CD or an ApplicationSet owns the Plan, live patches may be reverted by self-heal.\nFirst Stop: Make The Plan Match No Nodes Patch the Plan with an intentionally absent selector:\nkubectl patch plan agent-plan -n system-upgrade --type=merge -p \u0026#39; { \u0026#34;spec\u0026#34;: { \u0026#34;nodeSelector\u0026#34;: { \u0026#34;matchExpressions\u0026#34;: [ { \u0026#34;key\u0026#34;: \u0026#34;node-role.kubernetes.io/control-plane\u0026#34;, \u0026#34;operator\u0026#34;: \u0026#34;DoesNotExist\u0026#34; }, { \u0026#34;key\u0026#34;: \u0026#34;system-upgrade.example.com/agent-plan-paused\u0026#34;, \u0026#34;operator\u0026#34;: \u0026#34;In\u0026#34;, \u0026#34;values\u0026#34;: [\u0026#34;false\u0026#34;] } ] } } }\u0026#39; Delete active jobs and uncordon affected workers:\nkubectl delete job -n system-upgrade \\ -l upgrade.cattle.io/plan=agent-plan \\ --ignore-not-found=true kubectl uncordon worker-1 If GitOps Reverts The Patch If the Plan is restored and jobs come back, pause GitOps at the source if possible. If that is not fast enough during an incident, stop execution by scaling down the controller:\nkubectl scale deployment system-upgrade-controller \\ -n system-upgrade \\ --replicas=0 Then delete active jobs again and uncordon workers:\nkubectl delete job -n system-upgrade \\ -l upgrade.cattle.io/plan=agent-plan \\ --ignore-not-found=true kubectl uncordon worker-1 Verify The Stop sleep 20 kubectl get deploy -n system-upgrade system-upgrade-controller kubectl get jobs,pods -n system-upgrade -l upgrade.cattle.io/plan=agent-plan -o wide kubectl get nodes Expected emergency state:\nsystem-upgrade-controller replicas: 0 agent-plan jobs: none workers: Ready and schedulable unless independently unhealthy Find Why It Ran The controller may have started weeks ago and still be acting because workers never reached the desired version.\nCheck desired and actual versions:\nkubectl get plan agent-plan -n system-upgrade -o yaml kubectl get nodes -o wide Pattern:\ncontrol-plane nodes: desired version reached worker nodes: older version remains agent-plan: still selects workers That means the controller is not randomly starting. It is continuously reconciling unfinished desired state.\nAlso check whether the controller moved to a different worker because another worker became NotReady:\nkubectl get nodes -o wide kubectl describe node worker-2 If a worker entered NotReady due to certificate expiration, kubelet/RKE2 agent failure, or an unreachable node condition, the upgrade controller may select the last remaining schedulable worker. That is the dangerous point for management workloads.\nDurable Fix The emergency stop is not the durable fix.\nAfter management access is restored:\nupdate the Git source that renders the Plan. add a real pause flag or disable automated sync for the specific upgrade app. fix NotReady workers before allowing the worker Plan to resume. check certificate expiration and RKE2 agent health on every worker. restore controller replicas only when worker capacity is healthy. make sure at least one worker can be drained without losing management workloads. remove emergency live patches after Git reflects the desired state. Operating Rule If GitOps owns the Plan, live Kubernetes patches are temporary.\nDuring an outage, stop execution first. Then make the durable change in Git before turning the controller back on.\n","permalink":"https://trinidadmarroquin.com/field-notes/rancher-system-upgrade-controller-emergency-stop/","section":"field-notes","summary":"Use this when a Rancher system-upgrade-controller worker Plan is actively cordoning or draining nodes and a normal live-object patch does not stick because GitOps restores it.\nConfirm The Controller Is Active kubectl get plan -n system-upgrade kubectl get jobs,pods -n system-upgrade -o wide kubectl get events -n system-upgrade --sort-by=.lastTimestamp Look for jobs like:\napply-agent-plan-on-worker-1-... Repeated jobs can mean the upgrade job is timing out while trying to stop rke2-agent or rke2-server. In that case the controller may start over and attempt the shutdown again, keeping the selected worker in the disruption path.\n","tags":["rancher","rke2","kubernetes","gitops","upgrades","operations"],"title":"Emergency Stop For Rancher System Upgrade Controller"},{"categories":["DevOps Dirty Dozen"],"content":"Part 7 of the DevOps Dirty Dozen Series: Nullius boni sine socio iucunda possessio est — no good is enjoyable without a companion.\nInsight: Points to the value of teamwork over individual heroics.\nEvery engineering organization has a story about the person who saved production at 2 AM. They knew the undocumented system. They remembered the one flag. They could read logs nobody else understood. They jumped in, fixed the outage, and became the name everyone invoked the next time things went sideways.\nThat person may be talented. They may be dedicated. They may have prevented real damage. But when an organization repeatedly depends on that person, it is not witnessing excellence. It is exposing fragility.\nHero culture is what happens when operational success depends on exceptional individuals instead of sustainable systems. It feels reassuring in the moment because someone always seems able to save the day. Underneath, it is a slow-moving failure mode: knowledge concentrates, teams disengage, risk hides, and the hero burns out.\nIf one person is always under the spotlight, the system is already telling you where it is fragile.\nThe Anatomy Of Hero Culture Hero culture rarely starts with bad intent. It often starts with competence. Someone knows the system better than everyone else. They respond quickly. They care deeply. Leaders trust them. Teams route hard problems to them because it works.\nThe pattern becomes dangerous when that individual capability replaces team capability.\nHero culture looks like this:\nThe Same Names In Every Incident: The incident bridge starts with ten people and ends with two people actually doing the work. Everyone else watches because the real knowledge lives in a few heads.\nUndocumented Fixes: The hero fixes the issue faster than anyone can document it. The incident closes. The system remains just as mysterious as before.\nEscalation By Personality: Teams do not escalate to a role, runbook, or owning team. They escalate to a person. \u0026ldquo;Call Alex\u0026rdquo; becomes the operational model.\nPraise For Rescue, Silence For Prevention: The engineer who restores service gets visible recognition. The engineer who eliminated the class of failure before it happened receives little attention.\nKnowledge As A Bottleneck: Critical details about deployments, failover, networking, credentials, and recovery procedures live in private memory, old shell history, or undocumented chat threads.\nWhen all paths pass through one person, that person is no longer just helping. They are the bottleneck.\nThe Cost Of Hero Culture Hero culture is expensive because the bill arrives in burnout, fragility, and stalled learning.\nBurnout Becomes Inevitable: The hero cannot ever fully disconnect. Vacations are interrupted. Weekends are conditional. Sleep depends on whether the system behaves. Eventually, the person who cared most becomes the person most likely to leave.\nTeams Stop Learning: If every hard problem is handed to the expert, everyone else loses the chance to build judgment. The team becomes dependent not because people are incapable, but because the system routes learning away from them.\nRisk Is Hidden From Leadership: As long as the hero keeps saving the day, leadership sees recovery, not fragility. The organization mistakes survival for resilience.\nIncidents Become Harder To Reproduce And Prevent: A fix performed from memory is difficult to audit. If nobody can explain exactly what changed, the organization cannot reliably prevent recurrence.\nSuccession Becomes A Crisis: When the hero leaves, changes roles, gets sick, or is simply unavailable, the organization discovers that the recovery plan was a person.\nHeroics feel heroic once. Repeated heroics become exhaustion with a pager attached.\nA Real-World Example: The One Engineer Who Knew The Network A platform team I worked around had a recurring pattern during network incidents. Whenever routing behaved strangely, everyone waited for one senior engineer. He knew the history: the old firewall exceptions, the nonstandard BGP behavior, the one load balancer rule nobody wanted to touch, the Terraform module that had drifted from reality.\nHe was fast. Too fast, honestly. He could restore service before the rest of the bridge understood the failure mode. For a while, that looked like operational strength.\nThen he took a week off.\nDuring that week, a certificate rotation triggered a networking path that had not been documented. The team had dashboards, logs, and access. What they did not have was the mental map. The incident lasted hours longer than it should have because the real runbook was on vacation.\nThe painful part was not that one person knew too much. The painful part was that the organization had allowed that to remain true because his competence made the weakness easy to ignore.\nThe fix was not to blame him. He had been carrying the system. The fix was to convert hero knowledge into team knowledge: diagrams, failure-mode runbooks, paired incident reviews, ownership rotation, and explicit handoff of the undocumented paths. The goal was not to make him less valuable. It was to make the team more capable.\nWhy Hero Culture Persists Hero culture persists because it produces short-term wins that hide long-term damage.\nIt Works During The Incident: When production is down, speed matters. Calling the person who knows the answer is rational in the moment. The anti-pattern forms when the organization never follows up by distributing that knowledge.\nRecognition Systems Reward Rescue: Many organizations celebrate visible recovery more than invisible prevention. The person who fixes the outage is praised. The person who wrote the runbook that avoided three future outages is forgotten.\nDocumentation Feels Slower Than Doing: The hero can fix the problem in ten minutes. Writing the runbook, explaining the context, and pairing with another engineer takes an hour. Under pressure, the hour loses.\nLeaders Confuse Dependency With Ownership: A leader may say, \u0026ldquo;This system has an owner,\u0026rdquo; when what they really mean is, \u0026ldquo;Only one person understands it.\u0026rdquo; Ownership should create clarity. Dependency creates risk.\nTeams Normalize Interruptions: Once the hero is always available, the organization treats interruption as normal operating procedure. The cost is invisible because it is paid by one person\u0026rsquo;s attention, sleep, and health.\nHero culture is a loop: rescue, praise, forget, repeat.\nTurning Heroics Into Team Capability The answer is not to shame the hero. Most heroes are created by organizational gaps, not ego. The answer is to turn individual expertise into shared capability.\nPair During Incidents: If one person is driving the fix, another person should shadow and document. The goal is not to slow recovery. The goal is to ensure the same incident does not require the same person next time.\nWrite Runbooks From Real Incidents: The best runbooks come from actual recovery work. After an incident, capture the commands, decision points, rollback criteria, and verification steps while the context is fresh.\nRotate Ownership Deliberately: Ownership rotation does not mean everyone owns everything. It means critical systems should have more than one capable operator. Primary and secondary ownership should be explicit.\nReward Prevention Publicly: Celebrate deleted alerts, simplified architecture, successful game days, and completed postmortem action items. Make prevention visible enough to compete with rescue.\nMake Escalation Role-Based: Escalate to the owning team, on-call role, or incident function — not to a favorite individual. If the process only works when one person answers the phone, the process is not real.\nProtect Recovery Time: If someone carries a major incident, they need recovery time afterward. Sustainable operations require rest as deliberately as they require coverage.\nThe goal is not fewer experts. The goal is more shared capability.\nApplying The Scientific Method Hero culture can be measured and reduced like any other systemic risk.\nObserve: Who gets called during incidents? Which systems depend on one or two people? Which alerts require specific tribal knowledge?\nHypothesize: \u0026ldquo;If we pair during incidents and write runbooks from real recovery steps, repeat escalations to the same person will decrease over the next quarter.\u0026rdquo;\nTest: Pick one high-risk system. Assign primary and secondary ownership. Run a game day where the usual expert observes but does not drive.\nMeasure: Did the secondary owner recover the system? Was the runbook sufficient? Which steps still required private knowledge?\nIterate: Update the runbook, train another person, and repeat until the system is no longer dependent on one individual.\nCarl Sagan\u0026rsquo;s Baloney Detection Kit Hero culture survives on comforting myths. Challenge them directly:\n\u0026ldquo;Nobody else can do it.\u0026rdquo; — Is that true, or has nobody else been given time, access, and context?\n\u0026ldquo;Documentation will slow us down.\u0026rdquo; — Slower than a four-hour outage when the expert is unavailable?\n\u0026ldquo;They like being the expert.\u0026rdquo; — Maybe. But liking mastery is not the same as consenting to permanent interruption.\n\u0026ldquo;We have an owner.\u0026rdquo; — Do you have ownership, or dependency? Can the system be operated when that person is unreachable?\n\u0026ldquo;We will document it later.\u0026rdquo; — Later usually means never unless time is explicitly scheduled and protected.\nThe knowledge is not safe until it moves from private memory into shared practice.\nMoving Forward Together Hero culture is seductive because it gives the organization a story: when things go wrong, someone exceptional will save us. But sustainable DevOps and SRE practice is not built on exceptional rescue. It is built on systems that ordinary teams can operate under pressure.\nThe best engineers should absolutely be valued. But valuing them means refusing to turn them into single points of failure. It means giving them time to teach, document, simplify, and rest. It means rewarding the quiet work that makes future heroics unnecessary.\nThe measure of operational maturity is not how often the same hero saves production. It is how rarely production requires a hero at all.\nWho does your organization always call when things go wrong? What would happen if they were unavailable tomorrow? The answer is not a staffing concern. It is an architectural and operational risk.\nReferences The DevOps Handbook by Gene Kim, Patrick Debois, John Willis, and Jez Humble Google SRE Book — Being On-Call Google SRE Workbook — Postmortem Culture Team Topologies by Matthew Skelton and Manuel Pais DORA: Generative organizational culture The Bus Factor ","permalink":"https://trinidadmarroquin.com/posts/devops-dirty-dozen/hero-culture/","section":"posts","summary":"Part 7 of the DevOps Dirty Dozen Series: Nullius boni sine socio iucunda possessio est — no good is enjoyable without a companion.\nInsight: Points to the value of teamwork over individual heroics.\nEvery engineering organization has a story about the person who saved production at 2 AM. They knew the undocumented system. They remembered the one flag. They could read logs nobody else understood. They jumped in, fixed the outage, and became the name everyone invoked the next time things went sideways.\n","tags":["devops","sre","hero-culture","burnout","incidents","teamwork","operations"],"title":"No More Heroes: The DevOps Hero Culture Anti-Pattern"},{"categories":["notes"],"content":"Not every outage starts with a new deployment.\nSometimes the trigger is old desired state that never finished converging. A controller keeps watching. A GitOps reconciler keeps restoring. The cluster looks stable until capacity drops, and then the old desired state becomes active at the worst possible time.\nThat is the failure pattern I want to remember from this Rancher/RKE2 investigation.\nThe Shape Of The Problem The cluster had a Rancher-managed RKE2 system-upgrade workflow with separate plans:\nserver-plan -\u0026gt; control-plane / etcd nodes agent-plan -\u0026gt; worker nodes The server side had already completed. Control-plane nodes were on the desired RKE2 version.\nThe worker side had not completed. The agent-plan still targeted non-control-plane nodes and expected worker nodes to reach the newer RKE2 version.\nThat meant the upgrade was not historical. It was still live desired state.\nWhy It Happened Later The confusing part was timing. The plan had been synced weeks earlier, so why did it cause trouble later?\nBecause a Kubernetes Plan managed by system-upgrade-controller is declarative. It does not run once and disappear. It keeps reconciling until selected nodes satisfy the desired version.\nThe effective logic was:\nworker node selected by agent-plan worker version != desired version controller creates or retries upgrade job job attempts to stop rke2-agent or rke2-server shutdown does not complete before timeout controller retries the job job cordons and drains selected worker That was latent risk while there was enough worker capacity.\nIt became an outage when worker capacity collapsed:\none worker had been unreachable for a long time. another worker hit certificate expiration and became NotReady. the upgrade controller moved to the last available worker. the upgrade job cordoned and attempted to drain that remaining healthy worker. At that point, management workloads that needed schedulable workers lost placement. Rancher and Argo CD HTTP access disappeared at the same time.\nThe failed upgrade attempts were not clean one-shot failures. The job could get stuck while shutting down the RKE2 process, hit the upgrade timeout, then start over and try the shutdown again. That retry loop kept the Plan active and made worker selection dangerous once the cluster lost spare worker capacity.\nThe First Fix Was Not Durable The first mitigation was to patch the agent-plan so it matched no worker nodes. That stopped the immediate job briefly.\nBut the Plan was GitOps-managed. Argo CD self-heal restored the desired Plan, removed the manual pause, and the controller created another job.\nThat is the key lesson:\nPatching the live object is not durable if GitOps owns the object. If GitOps self-heal is enabled, the durable fix must happen in Git or in the GitOps control path. Otherwise the cluster will undo the emergency patch.\nEmergency Stop Sequence The safer stop path was layered:\npause or disable GitOps reconciliation for the system-upgrade app patch the Plan so it matches no nodes delete active upgrade jobs uncordon affected workers if reconciliation still wins, scale down system-upgrade-controller verify no jobs return The last step was necessary because the ApplicationSet/GitOps path continued to restore the Plan. Scaling the controller to zero cut off execution even while the declarative object still existed.\nThat is not the long-term fix. It is an emergency brake.\nCommands Worth Keeping Inspect Plans and jobs:\nkubectl get plan -n system-upgrade kubectl get jobs,pods -n system-upgrade -o wide kubectl get events -n system-upgrade --sort-by=.lastTimestamp Check whether GitOps owns the Plan:\nkubectl get plan agent-plan -n system-upgrade -o yaml kubectl get application -n argocd system-upgrade -o yaml kubectl get applicationset -n argocd system-upgrade -o yaml Pause execution when the controller is actively disrupting workers:\nkubectl scale deployment system-upgrade-controller \\ -n system-upgrade \\ --replicas=0 kubectl delete job -n system-upgrade \\ -l upgrade.cattle.io/plan=agent-plan \\ --ignore-not-found=true kubectl uncordon worker-1 Verify it stays stopped:\nkubectl get deploy -n system-upgrade system-upgrade-controller kubectl get jobs,pods -n system-upgrade -l upgrade.cattle.io/plan=agent-plan -o wide kubectl get nodes Cordoned Is Not NotReady During the investigation, one worker looked suspicious because it had been unusable for a long time. It was important to separate two states:\ncordoned / SchedulingDisabled -\u0026gt; spec.unschedulable=true NotReady / unreachable -\u0026gt; kubelet stopped reporting status or node is unreachable A node can be NotReady without being cordoned. Uncordoning does nothing for a node whose kubelet is not reporting.\nUseful check:\nkubectl get node worker-3 -o jsonpath=\u0026#39;{.spec.unschedulable}{\u0026#34;\\n\u0026#34;}\u0026#39; kubectl describe node worker-3 What I Would Add To The Runbook Before allowing an automated worker upgrade Plan to run:\nrequire at least N healthy schedulable workers after one worker is cordoned. block or pause upgrades when any worker has been NotReady longer than a threshold. block or pause worker upgrades when node certificates are near expiration or kubelet/RKE2 readiness is unstable. alert when an upgrade job repeatedly times out while stopping rke2-agent or rke2-server. alert on agent-plan jobs retrying for days or weeks. document whether the Plan is GitOps-owned and where the durable pause lives. verify Rancher and Argo CD workloads have enough placement redundancy. treat ApplicationSet self-heal as part of the control plane, not background noise. The Practical Lesson The system-upgrade controller did what it was told. Argo CD did what it was told. Kubernetes did what it was told.\nThe outage came from the gap between desired state and operational readiness:\nworkers still needed upgrade controller kept reconciling GitOps kept restoring the Plan worker 2 became NotReady after certificate expiration worker capacity dropped to one usable worker the remaining worker was cordoned and drained management workloads lost placement That is why upgrade automation needs a readiness gate, not just a desired version.\nRelated Field Note:\nEmergency Stop For Rancher System Upgrade Controller ","permalink":"https://trinidadmarroquin.com/posts/rancher-system-upgrade-controller-latent-worker-risk/","section":"posts","summary":"Not every outage starts with a new deployment.\nSometimes the trigger is old desired state that never finished converging. A controller keeps watching. A GitOps reconciler keeps restoring. The cluster looks stable until capacity drops, and then the old desired state becomes active at the worst possible time.\nThat is the failure pattern I want to remember from this Rancher/RKE2 investigation.\nThe Shape Of The Problem The cluster had a Rancher-managed RKE2 system-upgrade workflow with separate plans:\n","tags":["rancher","rke2","kubernetes","gitops","upgrades","operations"],"title":"When A Latent Rancher Worker Upgrade Becomes An Outage"},{"categories":["field-notes"],"content":"This field note documents the engineering workstation baseline used across the runbooks and field notes in this site. It is a living document — tools are added as they prove value and removed when they are retired.\nOS Selection Two distributions cover the majority of infrastructure engineering environments:\nDistribution Strengths Considerations Ubuntu LTS (24.04+) Broad package availability, largest community, NVidia driver support, first-class Snap/Flatpak support Canonical\u0026rsquo;s Snap push can be invasive; pin to LTS to avoid churn Debian Stable (12+) Rock-solid stability, no corporate backing, minimal pre-installed cruft Older packages; backports or third-party repos needed for newer tooling Both run the same toolchain. The choice is primarily about release cadence and package freshness vs. stability.\nHardware minimums for a daily-driver management workstation:\nCPU: 4+ cores (8 recommended for local container builds) RAM: 16 GB minimum, 32 GB recommended (multiple kubeconfigs, browser tabs, local kind/microk8s clusters) SSD: 256 GB minimum, 512 GB recommended (container images, multiple tool versions, git repos) Network: Reliable connectivity to cloud APIs, VPN, and internal infrastructure Core Toolchain Package Management # Ubuntu / Debian sudo apt update \u0026amp;\u0026amp; sudo apt install -y \\ curl wget git jq yq unzip \\ gnupg lsb-release ca-certificates \\ direnv tree htop iotop # Enable unattended security upgrades sudo dpkg-reconfigure --priority=low unattended-upgrades Kubernetes Tooling # kubectl — always match the cluster version or one minor version ahead curl -LO \u0026#34;https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl\u0026#34; sudo install kubectl /usr/local/bin/ \u0026amp;\u0026amp; rm kubectl # Helm curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash # Velero CLI curl -LO https://github.com/vmware-tanzu/velero/releases/latest/download/velero-v1.15.0-linux-amd64.tar.gz tar xzf velero-*-linux-amd64.tar.gz \u0026amp;\u0026amp; sudo mv velero-*/velero /usr/local/bin/ \u0026amp;\u0026amp; rm -rf velero-* # kustomize curl -fsSL https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh | bash sudo mv kustomize /usr/local/bin/ # stern — tail multiple pods and containers curl -Lo stern.tar.gz https://github.com/stern/stern/releases/latest/download/stern_*_linux_amd64.tar.gz tar xzf stern.tar.gz \u0026amp;\u0026amp; sudo mv stern /usr/local/bin/ \u0026amp;\u0026amp; rm stern.tar.gz # Popeye — cluster health scanner curl -Lo popeye.tar.gz https://github.com/derailed/popeye/releases/latest/download/popeye_linux_amd64.tar.gz tar xzf popeye.tar.gz \u0026amp;\u0026amp; sudo mv popeye /usr/local/bin/ \u0026amp;\u0026amp; rm popeye.tar.gz Infrastructure-As-Code # Terraform — use tfenv to pin per-project versions git clone https://github.com/tfutils/tfenv.git ~/.tfenv ~/.tfenv/bin/tfenv install latest echo \u0026#39;export PATH=\u0026#34;$HOME/.tfenv/bin:$PATH\u0026#34;\u0026#39; \u0026gt;\u0026gt; ~/.bashrc # Terragrunt curl -Lo terragrunt https://github.com/gruntwork-io/terragrunt/releases/latest/download/terragrunt_linux_amd64 sudo mv terragrunt /usr/local/bin/ \u0026amp;\u0026amp; chmod +x /usr/local/bin/terragrunt # tflint curl -s https://raw.githubusercontent.com/terraform-linters/tflint/master/install_linux.sh | bash # Checkov — policy-as-code scanner pip3 install checkov Cloud CLIs # AWS CLI v2 curl -fsSL https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip -o awscliv2.zip unzip awscliv2.zip \u0026amp;\u0026amp; sudo ./aws/install \u0026amp;\u0026amp; rm -rf aws awscliv2.zip # Vault CLI curl -fsSL https://releases.hashicorp.com/vault/1.18.0/vault_1.18.0_linux_amd64.zip -o vault.zip unzip vault.zip \u0026amp;\u0026amp; sudo mv vault /usr/local/bin/ \u0026amp;\u0026amp; rm vault.zip # govc (vSphere CLI) curl -Lo govc.tar.gz https://github.com/vmware/govmomi/releases/latest/download/govc_$(uname -s)_$(uname -m).tar.gz tar xzf govc.tar.gz \u0026amp;\u0026amp; sudo mv govc /usr/local/bin/ \u0026amp;\u0026amp; rm govc.tar.gz Containers # Docker CE curl -fsSL https://get.docker.com | bash sudo usermod -aG docker $USER # Dive — container layer inspection curl -Lo dive.tar.gz https://github.com/wagoodman/dive/releases/latest/download/dive_*_linux_amd64.tar.gz tar xzf dive.tar.gz \u0026amp;\u0026amp; sudo mv dive /usr/local/bin/ \u0026amp;\u0026amp; rm dive.tar.gz General Tooling # bat — cat with syntax highlighting sudo apt install -y bat # or: https://github.com/sharkdp/bat # fzf — fuzzy finder git clone --depth 1 https://github.com/junegunn/fzf.git ~/.fzf \u0026amp;\u0026amp; ~/.fzf/install # ripgrep — fast recursive grep sudo apt install -y ripgrep # httpie — human-friendly curl alternative pip3 install httpie # age — simple file encryption sudo apt install -y age Shell Hygiene Kubectl Aliases And Completions cat \u0026gt;\u0026gt; ~/.bashrc \u0026lt;\u0026lt; \u0026#39;EOF\u0026#39; source \u0026lt;(kubectl completion bash) alias k=\u0026#39;kubectl\u0026#39; alias kg=\u0026#39;kubectl get\u0026#39; alias kd=\u0026#39;kubectl describe\u0026#39; alias kdel=\u0026#39;kubectl delete\u0026#39; alias kl=\u0026#39;kubectl logs\u0026#39; alias kaf=\u0026#39;kubectl apply -f\u0026#39; alias kx=\u0026#39;kubectl ctx\u0026#39; # requires kubectx alias kn=\u0026#39;kubectl ns\u0026#39; # requires kubens # Switch context with tab completion complete -F __start_kubectl k EOF # kubectx / kubens sudo git clone https://github.com/ahmetb/kubectx /opt/kubectx sudo ln -s /opt/kubectx/kubectx /usr/local/bin/kubectx sudo ln -s /opt/kubectx/kubens /usr/local/bin/kubens KUBECONFIG Management The default ~/.kube/config merge approach becomes unwieldy with more than a few clusters.\n# Per-project kubeconfig directories mkdir -p ~/.kube/configs # Source the right config in each project via direnv cat \u0026gt;\u0026gt; ~/.bashrc \u0026lt;\u0026lt; \u0026#39;EOF\u0026#39; # KUBECONFIG merge helper — merges everything in ~/.kube/configs/ export KUBECONFIG=$(ls ~/.kube/configs/*.yaml 2\u0026gt;/dev/null | tr \u0026#39;\\n\u0026#39; \u0026#39;:\u0026#39;) EOF Create a .envrc per project to pin the kubeconfig:\n# In each project root echo \u0026#39;export KUBECONFIG=~/.kube/configs/production.yaml\u0026#39; \u0026gt; .envrc direnv allow Terraform Workspace Prompt # Show terraform workspace in the prompt cat \u0026gt;\u0026gt; ~/.bashrc \u0026lt;\u0026lt; \u0026#39;EOF\u0026#39; parse_terraform_workspace() { if [ -f .terraform/environment ]; then echo \u0026#34; (tf:$(cat .terraform/environment))\u0026#34; fi } PS1=\u0026#39;${debian_chroot:+($debian_chroot)}\\[\\033[01;32m\\]\\u@\\h\\[\\033[00m\\]:\\[\\033[01;34m\\]\\w\\[\\033[00m\\]$(parse_terraform_workspace)\\$ \u0026#39; EOF Session Persistence Infrastructure work spans long-running apply operations, multi-step incident responses, and context switches across terminals.\nTmux sudo apt install -y tmux # Minimal configuration cat \u0026gt;\u0026gt; ~/.tmux.conf \u0026lt;\u0026lt; \u0026#39;EOF\u0026#39; set -g default-terminal \u0026#34;screen-256color\u0026#34; set -g history-limit 50000 set -g mouse on bind | split-window -h bind - split-window -v bind r source-file ~/.tmux.conf \\; display-message \u0026#34;Reloaded\u0026#34; EOF Patterns:\nSituation Tmux Pattern Terraform plan/apply tmux new -s infra-apply — stay attached until complete Incident response tmux new -s incident-\u0026lt;ticket\u0026gt; with panes for logs, kubectl, runbook Long tail of tailing logs Detach (C-b d) and reattach later (tmux attach -t \u0026lt;session\u0026gt;) Multi-cluster operations One window per cluster, named by context Secret Hygiene Credentials live on the workstation temporarily and are never stored in plain text.\n# Age key generation — one per workstation, backed up offline age-keygen -o ~/.config/age/key.txt chmod 600 ~/.config/age/key.txt # Encrypt a secrets file age -e -r \u0026#34;$(cat ~/.config/age/key.txt | age-keygen -y)\u0026#34; -o secrets.env.age secrets.env # Decrypt inline age -d -i ~/.config/age/key.txt secrets.env.age # SOPS integration cat \u0026gt;\u0026gt; ~/.sops.yaml \u0026lt;\u0026lt; \u0026#39;EOF\u0026#39; creation_rules: - age: \u0026gt;- age1... EOF Never do these:\nStore cloud provider keys in ~/.bashrc or shell history Commit .env files to git Use the same SSH key for GitHub and infrastructure access Leave vault login tokens in the terminal scrollback Bootstrap Automation The entire workstation should be reproducible from a blank install.\n# ~/bootstrap/bootstrap.sh — idempotent workstation setup # Keep this in a private git repo #!/usr/bin/env bash set -euo pipefail sudo apt update \u0026amp;\u0026amp; sudo apt upgrade -y # Install all tools (functions above) # Clone dotfiles # Set up SSH keys (requires out-of-band transfer) # Install VS Code / extensions A bootstrap script should be idempotent — running it twice produces the same result. Use which \u0026lt;tool\u0026gt; guards or install -C for binaries.\nWhen To Update This Document A new CLI becomes part of the standard incident response workflow A tool in the chain reaches end-of-life or is superseded The team standardizes on a different container runtime or orchestrator A security practice changes (e.g., hardware SSH keys, new encryption scheme) References tmux: Productive Mouse-Free Development Direnv: Unclutter your .profile The Twelve-Factor App: Dev/prod parity Age: Simple, modern encryption Kubectx + Kubens: Kubernetes context/namespace switching ","permalink":"https://trinidadmarroquin.com/field-notes/devops-sre-linux-workstation/","section":"field-notes","summary":"This field note documents the engineering workstation baseline used across the runbooks and field notes in this site. It is a living document — tools are added as they prove value and removed when they are retired.\nOS Selection Two distributions cover the majority of infrastructure engineering environments:\nDistribution Strengths Considerations Ubuntu LTS (24.04+) Broad package availability, largest community, NVidia driver support, first-class Snap/Flatpak support Canonical\u0026rsquo;s Snap push can be invasive; pin to LTS to avoid churn Debian Stable (12+) Rock-solid stability, no corporate backing, minimal pre-installed cruft Older packages; backports or third-party repos needed for newer tooling Both run the same toolchain. The choice is primarily about release cadence and package freshness vs. stability.\n","tags":["workstation","linux","tools","devops","sre","productivity"],"title":"DevOps And SRE Linux Workstation"},{"categories":["DevOps Dirty Dozen"],"content":"Part 6 of the DevOps Dirty Dozen Series: Quod non reficitur, deficit — what is not renewed, deteriorates.\nInsight: Emphasizes the importance of constant improvement through feedback.\nEvery engineer has a story about the alert that fired too many times, the dashboard that showed a slow climb, the user complaint that got buried in a ticket queue. And every engineer has a story about the night that same ignored signal became a full-blown incident.\nIgnoring feedback loops is not a failure of tooling. It is a failure of response. The monitoring stack can be world-class, the dashboards immaculate, the on-call rotation perfectly staffed — and none of it matters if the organization has trained itself to look away.\nIn this sixth article of the DevOps Dirty Dozen, we examine how feedback loops atrophy, why teams rationalize ignoring signals, and what it takes to build a culture where feedback actually changes behavior.\nThe signal is usually there. The hard part is not detecting it. It is choosing to act.\nThe Anatomy Of Broken Feedback Loops A feedback loop is any mechanism that returns information about a system\u0026rsquo;s output back into its input. In DevOps, these loops include:\nMonitoring alerts and dashboards Incident postmortems and action items User complaints and feature requests Deployment failure rates and rollback triggers Performance regression detection Security scan findings A broken feedback loop looks the same regardless of the source: the signal arrives, and nothing changes.\nHere is what that looks like in practice:\nAlert Fatigue As Policy: Alarms fire so frequently that the on-call engineer checks the channel and, seeing nothing burning, looks away. The one genuine alert is indistinguishable from the twenty false ones. Teams respond by tuning alert severity upward until nothing is urgent — and then the real emergency has no severity high enough to break through.\nDashboards Without Decisions: The team builds beautiful dashboards reviewed in weekly meetings. Metrics are up, metrics are down. The meeting ends. No action is taken. The dashboards become screensavers — visually present, functionally inert.\nPostmortems That End At The Document: The incident is resolved. A postmortem is written. Action items are assigned. The document is filed. Six months later, the same incident happens again. The postmortem was written, but it was never read as a plan.\nThe Buried User Report: A user reports a subtle anomaly — increased latency at a specific time of day, an intermittent error in a non-critical path. The ticket is triaged as low priority. Weeks later, the anomaly surfaces as a production incident affecting the same path, now critical.\nThe Gradual Climb: CPU creeps up one percent per week. Memory fragmentation grows incrementally. Each individual change is too small to alert on. After six months, the system tips over during a routine deploy. The data was there the entire time. Nobody was watching for trends, only thresholds.\nThe disaster did not happen in an instant. It was measured in weeks of ignored data.\nThe Cost Of Ignoring Feedback The cost of broken feedback loops compounds because each ignored signal erodes the next one.\nSmall Problems Become Large Ones: Every production outage that could have been prevented by acting on a early warning represents debt that compounds in silence. The anomaly you ignore today is the P0 you wake up for tomorrow.\nTrust In Monitoring Collapses: When alerts are routinely ignored, the entire monitoring investment is wasted. Engineers stop trusting the tools, then stop looking at them. The observability stack becomes a line item on a budget rather than an operational lever.\nOn-Call Burnout Accelerates: Responding to real incidents is exhausting. Responding to incidents that should never have happened — because a signal was ignored weeks ago — is demoralizing. Teams that ignore feedback loops burn through on-call engineers faster.\nOrganizational Learning Stops: Feedback loops are how teams learn. Without them, every incident is a novel surprise. The same root causes recur. The organization plateaus while the systems around it grow more complex.\nUser Trust Erodes Slowly, Then All At Once: Users notice when their feedback disappears into a void. They stop reporting issues. The silence is not a sign that things are working — it is a sign that users have given up.\nEvery ignored signal is a crack in the organizational learning system.\nA Real-World Example: The Latency That Did Not Matter I worked with a team that ran a platform serving time-sensitive financial data. For three months, a monitoring dashboard showed a gradual latency increase in one API endpoint. The increase was small — fifty milliseconds over the baseline — and it only affected a non-critical reporting call used by internal teams.\nThe latency dashboard was reviewed in the weekly operations meeting. Each week, the same question: \u0026ldquo;Is this worth investigating?\u0026rdquo; Each week, the same answer: \u0026ldquo;It is not affecting customers. We will get to it.\u0026rdquo;\nThe latency was a symptom of a connection pool leak in a shared library. The leak was gradual. The connection pool had reserve capacity, so the leak only manifested as slower acquisition times, not failures. The team knew about the library — it had been flagged in a dependency audit six months prior — but replacing it was shelved for \u0026ldquo;when we have time.\u0026rdquo;\nWeek twelve was the failure point. The connection pool exhausted its reserves during a traffic spike triggered by a market event. The non-critical endpoint timed out. Then the timeout cascaded into the critical path because the shared library held connections across threads. The reporting endpoint took down the trading feed.\nThe incident cost more in engineering time alone than replacing the library would have cost ten times over. The signal was visible for three months. The team looked at it weekly. Nobody acted because the feedback loop ended at the dashboard.\nWhat fixed it was not better monitoring. The monitoring was already good enough. What fixed it was a practice: every dashboard reviewed in the weekly meeting had to have a decision attached. If a metric had been trending in the same direction for three consecutive meetings, it was automatically escalated. The feedback loop was not broken at the instrumentation layer. It was broken at the response layer.\nWhy Feedback Loops Break Feedback loops do not break because engineers are lazy or indifferent. They break because of systemic pressures:\nSignal-To-Noise Ratio Deteriorates: Every alert, dashboard, and report added to the ecosystem makes the next one harder to see. Teams add signals faster than they remove them. The noise drowns the signal.\nAction Requires Ownership: A signal without an owner is noise. If nobody is responsible for responding to a metric trend, the trend will be observed indefinitely without action. Ownership is the difference between data and intelligence.\nReactive Cultures Reward Heroism: In organizations that celebrate the engineer who \u0026ldquo;saved the weekend,\u0026rdquo; there is little incentive to prevent the weekend from needing saving. Ignoring early warnings allows incidents to become dramatic — and dramatic incidents generate recognition.\nTooling Replaces Practice: A team that buys a monitoring platform and declares observability \u0026ldquo;done\u0026rdquo; has confused the tool with the practice. The dashboard exists. The alert fires. But there is no habit of acting on the information. The loop is built but not closed.\nFeedback Fatigue: When every signal demands a response, teams develop a protective numbness. The only way to preserve energy is to ignore everything until something catches fire. This is not a failure of attention. It is a failure of signal prioritization.\nWhen everything is urgent, nothing is.\nClosing The Loop Fixing broken feedback loops does not require new tooling. It requires new habits.\nAudit Your Signals Quarterly: Every three months, review every alert, dashboard, and automated report. Which ones drove a decision in the last quarter? Remove or tune the rest. Signal count should trend down, not up.\nAttach Decisions To Dashboards: A dashboard without a decision attached is a screensaver. For every dashboard, define: \u0026ldquo;If this metric crosses X threshold, we do Y.\u0026rdquo; If there is no Y, the dashboard does not need to exist.\nCreate A Trend Review Practice: Threshold alerts catch spikes. Trend alerts catch declines. Add a regular review of metrics moving in the wrong direction slowly — the one percent per week problems that become catastrophic in month six.\nClose Postmortem Action Items: A postmortem without verified completion of action items is a diary entry. Track action items to closure. If an action item is not completed within two sprints, it needs a sponsor or it needs to be removed.\nMake User Feedback Visible: User-reported issues should appear in the same dashboards as system metrics. If the only way to see user complaints is to search the ticket system, the feedback loop is broken before it starts. Surface user sentiment alongside CPU and latency.\nCelebrate Prevention, Not Heroism: When an engineer identifies an early signal and prevents an incident, that should receive more recognition than the midnight heroics that could have been avoided. Shift the incentive from response to prevention.\nA closed feedback loop turns data into improvement. An open one just collects observations.\nApplying The Scientific Method Broken feedback loops are a failure of the observation-to-action cycle. The scientific method provides a structure for repair:\nObserve: What signals exist? Which ones are being ignored? Which ones drove action in the last month?\nHypothesize: \u0026ldquo;If we review this dashboard weekly with a decision requirement, we will catch regressions two weeks earlier on average.\u0026rdquo;\nTest: Implement the practice on one dashboard or one signal category for one month.\nMeasure: Did the practice change behavior? Were regressions caught earlier? Was an incident prevented?\nIterate: Expand the practice or adjust based on what was learned. The goal is not to monitor everything. It is to act on the right things.\nCarl Sagan\u0026rsquo;s Baloney Detection Kit When a signal is ignored, the justifications often sound reasonable. Apply critical thinking to evaluate them:\n\u0026ldquo;It is probably nothing.\u0026rdquo; — On what evidence? Has this specific pattern preceded incidents before? \u0026ldquo;Probably\u0026rdquo; is not data.\n\u0026ldquo;Nobody else is worried about it.\u0026rdquo; — Consensus is not evidence. If a metric is trending badly and nobody is concerned, the gap is in awareness, not in the metric.\n\u0026ldquo;We will get to it when we have time.\u0026rdquo; — When has that ever happened? If there is no scheduled time to investigate, the statement is a polite way of saying \u0026ldquo;never.\u0026rdquo;\n\u0026ldquo;The alert has been firing for weeks and nothing happened.\u0026rdquo; — Survival bias. The alert has been firing because the condition exists. One week of non-failure does not mean the condition is safe.\n\u0026ldquo;The dashboard looked fine.\u0026rdquo; — Did it, or did nobody look at the trend line instead of the current value? Most catastrophic signals are invisible at the last data point.\nLooking away does not make the signal stop. It just delays the response.\nMoving Forward Together Every significant incident I have been part of had a precursor. A metric that drifted. A ticket that sat. An alert that fired but was auto-acked. The signal was almost never missing. The response was.\nFeedback loops are the nervous system of an engineering organization. When they work, the organization detects and corrects before users notice. When they break, the organization operates blind until something crashes hard enough to be visible through the noise.\nClosing the loop is not a technical problem. It is a practice problem. It requires the discipline to look at the data, the courage to act on incomplete information, and the honesty to admit when a dashboard has become decoration.\nWhat signals is your organization ignoring right now? What metric has been trending in the wrong direction for weeks that nobody has escalated? The answer to that question might be the incident you prevent tomorrow.\nReferences The DevOps Handbook by Gene Kim, Patrick Debois, John Willis, and Jez Humble Accelerate: The Science of Lean Software and DevOps by Nicole Forsgren, Jez Humble, and Gene Kim Google SRE Workbook — Monitoring Distributed Systems DORA: Generative organizational culture The Three Ways: Principles Underpinning DevOps Honeycomb: Observability 101 Etsy\u0026rsquo;s Debriefing Facilitation Guide ","permalink":"https://trinidadmarroquin.com/posts/devops-dirty-dozen/ignoring-feedback-loops/","section":"posts","summary":"Part 6 of the DevOps Dirty Dozen Series: Quod non reficitur, deficit — what is not renewed, deteriorates.\nInsight: Emphasizes the importance of constant improvement through feedback.\nEvery engineer has a story about the alert that fired too many times, the dashboard that showed a slow climb, the user complaint that got buried in a ticket queue. And every engineer has a story about the night that same ignored signal became a full-blown incident.\n","tags":["devops","sre","feedback-loops","monitoring","anomalies","incidents","observability"],"title":"The Signal You Ignored: The DevOps Ignoring Feedback Loops Anti-Pattern"},{"categories":["field-notes"],"content":"An audit script can find contexts and still print zero rows. That may be success, not a parser failure.\nFor Calico node IP audits, many scripts intentionally emit only mismatches between a Kubernetes node\u0026rsquo;s InternalIP and the Calico node annotation:\nprojectcalico.org/IPv4Address If there are no mismatches, the target files can be empty.\nSymptom The audit finds contexts:\nContexts: site-a-ops-rke2 site-a-prod-rke2 site-a-uat-rke2 But reports zero targets:\nCounts for DC=site-a: all: 0 prod: 0 uat: 0 qa: 0 dev: 0 unknown: 0 Before assuming the node query broke, inspect the selection logic.\nCommon Mismatch Filter The script may only print rows matching this condition:\n.calico != \u0026#34;\u0026#34; and (.calico | split(\u0026#34;/\u0026#34;)[0]) != .nodeIP That means it ignores:\nnodes with no Calico IPv4 annotation. nodes where the Calico IPv4 host portion matches the Kubernetes InternalIP. So 0 can mean:\nno remediation targets not:\nno nodes checked Confirm The Raw Node Data Use a direct query to print every node:\nkubectl --context site-a-ops-rke2 get nodes -o json \\ | jq -r \u0026#39;.items[] | [ .metadata.name, (.status.addresses[]? | select(.type == \u0026#34;InternalIP\u0026#34;) | .address), (.metadata.annotations[\u0026#34;projectcalico.org/IPv4Address\u0026#34;] // \u0026#34;NONE\u0026#34;), (.metadata.annotations[\u0026#34;projectcalico.org/IPv4VXLANTunnelAddr\u0026#34;] // \u0026#34;NONE\u0026#34;) ] | @tsv\u0026#39; \\ | column -t Healthy shape:\nnode-a 192.0.2.10 192.0.2.10/24 192.0.2.50 node-b 192.0.2.11 192.0.2.11/24 192.0.2.51 The host portion of projectcalico.org/IPv4Address matches InternalIP.\nWhen It Is A Problem Investigate if you see:\nnode-a 192.0.2.10 198.51.100.10/24 That means Calico selected a different interface or subnet than Kubernetes considers the node InternalIP.\nTypical remediation is not to patch every node annotation by hand. Prefer fixing the Tigera Installation autodetection policy so Calico selects the intended CIDR, then restart or roll the affected Calico components according to the platform runbook.\nImprove Audit Output Mismatch-only scripts are useful for automation, but confusing for humans. Add an optional verbose mode that prints all checked nodes with status:\nMATCH MISMATCH MISSING_ANNOTATION UNREACHABLE_CONTEXT Example output:\ncontext node internal_ip calico_ipv4 status site-a-ops-rke2 node-a 192.0.2.10 192.0.2.10/24 MATCH site-a-ops-rke2 node-b 192.0.2.11 198.51.100.10 MISMATCH That keeps machine-readable target files small while giving operators confidence that the audit actually checked nodes.\nOne practical implementation is a --show-all flag that writes an additional report while preserving the existing mismatch-only outputs:\ncalico-ip-audit/site-a/mismatches.tsv calico-ip-audit/site-a/targets.tsv calico-ip-audit/site-a/targets-prod.tsv calico-ip-audit/site-a/all-nodes.tsv Useful counters:\nchecked nodes matching nodes missing annotations unreachable contexts mismatch targets Add a small fixture test with fake kubectl output so the audit keeps producing both target files and all-node status files after future edits.\nOperating Rule For mismatch-only audits, zero targets is not enough evidence by itself.\nConfirm at least once that contexts are reachable and raw node annotations match the intended relationship:\nKubernetes InternalIP == Calico IPv4Address host portion ","permalink":"https://trinidadmarroquin.com/field-notes/calico-ip-audit-zero-targets/","section":"field-notes","summary":"An audit script can find contexts and still print zero rows. That may be success, not a parser failure.\nFor Calico node IP audits, many scripts intentionally emit only mismatches between a Kubernetes node\u0026rsquo;s InternalIP and the Calico node annotation:\nprojectcalico.org/IPv4Address If there are no mismatches, the target files can be empty.\nSymptom The audit finds contexts:\nContexts: site-a-ops-rke2 site-a-prod-rke2 site-a-uat-rke2 But reports zero targets:\nCounts for DC=site-a: all: 0 prod: 0 uat: 0 qa: 0 dev: 0 unknown: 0 Before assuming the node query broke, inspect the selection logic.\n","tags":["calico","kubernetes","rke2","networking","audit","operations"],"title":"Calico IP Audit Zero Targets Does Not Mean Zero Nodes"},{"categories":["field-notes"],"content":"A DNS remediation script that works for environment-wide inventory can break when the workflow moves to per-cluster inventory files.\nThe script needs to solve four separate problems:\ntarget the right Ansible group. name audit output correctly. avoid local Vault dependencies during password-based testing. report incomplete final audits instead of pretending the cluster is clean. Per-Cluster Targeting If the command accepts either an environment or an inventory path, parse the second argument carefully:\n./scripts/fix-dns-search-domain.sh --audit-only site-a ./ansible/inventory/site-a-ops-rke2.yaml For that input, derive:\ninventory: /repo/ansible/inventory/site-a-ops-rke2.yaml group: site_a_ops_rke2 audit file: dns-search-audit-site-a-ops-rke2-\u0026lt;timestamp\u0026gt;.csv Do not generate duplicate site names in the audit filename, and do not target the old environment group by accident.\nVault-Backed Inventory Variables Ansible inventory may define credentials through Vault-backed group vars:\nansible_user: \u0026#34;{{ _vault_node_ssh.secret.username }}\u0026#34; ansible_password: \u0026#34;{{ _vault_node_ssh.secret.password }}\u0026#34; ansible_become_password: \u0026#34;{{ ansible_password }}\u0026#34; That is fine in AWX or a fully prepared operator shell, but it can break a local wrapper if the Python environment does not have the Vault client libraries installed.\nFor a local password-based remediation wrapper, collect credentials directly and pass them as high-precedence extra vars:\nread -rsp \u0026#39;SSH password: \u0026#39; SSH_PASSWORD read -rsp \u0026#39;BECOME password[defaults to SSH password]: \u0026#39; BECOME_PASSWORD Write a temporary extra-vars file with restrictive permissions:\ncredentials_file=$(mktemp) chmod 600 \u0026#34;$credentials_file\u0026#34; Quote YAML values because passwords can contain punctuation:\nlocal value=\u0026#34;$1\u0026#34; value=${value//\\\u0026#39;/\\\u0026#39;\\\u0026#39;} printf \u0026#34;\u0026#39;%s\u0026#39;\u0026#34; \u0026#34;$value\u0026#34; } Then pass the file to Ansible:\nansible \u0026#34;$GROUP\u0026#34; \\ -i \u0026#34;$INVENTORY\u0026#34; \\ -u \u0026#34;$SSH_USER\u0026#34; \\ -b \\ -e \u0026#34;@$credentials_file\u0026#34; \\ -m shell \\ -a \u0026#39;\u0026lt;audit or remediation command\u0026gt;\u0026#39; Remove the file on exit:\ntrap \u0026#39;rm -f \u0026#34;$credentials_file\u0026#34;\u0026#39; EXIT Remediation Shape For resolver search-domain drift where netplan has no search entry but systemd-resolved still exposes one, remediation should clear resolved domains explicitly:\n/etc/systemd/resolved.conf.d/99-clear-search-domains.conf Domains= Then restart systemd-resolved, apply or generate netplan as appropriate, and audit again.\nExpected clean state:\nresolv_conf_search: . netplan_search: [] or NONE, depending file presence resolv_conf_type: symlink:/run/systemd/resolve/stub-resolv.conf Final Audit Can Hang After remediation, a second full audit may stall on one host. That does not always mean remediation is still running.\nCheck whether the script already reached final audit:\n== Final audit == If so, the mutating phase is done and the current Ansible process is only collecting evidence.\nIf interrupted, the audit should still report the partial result:\nWARNING: ansible audit command exited with status 99 WARNING: audited 10 of 11 hosts. Check warnings/failures above. That row-count guardrail matters. Ten clean rows out of eleven is not a clean cluster.\nNotReady Nodes If one host is missing from the final audit, compare with Kubernetes readiness:\nkubectl --context site-a-ops-rke2 get nodes -o wide A NotReady node may still be present in inventory and targeted by Ansible. Keep it visible, but do not let it block interpretation of reachable hosts.\nRecommended follow-up:\nkubectl describe node site-a-worker-2 kubectl get node site-a-worker-2 -o jsonpath=\u0026#39;{range .status.conditions[*]}{.type}={.status} {.reason}{\u0026#34;\\n\u0026#34;}{end}\u0026#39; Operating Rule DNS remediation is not complete until the final audit covers every targeted host.\nIf a host is unreachable or NotReady, record the partial success, isolate the missing host, and rerun audit with an explicit limit or after node recovery.\n","permalink":"https://trinidadmarroquin.com/field-notes/dns-search-remediation-per-cluster-inventory/","section":"field-notes","summary":"A DNS remediation script that works for environment-wide inventory can break when the workflow moves to per-cluster inventory files.\nThe script needs to solve four separate problems:\ntarget the right Ansible group. name audit output correctly. avoid local Vault dependencies during password-based testing. report incomplete final audits instead of pretending the cluster is clean. Per-Cluster Targeting If the command accepts either an environment or an inventory path, parse the second argument carefully:\n","tags":["dns","ansible","rke2","inventory","systemd-resolved","operations"],"title":"DNS Search Remediation With Per-Cluster Ansible Inventory"},{"categories":["field-notes"],"content":"Inventory generation is only useful if the output matches the repo\u0026rsquo;s actual inventory contract.\nA script can successfully read kubectl get nodes and still generate inventory that the playbooks cannot use. The important details are usually group names, expected child groups, node roles, load balancer entries, and local conventions around readiness metadata.\nProblem A per-cluster RKE2 inventory workflow needed output like:\nsite_a_ops_rke2 rke2_servers rke2_agents api_lb But the generator output did not match the checked-in inventory shape. Common breaks included:\nkube context names with dashes where Ansible group vars expected underscores. missing rke2_servers and rke2_agents groups. missing API load balancer inventory entries. missing or inconsistent rke2_node_ready values. requiring explicit kubeconfig and environment flags even when the output filename already identified the target. Inputs Use Kubernetes as the live source for nodes:\nkubectl --context site-a-ops-rke2 get nodes -o wide Extract the facts the inventory actually needs:\nnode name internal IP Kubernetes roles Ready or NotReady condition cluster/context name environment inferred from output path or context Output Contract For a per-cluster inventory, generate stable groups first:\nall: children: site_a_ops_rke2: children: rke2_servers: rke2_agents: api_lb: Then classify nodes by role:\ncontrol-plane / master / etcd -\u0026gt; rke2_servers worker -\u0026gt; rke2_agents API load balancers -\u0026gt; api_lb Include readiness as metadata, not as a reason to silently omit the node:\nsite-a-worker-2: ansible_host: 192.0.2.20 rke2_node_ready: false That lets downstream tasks decide whether to skip, remediate, or alert.\nFilename-Based Inference If the caller writes to a per-cluster inventory path, infer what can be safely inferred:\n./ansible/generate-inventory.sh \\ ~/.kube/config \\ --output ansible/inventory/site-a-ops-rke2.yaml The script can derive:\ncontext: site-a-ops-rke2 group name: site_a_ops_rke2 environment: ops Do not require redundant flags unless ambiguity remains.\nTest With A Fake kubectl Inventory generators should have tests because small formatting changes break downstream playbooks.\nA simple fixture pattern:\nTMPDIR=$(mktemp -d) mkdir -p \u0026#34;$TMPDIR/bin\u0026#34; cat \u0026gt; \u0026#34;$TMPDIR/bin/kubectl\u0026#34; \u0026lt;\u0026lt;\u0026#39;EOF\u0026#39; #!/usr/bin/env bash case \u0026#34;$*\u0026#34; in *\u0026#34;config get-contexts\u0026#34;*) printf \u0026#39;site-a-ops-rke2\\n\u0026#39; ;; *\u0026#34;get nodes\u0026#34;*) cat ./testdata/nodes-site-a-ops.json ;; *) exit 1 ;; esac EOF chmod +x \u0026#34;$TMPDIR/bin/kubectl\u0026#34; PATH=\u0026#34;$TMPDIR/bin:$PATH\u0026#34; ./ansible/generate-inventory.sh --output /tmp/site-a-ops-rke2.yaml Then assert the output contains required groups and fields:\ngrep -q \u0026#39;site_a_ops_rke2:\u0026#39; /tmp/site-a-ops-rke2.yaml grep -q \u0026#39;rke2_servers:\u0026#39; /tmp/site-a-ops-rke2.yaml grep -q \u0026#39;rke2_agents:\u0026#39; /tmp/site-a-ops-rke2.yaml grep -q \u0026#39;api_lb:\u0026#39; /tmp/site-a-ops-rke2.yaml grep -q \u0026#39;rke2_node_ready:\u0026#39; /tmp/site-a-ops-rke2.yaml Validation After generation, validate Ansible can parse the file:\nansible-inventory -i ansible/inventory/site-a-ops-rke2.yaml --graph Then verify host counts against Kubernetes:\nkubectl --context site-a-ops-rke2 get nodes --no-headers | wc -l ansible site_a_ops_rke2 -i ansible/inventory/site-a-ops-rke2.yaml --list-hosts The counts do not need to match if inventory includes API load balancers, but the difference should be explainable.\nOperating Rule Do not generate inventory as a loose YAML export.\nGenerate the exact shape expected by playbooks and group_vars, test that shape with a fake kubectl, and keep NotReady nodes visible as inventory facts instead of hiding them.\n","permalink":"https://trinidadmarroquin.com/field-notes/rke2-per-cluster-inventory-generation/","section":"field-notes","summary":"Inventory generation is only useful if the output matches the repo\u0026rsquo;s actual inventory contract.\nA script can successfully read kubectl get nodes and still generate inventory that the playbooks cannot use. The important details are usually group names, expected child groups, node roles, load balancer entries, and local conventions around readiness metadata.\nProblem A per-cluster RKE2 inventory workflow needed output like:\nsite_a_ops_rke2 rke2_servers rke2_agents api_lb But the generator output did not match the checked-in inventory shape. Common breaks included:\n","tags":["ansible","kubernetes","rke2","inventory","automation","operations"],"title":"RKE2 Per-Cluster Inventory Generation From Kubeconfig"},{"categories":["field-notes"],"content":"A DNS audit can select the right hosts and still produce a misleading report.\nThe failure mode is usually a pipeline like this:\nansible \u0026#34;$GROUP\u0026#34; ... \\ | sed -n \u0026#39;s/.*CSV|//p\u0026#39; \\ | awk -F\u0026#39;|\u0026#39; \u0026#39;{print $1 \u0026#34;,\u0026#34; $2 \u0026#34;,\u0026#34; $3 \u0026#34;,\u0026#34; $4}\u0026#39; \\ \u0026gt;\u0026gt; \u0026#34;$OUT\u0026#34; That captures successful CSV|... markers, but it hides everything else. If half the hosts fail SSH or sudo, the CSV simply has fewer rows. The audit looks clean only because failed hosts disappeared.\nSymptom The host list is correct:\nhosts (13): cluster-a-cp-1 cluster-a-cp-2 cluster-a-cp-3 cluster-a-worker-1 ... cluster-a-worker-10 But the audit CSV has fewer data rows:\ntargeted hosts: 13 csv rows: 6 That is not partial success. That is an incomplete audit.\nFix The Pipeline Capture raw Ansible output first:\nAUDIT_RAW=\u0026#34;$(mktemp)\u0026#34; set +e ANSIBLE_NOCOLOR=1 \\ ANSIBLE_STDOUT_CALLBACK=default \\ ANSIBLE_HOST_KEY_CHECKING=False \\ ansible \u0026#34;$GROUP\u0026#34; \\ -i \u0026#34;$INVENTORY\u0026#34; \\ -u \u0026#34;$ANSIBLE_USER\u0026#34; \\ -b -kK \\ -m shell -a \u0026#39; ACTUAL_HOSTNAME=$(hostname -s) RESOLV_SEARCH=$(awk \u0026#34;/^search/{\\$1=\\\u0026#34;\\\u0026#34;; sub(/^ /,\\\u0026#34;\\\u0026#34;); print; exit} /^domain/{print \\$2; exit}\u0026#34; /etc/resolv.conf) [ -z \u0026#34;$RESOLV_SEARCH\u0026#34; ] \u0026amp;\u0026amp; RESOLV_SEARCH=\u0026#34;NONE\u0026#34; NETPLAN_SEARCH=$(grep -R \u0026#34;search:\u0026#34; /etc/netplan/*.yaml /etc/netplan/*.yml 2\u0026gt;/dev/null | sed \u0026#34;s/.*search:[[:space:]]*//\u0026#34; | paste -sd \u0026#34;;\u0026#34; -) [ -z \u0026#34;$NETPLAN_SEARCH\u0026#34; ] \u0026amp;\u0026amp; NETPLAN_SEARCH=\u0026#34;NONE\u0026#34; if [ -L /etc/resolv.conf ]; then RESOLV_TYPE=\u0026#34;symlink:$(readlink -f /etc/resolv.conf)\u0026#34; else RESOLV_TYPE=\u0026#34;static_file\u0026#34; fi printf \u0026#34;CSV|{{ inventory_hostname }}|%s|%s|%s|%s\\n\u0026#34; \u0026#34;$ACTUAL_HOSTNAME\u0026#34; \u0026#34;$RESOLV_SEARCH\u0026#34; \u0026#34;$NETPLAN_SEARCH\u0026#34; \u0026#34;$RESOLV_TYPE\u0026#34; \u0026#39; \u0026gt; \u0026#34;$AUDIT_RAW\u0026#34; 2\u0026gt;\u0026amp;1 AUDIT_RC=$? set -e Then parse CSV rows from the raw file:\nsed -n \u0026#39;s/.*CSV|//p\u0026#39; \u0026#34;$AUDIT_RAW\u0026#34; \\ | awk -F\u0026#39;|\u0026#39; \u0026#39;{print $1 \u0026#34;,\u0026#34; $2 \u0026#34;,\u0026#34; $3 \u0026#34;,\u0026#34; $4 \u0026#34;,\u0026#34; $5}\u0026#39; \\ | sort -t, -k1,1 \\ \u0026gt;\u0026gt; \u0026#34;$OUT\u0026#34; Print Failures Explicitly Do not discard the non-CSV output. Show it as audit evidence:\nif grep -v \u0026#39;CSV|\u0026#39; \u0026#34;$AUDIT_RAW\u0026#34; | grep -q \u0026#39;[^[:space:]]\u0026#39;; then echo echo \u0026#34;== Ansible warnings/failures ==\u0026#34; grep -v \u0026#39;CSV|\u0026#39; \u0026#34;$AUDIT_RAW\u0026#34; fi This exposes errors such as:\nUNREACHABLE! =\u0026gt; Permission denied (publickey,password,keyboard-interactive) Missing sudo password Failed to connect to the host via ssh Compare Targeted Hosts To CSV Rows Count targeted hosts before the audit:\nTARGETED_HOSTS=$(ansible \u0026#34;$GROUP\u0026#34; -i \u0026#34;$INVENTORY\u0026#34; --list-hosts \\ | awk \u0026#39;/hosts \\([0-9]+\\):/ {gsub(/[():]/, \u0026#34;\u0026#34;, $2); print $2}\u0026#39;) Count CSV data rows after the audit:\nCSV_ROWS=$(awk \u0026#39;NR \u0026gt; 1 {count++} END {print count + 0}\u0026#39; \u0026#34;$OUT\u0026#34;) Warn on mismatch:\nif [ \u0026#34;$CSV_ROWS\u0026#34; -lt \u0026#34;$TARGETED_HOSTS\u0026#34; ]; then echo echo \u0026#34;WARNING: audit produced $CSV_ROWS CSV rows for $TARGETED_HOSTS targeted hosts\u0026#34; echo \u0026#34;Some hosts failed before returning DNS state. Treat this audit as incomplete.\u0026#34; fi Interpret SSH And Become Flags For Ansible audit scripts, auth flags matter:\n-k ask for SSH password -K ask for sudo/become password -b enable become/sudo Useful combinations:\npassword SSH + password sudo: -b -kK SSH key + password sudo: -b -K SSH key + passwordless sudo: -b If -u ubuntu appears ignored, check inventory or group variables for ansible_user. A group var can force the remote user unless explicitly overridden with an extra var:\nansible \u0026#34;$GROUP\u0026#34; -i \u0026#34;$INVENTORY\u0026#34; -e ansible_user=ubuntu -m ping Audit Finish Criteria A DNS search-domain audit is complete only when:\nselected host count matches expected inventory CSV data row count equals selected host count raw Ansible output has no unreachable or failed hosts drift output is reviewed after row-count validation If the row count is short, fix SSH/bootstrap/sudo access first. Do not treat the DNS state as clean just because missing hosts did not write CSV rows.\nRelated: DNS Search Remediation With Per-Cluster Ansible Inventory covers per-cluster inventory arguments, local credential overrides for Vault-backed Ansible vars, and final-audit handling after remediation.\n","permalink":"https://trinidadmarroquin.com/field-notes/dns-search-domain-audit-failure-visibility/","section":"field-notes","summary":"A DNS audit can select the right hosts and still produce a misleading report.\nThe failure mode is usually a pipeline like this:\nansible \u0026#34;$GROUP\u0026#34; ... \\ | sed -n \u0026#39;s/.*CSV|//p\u0026#39; \\ | awk -F\u0026#39;|\u0026#39; \u0026#39;{print $1 \u0026#34;,\u0026#34; $2 \u0026#34;,\u0026#34; $3 \u0026#34;,\u0026#34; $4}\u0026#39; \\ \u0026gt;\u0026gt; \u0026#34;$OUT\u0026#34; That captures successful CSV|... markers, but it hides everything else. If half the hosts fail SSH or sudo, the CSV simply has fewer rows. The audit looks clean only because failed hosts disappeared.\n","tags":["dns","ansible","audit","netplan","systemd-resolved","operations"],"title":"DNS Search Domain Audit Failure Visibility"},{"categories":["field-notes"],"content":"A VM can have the correct FQDN intent and still receive the wrong resolver search suffix.\nThe trap is treating these as the same setting:\nvm_domain -\u0026gt; identity/FQDN domain dns_search -\u0026gt; resolver search suffix list They are related, but they are not the same control.\nSymptom An environment sets DNS search suffixes to empty:\ndns_search = \u0026#34;[]\u0026#34; But new vSphere VMs still boot with a resolver search domain such as:\nsearch corp.example.com The node audit shows drift even though the Terraform input looked correct:\nresolv_conf_search = corp.example.com netplan_search = [corp.example.com] Root Cause Pattern The environment may define dns_search, but the module might not consume it.\nThe broken pattern looks like this:\nmodule \u0026#34;vm_group\u0026#34; { source = \u0026#34;../../../modules/vsphere-vm-group\u0026#34; vm_domain = var.vm_domain dns_server_list = var.dns_server_list # dns_search exists in variables.tf, but is not passed here } Then the module derives vSphere guest customization search suffixes from vm_domain:\ncustomize { linux_options { host_name = each.value.name domain = var.vm_domain } network_interface { ipv4_address = each.value.ipv4_address ipv4_netmask = tonumber(each.value.ipv4_netmask) } ipv4_gateway = var.ipv4_gateway dns_server_list = var.dns_server_list dns_suffix_list = [var.vm_domain] } In that shape, dns_search = \u0026quot;[]\u0026quot; is a red herring. It exists, but it does not control anything.\nProve The Active Path Search the Terraform code first:\nrg \u0026#39;dns_search|dns_suffix_list|vm_domain|network-config|guestinfo\u0026#39; . Look for two possible paths:\nvSphere guest customization: customize.dns_suffix_list cloud-init guestinfo: network-config.yaml nameservers.search If guestinfo network config is disabled, the cloud-init template is not the active source even if it contains a search entry.\nInspect Plan JSON Without Reading Secrets If a plan JSON already exists, inspect only the resolved fields needed for evidence:\njq -r \u0026#39; .configuration.root_module.module_calls.vm_group.expressions.vm_domain.references, .variables.vm_domain, .variables.dns_search \u0026#39; tfplan.json Then inspect vSphere customization values:\njq -r \u0026#39; .planned_values.root_module.child_modules[]? | select(.address == \u0026#34;module.vm_group\u0026#34;) | .resources[]? | select(.type == \u0026#34;vsphere_virtual_machine\u0026#34;) | .values.clone[0].customize[0] | {dns_suffix_list, linux_options} \u0026#39; tfplan.json Useful evidence shape:\n{ \u0026#34;dns_search\u0026#34;: { \u0026#34;value\u0026#34;: \u0026#34;[]\u0026#34; }, \u0026#34;dns_suffix_list\u0026#34;: [\u0026#34;corp.example.com\u0026#34;], \u0026#34;linux_options\u0026#34;: [{ \u0026#34;domain\u0026#34;: \u0026#34;corp.example.com\u0026#34; }] } That proves the search suffix came from module logic, not from the intended empty dns_search input.\nMinimal Module Fix Pass dns_search into the module:\nmodule \u0026#34;vm_group\u0026#34; { source = \u0026#34;../../../modules/vsphere-vm-group\u0026#34; vm_domain = var.vm_domain dns_server_list = var.dns_server_list dns_search = var.dns_search } Add a module variable that preserves the existing bracketed string interface:\nvariable \u0026#34;dns_search\u0026#34; { type = string description = \u0026#34;DNS search suffixes in bracketed form, for example [] or [corp.example.com]\u0026#34; default = \u0026#34;[]\u0026#34; } Parse it once:\nlocals { dns_search_suffixes = compact([ for suffix in split(\u0026#34;,\u0026#34;, trim(var.dns_search, \u0026#34;[] \u0026#34;)) : trimspace(replace(suffix, \u0026#34;\\\u0026#34;\u0026#34;, \u0026#34;\u0026#34;)) ]) } Use it for vSphere customization:\ndns_suffix_list = local.dns_search_suffixes If cloud-init guestinfo network config is enabled, use the same parsed value there too. Do not maintain separate search suffix logic for vSphere customization and cloud-init.\nValidate The Behavior Run formatting and validation:\nterraform fmt main.tf ../../../modules/vsphere-vm-group/main.tf ../../../modules/vsphere-vm-group/variables.tf terraform validate Then inspect a new plan if provider credentials are available. If not, use existing plan/state evidence and document that a fresh plan was blocked by missing provider credentials.\nExpected behavior:\ndns_search = \u0026#34;[]\u0026#34; -\u0026gt; dns_suffix_list = [] dns_search = \u0026#34;[corp.example.com]\u0026#34; -\u0026gt; dns_suffix_list = [\u0026#34;corp.example.com\u0026#34;] vm_domain = \u0026#34;corp.example.com\u0026#34; -\u0026gt; still controls VM FQDN/domain intent Operating Rule Do not let a variable name create false confidence.\nFor infrastructure modules, every important input needs a traceable path:\nroot variable -\u0026gt; module argument -\u0026gt; module variable -\u0026gt; provider argument -\u0026gt; plan/state -\u0026gt; guest audit If the chain breaks anywhere, the variable is documentation, not behavior.\n","permalink":"https://trinidadmarroquin.com/field-notes/terraform-vsphere-dns-search-suffix-ownership/","section":"field-notes","summary":"A VM can have the correct FQDN intent and still receive the wrong resolver search suffix.\nThe trap is treating these as the same setting:\nvm_domain -\u0026gt; identity/FQDN domain dns_search -\u0026gt; resolver search suffix list They are related, but they are not the same control.\nSymptom An environment sets DNS search suffixes to empty:\ndns_search = \u0026#34;[]\u0026#34; But new vSphere VMs still boot with a resolver search domain such as:\nsearch corp.example.com The node audit shows drift even though the Terraform input looked correct:\n","tags":["terraform","vsphere","dns","netplan","cloud-init","operations"],"title":"Terraform vSphere DNS Search Suffix Ownership"},{"categories":["field-notes"],"content":"When NetBox inventory moves from DCIM devices to virtualization VMs, tags need their own audit.\nA tag can exist and still be wrong for the new object model. If a cluster tag is scoped only to DCIM devices or IP addresses, migrated virtualization VMs may not be taggable or discoverable through normal cluster filters.\nAudit Cluster Tag Scopes Set API variables:\nNETBOX_URL=\u0026#34;${NETBOX_URL:-https://netbox.example.com}\u0026#34; TOKEN=\u0026#34;${NETBOX_TOKEN:-}\u0026#34; case \u0026#34;$TOKEN\u0026#34; in nbt_*) AUTH=\u0026#34;Authorization: Bearer $TOKEN\u0026#34; ;; *) AUTH=\u0026#34;Authorization: Token $TOKEN\u0026#34; ;; esac List cluster-* tags missing virtualization VM scope:\ncurl -fsS -H \u0026#34;$AUTH\u0026#34; \u0026#34;$NETBOX_URL/api/extras/tags/?limit=1000\u0026#34; \\ | jq -r \u0026#39;.results[] | select(.slug | startswith(\u0026#34;cluster-\u0026#34;)) | [.slug, (.object_types | join(\u0026#34;,\u0026#34;))] | @tsv\u0026#39; \\ | awk -F \u0026#39;\\t\u0026#39; \u0026#39;$2 !~ /virtualization\\.virtualmachine/ {print}\u0026#39; Expected output after cleanup:\n\u0026lt;no output\u0026gt; Why It Matters Cluster tags are often used for:\nfiltering nodes in the NetBox UI. scoping scripts to one cluster. selecting inventory for CSV export. validating migrated VM counts. grouping Terraform import targets. If the tag does not support virtualization.virtualmachine, those workflows silently become incomplete.\nRepair Pattern Use a script or API patch that preserves existing object types and appends the VM type.\nTarget state:\ndcim.device ipam.ipaddress virtualization.virtualmachine Do not replace the whole object type list unless the script first reads the current tag and merges existing values.\nVerify Tagged VM Counts After scope repair and tag sync, verify VM counts by cluster tag:\nfor slug in cluster-site-a-prod-rke2 cluster-site-a-uat-rke2; do printf \u0026#39;%s\\t\u0026#39; \u0026#34;$slug\u0026#34; curl -fsS -H \u0026#34;$AUTH\u0026#34; \\ \u0026#34;$NETBOX_URL/api/virtualization/virtual-machines/?tag=$slug\u0026amp;limit=1\u0026#34; \\ | jq -r \u0026#39;.count\u0026#39; done Example output:\ncluster-site-a-prod-rke2 18 cluster-site-a-uat-rke2 14 The exact counts are less important than matching the expected cluster inventory.\nFinal Check Run both checks:\nmissing VM tag scope: none tagged VM counts: match expected inventory Only then treat tag migration as complete.\n","permalink":"https://trinidadmarroquin.com/field-notes/netbox-cluster-tag-scope-audit/","section":"field-notes","summary":"When NetBox inventory moves from DCIM devices to virtualization VMs, tags need their own audit.\nA tag can exist and still be wrong for the new object model. If a cluster tag is scoped only to DCIM devices or IP addresses, migrated virtualization VMs may not be taggable or discoverable through normal cluster filters.\nAudit Cluster Tag Scopes Set API variables:\nNETBOX_URL=\u0026#34;${NETBOX_URL:-https://netbox.example.com}\u0026#34; TOKEN=\u0026#34;${NETBOX_TOKEN:-}\u0026#34; case \u0026#34;$TOKEN\u0026#34; in nbt_*) AUTH=\u0026#34;Authorization: Bearer $TOKEN\u0026#34; ;; *) AUTH=\u0026#34;Authorization: Token $TOKEN\u0026#34; ;; esac List cluster-* tags missing virtualization VM scope:\n","tags":["netbox","kubernetes","ipam","audit","operations"],"title":"NetBox Cluster Tag Scope Audit"},{"categories":["field-notes"],"content":"Use a dry-run-first workflow when migrating VMware inventory from NetBox DCIM devices to virtualization VMs.\nThe script should be idempotent: running it again should report already-migrated or tags-ok, not create duplicates.\nRecommended Script Modes Useful modes for a migration helper:\n--cluster-tag \u0026lt;slug\u0026gt; process one cluster tag --all-cluster-devices discover all DCIM devices with cluster tags --all-matching-vms validate all matching migrated VMs --sync-tags-only sync tags without creating VMs --ensure-vm-tag-scope allow the selected cluster tag on VMs --ensure-all-cluster-tag-scopes repair all cluster tag scopes --apply perform writes after dry-run review Keep dry-run as the default. Make writes require an explicit flag.\nStep 1: Cluster-Scoped Dry Run python3 Generated/migrate-dcim-devices-to-vms.py \\ --cluster-tag cluster-site-a-prod-rke2 \\ --ensure-vm-tag-scope \\ \u0026gt; Generated/migrate-cluster-site-a-prod-rke2.dry-run.tsv 2\u0026gt;\u0026amp;1 Summarize the output:\nrg -c \u0026#39;^PLAN\\tcreate-vm\u0026#39; Generated/migrate-cluster-site-a-prod-rke2.dry-run.tsv rg -c \u0026#39;^DONE\\talready-migrated\u0026#39; Generated/migrate-cluster-site-a-prod-rke2.dry-run.tsv rg -c \u0026#39;^SKIP\u0026#39; Generated/migrate-cluster-site-a-prod-rke2.dry-run.tsv Review skips before applying.\nStep 2: Apply One Cluster python3 Generated/migrate-dcim-devices-to-vms.py \\ --cluster-tag cluster-site-a-prod-rke2 \\ --ensure-vm-tag-scope \\ --apply \\ \u0026gt; Generated/migrate-cluster-site-a-prod-rke2.apply.tsv 2\u0026gt;\u0026amp;1 Check writes:\nrg -c \u0026#39;^DONE\\tcreate-vm\u0026#39; Generated/migrate-cluster-site-a-prod-rke2.apply.tsv rg -c \u0026#39;^DONE\\talready-migrated\u0026#39; Generated/migrate-cluster-site-a-prod-rke2.apply.tsv rg -c \u0026#39;^ERROR\u0026#39; Generated/migrate-cluster-site-a-prod-rke2.apply.tsv Step 3: Sync Tags Separately VM creation and tag synchronization should be separate operations.\nDry run:\npython3 Generated/migrate-dcim-devices-to-vms.py \\ --cluster-tag cluster-site-a-prod-rke2 \\ --sync-tags-only \\ --ensure-vm-tag-scope \\ \u0026gt; Generated/sync-cluster-site-a-prod-rke2.dry-run.tsv 2\u0026gt;\u0026amp;1 Apply:\npython3 Generated/migrate-dcim-devices-to-vms.py \\ --cluster-tag cluster-site-a-prod-rke2 \\ --sync-tags-only \\ --ensure-vm-tag-scope \\ --apply \\ \u0026gt; Generated/sync-cluster-site-a-prod-rke2.apply.tsv 2\u0026gt;\u0026amp;1 Verify:\nrg -c \u0026#39;^DONE\\tsync-tags\u0026#39; Generated/sync-cluster-site-a-prod-rke2.apply.tsv rg -c \u0026#39;^DONE\\ttags-ok\u0026#39; Generated/sync-cluster-site-a-prod-rke2.apply.tsv Step 4: Re-Run For Idempotency After apply, run the dry run again.\nHealthy output trends:\ncreate-vm: 0 sync-tags: 0 already-migrated: expected count tags-ok: expected count errors: 0 This is the evidence that the migration can be resumed safely.\nScript Hardening Notes For long-running inventory scripts, add:\nimmediate log flushing. a reasonable HTTP timeout. tag scope caching. explicit dry-run/apply output prefixes. support for one-cluster scoping. support for tag-only repair. Those small features make the script operable during a real migration window.\nOperating Rule Do not migrate the whole fleet first.\nMigrate one cluster, sync tags, verify counts, rerun for idempotency, then expand the scope.\n","permalink":"https://trinidadmarroquin.com/field-notes/netbox-dcim-device-to-vm-migration-dry-run-pattern/","section":"field-notes","summary":"Use a dry-run-first workflow when migrating VMware inventory from NetBox DCIM devices to virtualization VMs.\nThe script should be idempotent: running it again should report already-migrated or tags-ok, not create duplicates.\nRecommended Script Modes Useful modes for a migration helper:\n--cluster-tag \u0026lt;slug\u0026gt; process one cluster tag --all-cluster-devices discover all DCIM devices with cluster tags --all-matching-vms validate all matching migrated VMs --sync-tags-only sync tags without creating VMs --ensure-vm-tag-scope allow the selected cluster tag on VMs --ensure-all-cluster-tag-scopes repair all cluster tag scopes --apply perform writes after dry-run review Keep dry-run as the default. Make writes require an explicit flag.\n","tags":["netbox","migration","automation","audit","operations"],"title":"NetBox DCIM Device To VM Migration Dry Run Pattern"},{"categories":["notes"],"content":"Inventory migrations are easy to underestimate.\nChanging a NetBox object from a DCIM device to a virtualization VM sounds like a data cleanup task. In practice, it touches ownership boundaries: IP assignments, primary IPs, cluster tags, Terraform imports, and the difference between audit-only work and infrastructure mutation.\nThe safe way to run this kind of migration is to treat it like an operational change, not a CSV edit.\nThe Problem Some VMware-backed Kubernetes nodes were represented in NetBox as DCIM devices. That worked for basic inventory lookup, but it did not match the automation model.\nTerraform was managing vSphere VMs through a shared module and using NetBox virtualization resources for source-of-truth state:\nnetbox_virtual_machine netbox_interface netbox_ip_address netbox_primary_ip That meant the NetBox object model had to move from:\nDCIM device + DCIM interface + IP to:\nvirtualization VM + VM interface + IP + primary IPv4 The migration also had to preserve cluster tags so operators could still answer simple questions like “which VMs belong to this cluster?”\nThe Control Plane For The Migration The migration script needed more than a create loop.\nThe useful controls were:\n--apply only when the dry run was reviewed. --cluster-tag to scope work to one cluster at a time. --all-cluster-devices for broad inventory discovery. --all-matching-vms for idempotent follow-up checks. --sync-tags-only to repair tags after VM creation. --ensure-vm-tag-scope to make a cluster tag usable on virtualization VMs. --ensure-all-cluster-tag-scopes for fleet-wide tag-scope hygiene. The important pattern is that create, tag scope, and tag sync are separate operations. Separating them makes the output easier to review and reduces the blast radius of each apply.\nDry Run First A dry run should produce evidence, not just a yes/no result.\nExample output classes:\nPLAN create-vm PLAN move-ip-to-vm-interface PLAN set-vm-primary-ip PLAN unset-device-primary-ip PLAN sync-tags DONE already-migrated DONE tags-ok SKIP missing-required-field That output lets you count planned work before making changes:\nrg -c \u0026#39;^PLAN\\tcreate-vm\u0026#39; Generated/migrate-cluster-a.dry-run.tsv rg -c \u0026#39;^DONE\\talready-migrated\u0026#39; Generated/migrate-cluster-a.dry-run.tsv rg -c \u0026#39;^SKIP\u0026#39; Generated/migrate-cluster-a.dry-run.tsv The goal is to make the dry run boring before the apply.\nApply In Small Batches For cluster-scoped migration, the operating loop is:\n1. dry run one cluster tag 2. review planned creates/skips 3. apply one cluster tag 4. dry run tag sync 5. apply tag sync 6. verify tagged virtualization VM count 7. move to the next cluster Do not jump from one successful cluster to a fleet-wide write unless the script has already proven idempotent behavior across the same object model.\nTags Are Part Of The Data Model One subtle failure mode is that a cluster tag may exist, but only be valid for DCIM devices or IP addresses.\nIf the tag is not scoped for virtualization VMs, the migrated VM can exist but not be discoverable through the same tag-based workflows operators used before.\nThe audit for that is simple:\ncurl -fsS -H \u0026#34;$AUTH\u0026#34; \u0026#34;$NETBOX_URL/api/extras/tags/?limit=1000\u0026#34; \\ | jq -r \u0026#39;.results[] | select(.slug | startswith(\u0026#34;cluster-\u0026#34;)) | [.slug, (.object_types | join(\u0026#34;,\u0026#34;))] | @tsv\u0026#39; \\ | awk -F \u0026#39;\\t\u0026#39; \u0026#39;$2 !~ /virtualization\\.virtualmachine/ {print}\u0026#39; Expected output:\n\u0026lt;no output\u0026gt; That means every cluster-* tag can be applied to virtualization VMs.\nTerraform Comes After The Inventory Shape Is Correct Once NetBox virtualization objects exist, Terraform state can be imported into the shared module shape.\nThe safe checkpoint looks like this:\nNetBox VM resources: no-op NetBox interface resources: no-op NetBox IP resources: no-op NetBox primary IP: no-op vSphere VM resources: update That is a useful result. It means NetBox is reconciled and the remaining drift is isolated to vSphere.\nIt does not mean the full plan should be applied.\nWhen Drift Means Replacement Existing VMs often carry old-template drift:\nCPU and memory hot-add settings. datastore placement. port group attachment. missing or different guestinfo metadata. clone/customize metadata differences. disk label mismatches. extra worker disks outside the module model. For Kubernetes nodes, especially control-plane and etcd members, applying those differences in place can be the wrong operational move. The better answer may be replacement-node lifecycle: build new nodes with the current module and template, join them, drain old nodes, and retire the legacy VMs one at a time.\nThe Practical Lesson A NetBox migration is not complete when the new objects exist.\nIt is complete when:\nobjects exist in the correct NetBox model IPs are attached to the correct VM interfaces primary IPs are set tags are synced tag scopes support the new object type Terraform imports are clean full-plan drift is classified unsafe vSphere changes are blocked That is the difference between data cleanup and operational control.\nRelated Field Notes:\nNetBox Cluster Tag Scope Audit NetBox DCIM Device To VM Migration Dry Run Pattern Replacement Node Workflow After Terraform Import Drift ","permalink":"https://trinidadmarroquin.com/posts/netbox-virtualization-migration-operational-pattern/","section":"posts","summary":"Inventory migrations are easy to underestimate.\nChanging a NetBox object from a DCIM device to a virtualization VM sounds like a data cleanup task. In practice, it touches ownership boundaries: IP assignments, primary IPs, cluster tags, Terraform imports, and the difference between audit-only work and infrastructure mutation.\nThe safe way to run this kind of migration is to treat it like an operational change, not a CSV edit.\nThe Problem Some VMware-backed Kubernetes nodes were represented in NetBox as DCIM devices. That worked for basic inventory lookup, but it did not match the automation model.\n","tags":["netbox","terraform","vsphere","kubernetes","migration","audit","operations"],"title":"Operating A NetBox Virtualization Migration Without Losing Control"},{"categories":["field-notes"],"content":"Importing existing vSphere VMs into Terraform can produce a clean source-of-truth checkpoint and still leave a plan that should not be applied.\nThat is common when legacy Kubernetes nodes were built from an older template or outside the current module conventions.\nAudit Checkpoint A useful checkpoint looks like this:\nNetBox resources: no-op vSphere resources: update destroy actions: none This means NetBox ownership is reconciled, but Terraform still sees vSphere drift.\nDrift That Should Trigger Caution Do not treat update in-place as automatically safe.\nFor existing Kubernetes nodes, review drift such as:\nCPU hot-add changing state. memory hot-add changing state. datastore ID changes. network port group changes. imported VM markers changing. clone/customize metadata being added. cloud-init guestinfo metadata being added. timeout setting changes. disk label mismatches. disk add/remove mismatches. extra worker disks not modeled in the module. Any of these can be manageable in isolation. Together, they are a sign that the VM shape does not match the module\u0026rsquo;s intended lifecycle model.\nDecision If the VM is a legacy old-template node, prefer replacement over in-place reconciliation.\nReplacement is slower but safer:\nnew VM gets current template and module shape old VM remains untouched until workload is drained cluster health is checked at each step rollback is clearer Worker Node Sequence Start with workers when possible.\n1. Provision replacement worker with Terraform. 2. Bootstrap OS and RKE2 prerequisites. 3. Join worker to the cluster. 4. Verify node Ready condition. 5. Verify CNI, kube-proxy, DNS, and storage behavior. 6. Cordon old worker. 7. Drain old worker. 8. Delete old node from Kubernetes. 9. Retire old VM and clean source-of-truth state. 10. Repeat one worker at a time. Useful checks:\nkubectl get nodes -o wide kubectl get pods -A -o wide --field-selector spec.nodeName=worker-1 kubectl drain worker-1 --ignore-daemonsets --delete-emptydir-data Control-Plane And Etcd Sequence Control-plane and etcd nodes need stricter sequencing.\nBefore each replacement:\nkubectl get nodes kubectl get --raw=\u0026#39;/readyz?verbose\u0026#39; Check etcd health using the platform\u0026rsquo;s supported tooling.\nOperational rules:\nreplace one member at a time. preserve quorum. wait for the replacement to become healthy before touching the next node. do not reuse hostname or IP until old membership is safely removed. keep an etcd backup before membership work. Tool Boundaries Use each tool for its layer:\nTerraform provisions the replacement VM and NetBox ownership. Ansible or bootstrap scripts prepare the OS and RKE2 configuration. Kubernetes handles cordon, drain, and node deletion. RKE2/etcd tooling handles control-plane membership checks. Rancher or the platform UI confirms final cluster visibility. Finish Criteria For each replaced node:\nreplacement node Ready critical pods healthy storage attach/mount works if applicable old node drained old node removed from Kubernetes old VM retired NetBox reflects current VM ownership Terraform plan remains intentional Operating Rule A drift audit is allowed to end with “do not apply.”\nWhen Terraform reveals that legacy nodes do not match the current module shape, replacement-node lifecycle is often the safer reconciliation path.\n","permalink":"https://trinidadmarroquin.com/field-notes/replacement-node-workflow-after-terraform-import-drift/","section":"field-notes","summary":"Importing existing vSphere VMs into Terraform can produce a clean source-of-truth checkpoint and still leave a plan that should not be applied.\nThat is common when legacy Kubernetes nodes were built from an older template or outside the current module conventions.\nAudit Checkpoint A useful checkpoint looks like this:\nNetBox resources: no-op vSphere resources: update destroy actions: none This means NetBox ownership is reconciled, but Terraform still sees vSphere drift.\n","tags":["terraform","vsphere","kubernetes","rke2","drift","operations"],"title":"Replacement Node Workflow After Terraform Import Drift"},{"categories":["field-notes"],"content":"Velero has two parts: the CLI is installed on a management machine (your jumpbox or workstation) and issues commands to the cluster via kubectl; the server and node-agent run as pods inside the Kubernetes cluster and carry out the actual backup and restore work. All velero backup, restore, and schedule commands below assume the CLI is installed on a machine with kubectl access to the target cluster.\nPrerequisites Velero backs up Kubernetes resources and, with the restic/kopia integration, persistent volume data. The backup target must be an S3-compatible object store.\n# Install Velero CLI (Linux) curl -LO https://github.com/vmware-tanzu/velero/releases/latest/download/velero-v1.15.0-linux-amd64.tar.gz tar xzf velero-*-linux-amd64.tar.gz sudo mv velero-*/velero /usr/local/bin/ # Install server with restic for PV backup velero install \\ --provider aws \\ --bucket \u0026lt;bucket-name\u0026gt; \\ --prefix \u0026lt;optional-prefix\u0026gt; \\ --backup-location-config region=\u0026lt;region\u0026gt;,s3ForcePathStyle=true,s3Url=\u0026lt;endpoint\u0026gt; \\ --snapshot-location-config region=\u0026lt;region\u0026gt; \\ --secret-file ./credentials-velero \\ --use-node-agent \\ --wait # Verify velero version velero client config set --namespace velero Backup Ad-Hoc Backup # Backup all resources in a namespace velero backup create \u0026lt;name\u0026gt; --include-namespaces \u0026lt;ns\u0026gt; # Backup with PV data (requires node-agent) velero backup create \u0026lt;name\u0026gt; \\ --include-namespaces \u0026lt;ns\u0026gt; \\ --default-volumes-to-fs-backup # Backup specific resource types velero backup create \u0026lt;name\u0026gt; \\ --include-namespaces \u0026lt;ns\u0026gt; \\ --include-resources deployments,configmaps,secrets,pvc # Exclude specific resources velero backup create \u0026lt;name\u0026gt; \\ --include-namespaces \u0026lt;ns\u0026gt; \\ --exclude-resources events,events.events.k8s.io # Backup entire cluster (use with caution in large clusters) velero backup create \u0026lt;name\u0026gt; --exclude-namespaces velero,kube-system Label-Based Backup # Backup resources matching label velero backup create \u0026lt;name\u0026gt; \\ --selector app=\u0026lt;app-name\u0026gt; \\ --include-namespaces \u0026lt;ns\u0026gt; # Or use a label selector for opt-in backup velero backup create \u0026lt;name\u0026gt; \\ --include-namespaces \u0026lt;ns\u0026gt; \\ --selector velero-backup=true Schedule # Daily backup with 7-day retention velero schedule create daily \\ --schedule \u0026#34;0 2 * * *\u0026#34; \\ --include-namespaces \u0026lt;ns\u0026gt; \\ --default-volumes-to-fs-backup \\ --ttl 168h # Hourly backup for critical namespaces velero schedule create hourly-critical \\ --schedule \u0026#34;0 * * * *\u0026#34; \\ --include-namespaces critical-ns \\ --default-volumes-to-fs-backup \\ --ttl 24h # Pause/resume schedule velero schedule pause daily velero schedule unpause daily Restore Basic Restore # Restore from latest backup velero restore create --from-backup \u0026lt;backup-name\u0026gt; # Restore to a different namespace velero restore create \\ --from-backup \u0026lt;backup-name\u0026gt; \\ --namespace-mappings original-ns:new-ns # Restore specific items velero restore create \\ --from-backup \u0026lt;backup-name\u0026gt; \\ --include-resources deployments,configmaps Restore With Options # Restore without restoring PV data velero restore create \\ --from-backup \u0026lt;backup-name\u0026gt; \\ --exclude-resources persistentvolumeclaims # Restore and skip existing resources velero restore create \\ --from-backup \u0026lt;backup-name\u0026gt; \\ --existing-resource-policy none # Restore to a cluster with different storage class velero restore create \\ --from-backup \u0026lt;backup-name\u0026gt; \\ --storage-class-mappings standard:fast-ssd Validation # List backups and check status velero backup get velero backup describe \u0026lt;backup-name\u0026gt; --details # List restores velero restore get velero restore describe \u0026lt;restore-name\u0026gt; --details # Check backup logs for warnings/errors velero backup logs \u0026lt;backup-name\u0026gt; | grep -E \u0026#34;warning|error|fail\u0026#34; # Verify backup integrity (comparison of actual restore) velero restore create \\ --from-backup \u0026lt;backup-name\u0026gt; \\ --dry-run \\ --namespace-mappings source-ns:verify-ns Common Failure Modes Backup Fails With \u0026ldquo;AccessDenied\u0026rdquo; The IAM credentials or S3 endpoint configuration is wrong. Verify the bucket exists and the credentials file is current.\n# Test S3 access directly aws s3 --endpoint-url \u0026lt;s3-endpoint\u0026gt; ls s3://\u0026lt;bucket\u0026gt;/ --profile velero Volume Backup Hangs Or Never Completes The node-agent pod may be resource-constrained or the PVC is not mounted on a schedulable node.\n# Check node-agent pod status kubectl -n velero get pods -l component=node-agent # Check if the PVC is mounted on a reachable node kubectl get pod -n \u0026lt;ns\u0026gt; -o wide | grep \u0026lt;pvc-name\u0026gt; # If the pod is stuck in Pending or the node is cordoned, PV backup stalls Restore Creates Resources But PVCs Stay Pending The storage class does not exist or the CSI driver is not installed on the target cluster.\n# Check storage class mapping velero restore describe \u0026lt;restore-name\u0026gt; | grep -A 5 \u0026#34;Storage Class Mapping\u0026#34; # Verify storage class exists on the target kubectl get storageclass Backup Succeeds But Restore Is Incomplete Resources with dependencies (e.g., a Deployment that depends on a ConfigMap) may fail if ordering is not preserved. Velero handles most ordering but custom resources may need manual ordering.\n# Check which resources failed velero restore describe \u0026lt;restore-name\u0026gt; | grep -A 10 \u0026#34;Warnings:\u0026#34; Namespace Already Exists On Target Velero will not overwrite existing namespaces. Use --existing-resource-policy update carefully or restore into a different namespace with --namespace-mappings.\nRetention Strategy Environment Schedule Retention PV Backup Production Every 4 hours 14 days Yes UAT/Staging Every 12 hours 7 days Recommended Development Daily 3 days Optional Backups are only useful when:\nThe restore process is rehearsed quarterly with a documented runbook. The object store credentials are rotated and stored outside Velero\u0026rsquo;s namespace. Cross-region replication is tested by performing a full restore. The backup schedule covers all namespaces with persistent data. The ttl on the backup schedule aligns with the recovery point objective. Node-agent resource requests are sized to handle the largest PVC. ","permalink":"https://trinidadmarroquin.com/field-notes/velero-workload-backup-restore/","section":"field-notes","summary":"Velero has two parts: the CLI is installed on a management machine (your jumpbox or workstation) and issues commands to the cluster via kubectl; the server and node-agent run as pods inside the Kubernetes cluster and carry out the actual backup and restore work. All velero backup, restore, and schedule commands below assume the CLI is installed on a machine with kubectl access to the target cluster.\nPrerequisites Velero backs up Kubernetes resources and, with the restic/kopia integration, persistent volume data. The backup target must be an S3-compatible object store.\n","tags":["kubernetes","velero","backup","recovery","storage"],"title":"Velero Workload Backup And Restore For Kubernetes"},{"categories":["field-notes"],"content":"After existing vSphere VMs are imported into Terraform state, expect drift.\nThe question is not whether drift exists. The question is whether applying that drift is safe.\nGenerate The Audit Plan terraform plan -out=audit.tfplan terraform show -json audit.tfplan \\ | jq -r \u0026#39;.resource_changes[]? | [.address, .type, (.change.actions | join(\u0026#34;,\u0026#34;))] | @tsv\u0026#39; Start with action types:\nno-op update create delete delete,create For imported production-like nodes, any delete, create, or delete,create action needs explicit review before apply.\nGood Checkpoint A good checkpoint after NetBox import may look like:\nnetbox_virtual_machine no-op netbox_interface no-op netbox_ip_address no-op netbox_primary_ip no-op vsphere_virtual_machine update That means source-of-truth inventory is clean and vSphere drift remains isolated.\nCommon Legacy Drift Categories Imported VMs often differ from the shared Terraform module in these areas:\nCPU hot-add setting. memory hot-add setting. datastore ID. network ID. imported VM marker or clone metadata. guestinfo metadata and userdata. wait timeout settings. disk labels. disk count and unit numbers. orphaned or extra disks. Some of these are harmless. Some can disrupt a Kubernetes node.\nDo Not Treat In-Place As Automatically Safe Terraform may show:\nupdate But an in-place update can still include risky changes:\nnetwork_id change datastore_id change disk label changes disk removal or orphan handling guestinfo metadata injection clone/customize block changes For Kubernetes nodes, especially control-plane or etcd members, do not apply these casually.\nInspect vSphere VM Details Use targeted JSON review:\nterraform show -json audit.tfplan \\ | jq \u0026#39;.resource_changes[]? | select(.type == \u0026#34;vsphere_virtual_machine\u0026#34;) | { address, actions: .change.actions, before: { datastore_id: .change.before.datastore_id, network: [.change.before.network_interface[]? | .network_id], disks: [.change.before.disk[]? | {label, unit_number, size}] }, after: { datastore_id: .change.after.datastore_id, network: [.change.after.network_interface[]? | .network_id], disks: [.change.after.disk[]? | {label, unit_number, size}] } }\u0026#39; Classify each change as:\nsafe metadata drift safe compute setting requires maintenance window unsafe on legacy node replace node instead Replacement-Node Decision If the imported node shape differs materially from the shared module, prefer replacement over in-place reconciliation.\nReplacement-node model:\n1. Provision replacement VM with current module/template. 2. Bootstrap OS and RKE2 prerequisites. 3. Join replacement node. 4. Drain old node. 5. Remove old node from Kubernetes/RKE2/etcd as appropriate. 6. Retire old VM and Terraform state. 7. Repeat one node at a time. For workers, add temporary capacity first when needed.\nFor control-plane or etcd nodes:\npreserve quorum. replace one node at a time. verify API health after each step. verify etcd health after each step. do not reuse hostname/IP until old membership is safely removed. For a more detailed rehearsal shape using prebuilt powered-off VMs, see Fast OS Template Node Replacement Rehearsal.\nSuggested Tool Split Use the right tool for each layer:\nTerraform: VM, disks, network intent, NetBox ownership. Ansible: OS prep, packages, kernel/sysctl, RKE2 config, service control. kubectl and RKE2 tooling: drain, node deletion, etcd checks. Rancher: cluster visibility and final health validation. Operating Rule A drift audit is successful when it tells you not to apply.\nIf NetBox is clean but vSphere drift is risky, stop at the checkpoint and move to a replacement-node runbook.\n","permalink":"https://trinidadmarroquin.com/field-notes/classifying-vsphere-drift-after-terraform-import/","section":"field-notes","summary":"After existing vSphere VMs are imported into Terraform state, expect drift.\nThe question is not whether drift exists. The question is whether applying that drift is safe.\nGenerate The Audit Plan terraform plan -out=audit.tfplan terraform show -json audit.tfplan \\ | jq -r \u0026#39;.resource_changes[]? | [.address, .type, (.change.actions | join(\u0026#34;,\u0026#34;))] | @tsv\u0026#39; Start with action types:\nno-op update create delete delete,create For imported production-like nodes, any delete, create, or delete,create action needs explicit review before apply.\n","tags":["terraform","vsphere","kubernetes","drift","audit","operations"],"title":"Classifying vSphere Drift After Terraform Import"},{"categories":["notes"],"content":"This work was not a greenfield deployment.\nThe cluster already existed. The VMs already existed in vSphere. NetBox already had inventory, but some of it was modeled as DCIM devices rather than virtualization virtual machines. Terraform had a reusable module that could manage VM lifecycle state, NetBox VM records, interfaces, IP addresses, and primary IPv4 relationships. The problem was getting from “existing infrastructure” to “managed state” without accidentally changing the running cluster.\nThat calls for a different kind of write-up: a drift audit report.\nThe goal was not to apply changes. The goal was to model the current site, import state carefully, and use Terraform plans as evidence.\nThe Starting Problem The first signal was simple: a NetBox API query against the virtualization VM endpoint returned nothing for a known Kubernetes node.\nThat led to an important distinction:\nDCIM device != virtualization virtual machine NetBox can represent a VMware VM as a DCIM device if that is how it was imported, but Terraform\u0026rsquo;s NetBox virtualization resources expect virtual machine objects. If the source of truth is going to participate in VM lifecycle automation, the model needs to match the provider\u0026rsquo;s resource model.\nThe audit therefore had two tracks:\nmigrate or recreate the inventory as NetBox virtualization VM objects. import matching vSphere and NetBox resources into Terraform state. Only after that could Terraform be used as a drift detector.\nThe Reconciliation Boundary The safe boundary was explicit:\nImport and audit first. Do not apply full vSphere changes to legacy nodes. That boundary matters because an imported VM may not look like a VM created by the shared module. Existing disks may have different labels. Network backing may differ. Datastore placement may differ. Clone metadata may be absent. Guestinfo metadata may not exist. Hot-add may be disabled.\nTerraform can detect all of that, but detection is not permission to apply.\nNetBox Migration Checkpoint After the NetBox virtualization objects were created and imported, the NetBox side reached a clean checkpoint:\nnetbox_virtual_machine: no-op netbox_interface: no-op netbox_ip_address: no-op netbox_primary_ip: no-op That is the useful milestone. It means Terraform and NetBox agree about the VM inventory, interfaces, IPs, and primary IP relationships.\nOne quirk showed up during this process: existing primary IP relationships may not import cleanly as standalone netbox_primary_ip resources. In practice, creating the synthetic Terraform primary-IP resource can also normalize related NetBox VM and IP metadata, such as vCPU count, memory, disk size, DNS name, and interface association.\nThat is acceptable only when the plan is reviewed and confirmed to be NetBox-only.\nPlan Review Saved The Site The important command was not terraform apply. It was plan classification:\nterraform plan -out=audit.tfplan terraform show -json audit.tfplan \\ | jq -r \u0026#39;.resource_changes[]? | [.address, .type, (.change.actions | join(\u0026#34;,\u0026#34;))] | @tsv\u0026#39; After NetBox was reconciled, the remaining full plan showed:\nPlan: 0 to add, 14 to change, 0 to destroy All remaining changes were vSphere VM in-place updates.\nThat sounds safe at first glance. It was not safe enough to apply casually.\nThe drift categories included:\nCPU hot-add and memory hot-add changing from disabled to enabled. datastore placement differences. network backing differences. imported VM metadata changing to module clone metadata. guestinfo metadata/userdata being added. disk label and disk shape mismatches. extra legacy worker disks not represented by the shared module. The key point: there were no NetBox changes left, but the vSphere changes were still operationally risky for existing Kubernetes nodes.\nThe Decision: Replace, Do Not Reconcile In Place At that point the audit had done its job.\nIt showed that NetBox and Terraform state could be reconciled cleanly, but the legacy VMs did not match the desired shared-module shape. Applying the module shape directly onto old-template Kubernetes nodes could touch network, datastore, clone, guestinfo, and disk attributes.\nThe better path is replacement-node lifecycle:\ncreate replacement VM with current module/template bootstrap OS and RKE2 prerequisites join replacement node drain old node remove old node from Kubernetes/RKE2/etcd as appropriate retire old VM and state repeat one node at a time For workers, temporary extra capacity can make migration easier. For control-plane and etcd nodes, the sequence needs stricter quorum and health checks. For the rehearsal pattern, see Fast OS Template Node Replacement Rehearsal.\nResponsibility Split This audit also clarified tool boundaries:\nTerraform owns VM, disks, network intent, and NetBox ownership. Ansible is a good fit for OS prep, packages, kernel/sysctl settings, RKE2 configuration, and service control. kubectl and RKE2 tooling own drain, node deletion, and etcd membership checks. Rancher confirms final cluster visibility and health. That split keeps Terraform from becoming an unsafe in-place remediation tool for legacy Kubernetes nodes.\nThe Practical Lesson Importing existing infrastructure into Terraform should start as an audit, not an apply.\nThe win is not “Terraform can now change all these VMs.” The win is knowing exactly which systems agree, which resources are clean, and which drift is risky enough to require replacement instead of reconciliation.\nFor this site, the checkpoint was:\nNetBox migration complete. Terraform state import complete for source-of-truth objects. NetBox plan clean. vSphere drift classified. Full apply intentionally blocked. Next step: replacement-node runbook. That is a healthy outcome for a drift audit.\nRelated Field Notes:\nNetBox DCIM To Virtualization VM Migration Terraform Import Workflow For Existing vSphere VMs Classifying vSphere Drift After Terraform Import ","permalink":"https://trinidadmarroquin.com/posts/netbox-terraform-drift-audit-existing-kubernetes-vms/","section":"posts","summary":"This work was not a greenfield deployment.\nThe cluster already existed. The VMs already existed in vSphere. NetBox already had inventory, but some of it was modeled as DCIM devices rather than virtualization virtual machines. Terraform had a reusable module that could manage VM lifecycle state, NetBox VM records, interfaces, IP addresses, and primary IPv4 relationships. The problem was getting from “existing infrastructure” to “managed state” without accidentally changing the running cluster.\n","tags":["terraform","vsphere","netbox","kubernetes","drift","audit","operations"],"title":"Drift Audit Report: Bringing Existing Kubernetes VMs Under Terraform And NetBox"},{"categories":["field-notes"],"content":"NetBox has more than one way to represent infrastructure.\nA VMware VM imported as a DCIM device may look usable in the UI, but it is not the same object model as a NetBox virtualization virtual machine. Terraform provider resources for NetBox virtualization expect the virtualization model.\nSymptom A known VM does not show up through the virtualization endpoint:\ncurl -s \\ -H \u0026#34;Authorization: Token $NETBOX_TOKEN\u0026#34; \\ -H \u0026#34;Accept: application/json\u0026#34; \\ \u0026#34;$NETBOX_URL/api/virtualization/virtual-machines/?name=$VM_NAME\u0026#34; \\ | jq \u0026#39;.count\u0026#39; Expected if it is modeled as a virtualization VM:\n1 If it returns 0, check whether it was modeled as a DCIM device:\ncurl -s \\ -H \u0026#34;Authorization: Token $NETBOX_TOKEN\u0026#34; \\ -H \u0026#34;Accept: application/json\u0026#34; \\ \u0026#34;$NETBOX_URL/api/dcim/devices/?name=$VM_NAME\u0026#34; \\ | jq \u0026#39;.results[] | { id, name, status: .status.value, site: .site.name, role: .role.name, device_type: .device_type.model, primary_ip4: .primary_ip4.address }\u0026#39; Environment Variable Mismatch Be consistent with token variable names.\nSome scripts use:\nNETBOX_URL NETBOX_TOKEN The Terraform NetBox provider commonly uses:\nNETBOX_SERVER_URL NETBOX_API_TOKEN If one command uses empty variables, it can look like NetBox has no records when the request is actually malformed or unauthenticated.\nMigration Target For Terraform-managed VM lifecycle work, model each VM as:\nvirtualization virtual machine virtualization interface IP address assigned to VM interface primary IPv4 relationship The object graph should line up with Terraform resources such as:\nnetbox_virtual_machine netbox_interface netbox_ip_address netbox_primary_ip Validation Checks Before importing or reconciling, validate:\nno duplicate VM names. no duplicate IP addresses. every IP has the expected prefix. every assigned IP has a VM/interface owner. every VM belongs to the expected virtualization cluster. every VM has the expected primary IPv4. VM compute metadata is either intentionally blank or ready to be managed. Primary IP Quirk Primary IPv4 assignment may require a separate import, API patch, or Terraform reconciliation step.\nIf an existing NetBox VM already has a primary IP, Terraform may still need a matching netbox_primary_ip state resource. Review the plan carefully before applying any primary-IP reconciliation.\nAcceptable targeted action shape:\nnetbox_primary_ip create netbox_virtual_machine update netbox_ip_address update netbox_interface no-op Only proceed if the plan contains no vSphere resources and the NetBox updates are expected metadata normalization.\nOperating Rule Do not treat DCIM device inventory and virtualization VM inventory as interchangeable.\nPick the NetBox object model that matches the automation you want Terraform to own, then import and audit before applying changes.\n","permalink":"https://trinidadmarroquin.com/field-notes/netbox-dcim-to-virtualization-vm-migration/","section":"field-notes","summary":"NetBox has more than one way to represent infrastructure.\nA VMware VM imported as a DCIM device may look usable in the UI, but it is not the same object model as a NetBox virtualization virtual machine. Terraform provider resources for NetBox virtualization expect the virtualization model.\nSymptom A known VM does not show up through the virtualization endpoint:\ncurl -s \\ -H \u0026#34;Authorization: Token $NETBOX_TOKEN\u0026#34; \\ -H \u0026#34;Accept: application/json\u0026#34; \\ \u0026#34;$NETBOX_URL/api/virtualization/virtual-machines/?name=$VM_NAME\u0026#34; \\ | jq \u0026#39;.count\u0026#39; Expected if it is modeled as a virtualization VM:\n","tags":["netbox","terraform","vsphere","ipam","audit","operations"],"title":"NetBox DCIM To Virtualization VM Migration"},{"categories":["field-notes"],"content":"Use this workflow when existing vSphere VMs need to be brought under Terraform state for audit and future lifecycle management.\nThe first goal is not to change the VMs. The first goal is to make Terraform aware of them and classify drift.\nSafety Boundary Set the operating rule before importing:\nImport state and audit only. Do not apply full vSphere changes to existing cluster nodes. This avoids turning a state migration into an accidental infrastructure mutation.\nRefactor To The Shared Module Move the environment from root-level VM resources to the shared module shape:\nmodule \u0026#34;vm_group\u0026#34; { source = \u0026#34;../../../modules/vsphere-vm-group\u0026#34; vms = local.vms netbox_enabled = true netbox_cluster_name = var.netbox_cluster_name # vSphere, template, network, DNS, and bootstrap inputs omitted } Keep VM definitions in a local map:\nlocals { vms = { \u0026#34;cp1\u0026#34; = { name = \u0026#34;cluster-a-cp-01\u0026#34; ipv4_address = \u0026#34;192.0.2.5\u0026#34; ipv4_netmask = tostring(var.netmask) cpu = 4 ram_gb = 16 disksize = 80 attach_data_disk = true data_disk_gb = 200 } } } Import State In Layers Import in small, reviewable layers:\n1. NetBox VM records 2. NetBox VM interfaces 3. NetBox IP addresses 4. NetBox primary IPv4 relationships 5. vSphere virtual machines After each layer, run a targeted plan or full audit plan and inspect action types.\nPlan Classification Command Generate an audit plan:\nterraform plan -out=audit.tfplan Classify every resource action:\nterraform show -json audit.tfplan \\ | jq -r \u0026#39;.resource_changes[]? | [.address, .type, (.change.actions | join(\u0026#34;,\u0026#34;))] | @tsv\u0026#39; Useful checkpoint output:\nnetbox_virtual_machine no-op netbox_interface no-op netbox_ip_address no-op netbox_primary_ip no-op vsphere_virtual_machine update That means NetBox is reconciled, but vSphere drift remains.\nTargeted Primary IP Reconciliation If primary IP resources do not import cleanly, use a targeted plan and inspect it before apply:\nterraform plan \\ -target=\u0026#39;module.vm_group.netbox_primary_ip.vm\u0026#39; \\ -out=primary-ips.tfplan terraform show -json primary-ips.tfplan \\ | jq -r \u0026#39;.resource_changes[]? | [.address, .type, (.change.actions | join(\u0026#34;,\u0026#34;))] | @tsv\u0026#39; Acceptable shape for a NetBox-only reconciliation:\nnetbox_interface no-op netbox_ip_address no-op or update netbox_virtual_machine no-op or update netbox_primary_ip create Do not apply if any vSphere VM appears in the targeted plan.\nAfter apply, verify count:\nterraform state list | grep \u0026#39;module.vm_group.netbox_primary_ip.vm\u0026#39; | wc -l Verify A Primary IP Resource terraform state show \u0026#39;module.vm_group.netbox_primary_ip.vm[\u0026#34;cp1\u0026#34;]\u0026#39; Then verify NetBox directly:\ncurl -s \\ -H \u0026#34;Authorization: Token $NETBOX_TOKEN\u0026#34; \\ -H \u0026#34;Accept: application/json\u0026#34; \\ \u0026#34;$NETBOX_URL/api/virtualization/virtual-machines/$NETBOX_VM_ID/\u0026#34; \\ | jq \u0026#39;{id,name,memory,disk,vcpus,primary_ip4}\u0026#39; Audit-Only Finish Line The first safe finish line is not a clean full plan. It may be:\nNetBox resources: no-op vSphere resources: update no destroys That is still useful. It means source-of-truth inventory is reconciled and remaining drift is isolated to legacy vSphere shape.\nOperating Rule Importing existing VMs into Terraform is an audit workflow first.\nDo not apply the full plan until every vSphere update is classified as safe, intentionally accepted, or replaced by a node migration plan.\n","permalink":"https://trinidadmarroquin.com/field-notes/terraform-import-existing-vsphere-vms/","section":"field-notes","summary":"Use this workflow when existing vSphere VMs need to be brought under Terraform state for audit and future lifecycle management.\nThe first goal is not to change the VMs. The first goal is to make Terraform aware of them and classify drift.\nSafety Boundary Set the operating rule before importing:\nImport state and audit only. Do not apply full vSphere changes to existing cluster nodes. This avoids turning a state migration into an accidental infrastructure mutation.\n","tags":["terraform","vsphere","netbox","state","audit","operations"],"title":"Terraform Import Workflow For Existing vSphere VMs"},{"categories":["notes"],"content":"Provisioning a VM is only the first part of ownership.\nThe more interesting operational question is what happens after the VM exists. Can Terraform continue to manage CPU, memory, disk sizing, NetBox metadata, idempotency, and teardown without turning every follow-up change into a manual vCenter edit?\nThat was the next step after adding VM/IP guardrails. The automation already knew how to prevent duplicate VM names, duplicate IPs, stale DNS collisions, active network IP reuse, and NetBox/vCenter drift before create. The next improvement was to keep the VM lifecycle managed after creation.\nThe useful pattern became:\nprovision -\u0026gt; record in NetBox -\u0026gt; manage compute and disks -\u0026gt; verify metadata -\u0026gt; confirm idempotency -\u0026gt; destroy -\u0026gt; verify cleanup The Ownership Boundary The module now treats a VM as more than a clone operation.\nTerraform owns the intended values for:\nvCPU count. memory size. primary disk size. optional secondary data disk attachment. optional secondary data disk size. CPU hot-add setting. memory hot-add setting. NetBox VM memory metadata. NetBox VM vCPU metadata. NetBox VM disk-size metadata. NetBox VM/interface/IP/primary-IP relationships. That matters because manual post-provisioning changes are where platform drift usually starts. A VM gets resized during a troubleshooting window. A disk gets grown in vCenter. NetBox still shows the old size. Terraform either wants to undo the change later or keeps reporting confusing drift.\nThe better operating model is to make the desired size change in the VM map, review the plan, apply the saved plan, and then verify both vSphere and NetBox reflect the same intent.\nExample State Change The VM map carries per-VM overrides for compute and disk sizing:\nlocals { vms = { \u0026#34;app-1\u0026#34; = { name = \u0026#34;cluster-a-app-01\u0026#34; ipv4_address = \u0026#34;192.0.2.10\u0026#34; ipv4_netmask = tostring(var.netmask) cpu = 4 ram_gb = 32 disksize = 80 attach_data_disk = true data_disk_gb = 200 } } } Those values drive both the vSphere VM and the NetBox metadata.\nIn vSphere, Terraform manages:\nnum_cpus memory disk[0].size disk[1].size, when a data disk is enabled In NetBox, Terraform records:\nvcpus memory_mb disk_size_mb The disk metadata is the sum of the primary disk and optional data disk, expressed in megabytes.\nSafe Change Workflow The workflow is intentionally conservative.\nGenerate a saved plan:\nterraform plan -out=compute-change.tfplan Inspect action types:\nterraform show -json compute-change.tfplan \\ | jq -r \u0026#39;.resource_changes[]? | [.address, .type, (.change.actions | join(\u0026#34;,\u0026#34;))] | @tsv\u0026#39; For a normal resize, the expected vSphere action is usually:\nupdate If the VM shows delete,create, stop and understand why Terraform wants replacement.\nInspect before and after values:\nterraform show -json compute-change.tfplan \\ | jq \u0026#39;.resource_changes[]? | select(.type == \u0026#34;vsphere_virtual_machine\u0026#34;) | { address, actions: .change.actions, before: { cpu: .change.before.num_cpus, memory_mb: .change.before.memory, disks: [.change.before.disk[]? | {label, size}] }, after: { cpu: .change.after.num_cpus, memory_mb: .change.after.memory, disks: [.change.after.disk[]? | {label, size}] } }\u0026#39; Inspect NetBox metadata changes too:\nterraform show -json compute-change.tfplan \\ | jq \u0026#39;.resource_changes[]? | select(.type == \u0026#34;netbox_virtual_machine\u0026#34;) | { address, actions: .change.actions, before: { vcpus: .change.before.vcpus, memory_mb: .change.before.memory_mb, disk_size_mb: .change.before.disk_size_mb }, after: { vcpus: .change.after.vcpus, memory_mb: .change.after.memory_mb, disk_size_mb: .change.after.disk_size_mb } }\u0026#39; Apply only the reviewed saved plan:\nterraform apply compute-change.tfplan Then check idempotency:\nterraform plan -detailed-exitcode The desired result is no changes.\nHot-Add Is Not A Substitute For Review CPU and memory hot-add make many changes less disruptive, but they do not remove the need for plan review.\nThe operator still needs to verify:\nthe VM is being updated, not replaced. the CPU and memory values are expected. the disk changes are growth-only unless replacement is intentional. NetBox metadata changes match the VM state change. no unrelated VM, IP, interface, or destroy action is present. Hot-add improves the execution path. It does not prove the change is safe.\nDisk Growth Has Two Layers Terraform can grow the virtual disk in vSphere. That does not always mean the guest operating system has expanded the partition, LVM volume, or filesystem.\nAfter increasing a disk, plan for guest verification:\nlsblk df -h sudo growpart \u0026lt;disk\u0026gt; \u0026lt;partition\u0026gt; sudo pvresize \u0026lt;device\u0026gt; sudo lvextend -r -l +100%FREE \u0026lt;logical-volume\u0026gt; The exact guest commands depend on the image layout. The important distinction is that vSphere disk size and guest usable filesystem size are separate layers.\nNetBox Verification After apply, verify NetBox reflects the intended lifecycle state.\nSet generic lookup values:\nexport VM_NAME=\u0026#34;cluster-a-app-01\u0026#34; export VM_IP=\u0026#34;192.0.2.10\u0026#34; export VM_DNS_NAME=\u0026#34;cluster-a-app-01.example.com\u0026#34; export NETBOX_CLUSTER_NAME=\u0026#34;cluster-a\u0026#34; Verify VM metadata:\ncurl -s \\ -H \u0026#34;Authorization: Token $NETBOX_API_TOKEN\u0026#34; \\ -H \u0026#34;Accept: application/json\u0026#34; \\ \u0026#34;$NETBOX_SERVER_URL/api/virtualization/virtual-machines/?name=$VM_NAME\u0026#34; \\ | jq \u0026#39;.results[] | { name, status: .status.value, cluster: .cluster.name, vcpus, memory_mb: .memory, disk_size_mb: .disk, primary_ip4: .primary_ip4.address }\u0026#39; Verify IP assignment:\ncurl -s \\ -H \u0026#34;Authorization: Token $NETBOX_API_TOKEN\u0026#34; \\ -H \u0026#34;Accept: application/json\u0026#34; \\ \u0026#34;$NETBOX_SERVER_URL/api/ipam/ip-addresses/?q=$VM_IP\u0026#34; \\ | jq \u0026#39;.results[] | { address, status: .status.value, dns_name, assigned_object_type, assigned_object: .assigned_object.name }\u0026#39; NetBox should not lag behind the hypervisor. If Terraform manages the VM size, the source-of-truth metadata should move with it.\nDestroy Is Part Of Lifecycle Management The lifecycle test is not complete until destroy is tested too.\nFor a disposable managed VM, review the destroy plan:\nterraform plan -destroy -out=destroy.tfplan terraform show -json destroy.tfplan \\ | jq -r \u0026#39;.resource_changes[]? | [.address, .type, (.change.actions | join(\u0026#34;,\u0026#34;))] | @tsv\u0026#39; Expected resources include the vSphere VM and the NetBox object graph:\nnetbox_primary_ip netbox_ip_address netbox_interface netbox_virtual_machine vsphere_virtual_machine After applying the destroy plan, verify cleanup:\nterraform apply destroy.tfplan terraform state list govc find / -type m -name \u0026#34;$VM_NAME\u0026#34; NetBox lookups for the VM and IP should return count 0.\nThat proves the full lifecycle:\ncreate -\u0026gt; manage -\u0026gt; verify idempotency -\u0026gt; destroy -\u0026gt; cleanup NetBox and vSphere The Practical Lesson The most useful automation is not the automation that creates something once. It is the automation that keeps ownership clear after the first successful apply.\nFor vSphere VM operations, that means Terraform should own the intended VM lifecycle state, NetBox should reflect that state, and operators should review plans as lifecycle changes, not just deployment events.\nRelated Field Notes:\nTerraform vSphere Compute Resize Checklist Post-Provision VM State Verification With NetBox ","permalink":"https://trinidadmarroquin.com/posts/managing-vsphere-vm-lifecycle-state-terraform-netbox/","section":"posts","summary":"Provisioning a VM is only the first part of ownership.\nThe more interesting operational question is what happens after the VM exists. Can Terraform continue to manage CPU, memory, disk sizing, NetBox metadata, idempotency, and teardown without turning every follow-up change into a manual vCenter edit?\nThat was the next step after adding VM/IP guardrails. The automation already knew how to prevent duplicate VM names, duplicate IPs, stale DNS collisions, active network IP reuse, and NetBox/vCenter drift before create. The next improvement was to keep the VM lifecycle managed after creation.\n","tags":["terraform","vsphere","netbox","automation","operations","sre"],"title":"Managing vSphere VM Lifecycle State With Terraform And NetBox"},{"categories":["field-notes"],"content":"After Terraform creates or resizes a vSphere VM, verify NetBox reflects the same lifecycle state.\nThis check is separate from preflight. Preflight prevents unsafe allocation before apply. Post-provision verification proves source-of-truth state matches what Terraform just changed.\nSet Lookup Values Use generic environment variables so the commands are reusable:\nexport VM_NAME=\u0026#34;cluster-a-app-01\u0026#34; export VM_IP=\u0026#34;192.0.2.10\u0026#34; export VM_DNS_NAME=\u0026#34;cluster-a-app-01.example.com\u0026#34; export NETBOX_CLUSTER_NAME=\u0026#34;cluster-a\u0026#34; NetBox API credentials:\nexport NETBOX_SERVER_URL=\u0026#34;https://netbox.example.com\u0026#34; export NETBOX_API_TOKEN=\u0026#34;...\u0026#34; Verify VM Metadata curl -s \\ -H \u0026#34;Authorization: Token $NETBOX_API_TOKEN\u0026#34; \\ -H \u0026#34;Accept: application/json\u0026#34; \\ \u0026#34;$NETBOX_SERVER_URL/api/virtualization/virtual-machines/?name=$VM_NAME\u0026#34; \\ | jq \u0026#39;.results[] | { id, name, status: .status.value, cluster: .cluster.name, vcpus, memory_mb: .memory, disk_size_mb: .disk, primary_ip4: .primary_ip4.address }\u0026#39; Confirm:\none VM is returned. cluster is expected. status is expected. vCPU count matches Terraform. memory matches Terraform in MB. disk size matches Terraform in MB. primary IPv4 is populated. Verify Interface Assignment curl -s \\ -H \u0026#34;Authorization: Token $NETBOX_API_TOKEN\u0026#34; \\ -H \u0026#34;Accept: application/json\u0026#34; \\ \u0026#34;$NETBOX_SERVER_URL/api/virtualization/interfaces/?virtual_machine=$VM_NAME\u0026#34; \\ | jq \u0026#39;.results[] | { id, name, enabled, vm: .virtual_machine.name }\u0026#39; Confirm the expected management interface exists and belongs to the VM.\nVerify IPAM Assignment curl -s \\ -H \u0026#34;Authorization: Token $NETBOX_API_TOKEN\u0026#34; \\ -H \u0026#34;Accept: application/json\u0026#34; \\ \u0026#34;$NETBOX_SERVER_URL/api/ipam/ip-addresses/?q=$VM_IP\u0026#34; \\ | jq \u0026#39;.results[] | { id, address, status: .status.value, dns_name, assigned_object_type, assigned_object: .assigned_object.name }\u0026#39; Confirm:\none IP is returned. address and prefix are expected. status is expected. DNS name is expected. assigned object points at the VM interface. Verify DNS Name Lookup In NetBox curl -s \\ -H \u0026#34;Authorization: Token $NETBOX_API_TOKEN\u0026#34; \\ -H \u0026#34;Accept: application/json\u0026#34; \\ \u0026#34;$NETBOX_SERVER_URL/api/ipam/ip-addresses/?dns_name=$VM_DNS_NAME\u0026#34; \\ | jq \u0026#39;.results[] | { id, address, dns_name, status: .status.value, assigned_object_type, assigned_object: .assigned_object.name }\u0026#39; This catches cases where the IP exists but the recorded DNS name is stale or missing.\nVerify Cluster Membership curl -s \\ -H \u0026#34;Authorization: Token $NETBOX_API_TOKEN\u0026#34; \\ -H \u0026#34;Accept: application/json\u0026#34; \\ \u0026#34;$NETBOX_SERVER_URL/api/virtualization/virtual-machines/?cluster=$(jq -rn --arg value \u0026#34;$NETBOX_CLUSTER_NAME\u0026#34; \u0026#39;$value|@uri\u0026#39;)\u0026#34; \\ | jq --arg vm_name \u0026#34;$VM_NAME\u0026#34; \u0026#39;.results[] | select(.name == $vm_name) | { id, name, status: .status.value, cluster: .cluster.name, vcpus, memory_mb: .memory, disk_size_mb: .disk, primary_ip4: .primary_ip4.address }\u0026#39; This proves the VM appears under the expected virtualization cluster, not just by global name search.\nVerify Cleanup After Destroy After applying a reviewed destroy plan, the VM and IP lookups should return zero records.\nVM cleanup check:\ncurl -s \\ -H \u0026#34;Authorization: Token $NETBOX_API_TOKEN\u0026#34; \\ -H \u0026#34;Accept: application/json\u0026#34; \\ \u0026#34;$NETBOX_SERVER_URL/api/virtualization/virtual-machines/?name=$VM_NAME\u0026#34; \\ | jq \u0026#39;.count\u0026#39; Expected:\n0 IP cleanup check:\ncurl -s \\ -H \u0026#34;Authorization: Token $NETBOX_API_TOKEN\u0026#34; \\ -H \u0026#34;Accept: application/json\u0026#34; \\ \u0026#34;$NETBOX_SERVER_URL/api/ipam/ip-addresses/?q=$VM_IP\u0026#34; \\ | jq \u0026#39;.count\u0026#39; Expected:\n0 Operating Rule NetBox verification should happen after create, after resize, and after destroy.\nIf Terraform owns VM lifecycle state, NetBox should show the same lifecycle state or be intentionally updated by the same apply.\n","permalink":"https://trinidadmarroquin.com/field-notes/post-provision-vm-state-verification-netbox/","section":"field-notes","summary":"After Terraform creates or resizes a vSphere VM, verify NetBox reflects the same lifecycle state.\nThis check is separate from preflight. Preflight prevents unsafe allocation before apply. Post-provision verification proves source-of-truth state matches what Terraform just changed.\nSet Lookup Values Use generic environment variables so the commands are reusable:\nexport VM_NAME=\u0026#34;cluster-a-app-01\u0026#34; export VM_IP=\u0026#34;192.0.2.10\u0026#34; export VM_DNS_NAME=\u0026#34;cluster-a-app-01.example.com\u0026#34; export NETBOX_CLUSTER_NAME=\u0026#34;cluster-a\u0026#34; NetBox API credentials:\nexport NETBOX_SERVER_URL=\u0026#34;https://netbox.example.com\u0026#34; export NETBOX_API_TOKEN=\u0026#34;...\u0026#34; Verify VM Metadata curl -s \\ -H \u0026#34;Authorization: Token $NETBOX_API_TOKEN\u0026#34; \\ -H \u0026#34;Accept: application/json\u0026#34; \\ \u0026#34;$NETBOX_SERVER_URL/api/virtualization/virtual-machines/?name=$VM_NAME\u0026#34; \\ | jq \u0026#39;.results[] | { id, name, status: .status.value, cluster: .cluster.name, vcpus, memory_mb: .memory, disk_size_mb: .disk, primary_ip4: .primary_ip4.address }\u0026#39; Confirm:\n","tags":["netbox","terraform","vsphere","ipam","automation","operations"],"title":"Post-Provision VM State Verification With NetBox"},{"categories":["field-notes"],"content":"Use this checklist when resizing Terraform-managed vSphere VMs after initial provisioning.\nThe goal is to update VM state without accidentally replacing the VM, losing NetBox alignment, or skipping guest OS disk follow-up.\nEdit Desired State Change the VM entry in the environment VM map:\nlocals { vms = { \u0026#34;app-1\u0026#34; = { name = \u0026#34;cluster-a-app-01\u0026#34; ipv4_address = \u0026#34;192.0.2.10\u0026#34; ipv4_netmask = tostring(var.netmask) cpu = 4 ram_gb = 32 disksize = 80 attach_data_disk = true data_disk_gb = 200 } } } Avoid manual vCenter edits for values Terraform owns.\nGenerate A Saved Plan terraform plan -out=compute-change.tfplan Use a saved plan so the reviewed plan is the applied plan.\nInspect Action Types terraform show -json compute-change.tfplan \\ | jq -r \u0026#39;.resource_changes[]? | [.address, .type, (.change.actions | join(\u0026#34;,\u0026#34;))] | @tsv\u0026#39; Expected for a normal resize:\nmodule.vm_group.vsphere_virtual_machine.vm[\u0026#34;app-1\u0026#34;] vsphere_virtual_machine update module.vm_group.netbox_virtual_machine.vm[\u0026#34;app-1\u0026#34;] netbox_virtual_machine update Stop if you see an unexpected replacement:\ndelete,create Stop if unrelated VMs or destroy actions appear.\nInspect vSphere Before And After Values terraform show -json compute-change.tfplan \\ | jq \u0026#39;.resource_changes[]? | select(.type == \u0026#34;vsphere_virtual_machine\u0026#34;) | { address, actions: .change.actions, before: { cpu: .change.before.num_cpus, memory_mb: .change.before.memory, disks: [.change.before.disk[]? | {label, size}] }, after: { cpu: .change.after.num_cpus, memory_mb: .change.after.memory, disks: [.change.after.disk[]? | {label, size}] } }\u0026#39; Confirm:\nCPU is the intended value. memory is the intended value in MB. primary disk size is expected. data disk exists only when intended. disk changes are growth-only unless replacement is explicitly approved. Inspect NetBox Metadata Changes terraform show -json compute-change.tfplan \\ | jq \u0026#39;.resource_changes[]? | select(.type == \u0026#34;netbox_virtual_machine\u0026#34;) | { address, actions: .change.actions, before: { vcpus: .change.before.vcpus, memory_mb: .change.before.memory_mb, disk_size_mb: .change.before.disk_size_mb }, after: { vcpus: .change.after.vcpus, memory_mb: .change.after.memory_mb, disk_size_mb: .change.after.disk_size_mb } }\u0026#39; Expected metadata relationship:\nmemory_mb = ram_gb * 1024 disk_size_mb = (primary_disk_gb + optional_data_disk_gb) * 1024 Apply The Reviewed Plan terraform apply compute-change.tfplan Do not regenerate the plan between review and apply unless you review the new plan too.\nConfirm Idempotency terraform plan -detailed-exitcode Expected:\nNo changes. Your infrastructure matches the configuration. Exit code expectations:\n0 = no changes 1 = error 2 = diff remains Guest Disk Follow-Up If a virtual disk grew, verify the guest sees the new size:\nlsblk df -h Common Linux follow-up patterns, depending on layout:\nsudo growpart \u0026lt;disk\u0026gt; \u0026lt;partition\u0026gt; sudo resize2fs \u0026lt;partition\u0026gt; or LVM:\nsudo pvresize \u0026lt;device\u0026gt; sudo lvextend -r -l +100%FREE \u0026lt;logical-volume\u0026gt; Do not assume vSphere disk growth means the guest filesystem expanded.\nOperating Rule Treat post-provision VM resizing as a lifecycle change, not a console task.\nReview the saved plan, apply the saved plan, verify NetBox metadata, and confirm idempotency.\n","permalink":"https://trinidadmarroquin.com/field-notes/terraform-vsphere-compute-resize-checklist/","section":"field-notes","summary":"Use this checklist when resizing Terraform-managed vSphere VMs after initial provisioning.\nThe goal is to update VM state without accidentally replacing the VM, losing NetBox alignment, or skipping guest OS disk follow-up.\nEdit Desired State Change the VM entry in the environment VM map:\nlocals { vms = { \u0026#34;app-1\u0026#34; = { name = \u0026#34;cluster-a-app-01\u0026#34; ipv4_address = \u0026#34;192.0.2.10\u0026#34; ipv4_netmask = tostring(var.netmask) cpu = 4 ram_gb = 32 disksize = 80 attach_data_disk = true data_disk_gb = 200 } } } Avoid manual vCenter edits for values Terraform owns.\n","tags":["terraform","vsphere","vmware","automation","operations","validation"],"title":"Terraform vSphere Compute Resize Checklist"},{"categories":["notes"],"content":"The dangerous failure mode was simple: Terraform could be technically correct and still create the wrong thing.\nA VM name might already exist in vCenter. An IP might already be active on the network but missing from NetBox. DNS might still resolve from an older system. NetBox might be unavailable. A plan might look routine while carrying enough ambiguity to clobber an existing workload.\nThe fix was not one control. It was a guardrail stack.\nThe goal was clear: before Terraform creates a vSphere VM, prove that the planned VM name and IP are safe across the systems that already know about infrastructure. After the test VM is no longer needed, prove Terraform also removes the NetBox and vSphere objects it created.\nThe Boundary That Matters For VM provisioning, Terraform usually owns desired state, but it is not the only source of truth.\nThe real environment also has:\nvCenter inventory. NetBox VM and IPAM records. DNS forward records. reverse DNS records. live hosts responding on the network. If Terraform only validates its own input map, it can catch duplicate values inside the plan but miss everything that already exists outside the plan.\nThat is why the guardrails were split into layers:\nTerraform variable validation -\u0026gt; NetBox provider resources -\u0026gt; preflight checks against NetBox, vCenter, DNS, and network liveness -\u0026gt; vSphere VM creation -\u0026gt; destroy plan verification and cleanup checks Each layer catches a different class of mistake.\nTerraform Validation Catches Local Mistakes The first layer was inside the Terraform module. VM names and IPv4 addresses must be unique inside the planned VM map.\nThat catches problems like:\nvm-a -\u0026gt; 192.0.2.10 vm-b -\u0026gt; 192.0.2.10 or:\nkey-a -\u0026gt; name = worker-01 key-b -\u0026gt; name = worker-01 Those failures should happen during terraform plan, before any provider starts creating infrastructure.\nThis layer is useful, but it only knows what is in the configuration.\nNetBox First, vSphere Second The stronger pattern was to make NetBox ownership explicit before creating the vSphere VM.\nThe module planned NetBox records for:\nvirtual machine. interface. IP address. primary IPv4 relationship. Then the vSphere VM depended on the NetBox primary IP relationship.\nThat dependency is important. It makes NetBox more than documentation. It becomes a provisioning gate.\nIf NetBox already has the VM name, the NetBox VM resource fails. If NetBox already has the IP address, the NetBox IP resource fails. If NetBox is down or the cluster lookup fails, Terraform fails before vSphere creation proceeds.\nThat is the desired fail-closed behavior.\nNetBox unavailable -\u0026gt; Terraform cannot resolve/create NetBox records -\u0026gt; vSphere VM is not created Why Preflight Still Matters NetBox-first provisioning is not enough by itself.\nThere are real situations where NetBox is incomplete or stale:\na VM exists in vCenter but not in NetBox. an IP responds on the network but has no IPAM record. a DNS record still resolves after a system was removed. reverse DNS points to an old hostname. govc is missing configuration and the vCenter check silently becomes useless. Those are exactly the cases a preflight script should catch.\nThe preflight workflow used the Terraform plan JSON as input:\nterraform plan -out=tfplan terraform show -json tfplan \u0026gt; tfplan.json ./scripts/preflight-vm-guardrails.sh --plan-json tfplan.json From that plan, the script can inspect the planned VM names and static IP addresses, then check external systems before apply.\nThe Checks That Paid Off The useful preflight checks were intentionally boring:\nquery NetBox for planned VM names. query NetBox for planned IP addresses. query vCenter with govc for planned VM names. resolve planned VM names with getent hosts. check reverse DNS with dig -x or getent. probe planned IPs with nmap -sn -n. fail explicitly if govc is required but not configured. The last point matters more than it looks.\nA skipped check is acceptable when it is explicit. A silently weakened check is not.\nIf vCenter checks are enabled and govc is not configured, the safe behavior is to fail with a clear message. Operators can rerun with --skip-govc when that is intentional.\nTeardown Hygiene Matters Too Creation guardrails prevent clobbering existing infrastructure. Teardown guardrails prevent leaving stale source-of-truth records behind.\nThe destroy path was tested as part of the same lifecycle. A disposable VM was created, managed, checked for idempotency, then destroyed. The expected destroy plan included both NetBox and vSphere resources:\nnetbox_primary_ip netbox_ip_address netbox_interface netbox_virtual_machine vsphere_virtual_machine After terraform apply destroy.tfplan, the cleanup checks verified:\nTerraform state no longer listed the managed resources. NetBox no longer returned the VM record. NetBox no longer returned the IP address record. govc find no longer returned the vSphere VM. That proves the full lifecycle, not just safe allocation:\ncreate -\u0026gt; manage -\u0026gt; verify idempotency -\u0026gt; destroy -\u0026gt; cleanup NetBox and vSphere What Was Tested The guardrail stack was tested against the failure modes that matter:\nclean new VM. duplicate IP inside Terraform config. duplicate VM name inside Terraform config. IP already exists in NetBox. VM name already exists in NetBox. VM exists in vCenter but not NetBox. IP is active on the network but not in NetBox. reverse DNS exists. forward DNS exists for the VM name. NetBox enabled without provider credentials. preflight without NetBox credentials. NetBox cluster missing. NetBox unreachable. teardown removes Terraform-managed NetBox and vSphere objects. The important result was not that every command succeeded. The important result was that each unsafe condition failed in the correct place.\nSome failures belong in terraform plan. Some belong in preflight. Some belong in provider lookups. Teardown hygiene belongs in destroy-plan review and post-destroy verification. The operator should know which layer is responsible for each class of risk.\nThe Practical Lesson Guardrails should be close to the action they protect.\nTerraform validation catches mistakes in Terraform input. NetBox provider resources protect source-of-truth ownership. vCenter, DNS, and nmap preflight checks catch reality outside Terraform and NetBox. Destroy verification closes the loop by proving the automation cleans up the same objects it created.\nThe strongest design was not “trust NetBox” or “trust Terraform.” It was:\nTrust each system only for the thing it can actually prove. Then make the VM creation step depend on those proofs, and make teardown verification part of the lifecycle test.\nRelated Field Notes:\nNetBox First Ownership For vSphere VM Provisioning Terraform vSphere VM Preflight Guardrails Terraform VM Guardrail Test Matrix ","permalink":"https://trinidadmarroquin.com/posts/terraform-vsphere-vm-guardrails-netbox/","section":"posts","summary":"The dangerous failure mode was simple: Terraform could be technically correct and still create the wrong thing.\nA VM name might already exist in vCenter. An IP might already be active on the network but missing from NetBox. DNS might still resolve from an older system. NetBox might be unavailable. A plan might look routine while carrying enough ambiguity to clobber an existing workload.\nThe fix was not one control. It was a guardrail stack.\n","tags":["terraform","vsphere","netbox","automation","sre","operations"],"title":"Building Guardrails Around Terraform vSphere VM Lifecycle"},{"categories":["field-notes"],"content":"When Terraform creates vSphere VMs, NetBox should not be an after-the-fact documentation step.\nUse NetBox as an ownership gate before vSphere VM creation, then verify Terraform removes the NetBox records when the managed VM is destroyed.\nDesired Order The safe order is:\n1. Resolve required NetBox objects 2. Create NetBox VM record 3. Create NetBox interface record 4. Create NetBox IP address record 5. Set NetBox primary IPv4 6. Create vSphere VM The vSphere VM should depend on the NetBox primary IP relationship, not just the VM record.\nThat proves the source-of-truth object graph exists before the hypervisor receives the create request.\nThe destroy path should remove the same object graph:\n1. Remove vSphere VM 2. Remove NetBox primary IPv4 relationship 3. Remove NetBox IP address record 4. Remove NetBox interface record 5. Remove NetBox VM record Exact ordering is provider-dependent, but the final state should be clean in both Terraform and NetBox.\nTerraform Pattern The shape is:\ndata \u0026#34;netbox_cluster\u0026#34; \u0026#34;cluster\u0026#34; { count = var.netbox_enabled ? 1 : 0 name = var.netbox_cluster_name } resource \u0026#34;netbox_virtual_machine\u0026#34; \u0026#34;vm\u0026#34; { for_each = var.netbox_enabled ? var.vms : {} name = each.value.name cluster_id = data.netbox_cluster.cluster[0].id status = \u0026#34;active\u0026#34; } resource \u0026#34;netbox_interface\u0026#34; \u0026#34;mgmt\u0026#34; { for_each = var.netbox_enabled ? var.vms : {} virtual_machine_id = netbox_virtual_machine.vm[each.key].id name = \u0026#34;mgmt0\u0026#34; enabled = true } resource \u0026#34;netbox_ip_address\u0026#34; \u0026#34;mgmt\u0026#34; { for_each = var.netbox_enabled ? var.vms : {} ip_address = \u0026#34;${each.value.ipv4_address}/${each.value.ipv4_prefix_length}\u0026#34; status = \u0026#34;active\u0026#34; dns_name = \u0026#34;${each.value.name}.example.com\u0026#34; interface_id = netbox_interface.mgmt[each.key].id } resource \u0026#34;netbox_primary_ip\u0026#34; \u0026#34;vm\u0026#34; { for_each = var.netbox_enabled ? var.vms : {} virtual_machine_id = netbox_virtual_machine.vm[each.key].id ip_address_id = netbox_ip_address.mgmt[each.key].id } resource \u0026#34;vsphere_virtual_machine\u0026#34; \u0026#34;vm\u0026#34; { for_each = var.vms name = each.value.name depends_on = [ netbox_primary_ip.vm, ] } Adapt resource arguments to the provider version in use. The important part is the dependency boundary.\nWhat This Protects This pattern protects against:\ncreating a VM when NetBox cannot be reached. creating a VM when the NetBox cluster lookup fails. creating a VM when the NetBox VM name already exists. creating a VM when the NetBox IP address already exists. creating a VM without a primary IP relationship in source of truth. leaving Terraform-managed NetBox records behind after destroying a disposable VM. What This Does Not Protect NetBox-first ownership does not prove:\nthe VM name is absent from vCenter. the IP is quiet on the network. forward DNS is clear. reverse DNS is clear. govc is configured correctly. Those need a separate preflight check.\nFail-Closed Tests Run these before trusting the pattern.\nMissing NetBox credentials:\nunset NETBOX_SERVER_URL unset NETBOX_API_TOKEN terraform plan -out=/tmp/missing-netbox-creds.tfplan Expected result:\nError: Missing required argument The argument \u0026#34;server_url\u0026#34; is required Error: Missing required argument The argument \u0026#34;api_token\u0026#34; is required Missing NetBox cluster:\nterraform plan -out=tfplan \\ -var=\u0026#39;netbox_cluster_name=missing-cluster-test\u0026#39; Expected result:\nError: no result with module.vm_group.data.netbox_cluster.cluster[0] Unreachable NetBox:\nNETBOX_SERVER_URL=\u0026#34;https://netbox-unreachable.invalid\u0026#34; \\ NETBOX_API_TOKEN=\u0026#34;dummy\u0026#34; \\ NETBOX_SKIP_VERSION_CHECK=true \\ terraform plan -out=/tmp/netbox-unreachable-test.tfplan Expected result:\nPlanning failed. Error: Get \u0026#34;https://netbox-unreachable.invalid/api/...\u0026#34; The desired behavior is:\nNetBox fails -\u0026gt; Terraform plan fails -\u0026gt; vSphere VM is not created Teardown Hygiene Test Use a disposable test VM that Terraform already created and owns.\nConfirm current state:\nterraform state list Expected resources:\nmodule.vm_group.netbox_interface.vm[\u0026#34;test-1\u0026#34;] module.vm_group.netbox_ip_address.vm[\u0026#34;test-1\u0026#34;] module.vm_group.netbox_primary_ip.vm[\u0026#34;test-1\u0026#34;] module.vm_group.netbox_virtual_machine.vm[\u0026#34;test-1\u0026#34;] module.vm_group.vsphere_virtual_machine.vm[\u0026#34;test-1\u0026#34;] Generate a destroy plan:\nterraform plan -destroy -out=destroy.tfplan Inspect planned deletes before applying:\nterraform show -json destroy.tfplan \\ | jq -r \u0026#39;.resource_changes[]? | [.address, .type, (.change.actions | join(\u0026#34;,\u0026#34;))] | @tsv\u0026#39; Expected action shape:\nmodule.vm_group.netbox_primary_ip.vm[\u0026#34;test-1\u0026#34;] netbox_primary_ip delete module.vm_group.netbox_ip_address.vm[\u0026#34;test-1\u0026#34;] netbox_ip_address delete module.vm_group.netbox_interface.vm[\u0026#34;test-1\u0026#34;] netbox_interface delete module.vm_group.netbox_virtual_machine.vm[\u0026#34;test-1\u0026#34;] netbox_virtual_machine delete module.vm_group.vsphere_virtual_machine.vm[\u0026#34;test-1\u0026#34;] vsphere_virtual_machine delete Apply the destroy plan:\nterraform apply destroy.tfplan Verify Terraform state no longer lists the managed resources:\nterraform state list Verify NetBox VM cleanup:\ncurl -s \\ -H \u0026#34;Authorization: Token $NETBOX_API_TOKEN\u0026#34; \\ -H \u0026#34;Accept: application/json\u0026#34; \\ \u0026#34;$NETBOX_SERVER_URL/api/virtualization/virtual-machines/?name=cluster-a-test-01\u0026#34; \\ | jq \u0026#39;.count\u0026#39; Expected:\n0 Verify NetBox IP cleanup:\ncurl -s \\ -H \u0026#34;Authorization: Token $NETBOX_API_TOKEN\u0026#34; \\ -H \u0026#34;Accept: application/json\u0026#34; \\ \u0026#34;$NETBOX_SERVER_URL/api/ipam/ip-addresses/?q=192.0.2.10\u0026#34; \\ | jq \u0026#39;.count\u0026#39; Expected:\n0 Verify vCenter cleanup:\ngovc find / -type m -name \u0026#39;cluster-a-test-01\u0026#39; Expected: no output.\nThe successful lifecycle result is:\ncreate -\u0026gt; manage -\u0026gt; verify idempotency -\u0026gt; destroy -\u0026gt; cleanup NetBox and vSphere Operating Rule If NetBox is the source of truth, make VM creation depend on NetBox ownership being established first.\nDo not let documentation happen after provisioning when the documentation system is supposed to prevent collisions. Do not skip teardown checks when the same documentation system needs to stay clean after destroy.\n","permalink":"https://trinidadmarroquin.com/field-notes/netbox-first-vsphere-vm-ownership/","section":"field-notes","summary":"When Terraform creates vSphere VMs, NetBox should not be an after-the-fact documentation step.\nUse NetBox as an ownership gate before vSphere VM creation, then verify Terraform removes the NetBox records when the managed VM is destroyed.\nDesired Order The safe order is:\n1. Resolve required NetBox objects 2. Create NetBox VM record 3. Create NetBox interface record 4. Create NetBox IP address record 5. Set NetBox primary IPv4 6. Create vSphere VM The vSphere VM should depend on the NetBox primary IP relationship, not just the VM record.\n","tags":["netbox","terraform","vsphere","ipam","automation","operations"],"title":"NetBox First Ownership For vSphere VM Provisioning"},{"categories":["field-notes"],"content":"Use this matrix to prove VM provisioning guardrails before trusting automation with real vSphere creates.\nThe point is not to make every test pass green. The point is to prove each unsafe condition fails at the correct layer.\nBaseline Commands Generate a plan and plan JSON:\nterraform init terraform plan -out=tfplan terraform show -json tfplan \u0026gt; tfplan.json Run preflight:\n./scripts/preflight-vm-guardrails.sh --plan-json tfplan.json Confirm clean post-apply state when a test intentionally creates a disposable VM:\nterraform apply tfplan terraform plan -detailed-exitcode Confirm teardown hygiene when the disposable VM is no longer needed:\nterraform plan -destroy -out=destroy.tfplan terraform show -json destroy.tfplan \\ | jq -r \u0026#39;.resource_changes[]? | [.address, .type, (.change.actions | join(\u0026#34;,\u0026#34;))] | @tsv\u0026#39; terraform apply destroy.tfplan 1. Clean New VM Test:\nPlan one new VM name and one unused IP address. Expected:\nterraform plan succeeds preflight passes NetBox resources are planned when enabled terraform apply succeeds for disposable test target follow-up plan shows no changes Proof to capture:\nVM name. IP and prefix. DNS name. planned NetBox VM/interface/IP/primary-IP resources. clean terraform plan -detailed-exitcode after apply. 2. Duplicate IP Inside Terraform Config Test:\nSet two planned VMs to the same ipv4_address. Command:\nterraform plan -out=/tmp/duplicate-ip-test.tfplan Expected failure:\nVM IPv4 addresses must be unique within var.vms. Layer responsible:\nTerraform variable validation 3. Duplicate VM Name Inside Terraform Config Test:\nSet two planned VMs to the same name. Command:\nterraform plan -out=/tmp/duplicate-name-test.tfplan Expected failure:\nVM names must be unique within var.vms. Layer responsible:\nTerraform variable validation 4. IP Already Exists In NetBox Test:\nUse an IP address that already exists in NetBox. Command:\n./scripts/preflight-vm-guardrails.sh --plan-json tfplan.json Expected failure:\nNetBox already has IP \u0026#39;192.0.2.10\u0026#39; (192.0.2.10/24 status=active dns=cluster-a-worker-01.example.com). Layer responsible:\nPreflight NetBox check NetBox provider resource on apply 5. VM Name Already Exists In NetBox Test:\nUse a VM name that already exists in NetBox. Command:\n./scripts/preflight-vm-guardrails.sh --plan-json tfplan.json Expected failure:\nNetBox already has VM \u0026#39;cluster-a-worker-01\u0026#39;. Layer responsible:\nPreflight NetBox check NetBox provider resource on apply 6. VM Exists In vCenter But Not NetBox Test:\nUse a VM name that exists in vCenter but does not exist in NetBox. Command:\n./scripts/preflight-vm-guardrails.sh \\ --plan-json tfplan.json \\ --skip-netbox \\ --skip-dns \\ --skip-nmap Expected failure:\nvCenter already has VM \u0026#39;cluster-a-worker-01\u0026#39;: /DC-Site-A/vm/K8s-Cluster/Prod/cluster-a-worker-01. Also test missing govc configuration:\nenv -u GOVC_URL -u GOVC_USERNAME -u GOVC_PASSWORD \\ ./scripts/preflight-vm-guardrails.sh \\ --plan-json tfplan.json \\ --skip-netbox \\ --skip-dns \\ --skip-nmap Expected failure:\nvCenter checks require a working govc configuration. Layer responsible:\nPreflight vCenter check 7. IP Active On Network But Not In NetBox Test:\nUse an IP that responds to nmap -sn -n but has no NetBox record. Command:\n./scripts/preflight-vm-guardrails.sh \\ --plan-json tfplan.json \\ --skip-netbox \\ --skip-govc \\ --skip-dns Expected failure:\nNetwork scan indicates planned IP \u0026#39;192.0.2.10\u0026#39; is already active. Layer responsible:\nPreflight nmap check Note: if the plan includes already-managed active VMs, the scan may flag those too. That is expected if the script scans all planned static IPs instead of only create/update actions.\n8. Reverse DNS Exists Test:\nUse an IP with an existing PTR record. Candidate checks:\ndig +short -x \u0026#39;192.0.2.10\u0026#39; getent hosts \u0026#39;192.0.2.10\u0026#39; Preflight command:\n./scripts/preflight-vm-guardrails.sh \\ --plan-json tfplan.json \\ --skip-netbox \\ --skip-govc \\ --skip-nmap Expected failure:\nReverse DNS/getent already resolves planned IP \u0026#39;192.0.2.10\u0026#39;. Layer responsible:\nPreflight reverse DNS check 9. Forward DNS Exists For VM Name Test:\nUse a VM name that already resolves in DNS. Candidate check:\ngetent hosts \u0026#39;cluster-a-worker-01\u0026#39; getent hosts \u0026#39;cluster-a-worker-01.example.com\u0026#39; Preflight command:\n./scripts/preflight-vm-guardrails.sh \\ --plan-json tfplan.json \\ --skip-netbox \\ --skip-govc \\ --skip-nmap Expected failure:\nDNS already resolves planned VM name \u0026#39;cluster-a-worker-01\u0026#39;. Layer responsible:\nPreflight forward DNS check 10. NetBox Enabled Without Provider Credentials Test:\nSet netbox_enabled = true and remove NETBOX_SERVER_URL / NETBOX_API_TOKEN. Command:\nunset NETBOX_SERVER_URL unset NETBOX_API_TOKEN terraform plan -out=/tmp/missing-netbox-creds.tfplan Expected failure:\nError: Missing required argument The argument \u0026#34;server_url\u0026#34; is required Error: Missing required argument The argument \u0026#34;api_token\u0026#34; is required Layer responsible:\nTerraform provider configuration 11. Preflight Without NetBox Credentials Test:\nRun preflight without NetBox credentials. Command:\nunset NETBOX_SERVER_URL unset NETBOX_API_TOKEN ./scripts/preflight-vm-guardrails.sh --plan-json tfplan.json --skip-govc Expected warning, if degraded mode is intentional:\nWarnings: - Skipping NetBox checks because NetBox URL/token environment variables or --netbox-url/--netbox-token were not provided. Expected behavior:\nDNS and nmap checks continue after the NetBox warning. Layer responsible:\nPreflight degraded-mode handling 12. NetBox Cluster Missing Test:\nSet netbox_enabled = true and use a non-existent netbox_cluster_name. Command:\nterraform plan -out=tfplan \\ -var=\u0026#39;netbox_cluster_name=missing-cluster-test\u0026#39; Expected failure:\nError: no result with module.vm_group.data.netbox_cluster.cluster[0] Layer responsible:\nTerraform NetBox data lookup 13. NetBox Unreachable Test:\nPoint the NetBox provider at an unreachable endpoint. Command:\nNETBOX_SERVER_URL=\u0026#34;https://netbox-unreachable.invalid\u0026#34; \\ NETBOX_API_TOKEN=\u0026#34;dummy\u0026#34; \\ NETBOX_SKIP_VERSION_CHECK=true \\ terraform plan -out=/tmp/netbox-unreachable-test.tfplan Expected failure:\nPlanning failed. Error: Get \u0026#34;https://netbox-unreachable.invalid/api/...\u0026#34; Layer responsible:\nTerraform NetBox provider/data lookup Desired outcome:\nNetBox unreachable -\u0026gt; Terraform plan fails -\u0026gt; vSphere VM is not created 14. Destroy Removes NetBox And vSphere Objects Test:\nDestroy a disposable Terraform-managed VM and verify Terraform removes its NetBox and vSphere records. Confirm current state:\nterraform state list Expected state includes:\nmodule.vm_group.netbox_interface.vm[\u0026#34;test-1\u0026#34;] module.vm_group.netbox_ip_address.vm[\u0026#34;test-1\u0026#34;] module.vm_group.netbox_primary_ip.vm[\u0026#34;test-1\u0026#34;] module.vm_group.netbox_virtual_machine.vm[\u0026#34;test-1\u0026#34;] module.vm_group.vsphere_virtual_machine.vm[\u0026#34;test-1\u0026#34;] Generate and inspect the destroy plan:\nterraform plan -destroy -out=destroy.tfplan terraform show -json destroy.tfplan \\ | jq -r \u0026#39;.resource_changes[]? | [.address, .type, (.change.actions | join(\u0026#34;,\u0026#34;))] | @tsv\u0026#39; Expected planned deletes:\nmodule.vm_group.netbox_primary_ip.vm[\u0026#34;test-1\u0026#34;] netbox_primary_ip delete module.vm_group.netbox_ip_address.vm[\u0026#34;test-1\u0026#34;] netbox_ip_address delete module.vm_group.netbox_interface.vm[\u0026#34;test-1\u0026#34;] netbox_interface delete module.vm_group.netbox_virtual_machine.vm[\u0026#34;test-1\u0026#34;] netbox_virtual_machine delete module.vm_group.vsphere_virtual_machine.vm[\u0026#34;test-1\u0026#34;] vsphere_virtual_machine delete Apply the destroy plan:\nterraform apply destroy.tfplan Verify Terraform state no longer lists the managed resources:\nterraform state list Verify the NetBox VM record is gone:\ncurl -s \\ -H \u0026#34;Authorization: Token $NETBOX_API_TOKEN\u0026#34; \\ -H \u0026#34;Accept: application/json\u0026#34; \\ \u0026#34;$NETBOX_SERVER_URL/api/virtualization/virtual-machines/?name=cluster-a-test-01\u0026#34; \\ | jq \u0026#39;.count\u0026#39; Expected:\n0 Verify the NetBox IP address record is gone:\ncurl -s \\ -H \u0026#34;Authorization: Token $NETBOX_API_TOKEN\u0026#34; \\ -H \u0026#34;Accept: application/json\u0026#34; \\ \u0026#34;$NETBOX_SERVER_URL/api/ipam/ip-addresses/?q=192.0.2.10\u0026#34; \\ | jq \u0026#39;.count\u0026#39; Expected:\n0 Verify the vSphere VM is gone:\ngovc find / -type m -name \u0026#39;cluster-a-test-01\u0026#39; Expected: no output.\nLayer responsible:\nTerraform destroy plus post-destroy NetBox and vCenter verification Successful result:\ncreate -\u0026gt; manage -\u0026gt; verify idempotency -\u0026gt; destroy -\u0026gt; cleanup NetBox and vSphere Temporary Test Cleanup For each destructive or collision test:\nadd only one temporary collision at a time. run the test. save the exact output. revert the temporary config immediately. confirm no diff remains. Cleanup checks:\ngit status --short -- path/to/test/env git diff -- path/to/test/env Operating Rule A VM guardrail is not proven until every unsafe path fails where you expect it to fail.\nDocument the test, the command, the observed output, and the cleanup state.\n","permalink":"https://trinidadmarroquin.com/field-notes/terraform-vm-guardrail-test-matrix/","section":"field-notes","summary":"Use this matrix to prove VM provisioning guardrails before trusting automation with real vSphere creates.\nThe point is not to make every test pass green. The point is to prove each unsafe condition fails at the correct layer.\nBaseline Commands Generate a plan and plan JSON:\nterraform init terraform plan -out=tfplan terraform show -json tfplan \u0026gt; tfplan.json Run preflight:\n./scripts/preflight-vm-guardrails.sh --plan-json tfplan.json Confirm clean post-apply state when a test intentionally creates a disposable VM:\n","tags":["terraform","vsphere","netbox","testing","validation","automation","operations"],"title":"Terraform VM Guardrail Test Matrix"},{"categories":["field-notes"],"content":"Terraform can validate its own input, but it cannot automatically prove that the outside world is clear.\nBefore applying a vSphere VM plan, run preflight checks against the systems that already know about names and addresses.\nAfter testing a disposable VM, verify destroy cleanup as a separate lifecycle check. Preflight protects allocation. Destroy verification protects source-of-truth hygiene.\nGenerate Plan JSON Use the plan as the preflight input:\nterraform plan -out=tfplan terraform show -json tfplan \u0026gt; tfplan.json Then run the guardrail script:\n./scripts/preflight-vm-guardrails.sh --plan-json tfplan.json Inputs To Extract From the plan JSON, extract each planned VM\u0026rsquo;s:\nTerraform address. VM name. IPv4 address. prefix length. DNS name, if generated. NetBox enabled/disabled state. Do not scrape HCL directly when a plan JSON is available. The plan reflects variables, locals, defaults, and module expansion after Terraform evaluation.\nNetBox Checks Check whether NetBox already has the planned VM name:\nGET /api/virtualization/virtual-machines/?name=\u0026lt;planned-name\u0026gt; Check whether NetBox already has the planned IP:\nGET /api/ipam/ip-addresses/?address=\u0026lt;planned-ip\u0026gt; Fail on exact matches:\nFailures: - NetBox already has VM \u0026#39;cluster-a-worker-01\u0026#39;. - NetBox already has IP \u0026#39;192.0.2.10\u0026#39; (192.0.2.10/24 status=active dns=cluster-a-worker-01.example.com). If NetBox credentials are missing, decide intentionally:\nfail if NetBox checks are mandatory for the environment. warn and continue only if the script explicitly supports degraded mode. Example warning for degraded mode:\nWarnings: - Skipping NetBox checks because NetBox URL/token environment variables or --netbox-url/--netbox-token were not provided. vCenter Checks Use govc to check whether the VM name already exists in vCenter:\ngovc find / -type m -name \u0026#39;cluster-a-worker-01\u0026#39; Failure example:\nFailures: - vCenter already has VM \u0026#39;cluster-a-worker-01\u0026#39;: /DC-Site-A/vm/K8s-Cluster/Prod/cluster-a-worker-01. Do not silently skip this check when govc is broken.\nIf vCenter checks are enabled, verify govc first:\ngovc about \u0026gt;/dev/null Failure example:\nFailures: - vCenter checks require a working govc configuration. Export GOVC_URL, GOVC_USERNAME, GOVC_PASSWORD, and GOVC_INSECURE as needed, or rerun with --skip-govc. Forward DNS Checks Check whether the planned VM name already resolves:\ngetent hosts \u0026#39;cluster-a-worker-01\u0026#39; getent hosts \u0026#39;cluster-a-worker-01.example.com\u0026#39; Failure example:\nFailures: - DNS already resolves planned VM name \u0026#39;cluster-a-worker-01\u0026#39;. Forward DNS catches stale names that may not exist in NetBox or vCenter anymore.\nReverse DNS Checks Check whether the planned IP has a PTR record:\ndig +short -x \u0026#39;192.0.2.10\u0026#39; Fallback if dig is unavailable:\ngetent hosts \u0026#39;192.0.2.10\u0026#39; Failure example:\nFailures: - Reverse DNS/getent already resolves planned IP \u0026#39;192.0.2.10\u0026#39;. Reverse DNS is useful because PTR records often outlive the systems they describe.\nNetwork Liveness Checks Check whether the planned IP responds on the network:\nnmap -sn -n \u0026#39;192.0.2.10\u0026#39; Failure example:\nFailures: - Network scan indicates planned IP \u0026#39;192.0.2.10\u0026#39; is already active. This catches the important case where an address is active but missing from NetBox.\nSkip Flags Skip flags are useful for isolated tests and controlled degraded mode:\n./scripts/preflight-vm-guardrails.sh \\ --plan-json tfplan.json \\ --skip-netbox \\ --skip-govc \\ --skip-dns \\ --skip-nmap But skip flags should be visible in the summary:\nPreflight VM guardrail summary Planned VMs checked: 2 NetBox checks: skipped vCenter checks: enabled DNS checks: enabled nmap checks: enabled Safe Summary Format Make the output explicit enough to paste into a change record:\nPreflight VM guardrail summary Planned VMs checked: 2 NetBox checks: enabled vCenter checks: enabled DNS checks: enabled nmap checks: enabled Warnings: 0 Failures: 0 On failure, include the exact proof:\nFailures: - vCenter already has VM \u0026#39;cluster-a-worker-01\u0026#39;: /DC-Site-A/vm/K8s-Cluster/Prod/cluster-a-worker-01. - DNS already resolves planned VM name \u0026#39;cluster-a-worker-01\u0026#39;. - Network scan indicates planned IP \u0026#39;192.0.2.10\u0026#39; is already active. Operating Rule Preflight does not replace teardown verification.\nFor disposable test VMs, finish with a destroy-plan review and post-destroy checks:\nterraform plan -destroy -out=destroy.tfplan terraform show -json destroy.tfplan \\ | jq -r \u0026#39;.resource_changes[]? | [.address, .type, (.change.actions | join(\u0026#34;,\u0026#34;))] | @tsv\u0026#39; terraform apply destroy.tfplan terraform state list Then verify the external systems are clean:\ncurl -s \\ -H \u0026#34;Authorization: Token $NETBOX_API_TOKEN\u0026#34; \\ -H \u0026#34;Accept: application/json\u0026#34; \\ \u0026#34;$NETBOX_SERVER_URL/api/virtualization/virtual-machines/?name=cluster-a-test-01\u0026#34; \\ | jq \u0026#39;.count\u0026#39; curl -s \\ -H \u0026#34;Authorization: Token $NETBOX_API_TOKEN\u0026#34; \\ -H \u0026#34;Accept: application/json\u0026#34; \\ \u0026#34;$NETBOX_SERVER_URL/api/ipam/ip-addresses/?q=192.0.2.10\u0026#34; \\ | jq \u0026#39;.count\u0026#39; govc find / -type m -name \u0026#39;cluster-a-test-01\u0026#39; Expected results:\nNetBox VM count: 0 NetBox IP count: 0 govc find: no output Terraform plan answers “what will Terraform try to do?”\nPreflight answers “is the outside world clear enough for Terraform to do it safely?”\nDestroy verification answers “did Terraform clean up the outside-world records it created?”\nRun all three before trusting vSphere VM automation end to end.\n","permalink":"https://trinidadmarroquin.com/field-notes/terraform-vsphere-vm-preflight-guardrails/","section":"field-notes","summary":"Terraform can validate its own input, but it cannot automatically prove that the outside world is clear.\nBefore applying a vSphere VM plan, run preflight checks against the systems that already know about names and addresses.\nAfter testing a disposable VM, verify destroy cleanup as a separate lifecycle check. Preflight protects allocation. Destroy verification protects source-of-truth hygiene.\nGenerate Plan JSON Use the plan as the preflight input:\nterraform plan -out=tfplan terraform show -json tfplan \u0026gt; tfplan.json Then run the guardrail script:\n","tags":["terraform","vsphere","netbox","govc","dns","automation","operations"],"title":"Terraform vSphere VM Preflight Guardrails"},{"categories":["field-notes"],"content":"AI agents can help with infrastructure work, but tool access does not make an answer trustworthy.\nUse guardrails when an agent is helping with NetBox, IPAM, Jira, Kubernetes, Terraform, or any other system where a wrong write creates cleanup work.\nRed Flags Slow down when the agent:\nsays it can write but cannot prove the target endpoint. assumes import behavior without testing one row. retries the same broken tool call repeatedly. switches from evidence to confident explanation. mixes generated examples with real environment values. proposes a bulk change without a dry run. cannot distinguish source data from desired state. ignores duplicate names or ambiguous matches. Required Behavior For infrastructure writes, require:\nplan before execution. exact target system and endpoint. dry-run output. object counts before and after. duplicate detection. explicit confirmation before write. post-write verification. clear rollback or cleanup path. Good instruction:\nDo not speculate. Do not guess. Show what was verified. If evidence is missing, stop and ask. Prefer Artifacts Over Chat For bulk changes, ask for artifacts:\nCSV files. scripts. validation reports. duplicate reports. import instructions. dry-run logs. Artifacts can be reviewed, versioned, and rerun. A long chat response is harder to audit.\nWhen To Start A New Session Start fresh when:\nthe agent keeps carrying forward a bad assumption. tool calls are malformed repeatedly. the session mixes too many unrelated tasks. the agent loses track of what was actually verified. you need a clean second opinion on generated files. A new session is not a failure. It is a context reset.\nOperating Rule Use AI to reduce toil, not to bypass operational discipline.\nThe human operator owns the source of truth, the write target, and the final verification.\n","permalink":"https://trinidadmarroquin.com/field-notes/ai-agent-guardrails-for-infrastructure-writes/","section":"field-notes","summary":"AI agents can help with infrastructure work, but tool access does not make an answer trustworthy.\nUse guardrails when an agent is helping with NetBox, IPAM, Jira, Kubernetes, Terraform, or any other system where a wrong write creates cleanup work.\nRed Flags Slow down when the agent:\nsays it can write but cannot prove the target endpoint. assumes import behavior without testing one row. retries the same broken tool call repeatedly. switches from evidence to confident explanation. mixes generated examples with real environment values. proposes a bulk change without a dry run. cannot distinguish source data from desired state. ignores duplicate names or ambiguous matches. Required Behavior For infrastructure writes, require:\n","tags":["ai","automation","sre","operations","infrastructure"],"title":"AI Agent Guardrails For Infrastructure Writes"},{"categories":["field-notes"],"content":"NetBox imports are safest when object relationships are created in dependency order.\nFor Kubernetes VM nodes, split the import into devices, interfaces, and IP addresses instead of trying to represent everything as one operation.\nImport Order Use this order:\n1. devices.csv 2. interfaces.csv 3. ip-addresses.csv Why:\ninterfaces need devices to exist first. IP addresses need interfaces to exist before assignment. primary IP selection is a device relationship and may need a later update. Device CSV Example shape:\nname,role,manufacturer,device_type,site,status,tags,comments cluster-a-cp-01,Server,VMware,Virtual Machine,site-a,active,\u0026#34;cluster-a-prod,k8s-node,control-plane\u0026#34;,Kubernetes control-plane node cluster-a-worker-01,Server,VMware,Virtual Machine,site-a,active,\u0026#34;cluster-a-prod,k8s-node,worker\u0026#34;,Kubernetes worker node Confirm these objects exist before import:\nsite. role. manufacturer. device type. tags. Do not assume the NetBox UI will create tags with the slugs you want. Pre-create important tags when naming consistency matters.\nInterface CSV Example shape:\ndevice,name,type,enabled cluster-a-cp-01,mgmt0,virtual,true cluster-a-worker-01,mgmt0,virtual,true For virtual machines, virtual is usually clearer than a physical media type unless your NetBox model intentionally tracks the emulated adapter type.\nKeep the management interface name consistent. mgmt0 is simple and predictable for automation.\nIP Address CSV Example shape:\naddress,status,dns_name,description,device,interface 192.0.2.10/24,active,cluster-a-cp-01.example.internal,Kubernetes control-plane management IP,cluster-a-cp-01,mgmt0 192.0.2.20/24,active,cluster-a-worker-01.example.internal,Kubernetes worker management IP,cluster-a-worker-01,mgmt0 192.0.2.2/24,reserved,cluster-a-api.example.internal,Kubernetes API VIP,, VIPs can be intentionally unassigned. Make that explicit with reserved status and blank device/interface fields.\nValidation Checklist Before import, verify:\nno duplicate device names. no duplicate IP addresses. no duplicate DNS names where uniqueness matters. every interface references a device in devices.csv. every assigned IP references a device/interface pair in the generated files. every IP belongs to the expected prefix. VIPs are intentionally unassigned. required NetBox objects already exist. Operating Rule Generate CSVs from source data, then validate the relationships before opening the NetBox UI.\nIf the validation finds duplicate device names or ambiguous ownership, stop and resolve the naming decision before import.\n","permalink":"https://trinidadmarroquin.com/field-notes/netbox-kubernetes-node-bulk-import/","section":"field-notes","summary":"NetBox imports are safest when object relationships are created in dependency order.\nFor Kubernetes VM nodes, split the import into devices, interfaces, and IP addresses instead of trying to represent everything as one operation.\nImport Order Use this order:\n1. devices.csv 2. interfaces.csv 3. ip-addresses.csv Why:\ninterfaces need devices to exist first. IP addresses need interfaces to exist before assignment. primary IP selection is a device relationship and may need a later update. Device CSV Example shape:\n","tags":["netbox","kubernetes","ipam","automation","operations"],"title":"NetBox Bulk Import Order For Kubernetes Nodes"},{"categories":["field-notes"],"content":"Importing an IP address and assigning it to an interface does not always mean the device\u0026rsquo;s primary_ip4 field is set.\nTreat primary IP assignment as a separate verification and update step.\nInput File Use a small mapping file:\nname,primary_ip4 cluster-a-cp-01,192.0.2.10/24 cluster-a-worker-01,192.0.2.20/24 The address should match the existing NetBox IP object, including prefix length if your NetBox query requires it.\nSafe Update Logic For each row:\nquery device by exact name. query IP address by exact address. require exactly one device match. require exactly one IP match. skip blank rows. skip ambiguous rows. support dry-run mode. patch only the device primary_ip4 field. Ambiguity should stop the row, not trigger guessing.\nExample API Flow Pseudo-flow:\nGET /api/dcim/devices/?name=\u0026lt;device-name\u0026gt; GET /api/ipam/ip-addresses/?address=\u0026lt;address\u0026gt; PATCH /api/dcim/devices/\u0026lt;id\u0026gt;/ {\u0026#34;primary_ip4\u0026#34;: \u0026lt;ip-id\u0026gt;} Dry-run output should show:\nDRY RUN: cluster-a-cp-01 -\u0026gt; 192.0.2.10/24 (device_id=123 ip_id=456) Only run the patch after the dry run shows the expected device/IP pairs.\nVerification After the update:\nopen a few representative devices in the UI. filter devices by cluster tag. confirm primary IPv4 is populated. confirm VIPs remain reserved and unassigned if that was intended. confirm no unexpected device count changes occurred. Operating Rule Primary IP assignment is a relationship update, not just an IPAM import.\nMake it idempotent, exact-match only, and safe to dry-run.\n","permalink":"https://trinidadmarroquin.com/field-notes/netbox-primary-ip-after-import/","section":"field-notes","summary":"Importing an IP address and assigning it to an interface does not always mean the device\u0026rsquo;s primary_ip4 field is set.\nTreat primary IP assignment as a separate verification and update step.\nInput File Use a small mapping file:\nname,primary_ip4 cluster-a-cp-01,192.0.2.10/24 cluster-a-worker-01,192.0.2.20/24 The address should match the existing NetBox IP object, including prefix length if your NetBox query requires it.\nSafe Update Logic For each row:\nquery device by exact name. query IP address by exact address. require exactly one device match. require exactly one IP match. skip blank rows. skip ambiguous rows. support dry-run mode. patch only the device primary_ip4 field. Ambiguity should stop the row, not trigger guessing.\n","tags":["netbox","ipam","automation","api","operations"],"title":"NetBox Primary IP Assignment After Import"},{"categories":["notes"],"content":"AI assistance is useful in infrastructure work when it accelerates the boring parts: parsing inventory, building CSVs, checking references, and producing repeatable commands.\nIt becomes dangerous when it gets confident without evidence.\nIn one NetBox/IPAM update, the task sounded simple: take Kubernetes node inventory, create the matching NetBox records, associate management IPs, and set each device\u0026rsquo;s primary IPv4 address. The work was not conceptually hard. The risk was in the details: object relationships, import order, duplicate names, existing tags, site mappings, and whether the tool was writing to the intended NetBox instance.\nThe first AI-assisted path was frustrating because the agent began to infer too much. It mixed real checks with assumptions about NetBox behavior. It had access to tools, but tool access did not automatically make the work safe.\nThe better path started when the workflow became evidence-driven again.\nThe Work To Be Done The desired model was straightforward:\nKubernetes node inventory -\u0026gt; NetBox devices -\u0026gt; management interfaces -\u0026gt; IP address assignments -\u0026gt; primary IPv4 on each device For each node, the useful data was:\nhostname. management IP and prefix length. site. device role. cluster tag. node function such as control plane, worker, etcd, monitor, or storage. whether an IP was assigned to a device or reserved as a VIP. That maps cleanly to NetBox, but not as one flat operation. Devices, interfaces, IP addresses, and primary IP fields are related objects. If those relationships are created in the wrong order, the import becomes noisy or incomplete.\nWhere The Agent Went Wrong The bot did some useful things. It identified needed fields, proposed dry-run behavior, and recognized that deletes and bulk changes needed confirmation.\nBut it also showed classic AI-agent failure modes:\nassuming NetBox would auto-create tags safely. assuming a CSV field would behave the same across NetBox versions. treating a tool\u0026rsquo;s claimed write access as proof that the write path was correct. trying to generate large downloadable files through a broken tool path. continuing to retry malformed tool calls instead of changing strategy. mixing “this should work” with “this was verified.” The uncomfortable part was not that the model was wrong once. It was that the model could be wrong confidently while still sounding operationally fluent.\nThat is the exact situation where an operator needs to slow the workflow down.\nThe Better Pattern The safer workflow used generated artifacts and validation instead of direct trust in the assistant.\nThe import was split into three CSVs:\ndevices.csv interfaces.csv ip-addresses.csv The order mattered:\n1. Devices 2. Interfaces 3. IP addresses Devices need to exist before interfaces can reference them. Interfaces need to exist before IP addresses can attach to them. Reserved VIPs can exist without device/interface assignment, but that should be intentional and visible in the data.\nAfter the import, primary IPv4 assignment was handled separately. That separated “create the object graph” from “set the device display relationship.”\nThat distinction matters. It makes verification easier and avoids turning one failed field into a failed bulk import.\nValidation Before Import Before importing, the generated CSVs were checked for relationship consistency:\nevery interface referenced a known device. every assigned IP referenced a known device and interface. VIP rows were intentionally unassigned and marked reserved. device names were unique. interface tuples were unique. IP addresses were unique. DNS names were unique where that mattered operationally. all addresses belonged to the expected prefix. required NetBox objects existed first: site, role, manufacturer, device type, and tags. The duplicate-name check was especially important. A repeated node-name pattern across two environments can look harmless in a text file but become a hard NetBox conflict. The validation step surfaced that before import.\nThat is exactly where AI is useful: not to “just do it,” but to help build mechanical checks around human decisions.\nPrimary IPs As A Separate Pass NetBox can associate an IP address to an interface and still not have that IP set as the device\u0026rsquo;s primary IPv4 address.\nThat is not a failure. It is a separate relationship.\nThe safer post-import approach was:\ndevice name, primary IPv4 address Then for each row:\nquery NetBox for exactly one matching device. query NetBox for exactly one matching IP address. skip the row if either lookup is ambiguous or missing. patch the device only after a dry run succeeds. This is a good example of operational automation being intentionally boring. The script does not need to be clever. It needs to refuse unsafe ambiguity.\nThe Most Important Guardrail The strongest guardrail was not code. It was language.\nThe useful instruction to the agent was effectively:\nDo not speculate. Do not guess. Show what you verified. Stop when the evidence is missing. That changed the shape of the work. Instead of asking the agent to own the outcome, the operator used it to produce artifacts that could be inspected:\nCSV files. validation summaries. duplicate reports. import instructions. dry-run output. API patch scripts. Those artifacts are reviewable. A fluent chat answer is not enough.\nLessons For AI-Assisted SRE Work The practical lessons are simple:\nTool access is not trust. A successful API call is not proof that the correct system was updated. Bulk writes need a plan and a verification path. Generated files are safer than huge inline responses. Dry runs should be explicit and boring. Ambiguous matches should skip, not guess. Starting a new session can be the right move when context gets polluted. AI can reduce toil in inventory and IPAM work. It can parse, normalize, generate, and validate faster than a person doing it by hand.\nBut the operator still owns the source of truth.\nThe win is not “the bot updated NetBox.” The win is “the bot helped produce a repeatable, validated workflow that a human could reason about before anything changed.”\nThat is the difference between automation and gambling.\n","permalink":"https://trinidadmarroquin.com/posts/ai-assisted-netbox-inventory-guardrails/","section":"posts","summary":"AI assistance is useful in infrastructure work when it accelerates the boring parts: parsing inventory, building CSVs, checking references, and producing repeatable commands.\nIt becomes dangerous when it gets confident without evidence.\nIn one NetBox/IPAM update, the task sounded simple: take Kubernetes node inventory, create the matching NetBox records, associate management IPs, and set each device\u0026rsquo;s primary IPv4 address. The work was not conceptually hard. The risk was in the details: object relationships, import order, duplicate names, existing tags, site mappings, and whether the tool was writing to the intended NetBox instance.\n","tags":["netbox","automation","sre","infrastructure","ai","operations"],"title":"When AI-Assisted Infrastructure Updates Need Guardrails"},{"categories":["field-notes"],"content":"When a flow collector reports a huge byte count, resist the urge to start with the loudest application log. Start by proving whether the node, interface, and time window support the story.\nThis checklist is useful when Kubernetes or Rancher appears to be involved in a network spike.\nIdentify The Conversation Ask for the flow detail, not only the top-talker summary:\nsource IP destination IP source port destination port protocol bytes sessions start time end time Important ports in Rancher/RKE2 environments include:\n443 Rancher, ingress, Kubernetes services 6443 Kubernetes API 9345 RKE2 supervisor 10250 kubelet 2379 etcd client 2380 etcd peer 5473 Calico Typha 9090 Prometheus 3100 Loki Without ports and timestamps, top talkers are only clues.\nMap IPs To Ownership For Kubernetes nodes:\nkubectl get nodes -o wide | grep \u0026lt;ip\u0026gt; kubectl get pods -A -o wide | grep \u0026lt;ip\u0026gt; For DNS-owned endpoints:\ngetent hosts \u0026lt;ip-or-name\u0026gt; nslookup \u0026lt;name\u0026gt; dig -x \u0026lt;ip\u0026gt; Classify each endpoint:\ncontrol-plane node. etcd node. worker node. Rancher load balancer. ingress node. backup host. monitoring system. NAT or proxy. Check Live Interface Health Find the interface used for the path:\nIFACE=$(ip route get \u0026lt;destination-ip\u0026gt; | awk \u0026#39;{print $5; exit}\u0026#39;) echo \u0026#34;$IFACE\u0026#34; Check counters:\nip -s link show \u0026#34;$IFACE\u0026#34; sudo ethtool \u0026#34;$IFACE\u0026#34; sudo ethtool -S \u0026#34;$IFACE\u0026#34; | egrep -i \u0026#39;drop|err|timeout|reset|miss|coll|crc|fifo|buf|fail\u0026#39; Look for:\nRX or TX errors. drops. CRC errors. carrier changes. TX timeouts. ring or buffer failures. No interface errors does not prove the network was never congested. It only rules out one class of local NIC problem.\nMeasure Current Throughput If sysstat is available:\nsar -n DEV 1 10 sar -n TCP,ETCP 1 10 Look for:\nhigh %ifutil. high retrans/s. high estres/s. high orsts/s. high isegerr/s. For cumulative TCP counters:\nnstat -az | egrep -i \u0026#39;Retrans|Timeout|Listen|Reset|TCPAbort|TCPLoss\u0026#39; Cumulative counters need context. A large number since boot does not prove a current storm.\nInspect Current Sockets Count established peers:\nsudo ss -tan state established | \\ awk \u0026#39;NR\u0026gt;1 {print $5}\u0026#39; | \\ sed \u0026#39;s/::ffff://g\u0026#39; | \\ cut -d: -f1 | \\ sort | uniq -c | sort -nr | head -30 Focus on a suspected endpoint:\nsudo ss -tanp | grep \u0026lt;ip\u0026gt; sudo ss -tan state time-wait | grep \u0026lt;ip\u0026gt; | wc -l A current socket snapshot is not historical proof. Use it to form the next question, not to close the case.\nCheck Kernel Logs journalctl -k --since \u0026#39;24 hours ago\u0026#39; | \\ egrep -i \u0026#39;eth|ens|link|nic|tx|rx|drop|timeout|reset|watchdog|NETDEV|soft lockup\u0026#39; This helps rule out link flaps, driver issues, and kernel-visible network failures.\nValidate Etcd Before Blaming Etcd If an etcd node appears noisy, check the actual event window:\nsudo journalctl -u rke2-server \\ --since \u0026#39;\u0026lt;event-start-local\u0026gt;\u0026#39; \\ --until \u0026#39;\u0026lt;event-end-local\u0026gt;\u0026#39; | \\ egrep -i \u0026#39;error|warn|timeout|reset|disconnect|etcd|snapshot|defrag|compact|leader|slow|trace\u0026#39; Check snapshots:\nsudo ls -ltrh /var/lib/rancher/rke2/server/db/snapshots A small local snapshot is not enough to explain hundreds of gigabytes of traffic.\nCheck Scheduled Work If the spike happened at a predictable time, check scheduled work early:\nkubectl get cronjobs -A kubectl get jobs -A | egrep -i \u0026#39;backup|snapshot|sync|replication|export\u0026#39; On backup hosts:\njournalctl --since \u0026#39;\u0026lt;event-start-local\u0026gt;\u0026#39; --until \u0026#39;\u0026lt;event-end-local\u0026gt;\u0026#39; Network saturation from backup or replication jobs can make Rancher, Kubernetes API watches, Prometheus, and GitOps controllers look guilty because they are latency-sensitive.\nOperating Rule Separate three questions:\nWhat emitted the symptom? What generated the bytes? What changed during the exact time window? Those are often three different systems.\n","permalink":"https://trinidadmarroquin.com/field-notes/network-saturation-evidence-checklist/","section":"field-notes","summary":"When a flow collector reports a huge byte count, resist the urge to start with the loudest application log. Start by proving whether the node, interface, and time window support the story.\nThis checklist is useful when Kubernetes or Rancher appears to be involved in a network spike.\nIdentify The Conversation Ask for the flow detail, not only the top-talker summary:\nsource IP destination IP source port destination port protocol bytes sessions start time end time Important ports in Rancher/RKE2 environments include:\n","tags":["networking","linux","kubernetes","observability","troubleshooting"],"title":"Network Saturation Evidence Checklist"},{"categories":["field-notes"],"content":"Rancher cluster-agent errors can look like Rancher is the root cause. Sometimes Rancher is only the first component sensitive enough to report a degraded network path.\nUse this note when Rancher-managed clusters show large management-plane traffic, intermittent disconnects, or websocket errors.\nCommon Symptoms In downstream cluster-agent logs:\nFailed to dial steve aggregation server Remotedialer proxy error websocket: close 1006 (abnormal closure) context deadline exceeded i/o timeout connection reset by peer In Rancher server logs:\ntunnel disconnect ReverseProxy read error during body copy unable to decode event from watch stream Failed to watch Namespace Failed to watch Secret TLS handshake timeout In ingress or load balancer logs:\n499 502 504 upstream timed out client disconnected websocket EOF These messages prove connection disruption. They do not prove Rancher generated the original traffic spike.\nConfirm The Rancher Endpoint Resolve the Rancher endpoint and identify whether it is a VIP, load balancer, or DNS round-robin target:\nnslookup rancher.example.internal dig rancher.example.internal dig -x \u0026lt;rancher-vip\u0026gt; If the VIP fronts HAProxy, Keepalived, ingress, or multiple Rancher nodes, inspect each layer separately.\nFind Cluster-Agent Placement On the downstream cluster:\nkubectl get pods -A -o wide | egrep \u0026#39;cattle|fleet|rancher|system-agent\u0026#39; kubectl -n cattle-system get pods -o wide Then check the noisy agent:\nkubectl -n cattle-system logs \u0026lt;cattle-cluster-agent-pod\u0026gt; --tail=300 \\ | egrep -i \u0026#39;error|warn|websocket|disconnect|reconnect|timeout|reset|steve|remotedialer\u0026#39; Compare replicas. One noisy replica and one quiet replica can point to node placement, pod network, or a backend path issue.\nTest From The Agent Pod Connectivity to Rancher should work from the same pod that reports errors:\nkubectl -n cattle-system exec \u0026lt;cattle-cluster-agent-pod\u0026gt; -- \\ curl -sk -o /dev/null -w \u0026#39;http=%{http_code} connect=%{time_connect} tls=%{time_appconnect} total=%{time_total}\\n\u0026#39; \\ https://rancher.example.internal/v3/connect/config An HTTP 401 can be a healthy unauthenticated response. Timeouts, slow connects, TLS delays, or intermittent failures are the important signals.\nIf the VIP has known backend members, test each one:\nfor ip in \u0026lt;lb-member-1\u0026gt; \u0026lt;lb-member-2\u0026gt;; do kubectl -n cattle-system exec \u0026lt;cattle-cluster-agent-pod\u0026gt; -- \\ curl -sk --resolve rancher.example.internal:443:${ip} \\ -o /dev/null -w \u0026#34;${ip} http=%{http_code} connect=%{time_connect} tls=%{time_appconnect} total=%{time_total}\\n\u0026#34; \\ https://rancher.example.internal/v3/connect/config done One slow or failing backend can create intermittent remotedialer churn.\nCheck The Load Balancer Layer On HAProxy or the Rancher load balancer:\nsudo journalctl -u haproxy --since \u0026#39;24 hours ago\u0026#39; sudo journalctl -u keepalived --since \u0026#39;24 hours ago\u0026#39; Summarize top clients:\nsudo journalctl -u haproxy --since \u0026#39;24 hours ago\u0026#39; \\ | awk \u0026#39;/rancher/ {print $6}\u0026#39; \\ | cut -d: -f1 \\ | sort | uniq -c | sort -nr | head -20 Check websocket-related timeouts:\ngrep -i timeout /etc/haproxy/haproxy.cfg For Rancher, pay attention to long-lived connection settings such as timeout tunnel, timeout client, and timeout server.\nCorrelate The Time Window Do not accept nearby logs as proof. Convert UTC and local time carefully.\nFor a suspected event window:\nkubectl -n cattle-system logs \u0026lt;cattle-cluster-agent-pod\u0026gt; \\ --since-time=\u0026#39;\u0026lt;event-start-utc\u0026gt;\u0026#39; \\ | egrep -i \u0026#39;timeout|websocket|remotedialer|disconnect|reset\u0026#39; Then compare with:\nbackup schedules. HAProxy logs. ingress logs. flow collector timestamps. node-level network counters. Rancher server logs. If the Rancher errors happened hours after the traffic spike, treat them as a separate symptom until proven otherwise.\nOperating Rule Rancher remotedialer errors are a signal to inspect the management path, not a final diagnosis.\nBefore blaming Rancher, check whether another workload saturated the network and Rancher simply reported the pain first.\n","permalink":"https://trinidadmarroquin.com/field-notes/rancher-remotedialer-network-symptoms/","section":"field-notes","summary":"Rancher cluster-agent errors can look like Rancher is the root cause. Sometimes Rancher is only the first component sensitive enough to report a degraded network path.\nUse this note when Rancher-managed clusters show large management-plane traffic, intermittent disconnects, or websocket errors.\nCommon Symptoms In downstream cluster-agent logs:\nFailed to dial steve aggregation server Remotedialer proxy error websocket: close 1006 (abnormal closure) context deadline exceeded i/o timeout connection reset by peer In Rancher server logs:\n","tags":["rancher","kubernetes","rke2","networking","troubleshooting"],"title":"Rancher Remotedialer Network Symptoms"},{"categories":["notes"],"content":"A large network graph is good at creating anxiety. It is not always good at explaining cause.\nIn one investigation, a Rancher API load balancer appeared to receive hundreds of gigabytes of traffic in a 24-hour window. One RKE2 control-plane node appeared as a major sender. The first instinct was reasonable: something in Rancher, Kubernetes, monitoring, or the cluster agents might be misbehaving.\nThat was the right place to start. It was not the right place to stop.\nBy the end of the investigation, the stronger explanation was not that Rancher generated the original event. The better explanation was that a backup workload started during the same late-night window, saturated the network path, and made Rancher control-plane traffic look broken. Rancher was not the root cause. Rancher was the canary.\nThe Initial Signal The flow summary had two uncomfortable facts:\na Rancher API load balancer VIP was the largest destination by bytes. an RKE2 control-plane node appeared to send a large amount of traffic. the session count was high enough to suggest control-plane chatter, not a simple one-time file copy. the destination was an internal Rancher endpoint, not an unknown public host. The Rancher endpoint mattered. Large byte counts to a Rancher API load balancer can come from several normal-but-expensive sources:\ndownstream cluster agents. Fleet agents. Kubernetes API watches. Argo CD or GitOps controllers. Prometheus scraping. ingress and remotedialer websocket traffic. reconnect churn after a degraded network path. That does not make the traffic harmless. It only means the investigation should begin with protocol, port, timestamp, and component ownership before assuming exfiltration or malware.\nFirst Hypothesis: Rancher Management Traffic The Rancher VIP resolved to the internal API load balancer. That made the first working theory straightforward:\ndownstream RKE2 cluster -\u0026gt; cattle-cluster-agent -\u0026gt; Rancher API load balancer -\u0026gt; Rancher server / ingress The control-plane node was not a random worker. It hosted core RKE2 components and one of the Rancher cluster-agent replicas. That made the node a plausible source for control-plane sessions.\nThe first check was pod placement:\nkubectl get pods -A -o wide | grep \u0026lt;node-ip\u0026gt; kubectl get pods -A -o wide | egrep \u0026#39;cattle|fleet|rancher|system-agent\u0026#39; The interesting pods were the cattle-cluster-agent replicas. They are expected to maintain long-lived connections back to Rancher. If those connections fail repeatedly, the bytes and sessions can grow quickly without any application workload moving data.\nThe Logs Looked Like Rancher Was Broken Cluster-agent logs showed the kind of messages that get an operator\u0026rsquo;s attention:\nFailed to dial steve aggregation server Remotedialer proxy error websocket: close 1006 (abnormal closure) context deadline exceeded i/o timeout The Rancher server side later showed matching symptoms:\ntunnel disconnect ReverseProxy read error during body copy unable to decode event from watch stream Failed to watch Namespace Failed to watch Secret TLS handshake timeout Those messages are useful evidence, but they are not a root cause by themselves. They tell us the Rancher remotedialer and watch streams were disrupted. They do not tell us why.\nAt this point there were several reasonable suspects:\nload balancer websocket timeout. HAProxy reload or backend health flap. firewall idle timeout. Rancher pod resource pressure. ingress-nginx timeout or reload behavior. packet loss or congestion between subnets. backup or replication traffic sharing the same path. Proving What Was Not Happening The important move was to check the node and network before blaming Rancher.\nOn the suspected RKE2 node, live interface counters were quiet:\nIFACE=$(ip route get \u0026lt;rancher-vip\u0026gt; | awk \u0026#39;{print $5; exit}\u0026#39;) ip -s link show \u0026#34;$IFACE\u0026#34; sar -n DEV 1 10 sar -n TCP,ETCP 1 10 nstat -az | egrep -i \u0026#39;Retrans|Timeout|Listen|Reset|TCPAbort|TCPLoss\u0026#39; journalctl -k --since \u0026#34;24 hours ago\u0026#34; | egrep -i \u0026#39;link|nic|tx|rx|drop|timeout|reset|watchdog|NETDEV\u0026#39; That ruled out several tempting explanations:\nno live NIC saturation on the node. no interface error pattern. no obvious packet-drop storm. no current TCP retransmission storm. no kernel log evidence of link instability. The node was not melting down.\nThe Etcd Detour A current socket snapshot showed many connections to another RKE2 node. That node turned out to be an etcd member. That was worth checking because etcd events can produce control-plane noise:\nsnapshots. compaction. defragmentation. leader election. raft catch-up after a member falls behind. The time-windowed logs did show an etcd snapshot around midnight, but it was small and local:\nsnapshot taken from localhost snapshot size roughly tens of MB snapshot completed quickly That ruled out the etcd snapshot as an explanation for hundreds of gigabytes of traffic.\nThis was an important false lead to eliminate. Without checking size, endpoint, and timing, it would have been easy to say \u0026ldquo;etcd snapshot at midnight\u0026rdquo; and stop too early.\nThe Timeline Broke The Rancher Theory The strongest correction came from timestamps.\nSome Rancher disconnects happened during the live investigation, around mid-morning local time. The large traffic event was reported around the late-night backup window.\nThat mismatch changed the story:\nRancher remotedialer symptoms existed. The symptoms were real. But the captured symptoms did not line up cleanly with the original traffic spike. This is where incident work often goes wrong. A system can have a real problem that is not the cause of the event being investigated. The logs can be true and still be the wrong explanation.\nThe Better Root Cause Candidate After talking with another engineer, the timeline lined up with a scheduled backup job. That explanation fit better than Rancher as the primary generator:\nbackup job starts late at night -\u0026gt; network path becomes congested -\u0026gt; Rancher websocket and watch traffic experiences delay or drops -\u0026gt; cluster agents log timeouts and websocket closures -\u0026gt; Rancher LB and ingress show elevated sessions and churn -\u0026gt; flow collector highlights Rancher endpoints as noisy In that model, Rancher was not the thing saturating the network. Rancher was the sensitive control-plane workload that complained when the network was saturated by something else.\nThat fits distributed systems well. Control-plane traffic often breaks noisily before the network looks fully down.\nWhy This Investigation Was Still Valuable The exercise built a useful mental model for future incidents.\nWhen Rancher-managed RKE2 clusters experience network contention, the symptoms may appear as:\nRemotedialer proxy error. websocket close 1006. Failed to dial steve aggregation server. context deadline exceeded. tunnel disconnect. watch stream decode failures. intermittent API or ingress errors. large Rancher LB byte counts with many sessions. Those symptoms should trigger a wider checklist:\nIs a backup, replication, or migration running? Did the traffic start at a scheduled batch window? Are Rancher errors aligned with the spike or just nearby? Are HAProxy/ingress disconnects cause or symptom? Are etcd events large enough to matter? Are interface counters showing live saturation, or only historical noise? Does the flow collector show ports and conversations, or only top talkers? The Operating Lesson The thing screaming is not always the thing broken.\nRancher was visible because it sits on the management path. Cluster agents, remotedialer sessions, API watches, ingress, and controllers all make the management plane sensitive to latency and packet loss.\nThe investigation worked because each hypothesis had to survive evidence:\nRancher agent churn was plausible. HAProxy and ingress were plausible. etcd was plausible. node saturation was plausible. backup-window congestion fit the timeline better than all of them. That is the practical value of the incident: not a perfect first guess, but a disciplined narrowing process.\nNext time a Rancher API load balancer appears as a top talker, I would still check Rancher. I would also check the backup schedule before assuming Rancher caused the spike.\n","permalink":"https://trinidadmarroquin.com/posts/rancher-network-saturation-symptoms/","section":"posts","summary":"A large network graph is good at creating anxiety. It is not always good at explaining cause.\nIn one investigation, a Rancher API load balancer appeared to receive hundreds of gigabytes of traffic in a 24-hour window. One RKE2 control-plane node appeared as a major sender. The first instinct was reasonable: something in Rancher, Kubernetes, monitoring, or the cluster agents might be misbehaving.\nThat was the right place to start. It was not the right place to stop.\n","tags":["rancher","kubernetes","networking","rke2","troubleshooting","sre"],"title":"When Network Saturation Looks Like Rancher Trouble"},{"categories":["projects"],"content":"An alert should earn the right to interrupt a human.\nGoogle SRE guidance emphasizes alerting on urgent, actionable, user-visible or imminently user-visible problems. That is the useful standard.\nAlert Review Every page should answer:\nWhat is broken? Who owns it? What is the impact? What action should the responder take? Can this wait until business hours? If the response is always robotic, automate it or downgrade it.\nGolden Signals Dashboards should expose the four golden signals where applicable:\nlatency. traffic. errors. saturation. For infrastructure, add platform health signals such as node readiness, storage attach failures, controller health, certificate expiry, and capacity pressure.\nDashboard Design Dashboards should support decisions:\nIs the service healthy? Are users affected? What changed recently? Which dependency is unhealthy? Is capacity becoming a constraint? Avoid dashboards that are just metric galleries.\nLocal SLI Labs A small Docker-based lab with an application, Prometheus, Grafana, and a container exporter is a useful way to practice SLI design. The lab should prove that metrics are scraped, dashboards answer operational questions, and credentials or host mounts are not mistaken for production-safe defaults.\nSee also: Secret Handling In Terraform Managed Labs.\nAcceptance Criteria Every paging alert has an owner and runbook. Alerts are actionable and urgent. Dashboards show symptoms first, causes second. Noisy alerts are reviewed and removed. On-call feedback changes alert behavior. References Google SRE Book: Monitoring Distributed Systems. Google SRE Book: Practical Alerting. ","permalink":"https://trinidadmarroquin.com/projects/observability-incident-response/actionable-alerts-dashboards/","section":"projects","summary":"An alert should earn the right to interrupt a human.\nGoogle SRE guidance emphasizes alerting on urgent, actionable, user-visible or imminently user-visible problems. That is the useful standard.\nAlert Review Every page should answer:\nWhat is broken? Who owns it? What is the impact? What action should the responder take? Can this wait until business hours? If the response is always robotic, automate it or downgrade it.\nGolden Signals Dashboards should expose the four golden signals where applicable:\n","tags":["observability","alerting","sre"],"title":"Actionable Alerts And Dashboards"},{"categories":["projects"],"content":"AWS operations differ from other providers in naming, service boundaries, and tooling defaults. The patterns below capture what is useful to remember without reaching for the console.\nAccount Structure AWS uses accounts as the hard isolation boundary. Use separate accounts for production, non-production, shared services, security, and sandbox. Organization-level SCPs enforce guardrails before IAM comes into play.\nKey differences from other clouds:\naccount is also a billing boundary. some services (CloudTrail, Config) are per-account by default and must be aggregated. VPCs are regional, not global. IAM roles are global, but trust policies reference specific accounts. EKS Cluster Operations An EKS cluster needs a VPC with at least two subnets in different AZs, an IAM role with AmazonEKSClusterPolicy, and a node group with an instance profile. The control plane is managed, but the node group lifecycle and CNI configuration still require operational attention.\nTerraform module shape:\nmodule \u0026#34;eks_cluster\u0026#34; { source = \u0026#34;./modules/eks-cluster\u0026#34; cluster_name = var.cluster_name subnet_ids = aws_subnet.private[*].id node_instance_type = var.node_instance_type desired_capacity = var.desired_capacity min_size = var.min_size max_size = var.max_size } Common issues:\nnode groups fail to join the cluster if the IAM instance profile role is missing the required trust policy. the CNI must be compatible with the instance type and available IP addresses in the subnet. cluster endpoint access settings control whether kubectl works from outside the VPC. version upgrades require a node group replacement or rolling update strategy. Application Deployment On EKS A Node.js or similar application deployed to EKS follows a standard path: containerize with Docker, package as a Helm chart, deploy with Terraform using helm_release.\nThe Helm chart should expose:\nreplicaCount for scaling. image.repository and image.tag for promotion. service.type for exposure model (ClusterIP vs LoadBalancer). Terraform manages the Helm release, but the chart templates stay in the application repository. This keeps deployment configuration with the team that owns the service.\nIoT And Edge AWS IoT Core uses X.509 certificates for device authentication. A local Docker container with device certificates mounted as volumes can simulate edge workloads during development.\nThe IoT data path:\ndevice -\u0026gt; MQTT (port 8883) -\u0026gt; AWS IoT Core -\u0026gt; rule action -\u0026gt; downstream service Local labs should use the same certificate paths as production to avoid configuration drift.\nWorkstation Tooling The AWS CLI, Session Manager plugin, and SDKs should be installed through the package manager, not downloaded once and forgotten. Regular updates matter because API versions and signing algorithms change.\nEssential tooling:\nAWS CLI v2 with named profiles for each account. Session Manager for SSH-less instance access. terraform with the AWS provider. kubectl with aws eks update-kubeconfig. linters: TFLint, Checkov, terrascan. infracost for cost awareness during planning. Observability CloudWatch is the default log and metric sink, but it is not always the best place to view operational state. Use CloudWatch for retention and alerting, and a separate observability stack (Prometheus + Grafana) for interactive debugging and dashboarding.\nExport CloudWatch metrics to Prometheus where cross-service dashboards need them. Do not rely on the CloudWatch console as the primary dashboard for operator response.\nAcceptance Criteria Operators can create an EKS cluster with known networking and IAM requirements. Helm-deployed applications follow a consistent chart structure. IoT device certificates are treated as secrets with rotation expectations. Workstation tooling is version-managed and reproducible. Observability data flows to the same dashboards used for other providers. ","permalink":"https://trinidadmarroquin.com/projects/cloud-based-platforms/aws-operational-patterns/","section":"projects","summary":"AWS operations differ from other providers in naming, service boundaries, and tooling defaults. The patterns below capture what is useful to remember without reaching for the console.\nAccount Structure AWS uses accounts as the hard isolation boundary. Use separate accounts for production, non-production, shared services, security, and sandbox. Organization-level SCPs enforce guardrails before IAM comes into play.\nKey differences from other clouds:\naccount is also a billing boundary. some services (CloudTrail, Config) are per-account by default and must be aggregated. VPCs are regional, not global. IAM roles are global, but trust policies reference specific accounts. EKS Cluster Operations An EKS cluster needs a VPC with at least two subnets in different AZs, an IAM role with AmazonEKSClusterPolicy, and a node group with an instance profile. The control plane is managed, but the node group lifecycle and CNI configuration still require operational attention.\n","tags":["aws","cloud","eks","iam","networking"],"title":"AWS Operational Patterns"},{"categories":["field-notes"],"content":"Azure Kubernetes Service follows the same managed-control-plane pattern as EKS and GKE, but the resource model, networking defaults, and identity system are different enough that provider-specific knowledge matters during provisioning.\nFor a runnable lab, see the azure directory in the IaC repository.\nResource Group Structure Everything in Azure lives in a resource group. The AKS cluster, VNet, and NSG are typically in the same group for a lab, but production should separate network infrastructure into a shared resource group owned by the platform team:\nresource \u0026#34;azurerm_resource_group\u0026#34; \u0026#34;aks_rg\u0026#34; { name = \u0026#34;aks-resources\u0026#34; location = \u0026#34;East US\u0026#34; } Resource groups are the boundary for role assignments, locks, and tags. A cluster in the wrong resource group inherits the wrong policies.\nNetwork Model Azure AKS uses the azure network plugin by default, which assigns pod IPs from the subnet directly. This differs from AWS (where the CNI manages ENIs) and GCP (where alias IPs are used):\nnetwork_profile { network_plugin = \u0026#34;azure\u0026#34; } With the Azure CNI, every pod gets an IP from the VNet subnet. This means subnet size limits cluster scale more than node count does. Plan the address space for the maximum expected pod count before provisioning.\nSystem-Assigned Identity AKS can use a system-assigned managed identity instead of a service principal:\nidentity { type = \u0026#34;SystemAssigned\u0026#34; } This removes the credential rotation problem of service principals. The identity is tied to the cluster lifecycle and does not require manual secret management. For production, use a user-assigned identity with pre-configured role assignments so the identity can be created and authorized before the cluster references it.\nNetwork Security Groups Azure applies NSG rules at the subnet or NIC level. The AKS lab opens HTTPS as an example:\nsecurity_rule { name = \u0026#34;allow-https\u0026#34; priority = 1000 direction = \u0026#34;Inbound\u0026#34; access = \u0026#34;Allow\u0026#34; protocol = \u0026#34;Tcp\u0026#34; destination_port_range = \u0026#34;443\u0026#34; source_address_prefix = \u0026#34;*\u0026#34; destination_address_prefix = \u0026#34;*\u0026#34; } AKS creates its own NSG rules for the node pool. Adding custom rules to the same subnet can conflict with AKS-managed rules. Use azurerm_subnet_network_security_group_association carefully and test rule priority ordering.\nTerraform State Backend Azure Blob Storage is a common Terraform state backend:\nterraform { backend \u0026#34;azurerm\u0026#34; { storage_account_name = \u0026#34;trinidadstorageacct\u0026#34; container_name = \u0026#34;terraform-state\u0026#34; key = \u0026#34;azure-storage-account/terraform.tfstate\u0026#34; } } The storage account uses LRS replication by default. For shared team access, enable blob soft delete and versioning on the state container. See also: Terraform Azure Backend Bootstrap.\nCross-Cloud Comparison Concern Azure AWS GCP Resource boundary Resource group Account Project Network scope Regional VNet Regional VPC Global VPC Pod networking Azure CNI (subnet IPs) VPC CNI (ENI IPs) Alias IPs Cluster identity Managed Identity IAM role Service Account State backend Blob Storage S3 GCS Acceptance Criteria AKS cluster deploys without service principal credential management. Pod CIDR and subnet size support the target node and pod count. Custom NSG rules do not conflict with AKS-managed rules. Terraform state is stored in a versioned, access-controlled backend. Cluster endpoint is reachable from authorized networks only. ","permalink":"https://trinidadmarroquin.com/field-notes/azure-aks-terraform-operations/","section":"field-notes","summary":"Azure Kubernetes Service follows the same managed-control-plane pattern as EKS and GKE, but the resource model, networking defaults, and identity system are different enough that provider-specific knowledge matters during provisioning.\nFor a runnable lab, see the azure directory in the IaC repository.\nResource Group Structure Everything in Azure lives in a resource group. The AKS cluster, VNet, and NSG are typically in the same group for a lab, but production should separate network infrastructure into a shared resource group owned by the platform team:\n","tags":["azure","aks","terraform","cloud"],"title":"Azure AKS Operations With Terraform"},{"categories":["field-notes"],"content":"Blue-green deployment keeps two environments running. At any time, one environment serves production traffic and the other waits for the next release.\nFor a runnable lab, see the blue-green-simulation directory in the IaC repository. Version 1 demonstrates a manual NGINX upstream switch. Version 2 adds weighted routing and named containers.\nThe switch is the critical moment. How it happens determines whether the pattern is fast rollback or just extra complexity.\nThe NGINX Upstream Switch The simplest blue-green switch is an NGINX upstream block with one active server and one standby:\nupstream backend { server blue:5000; # server green:5000; } Comment or uncomment the server line, then reload NGINX. Traffic moves to the active environment.\nReload, Not Restart docker exec nginx nginx -s reload The reload applies configuration changes without dropping active connections. A restart would terminate the worker processes and lose in-flight requests. Always reload when changing upstream targets.\nProxy Headers Matter When switching between backends, the proxy must forward the original client information:\nproxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; If these headers are missing or inconsistent, the new backend sees different client information than the old one, which can break logging, rate limiting, and application logic.\nManual Switch Failure Modes A manual upstream edit is the lowest-common-denominator blue-green pattern. It fails when:\nthe wrong server line is uncommented. a syntax error in nginx.conf prevents reload. the backend is not ready but the switch happens anyway. no one checks whether the new environment is healthy before switching. the operator forgets to reload after editing the config. The manual pattern teaches the mechanics. Automate the switch before relying on it in production.\nWeighted Routing As Blue-Green NGINX supports weighted upstream servers:\nupstream backend { server blue:5000 weight=5; server green:5000 weight=1; } This is not true blue-green. Weighted routing is a gradual shift closer to canary deployment. True blue-green is an atomic cutover: all traffic moves from blue to green at once. Weighted routing blends traffic across both environments.\nUse weighted routing when the goal is to test the new environment with a fraction of traffic before committing. Use a hard upstream switch when the goal is instant rollback capability.\nNamed Containers For Debugging In Docker Compose, set container_name on each service:\nservices: blue: container_name: blue green: container_name: green nginx: container_name: router Named containers make debugging predictable:\ndocker exec -it router curl http://blue:5000 docker logs blue docker logs green Without named containers, Docker assigns random names and every debug session starts with docker ps to find the right container.\nAcceptance Criteria Upstream switch moves all traffic to the target environment. nginx -s reload applies changes without dropping connections. Proxy headers are identical before and after the switch. The standby environment is running and healthy before the switch. Rollback is a single reload away. ","permalink":"https://trinidadmarroquin.com/field-notes/blue-green-deployment-nginx-swap/","section":"field-notes","summary":"Blue-green deployment keeps two environments running. At any time, one environment serves production traffic and the other waits for the next release.\nFor a runnable lab, see the blue-green-simulation directory in the IaC repository. Version 1 demonstrates a manual NGINX upstream switch. Version 2 adds weighted routing and named containers.\nThe switch is the critical moment. How it happens determines whether the pattern is fast rollback or just extra complexity.\nThe NGINX Upstream Switch The simplest blue-green switch is an NGINX upstream block with one active server and one standby:\n","tags":["deployment","nginx","cicd","docker"],"title":"Blue-Green Deployment With NGINX"},{"categories":["field-notes"],"content":"A canary deployment sends a small fraction of traffic to a new version while the stable version handles the rest. If the canary fails, only the test fraction is affected.\nFor a runnable lab, see the canary-deployment directory in the IaC repository. It uses HAProxy weighted routing with raw C API servers.\nHAProxy makes this pattern visible and controllable through weighted backend servers.\nWeight Ratio backend servers balance leastconn server v1 api_v1:8080 weight 10 check server v2 api_v2:8080 weight 1 check With weights 10 and 1, approximately 9% of requests reach v2. The weight proportion directly controls the blast radius:\nweight 10 : weight 1 = ~9% canary weight 10 : weight 10 = 50% (half the traffic, not a canary) weight 100 : weight 1 = ~1% canary weight 1 : weight 0 = disabled (v2 receives no traffic) Start with a small fraction and increase as confidence grows.\nBalance Strategy The balance leastconn directive distributes requests to the server with the fewest active connections. This matters for long-lived connections like WebSockets or streaming responses.\nFor short-lived request-response workloads, roundrobin works just as well and is more predictable. Choose leastconn when backend response time varies significantly.\nHealth Checks Are TCP By Default HAProxy\u0026rsquo;s check directive sends a TCP connection check by default. It only verifies that the port is open, not that the application is healthy.\nA real canary should have a dedicated health check endpoint:\noption httpchk GET /healthz Without an HTTP health check, HAProxy may route traffic to a v2 that accepts TCP connections but returns 500s on every request.\nReading The Logs Confirm which version served each request by checking response headers or structured logs. In a lab, the simplest approach is a version header:\nHTTP/1.1 200 OK X-Version: v2 If the canary is weighted at 10%, roughly 1 in 10 requests should show the v2 header.\nWhy Raw C In The Lab The IaC repo canary lab uses a raw C HTTP server with BSD sockets. No framework, no dependencies. This is deliberate: deployment strategy labs should test traffic behavior, not framework configuration. If the lab requires understanding a web framework before testing the canary, it misses the point.\nProduction Translation Lab canary:\nHAProxy -\u0026gt; v1 (weight 10) | v2 (weight 1) -\u0026gt; manual observation Production canary:\nservice mesh -\u0026gt; v1 (traffic) | v2 (traffic) -\u0026gt; metrics comparison -\u0026gt; auto-promote or auto-rollback The lab teaches the weight concept. Production requires automated analysis and decision gates.\nAcceptance Criteria Weighted traffic reaches both versions. Health checks catch an unresponsive canary before user impact. Request distribution matches the configured weight ratio. Rollback is a config reload away. Canary logs are distinguishable from stable logs. ","permalink":"https://trinidadmarroquin.com/field-notes/canary-deployment-haproxy-weighted-routing/","section":"field-notes","summary":"A canary deployment sends a small fraction of traffic to a new version while the stable version handles the rest. If the canary fails, only the test fraction is affected.\nFor a runnable lab, see the canary-deployment directory in the IaC repository. It uses HAProxy weighted routing with raw C API servers.\nHAProxy makes this pattern visible and controllable through weighted backend servers.\nWeight Ratio backend servers balance leastconn server v1 api_v1:8080 weight 10 check server v2 api_v2:8080 weight 1 check With weights 10 and 1, approximately 9% of requests reach v2. The weight proportion directly controls the blast radius:\n","tags":["deployment","haproxy","cicd","docker"],"title":"Canary Deployments With HAProxy Weighted Routing"},{"categories":["projects"],"content":"Cloud platform organization is an operational control, not just a billing structure.\nThe account or project boundary should make it obvious who owns the workload, what environment it belongs to, which policies apply, and how blast radius is contained.\nOperating Model Use separate top-level containers for production, non-production, shared services, security, and sandbox work. The exact cloud terms differ across AWS accounts, Azure subscriptions, and Google Cloud projects, but the principle is the same: production should not share an administrative or network boundary with experiments.\nMinimum metadata for each environment:\nowner team. cost center or billing tag. environment classification. data classification. support and escalation path. lifecycle state. Environment Boundaries Recommended boundaries:\nproduction: restricted access, change control, backup, monitoring, and incident expectations. non-production: realistic enough for testing, but isolated from production identity and data. shared services: DNS, artifact repositories, observability, security tooling, and central automation. sandbox: time-limited experimentation with quota and cleanup rules. Avoid using naming alone as the control. Names help humans, but policies, IAM, network segmentation, and budget controls enforce the boundary.\nReview Checklist Is the owner clear without opening a ticket? Can production access be audited separately from development access? Are logs and security events routed centrally? Are budgets, quotas, or alerts configured? Is there a retirement path for unused environments? Terraform State Foundations Cloud account and environment organization should include where Terraform state lives, who can access it, and how state is separated by environment or blast radius. A remote backend is not just a Terraform setting; it is shared operational infrastructure.\nSee also: Terraform Azure Backend Bootstrap.\nReferences AWS Well-Architected Framework. Google Cloud Architecture Framework. Microsoft Cloud Adoption Framework. ","permalink":"https://trinidadmarroquin.com/projects/cloud-based-platforms/account-environment-organization/","section":"projects","summary":"Cloud platform organization is an operational control, not just a billing structure.\nThe account or project boundary should make it obvious who owns the workload, what environment it belongs to, which policies apply, and how blast radius is contained.\nOperating Model Use separate top-level containers for production, non-production, shared services, security, and sandbox work. The exact cloud terms differ across AWS accounts, Azure subscriptions, and Google Cloud projects, but the principle is the same: production should not share an administrative or network boundary with experiments.\n","tags":["cloud","platform","governance"],"title":"Cloud Account And Environment Organization"},{"categories":["projects"],"content":"Cloud guardrails should prevent common mistakes without turning the platform into a ticket queue for every change.\nThe platform team should define the default network, identity, logging, and policy controls that every environment inherits.\nNetworking Network design should make approved communication easy and accidental exposure difficult.\nBaseline expectations:\nprivate networks by default. explicit ingress and egress paths. separate production and non-production routing. documented CIDR allocation. centralized DNS ownership. no unmanaged public endpoints. IAM IAM should be group-based, role-based, and reviewable.\nUse:\nfederated identity for humans. short-lived credentials for automation where possible. separate deploy, read-only, and break-glass roles. least privilege for service accounts. periodic review of privileged roles. Avoid long-lived keys unless there is a documented exception and rotation plan.\nLogging And Guardrails Each account, project, or subscription should send audit logs to a central location that application teams cannot disable.\nGuardrails worth standardizing:\nblock public storage buckets by default. require encryption where supported. restrict allowed regions. require required tags or labels. alert on privileged IAM changes. alert on disabled logging. Acceptance Criteria New environments inherit logging and security baselines. Human access is federated and group-based. Public exposure is explicit and reviewable. Automation uses scoped identities. Guardrail exceptions have owners and expiration dates. References AWS Well-Architected Security Pillar. Google Cloud Architecture Framework: Security. Microsoft Cloud Adoption Framework: Secure methodology. ","permalink":"https://trinidadmarroquin.com/projects/cloud-based-platforms/networking-iam-guardrails/","section":"projects","summary":"Cloud guardrails should prevent common mistakes without turning the platform into a ticket queue for every change.\nThe platform team should define the default network, identity, logging, and policy controls that every environment inherits.\nNetworking Network design should make approved communication easy and accidental exposure difficult.\nBaseline expectations:\nprivate networks by default. explicit ingress and egress paths. separate production and non-production routing. documented CIDR allocation. centralized DNS ownership. no unmanaged public endpoints. IAM IAM should be group-based, role-based, and reviewable.\n","tags":["cloud","iam","networking","security"],"title":"Cloud Networking IAM And Guardrails"},{"categories":["projects"],"content":"Cloud operations become easier when provider differences are documented as patterns, not rediscovered during incidents.\nThe useful pattern is to separate provider-specific commands from provider-neutral operating expectations.\nPattern Categories Document patterns by discipline:\nidentity and access. network routing and exposure. compute lifecycle. storage and backup. Kubernetes integration. observability and audit logging. cost and capacity review. incident response. For each provider, keep the same structure so operators can compare behavior quickly.\nProvider Notes Each provider page should answer:\nWhat is the account boundary called? What is the network boundary called? Where are audit logs configured? How are service identities created and rotated? Where are quotas and limits reviewed? How do private endpoints and public exposure work? Which services are approved for production? Runbook Shape Every operational pattern should include:\nintent. owner. safe read-only checks. common failure modes. remediation path. escalation path. Acceptance Criteria Operators can find provider-specific commands without rewriting the operating model. Patterns are organized by discipline, not by one-off incidents. Cloud differences are documented where they matter. Shared expectations remain consistent across AWS, Azure, Google Cloud, and private cloud. References AWS Well-Architected Framework. Google Cloud Architecture Framework. Microsoft Cloud Adoption Framework. ","permalink":"https://trinidadmarroquin.com/projects/cloud-based-platforms/provider-operational-patterns/","section":"projects","summary":"Cloud operations become easier when provider differences are documented as patterns, not rediscovered during incidents.\nThe useful pattern is to separate provider-specific commands from provider-neutral operating expectations.\nPattern Categories Document patterns by discipline:\nidentity and access. network routing and exposure. compute lifecycle. storage and backup. Kubernetes integration. observability and audit logging. cost and capacity review. incident response. For each provider, keep the same structure so operators can compare behavior quickly.\n","tags":["cloud","operations","platform"],"title":"Cloud Provider Operational Patterns"},{"categories":["projects"],"content":"Running Concourse CI/CD on Windows Docker is uncommon enough that the setup patterns deserve their own reference. The Terraform provider, key generation, and security hardening steps differ significantly from Linux-based deployments.\nFor a complete working example, see the concourse-terraform-windows directory in the IaC repository.\nDocker Transport Windows Docker uses named pipes instead of Unix sockets:\nprovider \u0026#34;docker\u0026#34; { host = \u0026#34;npipe:////./pipe/docker_engine\u0026#34; } This is the first thing to verify when a Terraform Docker provider fails on Windows. The connection string is different, and not all provider features work identically on the Windows engine.\nKey Generation Without ssh-keygen On a Windows host, ssh-keygen may not be available. Terraform\u0026rsquo;s tls_private_key resource generates the key natively:\nresource \u0026#34;tls_private_key\u0026#34; \u0026#34;worker_key\u0026#34; { algorithm = \u0026#34;RSA\u0026#34; rsa_bits = 4096 } The output is PEM format. Concourse needs SSH public key format for authorized_worker_keys. A PowerShell script bridges the gap using a Terraform external data source:\n# convert_to_ssh_format.ps1 $tempFile = [System.IO.Path]::GetTempFileName() $pemContent = $env:pem_private_key Set-Content -Path $tempFile -Value $pemContent $sshPublicKey = ssh-keygen -y -f $tempFile 2\u0026gt;\u0026amp;1 Remove-Item $tempFile return ($sshPublicKey | ConvertTo-Json) data \u0026#34;external\u0026#34; \u0026#34;worker_public_key_ssh\u0026#34; { program = [\u0026#34;pwsh\u0026#34;, \u0026#34;./scripts/convert_to_ssh_format.ps1\u0026#34;] query = { pem_private_key = tls_private_key.worker_key.private_key_pem } } Permission Hardening Windows file permissions are not restrictive enough by default for private keys:\n# set_permissions.ps1 icacls $keyPath /inheritance:r icacls $keyPath /grant \u0026#34;${env:USERNAME}:(R,D)\u0026#34; The inheritance:r flag removes all inherited permissions. The (R,D) grant gives the current user read and delete access only. Without this, the Concourse process or other users on the machine could read the key material.\nArchitecture The Terraform config provisions three containers:\nconcourse-db (postgres:13) -\u0026gt; concourse-web (custom image, port 8080) -\u0026gt; concourse-worker (privileged, TSA to web:2222) The web container entrypoint uses dumb-init for proper PID 1 handling, which is especially important on Windows containers where signal propagation can be unpredictable.\nProduction Pipeline Example The repository includes three pipeline YAMLs. The full pipeline (pipeline_full_stubbed.yaml) demonstrates a production-grade Concourse pipeline for what appears to be a Cadence API service:\nResources: Git, ECR, semver, pull request. Jobs: unit test (with race detection), lint, build, integration, deploy staging, deploy production. Patterns: parallel test execution, semantic versioning with semver resource, PR status checks, Golang race detection. Acceptance Criteria Terraform applies on Windows Docker without Linux compatibility layer. Keys are generated by tls_private_key without ssh-keygen. Private key files have Windows permissions restricting access to the service account. Workers authenticate to the web TSA using the generated keys. Pipeline YAMLs can be loaded and triggered. Cleanup removes containers, keys, and local state. ","permalink":"https://trinidadmarroquin.com/projects/cicd-pipeline-design/concourse-windows-deployment/","section":"projects","summary":"Running Concourse CI/CD on Windows Docker is uncommon enough that the setup patterns deserve their own reference. The Terraform provider, key generation, and security hardening steps differ significantly from Linux-based deployments.\nFor a complete working example, see the concourse-terraform-windows directory in the IaC repository.\nDocker Transport Windows Docker uses named pipes instead of Unix sockets:\nprovider \u0026#34;docker\u0026#34; { host = \u0026#34;npipe:////./pipe/docker_engine\u0026#34; } This is the first thing to verify when a Terraform Docker provider fails on Windows. The connection string is different, and not all provider features work identically on the Windows engine.\n","tags":["concourse","cicd","windows","terraform","docker","key-management"],"title":"Concourse CI/CD On Windows With Terraform"},{"categories":["projects"],"content":"Concourse works well for infrastructure when the pipeline graph tells the truth about the change flow.\nJobs should be small enough to understand and resources should capture the actual inputs.\nPipeline Shape Common jobs:\nlint. validate. plan. plan review or approval. apply non-production. apply production. post-apply verification. Use serial groups for stateful targets so two applies do not compete for the same backend or environment.\nTask Design Tasks should:\nrun from versioned scripts. print the exact target environment. fail loudly on missing variables. keep secrets out of logs. publish plan or verification artifacts. Resource Patterns Useful resources:\nGit repository. versioned image for task runtime. artifact store for plans. notification or issue tracker integration. environment promotion marker. Failure Output A failed job should show:\ncommand executed. environment target. safe error output. link to plan or logs. owner or next action. Lab-To-Production Checks Local Concourse labs are useful for learning pipeline shape, worker behavior, and Terraform-driven platform setup. Before reusing the pattern in shared infrastructure, review secret handling, generated keys, local users, privileged workers, and exposed ports.\nSee also: Secret Handling In Terraform Managed Labs.\nAcceptance Criteria Pipeline graph matches promotion flow. Stateful applies are serialized. Plans are retained. Secrets are not printed. Operators can rerun safely after fixing inputs. ","permalink":"https://trinidadmarroquin.com/projects/cicd-pipeline-design/concourse-infrastructure-workflows/","section":"projects","summary":"Concourse works well for infrastructure when the pipeline graph tells the truth about the change flow.\nJobs should be small enough to understand and resources should capture the actual inputs.\nPipeline Shape Common jobs:\nlint. validate. plan. plan review or approval. apply non-production. apply production. post-apply verification. Use serial groups for stateful targets so two applies do not compete for the same backend or environment.\nTask Design Tasks should:\nrun from versioned scripts. print the exact target environment. fail loudly on missing variables. keep secrets out of logs. publish plan or verification artifacts. Resource Patterns Useful resources:\n","tags":["cicd","concourse","infrastructure"],"title":"Concourse Infrastructure Workflows"},{"categories":["field-notes"],"content":"Concourse requires a set of RSA keys for TSA (Transport Security Authority) authentication between web and worker nodes. Managing these keys is a bootstrapping problem: Concourse needs keys to start, but the keys should live in a secrets store.\nFor a runnable lab, see the concourse-terraform-unix directory in the IaC repository.\nTSA Key Architecture Concourse uses four key pairs:\nTSA host key (tsa_host_key + tsa_host_key.pub): identifies the web node to workers. Worker key (worker_key + worker_key.pub): identifies workers to the web node. Authorized worker keys (authorized_worker_keys): the public keys of permitted workers. Session signing key (session_signing_key): signs session tokens for the ATC API. The web node holds the TSA host key and authorized worker keys. Workers connect using their worker key. If any key pair mismatches, the worker cannot authenticate and stays disconnected.\nVault Integration Pattern Terraform reads keys from Vault using vault_generic_secret data sources, then writes them to local files and passes them to Docker containers:\ndata \u0026#34;vault_generic_secret\u0026#34; \u0026#34;tsa_host_key\u0026#34; { path = \u0026#34;secret/concourse/tsa_host_key\u0026#34; } The key files are written to disk with local_file resources and mounted into containers. The containers also retrieve keys at runtime via the Vault CLI in their entrypoint:\nvault login ${VAULT_TOKEN} \u0026amp;\u0026amp; \\ vault kv get -field=value secret/concourse/tsa_host_key \u0026gt; /concourse-keys/tsa_host_key \u0026amp;\u0026amp; \\ dumb-init /usr/local/concourse/bin/concourse web The Bootstrapping Problem The containers need VAULT_TOKEN to start, but the token must be available before Vault can provide the keys. This creates a circular dependency:\nVault needs to be running -\u0026gt; token must exist -\u0026gt; container starts -\u0026gt; logs into Vault -\u0026gt; retrieves keys -\u0026gt; Concourse starts In this lab, the token is passed as an environment variable. This works for testing but has operational risks:\nVAULT_TOKEN in env is visible via docker inspect. if Vault is unreachable at container startup, the entrypoint fails and the container crashes. token rotation requires container restart. For production, use a short-lived token generated by a trusted orchestrator or a Vault agent sidecar that handles token lifecycle separately.\nEntrypoint Design The entrypoint chains commands with \u0026amp;\u0026amp;. If any step fails, the container stops:\n/bin/sh -c \u0026#34;vault login ${VAULT_TOKEN} \u0026amp;\u0026amp; vault kv get ... \u0026amp;\u0026amp; dumb-init concourse web\u0026#34; This is correct behavior: a container that cannot retrieve its keys should not start. If the entrypoint swallowed errors, the container would run with missing keys and fail in harder-to-debug ways.\nDependencies Terraform models the startup order explicitly:\nconcourse-db -\u0026gt; concourse-web -\u0026gt; concourse-worker The database starts first because Concourse web needs it to migrate schemas. The worker starts after the web node because it needs the TSA endpoint.\nThe worker uses CONCOURSE_TSA_HOST=concourse-web:2222 to find the web node. If the web node restarts and its TSA host key changes, all workers must be updated with the new key.\nAcceptance Criteria Key files exist on disk before Concourse starts. Web node reaches Vault and retrieves its keys at startup. Worker authenticates to the web TSA and appears in the Concourse dashboard. Container restart does not change key material. VAULT_TOKEN is not visible in container logs. ","permalink":"https://trinidadmarroquin.com/field-notes/concourse-vault-key-bootstrap/","section":"field-notes","summary":"Concourse requires a set of RSA keys for TSA (Transport Security Authority) authentication between web and worker nodes. Managing these keys is a bootstrapping problem: Concourse needs keys to start, but the keys should live in a secrets store.\nFor a runnable lab, see the concourse-terraform-unix directory in the IaC repository.\nTSA Key Architecture Concourse uses four key pairs:\nTSA host key (tsa_host_key + tsa_host_key.pub): identifies the web node to workers. Worker key (worker_key + worker_key.pub): identifies workers to the web node. Authorized worker keys (authorized_worker_keys): the public keys of permitted workers. Session signing key (session_signing_key): signs session tokens for the ATC API. The web node holds the TSA host key and authorized worker keys. Workers connect using their worker key. If any key pair mismatches, the worker cannot authenticate and stays disconnected.\n","tags":["concourse","vault","cicd","terraform","docker"],"title":"Concourse Key Management With Vault Bootstrap"},{"categories":["projects"],"content":"Deployment strategy labs are useful when they make failure handling visible.\nThe goal is not to prove that blue-green, canary, feature toggles, or rollback scripts are fashionable. The goal is to understand which failure mode each strategy reduces and which operational cost it adds.\nStrategy Comparison Useful lab scenarios include:\nblue-green switching for fast rollback to a known environment. canary rollout for limiting user exposure. feature toggles for disabling behavior without redeploying. rollback packages or manifests for restoring a previous version. health checks that block promotion or trigger investigation. Each strategy should be tested with a deliberate failure. If the lab only demonstrates a successful deployment, it misses the point.\nOperator Questions For each deployment pattern, answer:\nHow is traffic shifted? What proves the new version is healthy? What metric or alert stops the rollout? How quickly can the previous version be restored? What state changes cannot be rolled back safely? Who owns the decision to continue, pause, or revert? Production Translation In production, the deployment strategy should be tied to service risk.\nLow-risk internal services may only need rolling updates and clear health checks. Customer-facing services with high blast radius may need canaries, automated analysis, and fast rollback. Database changes require a separate migration and recovery plan regardless of the application rollout strategy.\nRelated Field Notes Blue-Green Deployment With NGINX \u0026ndash; upstream switching mechanics, weighted routing, reload vs restart. Canary Deployments With HAProxy Weighted Routing \u0026ndash; weight ratio math, health checks, balance strategy. Feature Toggles With Environment Variables \u0026ndash; cross-language patterns, limitations, progression path. Rollback Strategies With Sentinel Files And Package Management \u0026ndash; failure detection, dependency chaining, production checklist. Acceptance Criteria Strategy is matched to service risk. Health checks are meaningful, not just process checks. Rollback is rehearsed. Monitoring can detect the failure the strategy is meant to contain. Database and external dependency changes are handled separately. ","permalink":"https://trinidadmarroquin.com/projects/cicd-pipeline-design/deployment-strategy-labs/","section":"projects","summary":"Deployment strategy labs are useful when they make failure handling visible.\nThe goal is not to prove that blue-green, canary, feature toggles, or rollback scripts are fashionable. The goal is to understand which failure mode each strategy reduces and which operational cost it adds.\nStrategy Comparison Useful lab scenarios include:\nblue-green switching for fast rollback to a known environment. canary rollout for limiting user exposure. feature toggles for disabling behavior without redeploying. rollback packages or manifests for restoring a previous version. health checks that block promotion or trigger investigation. Each strategy should be tested with a deliberate failure. If the lab only demonstrates a successful deployment, it misses the point.\n","tags":["cicd","deployment","validation","automation"],"title":"Deployment Strategy Labs"},{"categories":["field-notes"],"content":"Check Which Layer Is Injecting The Search Domain # 1. Is /etc/resolv.conf managed by systemd-resolved? ls -l /etc/resolv.conf # Expected: /etc/resolv.conf -\u0026gt; /run/systemd/resolve/stub-resolv.conf # 2. Show all search domains (global and per-link) resolvectl domain # Look for \u0026#34;Link \u0026lt;N\u0026gt; (\u0026lt;iface\u0026gt;): \u0026lt;domain\u0026gt;\u0026#34; — not just Global section # 3. What does the active resolver actually show? grep \u0026#39;^search\u0026#39; /etc/resolv.conf || echo \u0026#34;no search line\u0026#34; # 4. Which interface is injecting the domain? resolvectl domain | grep -B1 \u0026#39;\\.\u0026#39; # 5. Is DHCP the source? networkctl status eth0 | grep -i domain The Three Layers Layer Inspect With Fix Netplan grep -R \u0026quot;search:\u0026quot; /etc/netplan/ Edit yaml, netplan generate, netplan apply systemd-resolved (global) resolvectl domain (Global:) resolvectl domain \u0026quot;\u0026quot; systemd-resolved (per-link) resolvectl domain (Link N:) resolvectl domain eth0 \u0026quot;\u0026quot; Runtime Fix (Per-Link Domain) Clearing a per-link search domain without restarting systemd-resolved:\nsudo resolvectl domain eth0 \u0026#34;\u0026#34; Do not restart systemd-resolved after this. Restarting re-reads the link config and reapplies the domain.\nSafe alternative if you need to flush caches:\nsudo resolvectl flush-caches Verify The Fix resolvectl domain cat /etc/resolv.conf grep \u0026#39;^search\u0026#39; /etc/resolv.conf || echo \u0026#34;no search line\u0026#34; Expected clean state:\nresolvectl domain -\u0026gt; (no domains listed) /etc/resolv.conf -\u0026gt; search . search . is expected — it means \u0026ldquo;no search domain\u0026rdquo; in systemd-resolved.\nNetplan Inspection # Show effective netplan config sudo netplan get # Find search domain in all netplan files grep -R \u0026#34;search:\u0026#34; /etc/netplan/ # Validate without applying sudo netplan generate Ansible Audit (CSV Output) OUT=\u0026#34;dns-search-audit-$(date +%Y%m%d-%H%M%S).csv\u0026#34; echo \u0026#39;inventory_host,actual_hostname,resolv_conf_search,netplan_search\u0026#39; \u0026gt; \u0026#34;$OUT\u0026#34; ANSIBLE_NOCOLOR=1 \\ ANSIBLE_STDOUT_CALLBACK=default \\ ansible all -i inventory/prod.yaml -u operator -b -kK \\ -m shell -a \u0026#39; ACTUAL_HOSTNAME=$(hostname -s) RESOLV_SEARCH=$(awk \u0026#34;/^search/{\\$1=\\\u0026#34;\\\u0026#34;; sub(/^ /,\\\u0026#34;\\\u0026#34;); print; exit} /^domain/{print \\$2; exit}\u0026#34; /etc/resolv.conf) [ -z \u0026#34;$RESOLV_SEARCH\u0026#34; ] \u0026amp;\u0026amp; RESOLV_SEARCH=\u0026#34;NONE\u0026#34; NETPLAN_SEARCH=$(grep -R \u0026#34;search:\u0026#34; /etc/netplan/*.yaml /etc/netplan/*.yml 2\u0026gt;/dev/null | sed \u0026#34;s/.*search:[[:space:]]*//\u0026#34; | paste -sd \u0026#34;;\u0026#34; -) [ -z \u0026#34;$NETPLAN_SEARCH\u0026#34; ] \u0026amp;\u0026amp; NETPLAN_SEARCH=\u0026#34;NONE\u0026#34; printf \u0026#34;CSV|{{ inventory_hostname }}|%s|%s|%s\\n\u0026#34; \u0026#34;$ACTUAL_HOSTNAME\u0026#34; \u0026#34;$RESOLV_SEARCH\u0026#34; \u0026#34;$NETPLAN_SEARCH\u0026#34; \u0026#39; \\ | sed -n \u0026#39;s/.*CSV|//p\u0026#39; \\ | awk -F\u0026#39;|\u0026#39; \u0026#39;{print $1 \u0026#34;,\u0026#34; $2 \u0026#34;,\u0026#34; $3 \u0026#34;,\u0026#34; $4}\u0026#39; \\ | sort -t, -k1,1 \\ \u0026gt;\u0026gt; \u0026#34;$OUT\u0026#34; Do not rely on the CSV alone. If the pipeline only keeps CSV|... rows, failed or unreachable hosts are filtered out. Capture raw Ansible output, print non-CSV warnings/failures, and compare targeted host count to CSV data rows.\nSee DNS Search Domain Audit Failure Visibility for the safer audit pattern.\nIf Terraform and vSphere are creating the VM, also verify that DNS search suffixes are not being derived from the VM identity domain. See Terraform vSphere DNS Search Suffix Ownership.\nDrift Detection Show anything not matching desired baseline (. / []):\nawk -F, \u0026#39;NR==1 || $3!=\u0026#34;.\u0026#34; || $4!=\u0026#34;[]\u0026#34;\u0026#39; \u0026#34;$OUT\u0026#34; | column -s, -t Count distinct resolver patterns:\nawk -F, \u0026#39;NR\u0026gt;1 {print $3 \u0026#34;|\u0026#34; $4}\u0026#39; \u0026#34;$OUT\u0026#34; | sort | uniq -c | sort -nr Important Shell Detail Ansible shell module runs under /bin/sh by default. set -o pipefail is not available on Ubuntu\u0026rsquo;s /bin/sh and will fail silently or with:\n/bin/sh: 2: set: Illegal option -o pipefail Use set -eu instead of set -euo pipefail inside Ansible shell tasks.\nTroubleshooting Flow grep '^search' /etc/resolv.conf — observe the active value resolvectl domain — check if it is global or per-link grep -R \u0026quot;search:\u0026quot; /etc/netplan/ — check netplan networkctl status eth0 — check DHCP domain injection resolvectl domain eth0 \u0026quot;\u0026quot; — clear per-link at runtime (no restart) echo 'search .' | diff - /etc/resolv.conf — confirm clean state ","permalink":"https://trinidadmarroquin.com/field-notes/dns-search-domain-debugging/","section":"field-notes","summary":"Check Which Layer Is Injecting The Search Domain # 1. Is /etc/resolv.conf managed by systemd-resolved? ls -l /etc/resolv.conf # Expected: /etc/resolv.conf -\u0026gt; /run/systemd/resolve/stub-resolv.conf # 2. Show all search domains (global and per-link) resolvectl domain # Look for \u0026#34;Link \u0026lt;N\u0026gt; (\u0026lt;iface\u0026gt;): \u0026lt;domain\u0026gt;\u0026#34; — not just Global section # 3. What does the active resolver actually show? grep \u0026#39;^search\u0026#39; /etc/resolv.conf || echo \u0026#34;no search line\u0026#34; # 4. Which interface is injecting the domain? resolvectl domain | grep -B1 \u0026#39;\\.\u0026#39; # 5. Is DHCP the source? networkctl status eth0 | grep -i domain The Three Layers Layer Inspect With Fix Netplan grep -R \u0026quot;search:\u0026quot; /etc/netplan/ Edit yaml, netplan generate, netplan apply systemd-resolved (global) resolvectl domain (Global:) resolvectl domain \u0026quot;\u0026quot; systemd-resolved (per-link) resolvectl domain (Link N:) resolvectl domain eth0 \u0026quot;\u0026quot; Runtime Fix (Per-Link Domain) Clearing a per-link search domain without restarting systemd-resolved:\n","tags":["dns","netplan","systemd-resolved","linux","kubernetes"],"title":"DNS Search Domain Debugging With systemd-resolved"},{"categories":["projects"],"content":"RKE2 clusters across multiple sites had inconsistent DNS search domain behavior. Some nodes appended internal domains to all lookups, some appended data center-specific domains, and a few were clean. In a Kubernetes cluster, uncontrolled search domain expansion causes:\nPod DNS lookups that should resolve as-is getting unexpected suffix expansion. Inconsistent behavior across sites for the same service name. Debugging sessions that waste time on DNS before finding the real issue. Desired State Layer Target Netplan search: [] systemd-resolved No per-link or global domains Active resolver (/etc/resolv.conf) search . search . is systemd-resolved shorthand for \u0026ldquo;no search domains\u0026rdquo; and is the correct end state.\nOperating Pattern 1. Audit Run against all nodes to produce a CSV of current state:\n./scripts/fix-dns-search.sh --audit-only \u0026lt;site\u0026gt; \u0026lt;env\u0026gt; \u0026lt;inventory\u0026gt; The audit must also prove coverage. Compare the targeted Ansible host count with the CSV data row count, and print raw Ansible failures when a host does not return a row. Otherwise SSH or sudo failures can disappear behind a sed filter that only keeps successful CSV|... markers.\nExample output:\nresolv_conf_search netplan_search Meaning . [] Clean internal.corp.example [] Netplan staged, not applied dc.corp.example NONE Link-level injection (DHCP/systemd-networkd) 2. Stage For netplan-managed nodes, validate configs without applying:\n./scripts/fix-dns-search.sh --stage \u0026lt;site\u0026gt; \u0026lt;env\u0026gt; \u0026lt;inventory\u0026gt; Uses netplan generate only — no runtime impact.\n3. Runtime Fix (If Needed) For nodes with per-link injection (netplan shows NONE):\nresolvectl domain eth0 \u0026#34;\u0026#34; Do not restart systemd-resolved afterward.\n4. Apply (Maintenance Window) ./scripts/fix-dns-search.sh --apply \u0026lt;site\u0026gt; \u0026lt;env\u0026gt; \u0026lt;inventory\u0026gt; 5. Verify Drift awk -F, \u0026#39;NR==1 || $3!=\u0026#34;.\u0026#34; || $4!=\u0026#34;[]\u0026#34;\u0026#39; \u0026lt;audit-csv\u0026gt; | column -s, -t Header-only output = clean.\nEdge Cases Encountered Subiquity YAML indentation: Ubuntu installer sometimes writes search: at wrong indentation. Add a YAML normalization step to any netplan automation. set -o pipefail in Ansible: /bin/sh on Ubuntu does not support it. Use set -eu inside shell tasks. stale /etc/resolv.conf after netplan fix: If the file is a symlink to stub-resolv.conf, do not edit it directly. The change comes from systemd-resolved, not the file. short audit CSV: If the host list has 13 nodes but the CSV has 6 rows, the audit is incomplete. Capture raw Ansible output, print unreachable/failed hosts, and warn on host-count versus row-count mismatch before reading drift results. Related DNS Search Domain Debugging With systemd-resolved — field note with commands and troubleshooting flow. DNS Search Domain Audit Failure Visibility — field note for making audit scripts expose unreachable hosts and row-count mismatches. Terraform vSphere DNS Search Suffix Ownership — field note for separating VM identity domain from resolver search suffixes in Terraform modules. DNS Search Remediation With Per-Cluster Ansible Inventory — field note for remediation wrappers that target per-cluster inventories and avoid local Vault dependency failures. ","permalink":"https://trinidadmarroquin.com/projects/kubernetes-platform-operations/dns-search-domain-hygiene/","section":"projects","summary":"RKE2 clusters across multiple sites had inconsistent DNS search domain behavior. Some nodes appended internal domains to all lookups, some appended data center-specific domains, and a few were clean. In a Kubernetes cluster, uncontrolled search domain expansion causes:\nPod DNS lookups that should resolve as-is getting unexpected suffix expansion. Inconsistent behavior across sites for the same service name. Debugging sessions that waste time on DNS before finding the real issue. Desired State Layer Target Netplan search: [] systemd-resolved No per-link or global domains Active resolver (/etc/resolv.conf) search . search . is systemd-resolved shorthand for \u0026ldquo;no search domains\u0026rdquo; and is the correct end state.\n","tags":["dns","netplan","systemd-resolved","ansible","rke2","kubernetes"],"title":"DNS Search Domain Hygiene Across Multi-Site Clusters"},{"categories":["field-notes"],"content":"Docker can break silently when stale binaries in /usr/local/bin shadow the package-managed versions after an upgrade. The daemon starts, docker ps works, but containers fail to run with no obvious error in the Docker CLI.\nThe Pattern docker ps # works, lists containers docker run hello-world # hangs or fails silently The daemon itself is healthy. The break is in the runtime path. After a Docker or containerd upgrade, the new daemon expects a compatible runc and shim, but an old copy in /usr/local/bin gets resolved first because it appears earlier in $PATH than the packaged binary.\nDetection Check which runc is actually in use:\nwhich runc runc --version If it points to /usr/local/bin/runc and the version predates the installed Docker or containerd, that is the problem. Compare against the packaged version:\ndpkg -l runc # or rpm -q runc Fix Move the stale binaries out of the path, then restart the daemons:\nsudo mkdir -p /usr/local/bin/disabled sudo mv /usr/local/bin/runc /usr/local/bin/disabled/ sudo mv /usr/local/bin/containerd-shim-runc-v2 /usr/local/bin/disabled/ sudo systemctl restart containerd docker Verify:\ndocker run --rm hello-world After the fix, which runc should resolve to /usr/bin/runc (or wherever the package manager installed it).\nWhy It Happens Docker Engine and containerd are often installed via the official convenience script or a manual tarball that drops files into /usr/local/bin. When the system package manager later installs an updated version (or when Docker is reinstalled from the official apt repository), the $PATH shadowing persists because /usr/local/bin takes precedence over /usr/bin.\nThe same can happen with any binary that both a package and a manual installation place in the path. The fix is not specific to runc.\nPrevention Avoid mixing installation methods. Pick one:\napt / yum / dnf for system-managed installations. official convenience script only when the package manager does not offer the required version, and audit /usr/local/bin afterward. If /usr/local/bin has binaries from a prior installation, clean them out before upgrading the package-managed installation:\nsudo rm /usr/local/bin/runc /usr/local/bin/containerd-shim-runc-v2 sudo systemctl restart containerd docker Acceptance Criteria docker run --rm hello-world completes. which runc does not return /usr/local/bin/runc. Existing containers can start and stop normally. docker ps shows the same containers before and after the fix. ","permalink":"https://trinidadmarroquin.com/field-notes/docker-stale-binaries-path/","section":"field-notes","summary":"Docker can break silently when stale binaries in /usr/local/bin shadow the package-managed versions after an upgrade. The daemon starts, docker ps works, but containers fail to run with no obvious error in the Docker CLI.\nThe Pattern docker ps # works, lists containers docker run hello-world # hangs or fails silently The daemon itself is healthy. The break is in the runtime path. After a Docker or containerd upgrade, the new daemon expects a compatible runc and shim, but an old copy in /usr/local/bin gets resolved first because it appears earlier in $PATH than the packaged binary.\n","tags":["docker","containerd","runc","troubleshooting","linux"],"title":"Docker Stale Binaries In Usr Local Bin"},{"categories":["field-notes"],"content":"Snapshot Manual Snapshot ETCDCTL_ENDPOINTS=https://127.0.0.1:2379 \\ ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt \\ ETCDCTL_CERT=/etc/kubernetes/pki/etcd/server.crt \\ ETCDCTL_KEY=/etc/kubernetes/pki/etcd/server.key \\ etcdctl snapshot save /backup/etcd-snapshot-$(date +%Y%m%d-%H%M%S).db Automated Snapshot (RKE2) RKE2 includes etcd-snapshot as a subcommand:\nrke2 etcd-snapshot save \\ --node-name \u0026lt;node-name\u0026gt; \\ --s3 \\ --s3-bucket=\u0026lt;bucket\u0026gt; \\ --s3-region=\u0026lt;region\u0026gt; \\ --s3-access-key=\u0026lt;key\u0026gt; \\ --s3-secret-key=\u0026lt;secret\u0026gt; Automated snapshots can be configured via the RKE2 config file with etcd-snapshot-schedule-cron and etcd-snapshot-retention.\nVerify Snapshot ETCDCTL_ENDPOINTS=https://127.0.0.1:2379 \\ etcdctl snapshot status /backup/etcd-snapshot-\u0026lt;date\u0026gt;.db -w table Verify the snapshot is not corrupt and check the revision and hash. A snapshot with zero revisions or a mismatched hash indicates corruption.\nRestore Restore To A New Cluster # Stop etcd on all members systemctl stop rke2-server # Restore the snapshot (creates a new data directory) ETCDCTL_ENDPOINTS=https://127.0.0.1:2379 \\ etcdctl snapshot restore /backup/etcd-snapshot-\u0026lt;date\u0026gt;.db \\ --name=\u0026lt;node-name\u0026gt; \\ --initial-cluster=\u0026lt;node-name\u0026gt;=https://\u0026lt;peer-ip\u0026gt;:2380 \\ --initial-advertise-peer-urls=https://\u0026lt;peer-ip\u0026gt;:2380 \\ --data-dir=/var/lib/rancher/rke2/server/db/etcd The restore command creates a new cluster with a new cluster ID. All members must restore from the same snapshot to form the new cluster.\nRestore A Single Member If one member\u0026rsquo;s data is corrupt but the cluster is still healthy, remove the member and re-add it. The new member will stream data from the existing leader. No snapshot restore is needed.\nIf the entire cluster is lost, restore each member from the same snapshot.\nValidation After Restore # Cluster health etcdctl endpoint health -w table --cluster # Verify known keys etcdctl get /registry/namespaces/default -w json | jq .metadata.name # Verify cluster ID matches across all members etcdctl endpoint status --cluster -w table All members must report the same cluster ID.\nRetention Strategy Environment Snapshot Frequency Retention Storage Production Every 4 hours 14 days S3-compatible bucket UAT/Staging Every 12 hours 7 days S3-compatible bucket DR site Cross-region replication 30 days Separate bucket Snapshots are only useful if:\nThey are stored off the member node (single node loss takes the snapshots with it). The restore process is rehearsed at least quarterly. The S3 credentials used for snapshot upload are rotated and monitored. Cross-region replication is tested, not just configured. Disaster Recovery Procedure Total Cluster Loss Provision new nodes with the same IPs or update DNS. Restore the most recent snapshot on the first member. Start the first member and verify health. Add remaining members via member add and start them with initial-cluster-state=existing. Verify all members are healthy with the same cluster ID. Validate Kubernetes resources are accessible. Restart any workloads that depend on API server availability. Partial Failure (Minority Lost) Remove failed members. Add replacement members. Replacements sync from the existing majority. No snapshot needed. Quorum Loss Identify the member with the most recent data (highest revision). Force-remove the other members from that member\u0026rsquo;s perspective. Restart the lone member as a single-node cluster. Add new members to restore quorum. Accept that any writes accepted by the lost members after the last sync are gone. ","permalink":"https://trinidadmarroquin.com/field-notes/etcd-backup-disaster-recovery/","section":"field-notes","summary":"Snapshot Manual Snapshot ETCDCTL_ENDPOINTS=https://127.0.0.1:2379 \\ ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt \\ ETCDCTL_CERT=/etc/kubernetes/pki/etcd/server.crt \\ ETCDCTL_KEY=/etc/kubernetes/pki/etcd/server.key \\ etcdctl snapshot save /backup/etcd-snapshot-$(date +%Y%m%d-%H%M%S).db Automated Snapshot (RKE2) RKE2 includes etcd-snapshot as a subcommand:\nrke2 etcd-snapshot save \\ --node-name \u0026lt;node-name\u0026gt; \\ --s3 \\ --s3-bucket=\u0026lt;bucket\u0026gt; \\ --s3-region=\u0026lt;region\u0026gt; \\ --s3-access-key=\u0026lt;key\u0026gt; \\ --s3-secret-key=\u0026lt;secret\u0026gt; Automated snapshots can be configured via the RKE2 config file with etcd-snapshot-schedule-cron and etcd-snapshot-retention.\nVerify Snapshot ETCDCTL_ENDPOINTS=https://127.0.0.1:2379 \\ etcdctl snapshot status /backup/etcd-snapshot-\u0026lt;date\u0026gt;.db -w table Verify the snapshot is not corrupt and check the revision and hash. A snapshot with zero revisions or a mismatched hash indicates corruption.\n","tags":["etcd","kubernetes","rke2","backup","recovery"],"title":"etcd Backup, Restore, And Disaster Recovery"},{"categories":["field-notes"],"content":"Check Cluster Health # Endpoint health (all members) etcdctl endpoint health -w table --cluster # Endpoint status with version and DB size etcdctl endpoint status -w table --cluster # Member list etcdctl member list -w table # Leader and term etcdctl endpoint status --cluster -w table | awk \u0026#39;{print $1, $4, $5}\u0026#39; Member Lifecycle Add A New Member # On an existing member, add the new peer etcdctl member add \u0026lt;node-name\u0026gt; --peer-urls=https://\u0026lt;new-peer-ip\u0026gt;:2380 # On the new node, start etcd with the initial cluster state set to \u0026#34;existing\u0026#34; # Then verify etcdctl member list -w table etcdctl endpoint health -w table --cluster Remove A Member etcdctl member remove \u0026lt;member-id\u0026gt; After removal, verify quorum and health. The removed member\u0026rsquo;s data directory can be cleaned up.\nReplace An Unhealthy Member # 1. Remove the unhealthy member etcdctl member remove \u0026lt;unhealthy-member-id\u0026gt; # 2. Add the replacement (same name, new peer URL) etcdctl member add \u0026lt;node-name\u0026gt; --peer-urls=https://\u0026lt;new-peer-ip\u0026gt;:2380 # 3. Start etcd on the replacement node with initial cluster state \u0026#34;existing\u0026#34; Quorum Safety Members Quorum Tolerated Failures 1 1 0 3 2 1 5 3 2 7 4 3 Never operate a cluster with 2 members (quorum is 2, failure tolerance is 0). When replacing a member in a 3-node cluster, do not remove and add simultaneously — one node at a time, verifying quorum after each step.\nAlarm Management # List alarms etcdctl alarm list # Disarm (if resolved) etcdctl alarm disarm Common alarms:\nNOSPACE — DB size exceeded the quota. Compact and defragment, then disarm. CORRUPT — Data corruption detected. Restore from snapshot. DB Size And Compaction # Current DB size per member etcdctl endpoint status -w table --cluster | awk \u0026#39;{print $1, $6}\u0026#39; # Compact all historical revisions up to N etcdctl compact \u0026lt;revision\u0026gt; # Defragment (run per member, one at a time) etcdctl defrag --cluster Compaction only frees storage for future writes. Defragmentation reorganizes the DB file to reclaim space. Run defrag during maintenance windows and one member at a time.\nCommon Failure Modes Member removed but process still running. The process will repeatedly log connection errors. Stop and remove the data directory on the evicted node. Clock skew. etcd is sensitive to clock drift. Validate NTP sync across all members. Network partition. A minority partition will not accept writes. It will rejoin and catch up when the partition heals. Slow disk. A member with high fsync latency will degrade the entire cluster. Check disk latency if the leader is stable but followers cannot keep up. ","permalink":"https://trinidadmarroquin.com/field-notes/etcd-member-management/","section":"field-notes","summary":"Check Cluster Health # Endpoint health (all members) etcdctl endpoint health -w table --cluster # Endpoint status with version and DB size etcdctl endpoint status -w table --cluster # Member list etcdctl member list -w table # Leader and term etcdctl endpoint status --cluster -w table | awk \u0026#39;{print $1, $4, $5}\u0026#39; Member Lifecycle Add A New Member # On an existing member, add the new peer etcdctl member add \u0026lt;node-name\u0026gt; --peer-urls=https://\u0026lt;new-peer-ip\u0026gt;:2380 # On the new node, start etcd with the initial cluster state set to \u0026#34;existing\u0026#34; # Then verify etcdctl member list -w table etcdctl endpoint health -w table --cluster Remove A Member etcdctl member remove \u0026lt;member-id\u0026gt; After removal, verify quorum and health. The removed member\u0026rsquo;s data directory can be cleaned up.\n","tags":["etcd","kubernetes","rke2","operations"],"title":"etcd Member Management And Cluster Health"},{"categories":["field-notes"],"content":"Environment variable toggles are the simplest form of feature flag. No SDK, no external service, no runtime dependency. The application reads an env var at startup and enables or disables behavior accordingly.\nFor a runnable lab, see the feature-toggle directory in the IaC repository. It demonstrates the same toggle pattern in both C and Python.\nThe Pattern Python:\nimport os feature_enabled = os.getenv(\u0026#34;FEATURE_ENABLED\u0026#34;, \u0026#34;false\u0026#34;).lower() == \u0026#34;true\u0026#34; if feature_enabled: # new behavior else: # old behavior C:\n#include \u0026lt;stdlib.h\u0026gt; #include \u0026lt;string.h\u0026gt; const char* val = getenv(\u0026#34;FEATURE_ENABLED\u0026#34;); int feature_enabled = (val != NULL \u0026amp;\u0026amp; strcmp(val, \u0026#34;true\u0026#34;) == 0); Both implementations check a single environment variable. The toggle is consistent across languages.\nCross-Language Gotchas C\u0026rsquo;s strcmp is case-sensitive. FEATURE_ENABLED=True evaluates to false. Python\u0026rsquo;s .lower() normalizes the input. Standardize on lowercase true/false values across all services to avoid surprises.\nNo Runtime Reload An env-var toggle requires a process restart to take effect. The container must be redeployed with the new variable value.\nThis is the biggest limitation of the pattern. True feature flag platforms offer runtime reconfiguration, gradual rollout, targeting, and kill switches without redeployment. Env-var toggles are static.\nWhen env-var toggles are appropriate:\nCI/CD gating (enable a step only in certain pipelines). deployment-specific behavior (debug logging, alternative endpoints). off-switch for a risky feature that can wait for a redeploy. lab and development environments where external dependencies add friction. When they are not:\nincident response (you cannot redeploy during an outage). gradual rollouts (you need targeting or percentage-based flags). tenant-specific behavior (each customer needs different flags). Docker Compose Integration services: app: environment: FEATURE_ENABLED: \u0026#34;true\u0026#34; Change the value, rebuild, and redeploy. The toggle is visible in the compose file alongside other configuration.\nThe Progression env var toggle -\u0026gt; config server -\u0026gt; dedicated feature flag platform Start with env vars when the team is small and the deployment cadence is slow. Move to a dedicated platform when you need runtime control, gradual rollout, or audit trails.\nAcceptance Criteria Toggle behavior works identically in all language implementations. Default state (no env var set) is safe. Toggle is documented alongside the feature it controls. Toggle is removed when the feature is fully adopted. Toggle state is visible in logs or metrics during debugging. ","permalink":"https://trinidadmarroquin.com/field-notes/feature-toggles-environment-variables/","section":"field-notes","summary":"Environment variable toggles are the simplest form of feature flag. No SDK, no external service, no runtime dependency. The application reads an env var at startup and enables or disables behavior accordingly.\nFor a runnable lab, see the feature-toggle directory in the IaC repository. It demonstrates the same toggle pattern in both C and Python.\nThe Pattern Python:\nimport os feature_enabled = os.getenv(\u0026#34;FEATURE_ENABLED\u0026#34;, \u0026#34;false\u0026#34;).lower() == \u0026#34;true\u0026#34; if feature_enabled: # new behavior else: # old behavior C:\n","tags":["deployment","cicd","docker","python"],"title":"Feature Toggles With Environment Variables"},{"categories":["notes"],"content":"Local infrastructure labs are valuable when they are treated honestly.\nThey are not production. They are a controlled way to isolate a workflow, prove sequencing, expose assumptions, and build operational muscle memory before the real platform adds more failure modes.\nThe mistake is promoting a lab by copying it directly into production. The better path is to promote the pattern, not the implementation.\nWhat Local Labs Are Good For Local IaC labs are especially useful for learning dependencies.\nA Terraform Docker lab can show that Kafka topic creation needs a broker readiness boundary. A Prometheus and Grafana lab can show that a dashboard is only as useful as the metrics pipeline behind it. A Helm and Terraform lab can show where chart rendering, Kubernetes validation, and release ownership should be separated.\nThose are not toy lessons. They are the same categories of problems that appear in production.\nThe lab makes them cheaper to see.\nWhat Should Not Be Promoted Directly Lab code often contains shortcuts that are acceptable for learning and unacceptable for production:\nhardcoded credentials. local ports as integration contracts. single-node assumptions. latest image tags. local-exec readiness checks. no backup or restore path. no identity model. no alert routing. These are not moral failures. They are reminders that the lab was built for speed and visibility, not long-term ownership.\nPromote The Operating Pattern When moving from lab to production, translate the idea into production controls.\nFor Terraform:\nmove state to a remote backend. define environment roots and module boundaries. pin providers. review plans before apply. make drift detection visible. For Docker-based labs:\nreplace local ports with platform service discovery. replace local volumes with managed persistence or explicit storage classes. define health checks and restart behavior. use image tags that can be traced back to a build. For Helm:\nlint and render charts before release. validate generated manifests. keep values files environment-specific and reviewable. avoid using Helm as a dumping ground for unrelated platform decisions. For observability:\ndecide which SLIs matter. make scrape targets and dashboards versioned. attach alerts to ownership and response expectations. test what happens when the monitored service fails. The Useful Promotion Question The most useful question is not:\nCan this lab run in production? The better question is:\nWhich operating behavior did this lab prove, and what production control should carry that behavior forward? A Kafka readiness loop becomes startup probes, broker health checks, and topic lifecycle ownership. A local Prometheus scrape config becomes service discovery, retention policy, access control, and alert review. A Terraform Docker module becomes a deployment pattern with remote state, pipeline validation, and environment boundaries.\nKeep The Lab Around Do not throw the lab away after production exists.\nSmall labs remain useful for:\nreproducing sequencing bugs. testing provider behavior. explaining system boundaries to teammates. validating dashboard queries. rehearsing failure cases without touching production. The lab should stay small enough that it can be reset quickly. If it becomes a second production environment, it loses its value.\nFinal Rule Local labs should produce judgment, not just code.\nIf the lab helped identify readiness checks, validation gates, state boundaries, secret handling, or rollback expectations, it did its job. The production implementation should preserve those lessons while replacing the shortcuts with controls that match the blast radius.\n","permalink":"https://trinidadmarroquin.com/posts/local-iac-labs-to-production-patterns/","section":"posts","summary":"Local infrastructure labs are valuable when they are treated honestly.\nThey are not production. They are a controlled way to isolate a workflow, prove sequencing, expose assumptions, and build operational muscle memory before the real platform adds more failure modes.\nThe mistake is promoting a lab by copying it directly into production. The better path is to promote the pattern, not the implementation.\nWhat Local Labs Are Good For Local IaC labs are especially useful for learning dependencies.\n","tags":["terraform","docker","helm","kafka","observability","sre"],"title":"From Local IaC Labs To Production-Ready Patterns"},{"categories":["projects"],"content":"GCP organizes resources into projects, which are lighter and more numerous than AWS accounts. This changes how isolation, IAM, and networking are approached.\nProject Structure A GCP project is the primary boundary for resources, IAM, and billing. Use separate projects for production, non-production, shared services, and sandbox work.\nKey differences from other providers:\nprojects belong to a hierarchy: organization -\u0026gt; folder -\u0026gt; project. IAM policies are inherited from parent nodes in the hierarchy. service accounts are project-scoped but can be shared across projects. VPCs are global, not regional. projects have a quota and API enablement surface that must be managed. GKE Cluster Operations GKE offers three modes: Autopilot, Standard (regional), and Standard (zonal). Regional clusters are the recommended default for production because the control plane and nodes are replicated across zones.\nEssential checks during GKE provisioning:\nthe cluster is regional, not zonal. the node pool uses a stable channel and a supported Kubernetes minor version. Workload Identity is enabled for pod-to-GCP authentication. VPC-native (alias IP) is enabled for pod networking. private cluster is enabled for control plane access. Common failure patterns:\nnode auto-upgrade can break compatibility with custom daemonsets or node-level configuration. Workload Identity requires both the GKE metadata server and the IAM binding between the Kubernetes service account and the Google service account. IP address exhaustion in the pod CIDR range causes scheduling failures. regional clusters are more expensive but survive zone failures without manual intervention. IAM And Service Accounts GCP IAM uses roles (primitive, predefined, custom) attached to principals. Service accounts are the identity for workloads, not humans.\nPatterns:\none service account per microservice, not one shared service account per project. use Workload Identity on GKE instead of mounting service account keys. use the Secret Manager or a Vault integration for service account keys that must run outside GCP. audit IAM using the Policy Analyzer, not ad hoc scripts. Cloud IAM conditionals can restrict access by resource, IP range, or time, reducing the need for separate projects for fine-grained access control.\nNetworking GCP VPCs are global. A single VPC can span regions, which simplifies hub-and-spoke designs but requires careful CIDR planning.\nKey networking expectations:\nVPC firewall rules are evaluated in order, with an implicit deny at the end. Cloud NAT is required for private instances to reach the internet. Private Google Access allows on-premises and VM-based access to Google APIs without public IPs. Shared VPC lets the host project own the network while service projects consume subnets. SRE Discipline On GCP Google Cloud publishes SRE resources that map directly to operational maturity. The Google Cloud Architecture Framework covers design, security, privacy, reliability, cost optimization, performance, operations, and sustainability.\nUseful GCP-native observability tools:\nCloud Monitoring for metrics and alerting. Cloud Logging for log aggregation and querying (Logging Query Language). Error Reporting for application error aggregation. Cloud Trace for distributed tracing. For multi-cloud teams, use a consistent observability stack (Prometheus + Grafana) across all providers and treat Cloud Monitoring as a backup sink, not the primary dashboard.\nQuota And Capacity GCP projects have per-service quotas that are not always obvious until a deployment fails.\nCheck before expanding infrastructure:\ncompute engine API capacity in the target region. static IP address quota. GKE cluster and node pool quota. Cloud Load Balancing forwarding rules. Quota increases require a support ticket and should be requested before the capacity is needed.\nAcceptance Criteria Project hierarchy matches team and environment ownership. GKE clusters are regional with Workload Identity and VPC-native networking. Service account keys are not used where Workload Identity can replace them. VPC design accounts for the global scope and firewall evaluation order. Quota monitoring is in place before production deployment. ","permalink":"https://trinidadmarroquin.com/projects/cloud-based-platforms/gcp-operational-patterns/","section":"projects","summary":"GCP organizes resources into projects, which are lighter and more numerous than AWS accounts. This changes how isolation, IAM, and networking are approached.\nProject Structure A GCP project is the primary boundary for resources, IAM, and billing. Use separate projects for production, non-production, shared services, and sandbox work.\nKey differences from other providers:\nprojects belong to a hierarchy: organization -\u0026gt; folder -\u0026gt; project. IAM policies are inherited from parent nodes in the hierarchy. service accounts are project-scoped but can be shared across projects. VPCs are global, not regional. projects have a quota and API enablement surface that must be managed. GKE Cluster Operations GKE offers three modes: Autopilot, Standard (regional), and Standard (zonal). Regional clusters are the recommended default for production because the control plane and nodes are replicated across zones.\n","tags":["gcp","cloud","gke","iam","networking"],"title":"GCP Operational Patterns"},{"categories":["projects"],"content":"GitOps for a platform team is not about syncing a Kubernetes manifest directory. It is about making Git the source of truth for infrastructure state and using pipelines to enforce that state across environments, sites, and provider boundaries.\nThis page collects patterns, pipeline shapes, and operating model decisions for GitOps in infrastructure teams.\nScope These patterns apply to:\nMulti-site RKE2 cluster fleets. vSphere and cloud provider resource lifecycle. Template and image pipeline workflows. Platform tenant onboarding. Secrets and certificate lifecycle management. Configuration drift detection and remediation. The common thread is that Git is the single entry point for change, and pipelines are the only path to production.\nPipeline Shapes Infrastructure Change Pipeline The standard promotion path for infrastructure repositories:\npull request → lint → validate → plan → plan review → apply (non-prod) → apply (prod) → verify Key properties:\nPlan output is retained as a pipeline artifact. Apply stages are serialized per environment backend. Verification runs after apply and fails the job if the declared state does not match the target. Fleet-Wide Change Pipeline For changes that must roll across multiple data centers (image updates, configuration baseline changes, credential rotations):\npipeline triggers → per-site jobs (parallel or serial) → per-site verification → aggregate summary Each site job runs independently and can be retried without rerunning completed sites.\nPlatform Tenant Pipeline For onboarding a new cluster, environment, or project:\ntenant request (PR) → validate tenant manifest → generate resources → apply → notify tenant The tenant manifest is a declarative document (YAML or HCL) that captures the tenant\u0026rsquo;s requirements. The pipeline translates it into platform resources.\nOperating Model Git As The Entry Point Every infrastructure change starts with a pull request. There is no SSH + ad-hoc change path for platform state. This includes:\nCluster node configuration. Pipeline configuration and secrets binding. Image template versions. DNS and load balancer configuration. Monitoring and alerting rules. Pipelines Are The Only Author No human runs terraform apply or ansible-playbook directly against production. The pipeline is the only entity with credentials to apply changes.\nThis means:\nPipeline credentials are scoped per environment. Apply jobs require an explicit approval gate for production. Rollback is a Git revert followed by a pipeline run. State Is In Git, Not In A Backend Terraform state files, Ansible inventory, and configuration registries are treated as artifacts derived from Git. If the Git repo is lost, the ability to manage infrastructure is lost. Backup and disaster recovery procedures must account for the Git repository first, not the remote state backend.\nRepository Structure A common pattern for infrastructure GitOps:\ninfra-repo/ ├── environments/ │ ├── dev/ │ ├── uat/ │ └── prod/ ├── modules/ │ ├── terraform/ │ ├── ansible/ │ └── helm/ ├── pipelines/ │ ├── change-pipeline.yml │ └── fleet-rollout.yml └── tenants/ └── team-a-cluster.yml Environments are self-contained roots with their own backend configuration and variable files. Modules are shared and versioned through the repo, not through a separate registry.\nKey Differences From Application GitOps Aspect Application GitOps Infrastructure GitOps State backend Manifests in repo Terraform state, inventory, secrets Apply frequency Continuous sync Gated per change Rollback Revert manifest Revert + reapply, may need manual cleanup Secrets External sealed/secrets store Pipeline-bound, never in repo Target Kubernetes cluster Clusters, vSphere, cloud APIs, DNS Blast radius Namespace or app Environment, site, or fleet Acceptance Criteria Every production change is traceable to a Git commit. Pipelines are the only path to apply infrastructure changes. Plan output is reviewed before apply. Rollback is a documented and practiced procedure. Fleet-wide changes can be scoped, serialized, and retried per site. New sites or tenants can be onboarded through a pull request. ","permalink":"https://trinidadmarroquin.com/projects/cicd-pipeline-design/gitops-infrastructure-teams/","section":"projects","summary":"GitOps for a platform team is not about syncing a Kubernetes manifest directory. It is about making Git the source of truth for infrastructure state and using pipelines to enforce that state across environments, sites, and provider boundaries.\nThis page collects patterns, pipeline shapes, and operating model decisions for GitOps in infrastructure teams.\nScope These patterns apply to:\nMulti-site RKE2 cluster fleets. vSphere and cloud provider resource lifecycle. Template and image pipeline workflows. Platform tenant onboarding. Secrets and certificate lifecycle management. Configuration drift detection and remediation. The common thread is that Git is the single entry point for change, and pipelines are the only path to production.\n","tags":["gitops","cicd","concourse","infrastructure","platform"],"title":"GitOps For Infrastructure Teams"},{"categories":["field-notes"],"content":"Make Git The Only Entry Point If a change can be made without opening a pull request, it will eventually be made without a pull request. The rule is simple: no PR, no change.\nThis applies to:\nTerraform and Ansible runs. Image template version bumps. Pipeline configuration changes. DNS and load balancer records. Monitoring and alerting rules. Pipeline Shapes Change Pipeline PR → lint → validate → plan → plan review → apply non-prod → apply prod → verify Plan output must be retained as an artifact. Apply stages must be serialized per state backend.\nFleet Rollout Pipeline trigger → per-site jobs (run in parallel or serial groups) → per-site verify → summary Each site runs independently. A failure in one site does not block others. Retry individual sites without rerunning the entire fleet.\nTenant Onboarding Pipeline tenant PR → validate manifest → generate resources → apply → notify The tenant provides a declarative manifest. The pipeline translates it into platform resources. No platform engineer touches a terminal for standard onboarding.\nProduction Rules Pipelines hold the only credentials that can apply changes. No human runs terraform apply or ansible-playbook against production. Production apply jobs require an explicit approval gate. Rollback is git revert + pipeline run. Practice it. If the Git repo is lost, the ability to manage infrastructure is lost. Back up the repo before the Terraform backend. Verification After every apply, verify:\nPipeline job: verify - Does the target match the declared state? - Are checksums or version tags consistent? - Did the apply complete without partial failure? - Does the verification itself produce an auditable result? If verification fails, the pipeline stops. Do not proceed to the next environment or site until verification passes.\nCommon Mistakes Allowing SSH access for debugging. Debugging sessions turn into ad-hoc changes. Use pipeline debug jobs or ephemeral shells that leave no state. Sharing credentials across environments. The pipeline should assume different identities per environment. If the dev credential is compromised, prod should not be reachable. Bypassing plan review for urgent changes. Urgency is when processes matter most. Use expedited review, not no review. Treating rollback as a revert only. Some infrastructure changes (database migrations, storage expansions, certificate rotations) cannot be cleanly reverted. Rollback planning must happen before apply. ","permalink":"https://trinidadmarroquin.com/field-notes/gitops-pipeline-patterns/","section":"field-notes","summary":"Make Git The Only Entry Point If a change can be made without opening a pull request, it will eventually be made without a pull request. The rule is simple: no PR, no change.\nThis applies to:\nTerraform and Ansible runs. Image template version bumps. Pipeline configuration changes. DNS and load balancer records. Monitoring and alerting rules. Pipeline Shapes Change Pipeline PR → lint → validate → plan → plan review → apply non-prod → apply prod → verify Plan output must be retained as an artifact. Apply stages must be serialized per state backend.\n","tags":["gitops","cicd","concourse","infrastructure","platform"],"title":"GitOps Pipeline Patterns For Platform Teams"},{"categories":["field-notes"],"content":"The boundary between Terraform and Helm is a common source of confusion. Terraform provisions infrastructure. Helm deploys applications. Terraform\u0026rsquo;s helm_release resource bridges them, but the chart templates stay in the application repository.\nFor a runnable lab, see the helm-terraform-js-app directory in the IaC repository.\nThe Pattern Terraform manages the Helm release with set blocks that inject environment-specific values:\nresource \u0026#34;helm_release\u0026#34; \u0026#34;my_app\u0026#34; { name = \u0026#34;my-app\u0026#34; chart = \u0026#34;${path.module}/../helm/myapp\u0026#34; namespace = kubernetes_namespace.my_app.metadata[0].name set { name = \u0026#34;image.repository\u0026#34; value = var.docker_image_repository } set { name = \u0026#34;image.tag\u0026#34; value = var.docker_image_tag } set { name = \u0026#34;replicaCount\u0026#34; value = var.replica_count } } The Helm chart stays portable. Environment-specific values live in Terraform variables.\nChart Structure The Helm chart should follow the standard layout and expose the values Terraform needs to override:\nhelm/myapp/ Chart.yaml values.yaml templates/ deployment.yaml service.yaml values.yaml defines defaults:\nreplicaCount: 1 image: repository: my-app tag: latest pullPolicy: Always service: type: ClusterIP port: 80 The chart does not need to know about environments. Terraform overrides what changes per environment.\nValidation Before Apply Run helm template against the chart to validate it before Terraform applies:\nhelm template my-app helm/myapp --values helm/myapp/values.yaml | kubectl apply --dry-run=client -f - This catches syntax errors, missing template variables, and Kubernetes API validation issues before the release is attempted.\nDeployment Pipeline The lab includes scripts for the full pipeline:\n./docker_build.sh # build and tag the image ./docker_push.sh # push to ECR or registry terraform apply # create or update the Helm release The pipeline should:\nbuild and push the image first. run helm template validation. run terraform plan and review. apply with the new image tag. Image Tag Strategy Pass the image tag as a Terraform variable:\nvariable \u0026#34;docker_image_tag\u0026#34; { description = \u0026#34;Docker image tag for the application\u0026#34; type = string } Each deployment gets a unique tag. Avoid latest. Use commit SHAs, semantic versions, or build numbers so every release is identifiable and rollback is unambiguous.\nAcceptance Criteria Terraform creates or updates the Helm release without modifying the chart. Image tag overrides are injected via set blocks. helm template validation passes before Terraform apply. Rollback restores the previous image tag. Chart is versioned in the application repository, not the infrastructure repository. ","permalink":"https://trinidadmarroquin.com/field-notes/helm-terraform-eks-deployment/","section":"field-notes","summary":"The boundary between Terraform and Helm is a common source of confusion. Terraform provisions infrastructure. Helm deploys applications. Terraform\u0026rsquo;s helm_release resource bridges them, but the chart templates stay in the application repository.\nFor a runnable lab, see the helm-terraform-js-app directory in the IaC repository.\nThe Pattern Terraform manages the Helm release with set blocks that inject environment-specific values:\nresource \u0026#34;helm_release\u0026#34; \u0026#34;my_app\u0026#34; { name = \u0026#34;my-app\u0026#34; chart = \u0026#34;${path.module}/../helm/myapp\u0026#34; namespace = kubernetes_namespace.my_app.metadata[0].name set { name = \u0026#34;image.repository\u0026#34; value = var.docker_image_repository } set { name = \u0026#34;image.tag\u0026#34; value = var.docker_image_tag } set { name = \u0026#34;replicaCount\u0026#34; value = var.replica_count } } The Helm chart stays portable. Environment-specific values live in Terraform variables.\n","tags":["terraform","helm","eks","kubernetes","cicd"],"title":"Helm And Terraform Boundary On EKS"},{"categories":["projects"],"content":"When Terraform manages Helm releases, validation needs to happen at more than one layer.\nTerraform can show that the helm_release resource is configured. Helm can render templates. Kubernetes can reject invalid manifests. Production can still fail if the workload has no useful health checks or the image tag points at the wrong build.\nValidation Layers A practical validation sequence is:\nterraform fmt -\u0026gt; terraform validate -\u0026gt; helm lint -\u0026gt; helm template -\u0026gt; kubectl dry-run -\u0026gt; terraform plan Each step answers a different question:\nTerraform validation checks infrastructure syntax and provider configuration. Helm lint checks chart structure and template conventions. Helm template shows the rendered Kubernetes objects. Kubernetes dry-run checks whether the API server accepts the manifests. Terraform plan shows what release change Terraform intends to make. Review Artifacts For deployment review, retain:\nrendered manifests. values files or explicit set overrides. Terraform plan output. image repository and tag. namespace and release name. rollback command or previous release reference. The rendered manifest is especially useful because it removes ambiguity. Reviewers should not have to mentally evaluate templates during an incident or change window.\nAcceptance Criteria Chart linting passes. Rendered manifests are captured. Kubernetes dry-run passes against the intended cluster version when possible. Terraform plan is reviewed before apply. Image tags are specific enough to audit. Rollback path is known before deployment. ","permalink":"https://trinidadmarroquin.com/projects/cicd-pipeline-design/helm-terraform-validation/","section":"projects","summary":"When Terraform manages Helm releases, validation needs to happen at more than one layer.\nTerraform can show that the helm_release resource is configured. Helm can render templates. Kubernetes can reject invalid manifests. Production can still fail if the workload has no useful health checks or the image tag points at the wrong build.\nValidation Layers A practical validation sequence is:\nterraform fmt -\u0026gt; terraform validate -\u0026gt; helm lint -\u0026gt; helm template -\u0026gt; kubectl dry-run -\u0026gt; terraform plan Each step answers a different question:\n","tags":["cicd","helm","terraform","kubernetes","validation"],"title":"Helm And Terraform Validation Strategy"},{"categories":["projects"],"content":"The image factory produces the node template that every cluster in every site boots from. If the factory is manual or inconsistent, every cluster inherits drift from its template.\nThis page collects the pipeline shape, versioning strategy, validation gates, and operating model for a Packer-based image factory with vSphere.\nPipeline Shape graph LR TRIGGER[Git Tag / Schedule] --\u0026gt; BUILD[Packer Build] BUILD --\u0026gt; VALIDATE[Boot \u0026#43; Validate] VALIDATE --\u0026gt; HARDEN[Apply Baselines] HARDEN --\u0026gt; TEMPLATE[Convert To Template] TEMPLATE --\u0026gt; TEST[Test Instance] TEST --\u0026gt; PROMOTE[Promote Template] PROMOTE --\u0026gt; CLEANUP[Cleanup Temp Resources] Show Mermaid source graph LR TRIGGER[Git Tag / Schedule] --\u0026gt; BUILD[Packer Build] BUILD --\u0026gt; VALIDATE[Boot \u0026#43; Validate] VALIDATE --\u0026gt; HARDEN[Apply Baselines] HARDEN --\u0026gt; TEMPLATE[Convert To Template] TEMPLATE --\u0026gt; TEST[Test Instance] TEST --\u0026gt; PROMOTE[Promote Template] PROMOTE --\u0026gt; CLEANUP[Cleanup Temp Resources] Template Versioning Date-based versioning works for node templates because consumers (cluster autoscaler, Terraform modules) select images by a version string, not by semantic compatibility:\nubuntu-2204-v2026.06.01 ubuntu-2404-v2026.05.15 The version is the build date. If a build fails validation, the previous version remains current. Rollback is selecting the previous date tag.\nA current marker (e.g., folder or tag) is updated on each successful promotion. Consumers reference current and get the latest verified template.\nBuild Image Source The base image for Packer builds is a minimal OS ISO, not a previous template build. This avoids accumulating configuration drift across template generations.\nEach build:\nMounts the OS ISO via vSphere. Runs automated OS installation (autoinstall or kickstart). Applies baseline configuration (agents, security policies, kernel parameters). Hardens the image (remove unused packages, apply STIG or CIS baseline). Converts to a vSphere template. Boots a test instance from the template. Runs validation checks against the test instance. Promotes the template to production folders. Validation Gates A build that fails any gate does not become a template:\nGate Check Boot VM boots and is reachable via SSH within timeout Kernel Expected kernel version and parameters Networking Correct interface names, DNS resolution, NTP sync Agents Required agents installed and running (VMTools, monitoring, security) Storage iSCSI initiator configured and can reach targets Containerd Installed and responds to ctr version CIS/STIG Baseline security checks pass Cleanup No build artifacts, SSH host keys rotated Cross-Site Distribution If the image factory serves multiple vSphere sites, each site should have a local copy of the template:\nBuild in the central vSphere instance. Clone the template to each site\u0026rsquo;s vSphere. Verify the cloned template boots and passes validation in each site. Update the site-local current pointer. Site-local templates protect against vSphere link latency and provide a fallback if the central vSphere is unavailable.\nOperating Model Activity Cadence OS patch release Build within 5 business days CVSS 9+ kernel fix Build within 24 hours Agent version update Build on request Template refresh (no changes) Monthly Rollback drill Quarterly Site template validation Weekly Acceptance Criteria Every cluster node boots from a versioned, validated template. Template versions are traceable to a pipeline run and a Git commit. CIS/STIG baseline checks pass on every build. Site-local templates are independently validated. Rollback is selecting the previous date tag and rerunning the site distribution step. ","permalink":"https://trinidadmarroquin.com/projects/packer-image-pipelines/image-factory-workflow/","section":"projects","summary":"The image factory produces the node template that every cluster in every site boots from. If the factory is manual or inconsistent, every cluster inherits drift from its template.\nThis page collects the pipeline shape, versioning strategy, validation gates, and operating model for a Packer-based image factory with vSphere.\nPipeline Shape graph LR TRIGGER[Git Tag / Schedule] --\u0026gt; BUILD[Packer Build] BUILD --\u0026gt; VALIDATE[Boot \u0026#43; Validate] VALIDATE --\u0026gt; HARDEN[Apply Baselines] HARDEN --\u0026gt; TEMPLATE[Convert To Template] TEMPLATE --\u0026gt; TEST[Test Instance] TEST --\u0026gt; PROMOTE[Promote Template] PROMOTE --\u0026gt; CLEANUP[Cleanup Temp Resources] Show Mermaid source graph LR TRIGGER[Git Tag / Schedule] --\u0026gt; BUILD[Packer Build] BUILD --\u0026gt; VALIDATE[Boot \u0026#43; Validate] VALIDATE --\u0026gt; HARDEN[Apply Baselines] HARDEN --\u0026gt; TEMPLATE[Convert To Template] TEMPLATE --\u0026gt; TEST[Test Instance] TEST --\u0026gt; PROMOTE[Promote Template] PROMOTE --\u0026gt; CLEANUP[Cleanup Temp Resources] Template Versioning Date-based versioning works for node templates because consumers (cluster autoscaler, Terraform modules) select images by a version string, not by semantic compatibility:\n","tags":["packer","vsphere","images","automation","pipeline"],"title":"Image Factory Workflow With vSphere And Packer"},{"categories":["field-notes"],"content":"IoT device containers need TLS certificates to authenticate with AWS IoT Core. The certificate path is the critical configuration — if the container cannot find its credentials, it does not connect.\nFor a runnable lab, see the terraform-docker-iot directory in the IaC repository.\nCertificate Injection Pattern Mount certificates from the host using Terraform volume mounts. Do not bake certs into the image:\nresource \u0026#34;docker_container\u0026#34; \u0026#34;iot_device\u0026#34; { image = docker_image.iot_device_image.name volumes { host_path = abspath(\u0026#34;${path.module}/certs/root-CA.crt\u0026#34;) container_path = \u0026#34;/certs/root-CA.crt\u0026#34; } volumes { host_path = abspath(\u0026#34;${path.module}/certs/iot-thing.cert.pem\u0026#34;) container_path = \u0026#34;/certs/iot-thing.cert.pem\u0026#34; } volumes { host_path = abspath(\u0026#34;${path.module}/certs/iot-thing.private.key\u0026#34;) container_path = \u0026#34;/certs/iot-thing.private.key\u0026#34; } } abspath(path.module) resolves the relative cert path to an absolute path relative to the Terraform module directory. This avoids path ambiguity when Terraform runs from different working directories.\nWhy Volume Mounts Instead Of Baking Certs Baking certificates into the image means:\nevery environment needs a separate image. cert rotation requires a full image rebuild and redeploy. anyone with image pull access has the certs. Volume mounts keep certs outside the image lifecycle. The same image runs in dev, staging, and production with different mounted certificates.\nMQTT Over TLS AWS IoT Core uses port 8883 for MQTT over TLS:\nports { internal = 8883 external = 8883 } The device connects using mqtts:// protocol with the mounted CA and device certificates. If the connection fails, check the certificate file paths inside the container first.\nContainer Resilience IoT devices should restart automatically when the connection drops:\nrestart = \u0026#34;always\u0026#34; The application inside the container should implement exponential backoff for reconnection. Docker\u0026rsquo;s restart policy handles the container, but the application still needs to handle transient MQTT disconnections gracefully.\nCommon Failure Modes certificate file paths inside the container do not match the application configuration. the CA certificate is missing or expired. the device certificate is not registered and activated in AWS IoT Core. the MQTT client uses the wrong port (8883 vs 8884 for HTTPS). private key permissions inside the container prevent the application from reading the file. abspath resolves to a different directory than expected when Terraform runs from a pipeline. Acceptance Criteria Container starts without certificate-related errors. MQTT connection to AWS IoT Core succeeds. Certificate rotation requires only a file replacement and container restart. Same image runs across environments with different mounted certs. Application logs show successful connection and message flow. ","permalink":"https://trinidadmarroquin.com/field-notes/iot-docker-terraform-certificates/","section":"field-notes","summary":"IoT device containers need TLS certificates to authenticate with AWS IoT Core. The certificate path is the critical configuration — if the container cannot find its credentials, it does not connect.\nFor a runnable lab, see the terraform-docker-iot directory in the IaC repository.\nCertificate Injection Pattern Mount certificates from the host using Terraform volume mounts. Do not bake certs into the image:\nresource \u0026#34;docker_container\u0026#34; \u0026#34;iot_device\u0026#34; { image = docker_image.iot_device_image.name volumes { host_path = abspath(\u0026#34;${path.module}/certs/root-CA.crt\u0026#34;) container_path = \u0026#34;/certs/root-CA.crt\u0026#34; } volumes { host_path = abspath(\u0026#34;${path.module}/certs/iot-thing.cert.pem\u0026#34;) container_path = \u0026#34;/certs/iot-thing.cert.pem\u0026#34; } volumes { host_path = abspath(\u0026#34;${path.module}/certs/iot-thing.private.key\u0026#34;) container_path = \u0026#34;/certs/iot-thing.private.key\u0026#34; } } abspath(path.module) resolves the relative cert path to an absolute path relative to the Terraform module directory. This avoids path ambiguity when Terraform runs from different working directories.\n","tags":["iot","docker","terraform","security"],"title":"IoT Device Container With Terraform And Certificate Mounts"},{"categories":["field-notes"],"content":"Kafka without Zookeeper runs in KRaft mode using the Raft protocol for metadata consensus. This is the default in Kafka 4.x and available since Kafka 3.x. For local labs, KRaft mode removes the complexity of managing a separate Zookeeper container.\nFor a runnable reference, see the docker/kafka/docker-compose.yml file in the IaC repository.\nMinimal KRaft Configuration A single-broker KRaft setup needs a few key environment variables that differ from Zookeeper-based Kafka:\nservices: kafka: image: confluentinc/cp-kafka:latest environment: KAFKA_KRAFT_MODE: \u0026#34;true\u0026#34; KAFKA_PROCESS_ROLES: \u0026#34;controller,broker\u0026#34; KAFKA_NODE_ID: \u0026#34;1\u0026#34; KAFKA_CONTROLLER_QUORUM_VOTERS: \u0026#34;1@localhost:9093\u0026#34; KAFKA_LISTENERS: \u0026#34;PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093\u0026#34; KAFKA_ADVERTISED_LISTENERS: \u0026#34;PLAINTEXT://localhost:9092\u0026#34; KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: \u0026#34;1\u0026#34; KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: \u0026#34;0\u0026#34; CLUSTER_ID: \u0026#34;MkU3OEVBNTcwNTJENDM2Qk\u0026#34; Key Variables KAFKA_KRAFT_MODE=true — enables the Raft-based metadata mode. KAFKA_PROCESS_ROLES=controller,broker — a single node acts as both controller and broker. Production splits these roles across separate nodes. KAFKA_CONTROLLER_QUORUM_VOTERS=1@localhost:9093 — defines the controller ensemble. In a single-node lab, one voter is sufficient. CLUSTER_ID — a unique identifier generated once. If it changes, the broker treats the metadata as a new cluster. Single-Node Tuning Lab-specific settings that should not reach production:\nKAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 — single replica is fine for testing. KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS=0 — no rebalance delay speeds up consumer group testing. Network A static IP on a custom bridge network avoids hostname resolution issues:\nnetworks: kafka-net: driver: bridge ipam: config: - subnet: 172.19.0.0/16 services: kafka: networks: kafka-net: ipv4_address: 172.19.0.2 Static IPs are fine for a lab. Production should use DNS-based service discovery.\nWhen To Use KRaft KRaft mode is appropriate for:\nlocal development and testing. CI pipelines that need a real Kafka broker. learning Kafka without the Zookeeper overhead. Not appropriate for:\nproduction clusters that need a proven controller quorum implementation. Zookeeper-based Kafka is still the production standard for most teams until KRaft has broader production adoption. Acceptance Criteria Kafka broker starts without Zookeeper dependency. Topics can be created and messages produced and consumed. Repeated restarts do not corrupt metadata. Consumer group rebalancing works with single-broker constraints. Clean shutdown leaves the data directory consistent. ","permalink":"https://trinidadmarroquin.com/field-notes/kafka-kraft-docker-compose/","section":"field-notes","summary":"Kafka without Zookeeper runs in KRaft mode using the Raft protocol for metadata consensus. This is the default in Kafka 4.x and available since Kafka 3.x. For local labs, KRaft mode removes the complexity of managing a separate Zookeeper container.\nFor a runnable reference, see the docker/kafka/docker-compose.yml file in the IaC repository.\nMinimal KRaft Configuration A single-broker KRaft setup needs a few key environment variables that differ from Zookeeper-based Kafka:\nservices: kafka: image: confluentinc/cp-kafka:latest environment: KAFKA_KRAFT_MODE: \u0026#34;true\u0026#34; KAFKA_PROCESS_ROLES: \u0026#34;controller,broker\u0026#34; KAFKA_NODE_ID: \u0026#34;1\u0026#34; KAFKA_CONTROLLER_QUORUM_VOTERS: \u0026#34;1@localhost:9093\u0026#34; KAFKA_LISTENERS: \u0026#34;PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093\u0026#34; KAFKA_ADVERTISED_LISTENERS: \u0026#34;PLAINTEXT://localhost:9092\u0026#34; KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: \u0026#34;1\u0026#34; KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: \u0026#34;0\u0026#34; CLUSTER_ID: \u0026#34;MkU3OEVBNTcwNTJENDM2Qk\u0026#34; Key Variables KAFKA_KRAFT_MODE=true — enables the Raft-based metadata mode. KAFKA_PROCESS_ROLES=controller,broker — a single node acts as both controller and broker. Production splits these roles across separate nodes. KAFKA_CONTROLLER_QUORUM_VOTERS=1@localhost:9093 — defines the controller ensemble. In a single-node lab, one voter is sufficient. CLUSTER_ID — a unique identifier generated once. If it changes, the broker treats the metadata as a new cluster. Single-Node Tuning Lab-specific settings that should not reach production:\n","tags":["kafka","docker","local-dev"],"title":"Kafka KRaft Mode In Docker Compose"},{"categories":["field-notes"],"content":"Kafka labs are good at exposing the difference between a container being started and a broker being usable.\nTerraform can create the Docker network, broker container, application image, and topics, but it needs an explicit readiness boundary. Without that boundary, the next resource may try to create topics or start a processor before Kafka is listening.\nReadiness Boundary For a local lab, a simple port check is often enough to avoid racing the broker startup:\nwhile ! nc -zv localhost 9092; do sleep 5 done That check only proves that something is listening. It does not prove that the broker metadata path, topic APIs, or consumer group behavior are healthy.\nUse it as a first gate, not the final validation.\nBetter Verification After the port is reachable, verify Kafka itself:\nkafka-topics.sh --bootstrap-server localhost:9092 --list Then verify the topics expected by the lab:\nkafka-topics.sh --bootstrap-server localhost:9092 --describe --topic test-input-topic-1 If a stream processor is part of the lab, check its logs separately:\ndocker logs stream_processor Do not treat a successful Terraform apply as proof that messages are flowing.\nTerraform Lab Pattern A practical local pattern is:\nnetwork -\u0026gt; broker container -\u0026gt; readiness check -\u0026gt; topics -\u0026gt; app container -\u0026gt; message-flow test The readiness check should be visible in the graph through dependencies. Topic creation should not run until the broker passes the first gate.\nFor labs, null_resource and local-exec can be acceptable glue. In production, prefer platform-native health checks, managed service readiness, and deployment controllers that understand lifecycle state.\nCommon Failure Modes advertised listeners point clients at the wrong host or port. topics are created before the broker is ready. the stream processor subscribes to a topic name that differs from the created topic. local persisted data carries state from a previous failed run. single-broker defaults hide replication and availability assumptions. The fastest debug path is to separate broker readiness, topic existence, and application processing. They are related, but they fail differently.\nPromotion Notes Before moving a Kafka lab pattern toward production, revisit:\nlistener and network design. topic ownership and creation process. retention and cleanup policy. consumer group behavior. broker persistence and recovery. monitoring for lag, ISR, controller health, and failed produce or consume paths. A lab can teach sequencing. Production Kafka requires durability, capacity, security, and operational ownership.\nAcceptance Criteria Broker container is running. Kafka port readiness gate passes. Topic list and topic describe commands work. Producer and consumer configuration use the same bootstrap path. Stream processor logs show consumption or a clear connection failure. Cleanup removes local state when repeatability matters. ","permalink":"https://trinidadmarroquin.com/field-notes/kafka-readiness-terraform-docker-labs/","section":"field-notes","summary":"Kafka labs are good at exposing the difference between a container being started and a broker being usable.\nTerraform can create the Docker network, broker container, application image, and topics, but it needs an explicit readiness boundary. Without that boundary, the next resource may try to create topics or start a processor before Kafka is listening.\nReadiness Boundary For a local lab, a simple port check is often enough to avoid racing the broker startup:\n","tags":["kafka","terraform","docker","validation","automation"],"title":"Kafka Readiness Checks In Terraform Docker Labs"},{"categories":["projects"],"content":"A local Kafka Streams pipeline managed by Terraform is useful for understanding the full data path before production dependencies are involved. Terraform handles the broker, topics, stream processor image, and container lifecycle — all through the Docker provider.\nFor a complete working example, see the kafka-streams-pipeline directory in the IaC repository. The errors.txt file in the repository contains the exact build failure and resolution for the librdkafka C extension, which is the most common issue when containerizing Python Kafka applications.\nArchitecture Terraform provisions the full pipeline:\nKafka broker (KRaft mode) -\u0026gt; topic creation -\u0026gt; stream processor image -\u0026gt; processor container All resources are managed by Terraform and cleaned up with terraform destroy.\nBroker Provisioning The broker uses KRaft mode (no Zookeeper) with a custom image that adds nc for health checks. The base image is confluentinc/cp-kafka with yum install -y nc in a Red Hat-compatible Dockerfile.\nThe key configuration:\nKAFKA_KRAFT_MODE = \u0026#34;true\u0026#34; KAFKA_PROCESS_ROLES = \u0026#34;controller,broker\u0026#34; KAFKA_NODE_ID = \u0026#34;1\u0026#34; KAFKA_CONTROLLER_QUORUM_VOTERS = \u0026#34;1@localhost:9093\u0026#34; Readiness Checks Terraform uses a null_resource with local-exec to wait for Kafka to be ready before creating topics:\nresource \u0026#34;null_resource\u0026#34; \u0026#34;wait_for_kafka\u0026#34; { provisioner \u0026#34;local-exec\u0026#34; { command = \u0026lt;\u0026lt;-EOC until nc -zv localhost 9092; do sleep 5 done EOC interpreter = [\u0026#34;/bin/bash\u0026#34;, \u0026#34;-c\u0026#34;] } } This is a port-level check. It proves something is listening, not that the broker metadata is initialized. See also: Kafka Readiness Checks In Terraform Docker Labs.\nTopic Creation Topics are created with kafka-topics.sh inside the broker container, triggered by another null_resource:\nresource \u0026#34;null_resource\u0026#34; \u0026#34;create_topics\u0026#34; { depends_on = [null_resource.wait_for_kafka] provisioner \u0026#34;local-exec\u0026#34; { command = \u0026lt;\u0026lt;-EOC docker exec kafka bash -c \u0026#34; /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 \\ --create --topic input-topic \\ --config cleanup.policy=delete \\ --config compression.type=gzip \\ --config delete.retention.ms=86400000 \\ --config min.cleanable.dirty.ratio=0.01 \\ --if-not-exists \u0026#34; EOC } } Topic configuration matters even in a lab. Setting cleanup.policy=delete and compression.type=gzip mirrors production defaults. delete.retention.ms=86400000 keeps data for 24 hours before cleanup, which is enough for local testing.\nStream Processor The Python stream processor uses the confluent_kafka library inside a Docker container. The Dockerfile installs librdkafka-dev which requires the C extension build tools. If the build fails, check:\ngcc and python3-dev are installed in the build image. librdkafka-dev version matches the confluent_kafka Python package version. The base image includes yum groupinstall \u0026quot;Development Tools\u0026quot; for Red Hat-based images. Data Flow producer -\u0026gt; input-topic -\u0026gt; stream_processor -\u0026gt; output-topic -\u0026gt; consumer The stream processor reads from input-topic, applies transformation logic, and writes to output-topic. The processor logs show consumption, transformation, and production latency.\nAcceptance Criteria Kafka broker starts in KRaft mode without Zookeeper. Readiness check passes before topic creation runs. Topics are created with the correct configuration. Stream processor image builds without C extension errors. Processor reads from input topic and writes to output topic. terraform destroy removes all containers, images, and local state. ","permalink":"https://trinidadmarroquin.com/projects/cicd-pipeline-design/kafka-streams-terraform-pipeline/","section":"projects","summary":"A local Kafka Streams pipeline managed by Terraform is useful for understanding the full data path before production dependencies are involved. Terraform handles the broker, topics, stream processor image, and container lifecycle — all through the Docker provider.\nFor a complete working example, see the kafka-streams-pipeline directory in the IaC repository. The errors.txt file in the repository contains the exact build failure and resolution for the librdkafka C extension, which is the most common issue when containerizing Python Kafka applications.\n","tags":["kafka","terraform","data-pipeline","python","docker"],"title":"Kafka Streams Pipeline With Terraform"},{"categories":["projects"],"content":"Kubernetes gives teams enough flexibility to create drift quickly. Platform conventions are the small set of decisions that keep clusters operable when many teams share them.\nThe goal is not to standardize everything. The goal is to standardize the parts that affect troubleshooting, access, security, cost, and recovery.\nNamespace Ownership Every namespace should have an owner, purpose, and lifecycle expectation.\nUse consistent labels:\nmetadata: labels: owner: team-platform environment: prod workload-tier: platform data-classification: internal Use annotations for human-facing metadata:\nmetadata: annotations: owner/contact: platform-ops@example.com runbook/url: https://example.com/runbooks/service escalation/url: https://example.com/oncall/team-platform Minimum namespace contract:\nnamed owning team. support and escalation path. intended environment. resource quota expectation. default network posture. Pod Security Admission level. backup expectation for stateful workloads. Resource Controls Namespaces should not be unlimited by default.\nUse ResourceQuota to prevent one namespace from consuming shared cluster capacity, and use LimitRange to provide defaults or bounds where appropriate.\nReview:\nkubectl get resourcequota -A kubectl get limitrange -A kubectl describe namespace \u0026lt;namespace\u0026gt; Avoid setting defaults so low that teams cargo-cult tiny requests. The point is to make resource ownership visible, not to create noisy throttling incidents.\nIngress Conventions Ingress should be boring and predictable.\nDefine:\nsupported ingress class names. TLS ownership and certificate issuer expectations. DNS naming patterns. allowed public versus internal exposure. required annotations for timeouts, body size, redirects, or auth integrations. where ingress controller logs and metrics are reviewed. Recommended defaults:\nevery production ingress uses TLS. hostnames follow environment and ownership naming rules. public exposure requires explicit approval. ingress class is explicit, not assumed. app teams own route intent; platform owns controller behavior. Useful checks:\nkubectl get ingress -A kubectl get ingressclass kubectl describe ingress -n \u0026lt;namespace\u0026gt; \u0026lt;name\u0026gt; Storage Conventions Storage drift is expensive to debug. Make the intended storage path explicit.\nDefine:\napproved StorageClasses. default StorageClass policy. reclaim policy expectations. volume expansion support. snapshot and restore expectations. which workloads require backup outside Kubernetes. node or VM prerequisites for CSI drivers. For vSphere CSI, worker VMs should meet the CSI prerequisites, including disk UUID visibility where required:\ndisk.EnableUUID = \u0026#34;TRUE\u0026#34; Useful checks:\nkubectl get storageclass kubectl get pvc -A -o wide kubectl get pv -o wide kubectl get volumeattachments -o wide PVCs should specify the intended class when the default is ambiguous or has changed over the cluster lifetime.\nPolicy Conventions Policy should protect shared infrastructure without surprising application teams.\nBaseline policies to define:\nPod Security Admission labels by namespace type. NetworkPolicy default stance. image registry allowlist or private registry behavior. required workload labels. secret handling expectations. admission exceptions and ownership. For Pod Security Admission, namespace labels make the enforcement level visible:\nmetadata: labels: pod-security.kubernetes.io/enforce: restricted pod-security.kubernetes.io/audit: restricted pod-security.kubernetes.io/warn: restricted Not every namespace can start at restricted. Platform namespaces for ingress, CSI, CNI, monitoring, and node agents often need elevated permissions. The convention should document exceptions rather than hide them.\nNetworkPolicy Conventions NetworkPolicy behavior depends on having a policy-capable CNI. If the CNI does not enforce NetworkPolicy, policy manifests are documentation only.\nDefine:\nwhether default deny is required. how namespaces allow DNS. how ingress controller traffic reaches workloads. how monitoring and logging agents scrape or collect data. how cross-namespace service calls are approved. Useful checks:\nkubectl get networkpolicy -A kubectl describe networkpolicy -n \u0026lt;namespace\u0026gt; \u0026lt;name\u0026gt; Labels And Naming Labels should support operations, not just organization charts.\nRecommended common labels:\napp.kubernetes.io/name app.kubernetes.io/instance app.kubernetes.io/component app.kubernetes.io/part-of app.kubernetes.io/managed-by owner environment Use labels for selectors and automation. Use annotations for URLs, descriptions, and long-form metadata.\nGitOps Boundary Platform-owned conventions should live in Git:\nnamespace definitions. quotas and limit ranges. baseline policies. ingress controller configuration. storage class definitions, when platform-managed. monitoring and logging add-ons. Avoid letting long-lived manual changes become the real standard. If a live patch fixes the platform, the follow-up is to commit the desired state.\nAcceptance Criteria A cluster has useful platform conventions when:\nevery namespace has an owner and support path. ingress, DNS, and TLS rules are predictable. approved StorageClasses are documented and visible. stateful workloads have backup and restore expectations. Pod Security Admission levels are labeled and exceptions are intentional. NetworkPolicy posture is understood and enforced by the CNI. common labels support troubleshooting and ownership. GitOps preserves the conventions after sync. References Kubernetes documentation: Namespaces. Kubernetes documentation: Recommended Labels. Kubernetes documentation: Ingress and IngressClass. Kubernetes documentation: StorageClasses and PersistentVolumes. Kubernetes documentation: Pod Security Standards and Pod Security Admission. Kubernetes documentation: Network Policies. ","permalink":"https://trinidadmarroquin.com/projects/kubernetes-platform-operations/platform-conventions/","section":"projects","summary":"Kubernetes gives teams enough flexibility to create drift quickly. Platform conventions are the small set of decisions that keep clusters operable when many teams share them.\nThe goal is not to standardize everything. The goal is to standardize the parts that affect troubleshooting, access, security, cost, and recovery.\nNamespace Ownership Every namespace should have an owner, purpose, and lifecycle expectation.\nUse consistent labels:\nmetadata: labels: owner: team-platform environment: prod workload-tier: platform data-classification: internal Use annotations for human-facing metadata:\n","tags":["kubernetes","platform","storage","ingress","policy"],"title":"Kubernetes Platform Conventions"},{"categories":["projects"],"content":"Kubernetes upgrades should be treated as controlled production changes, not package updates. The same applies to node operating-system patching when automatic updates can restart services, touch device handling, or trigger storage churn. See Ubuntu Unattended Upgrades Are Kubernetes Node Changes for the node patching failure pattern.\nThe hard part is rarely clicking upgrade. The hard part is sequencing the change so operators know what can fail, what is safe to continue, and when to stop.\nRelated incident pattern: When A Latent Rancher Worker Upgrade Becomes An Outage shows how an unfinished worker system-upgrade-controller Plan can become disruptive when worker capacity collapses and GitOps keeps restoring the Plan. For management-cluster upgrade preparation, see Rancher Management Cluster Upgrades Need More Than A Version Target. For post-preflight execution, see Rancher RKE2 Minor Hops When UI Metadata And GitOps Plans Disagree. For node reboot sequencing under storage pressure, see RKE2 Node Reboots When Longhorn Is Already Degraded. For replacement-node rehearsal with prebuilt powered-off VMs, see Fast OS Template Node Replacement Rehearsal.\nUpgrade Principles Use a conservative model:\nupgrade non-production first. upgrade one minor version at a time unless the vendor path explicitly supports otherwise. keep control-plane, etcd, and worker sequencing explicit. drain nodes intentionally rather than relying on surprise evictions. validate platform add-ons before application teams validate workloads. define rollback and stop criteria before the maintenance window starts. For Rancher-managed clusters, also validate Rancher support for the target Kubernetes/RKE2/K3s version before scheduling the work.\nPre-Upgrade Readiness Before the window, capture the current state:\nkubectl version kubectl get nodes -o wide kubectl get pods -A --field-selector=status.phase!=Running kubectl get events -A --sort-by=.lastTimestamp kubectl get pdb -A kubectl get storageclass kubectl get pv,pvc -A Check the platform add-ons:\nkubectl get pods -n cattle-system kubectl get pods -n fleet-system kubectl get pods -n kube-system kubectl get pods -n cert-manager kubectl get pods -n vmware-system-csi The cluster should not enter an upgrade with unresolved node pressure, broken CSI, pending system controllers, expired certificates, or a backlog of failed platform pods.\nWorkload Disruption Review Node upgrades eventually become workload movement.\nBefore draining nodes, check which workloads can tolerate voluntary disruption:\nkubectl get pdb -A kubectl get deployments,statefulsets,daemonsets -A PodDisruptionBudgets are important because kubectl drain respects the eviction API. That is useful protection, but it can also block maintenance if budgets are too strict or replicas are already unhealthy.\nReview for:\nsingleton workloads with no maintenance plan. StatefulSets with storage that cannot move cleanly. PDBs requiring more available replicas than currently exist. workloads pinned to a single worker. DaemonSets that are expected to remain during drain. If the workload cannot move during an upgrade, the maintenance plan should say that explicitly.\nSuggested Sequence For a highly available Rancher-managed cluster, use this general order:\nValidate backups and restore expectations. Confirm Rancher supports the target cluster version. Upgrade a non-production cluster first. Pause or hold unrelated GitOps changes. Upgrade control-plane and etcd components according to the Rancher/RKE2 plan. Upgrade worker pools one node or one controlled batch at a time. Validate platform services. Validate application workloads. Re-enable normal GitOps flow. Capture post-upgrade notes and follow-ups. The exact implementation depends on Rancher, RKE2, K3s, or managed Kubernetes provider behavior. The operational point is that each stage has a checkpoint.\nNode Drain Pattern For manual or assisted worker maintenance, the safe pattern is:\nkubectl cordon \u0026lt;node\u0026gt; kubectl drain \u0026lt;node\u0026gt; --ignore-daemonsets --delete-emptydir-data Then perform the node upgrade or VM maintenance.\nAfter the node returns:\nkubectl uncordon \u0026lt;node\u0026gt; kubectl get node \u0026lt;node\u0026gt; -o wide kubectl describe node \u0026lt;node\u0026gt; Validate that the node is Ready, has no unexpected taints, and that DaemonSets returned.\nMaintenance Window Rules A maintenance window should include:\ntarget clusters and versions. expected order of operations. named operator and reviewer. communication channel. stop criteria. rollback or restore decision point. validation commands. known workloads with disruption risk. Avoid combining an upgrade with unrelated changes. If GitOps, storage, ingress, and node image changes are all moving at the same time, troubleshooting becomes guesswork.\nStop Criteria Stop the upgrade if any of these appear:\netcd health is uncertain. more than one control-plane node is unhealthy in an HA cluster. Rancher agents stop reporting correctly. CSI attach or mount behavior fails after a worker upgrade. CNI or DNS fails cluster-wide. platform controllers are crashlooping. application disruption exceeds the agreed window. Stopping is not failure. Continuing without a stable checkpoint is the failure.\nPost-Upgrade Validation After the upgrade, validate from the platform outward:\nkubectl get nodes -o wide kubectl get pods -A --field-selector=status.phase!=Running kubectl get events -A --sort-by=.lastTimestamp kubectl get deployments,statefulsets,daemonsets -A kubectl get pdb -A kubectl get volumeattachments -A 2\u0026gt;/dev/null || kubectl get volumeattachments For Rancher and Fleet:\nkubectl get pods -n cattle-system kubectl get pods -n fleet-system kubectl get gitrepos -A kubectl get bundles -A kubectl get bundledeployments -A Also verify functional paths:\nRancher UI login through AD. local break-glass login still works if tested as part of the runbook. workload ingress responds. DNS resolves service names. CSI can attach and mount a test volume. monitoring receives fresh samples after the window. Rollback Planning Rollback for Kubernetes upgrades is often constrained. That is why pre-upgrade backups and stop points matter.\nThe plan should define:\netcd snapshot location and restore owner. Rancher backup location and restore owner. node image or VM snapshot policy, if used. whether workload rollback means cluster rollback, application rollback, or both. who can approve restore. Do not rely on a generic \u0026ldquo;rollback if needed\u0026rdquo; line. Write the actual decision tree.\nAcceptance Criteria An upgrade is complete when:\nall nodes are Ready and schedulable unless intentionally cordoned. control-plane and worker versions match the target plan. Rancher and Fleet controllers are healthy. platform add-ons are healthy. no new cluster-wide event pattern is emerging. storage, ingress, DNS, and monitoring have been functionally tested. application owners have completed their agreed validation. follow-up work is captured outside the maintenance window. References Kubernetes documentation: Upgrade A Cluster. Kubernetes documentation: Safely Drain a Node. Kubernetes documentation: Specifying a Disruption Budget for your Application. Rancher documentation: Cluster administration and upgrade guidance for managed clusters. ","permalink":"https://trinidadmarroquin.com/projects/kubernetes-platform-operations/upgrade-sequencing/","section":"projects","summary":"Kubernetes upgrades should be treated as controlled production changes, not package updates. The same applies to node operating-system patching when automatic updates can restart services, touch device handling, or trigger storage churn. See Ubuntu Unattended Upgrades Are Kubernetes Node Changes for the node patching failure pattern.\nThe hard part is rarely clicking upgrade. The hard part is sequencing the change so operators know what can fail, what is safe to continue, and when to stop.\n","tags":["kubernetes","rancher","upgrades","operations"],"title":"Kubernetes Upgrade Sequencing"},{"categories":["field-notes"],"content":"A local SLI lab is useful when the goal is to understand the signal path before introducing production platform complexity.\nFor a live local demo, see the sli_app lab in the IaC repository. If the lab does not run as expected, open a bug fix request against that repository with the failing command, host OS, Docker version, Terraform version, and relevant logs.\nThe important part is not that everything runs on one machine. The important part is that the lab has the same basic observability chain operators rely on later:\ninstrumented app -\u0026gt; Prometheus scrape -\u0026gt; dashboard -\u0026gt; operational question Lab Shape A small SLI lab can be built from four moving pieces:\nan application that exposes Prometheus metrics. Prometheus with explicit scrape targets. Grafana with a pre-provisioned data source and dashboard. cAdvisor for container-level resource visibility. Terraform can wire the containers together with the Docker provider, but it should not hide the operational dependencies. Prometheus still needs reachable targets. Grafana still needs a healthy data source. Dashboards are only useful if the metric labels and queries match the application.\nFirst Checks After apply, start with reachability rather than dashboard screenshots:\ndocker ps --format \u0026#39;table {{.Names}}\\t{{.Status}}\\t{{.Ports}}\u0026#39; Check that Prometheus can see its targets:\nhttp://localhost:9090/targets The app and cAdvisor targets should be UP. If they are not, fix scraping before debugging Grafana.\nMetrics To Prove First For an HTTP service, prove the basic SLI inputs before creating complex dashboards:\nrequest count by endpoint and status. request latency by endpoint. error responses separated from successful responses. container CPU and memory signals from cAdvisor. Example Prometheus checks:\nrequest_count_total request_latency_seconds_bucket container_memory_usage_bytes container_cpu_usage_seconds_total The names must match the actual instrumentation. A dashboard with stale query names is worse than no dashboard because it creates false confidence.\nOperator Notes Local SLI labs usually fail in predictable ways:\nPrometheus is healthy, but scrape targets are down. the app exposes metrics on a different port than the service port. Grafana starts before its data source is ready. dashboards import successfully but query labels do not match. cAdvisor can run but lacks the host mounts needed for useful container visibility. Treat these as signal-chain failures. Move from producer to collector to visualization instead of jumping straight to the UI.\nWhat To Carry Forward Before promoting the pattern beyond a lab, decide:\nwhich SLIs represent user experience. which labels are stable enough for dashboards and alerts. which scrape intervals are useful without creating unnecessary load. how dashboards are provisioned, versioned, and reviewed. what alert would actually page a human. The lab proves mechanics. Production requires ownership, retention, access control, alert routing, and service-level intent.\nAcceptance Criteria App metrics endpoint is reachable. Prometheus target page shows expected targets as UP. Grafana data source connects to Prometheus. Dashboard panels answer a specific operating question. Resource metrics and application metrics can be correlated during a test failure. ","permalink":"https://trinidadmarroquin.com/field-notes/local-sli-labs-prometheus-grafana-cadvisor/","section":"field-notes","summary":"A local SLI lab is useful when the goal is to understand the signal path before introducing production platform complexity.\nFor a live local demo, see the sli_app lab in the IaC repository. If the lab does not run as expected, open a bug fix request against that repository with the failing command, host OS, Docker version, Terraform version, and relevant logs.\nThe important part is not that everything runs on one machine. The important part is that the lab has the same basic observability chain operators rely on later:\n","tags":["sre","observability","prometheus","grafana","terraform","docker"],"title":"Local SLI Labs With Prometheus Grafana And cAdvisor"},{"categories":["projects"],"content":"Telemetry is only useful if operators can find and compare it during pressure.\nNaming conventions reduce the translation tax between teams, dashboards, alerts, and incident notes.\nLog Fields Useful common fields:\ntimestamp level service environment cluster namespace pod trace_id request_id user_safe_error_code Do not put secrets or personal data in logs. Redaction should be a platform expectation, not an afterthought.\nMetric Names Metric names should describe the measured thing and unit.\nExamples:\nhttp_request_duration_seconds http_requests_total queue_depth volume_attach_failures_total Labels should support aggregation without exploding cardinality.\nAvoid labels for unbounded values such as user IDs, raw paths, request IDs, or pod UIDs unless there is a deliberate high-cardinality system for them.\nOwnership Labels Telemetry should carry ownership context:\nservice. team. environment. region. cluster. This makes alerts routable and dashboards filterable.\nAcceptance Criteria Logs can be searched by service, environment, and request correlation ID. Metric units are clear. Labels do not create uncontrolled cardinality. Sensitive data is not logged. Alert labels route to the right owner. ","permalink":"https://trinidadmarroquin.com/projects/observability-incident-response/log-metric-naming/","section":"projects","summary":"Telemetry is only useful if operators can find and compare it during pressure.\nNaming conventions reduce the translation tax between teams, dashboards, alerts, and incident notes.\nLog Fields Useful common fields:\ntimestamp level service environment cluster namespace pod trace_id request_id user_safe_error_code Do not put secrets or personal data in logs. Redaction should be a platform expectation, not an afterthought.\nMetric Names Metric names should describe the measured thing and unit.\n","tags":["observability","logging","metrics"],"title":"Log And Metric Naming Conventions"},{"categories":["field-notes"],"content":"Migration Strategies Attach-And-Clone (Lowest Downtime) For platforms that support concurrent attachment:\n1. Deploy new StorageClass targeting enterprise storage. 2. Create a clone of the existing PVC on the new StorageClass. 3. Attach the clone to a verification pod and validate data integrity. 4. Scale down the workload, detach old PVC, attach new PVC. 5. Scale up the workload. Downtime is limited to the time between scale-down and attach. Data integrity is verified before the cutover.\nBackup-And-Restore (Highest Safety) 1. Take a backup of the Longhorn volume (Velero or native snapshot). 2. Restore the backup to a new PVC on the enterprise StorageClass. 3. Verify the restored data in a separate pod. 4. Scale down the workload, delete old PVC, update PVC claim to new name. 5. Scale up the workload. Longhorn backups can be exported to S3-compatible storage and restored elsewhere. This is the safest pattern but requires the most time and storage overhead.\nIn-Place Migration (StatefulSet Only) For StatefulSets, the PVC template can be updated and pods recreated one at a time:\n1. Create new StorageClass for enterprise storage. 2. Update StatefulSet volumeClaimTemplates to reference new StorageClass. 3. Delete each pod one at a time (start from N-1 down to 0). 4. Each recreated pod gets a new PVC on the new StorageClass. 5. Data migration per pod is handled by the application (if replicas exist). This only works if the application replicates data between replicas (Kafka, Cassandra, Elasticsearch). Do not use this for single-replica databases.\nPre-Migration Validation # List all PVCs on the Longhorn StorageClass kubectl get pvc --all-namespaces -o json | \\ jq -r \u0026#39;.items[] | select(.spec.storageClassName==\u0026#34;longhorn\u0026#34;) | .metadata.namespace + \u0026#34;/\u0026#34; + .metadata.name\u0026#39; # Check which workloads use them kubectl get pods --all-namespaces -o json | \\ jq -r \u0026#39;.items[] | select(.spec.volumes[].persistentVolumeClaim?) | .metadata.namespace + \u0026#34;/\u0026#34; + .metadata.name + \u0026#34; -\u0026gt; \u0026#34; + (.spec.volumes[] | select(.persistentVolumeClaim?) | .persistentVolumeClaim.claimName)\u0026#39; # Validate enterprise StorageClass exists and is functional kubectl describe sc enterprise-storage kubectl get pvc -n default test-enterprise-pvc Volume Size And Access Mode Longhorn supports ReadWriteOnce and ReadWriteMany. Enterprise storage may have different access mode support. Verify before migration:\nkubectl get pvc \u0026lt;pvc-name\u0026gt; -o json | jq \u0026#39;.spec.accessModes\u0026#39; If the target StorageClass does not support the required access mode, the PVC will not bind.\nPost-Migration Validation # PVC is Bound kubectl get pvc \u0026lt;pvc-name\u0026gt; -o json | jq \u0026#39;.status.phase\u0026#39; # Data is accessible kubectl exec \u0026lt;pod\u0026gt; -- ls -la /data # Performance meets expectations kubectl exec \u0026lt;pod\u0026gt; -- dd if=/dev/zero of=/data/test bs=1M count=100 conv=fdatasync # Clean up test file kubectl exec \u0026lt;pod\u0026gt; -- rm /data/test Rollback Plan 1. Keep the original Longhorn PVC (do not delete). 2. If the new PVC fails, scale down the workload. 3. Delete the new PVC. 4. Recreate the original PVC claim if needed. 5. Scale up the workload using the original PVC. Retain Longhorn volumes and snapshots for at least one full retention cycle after migration completes successfully.\nRelated Operations Note If a Longhorn volume fails to attach with no scheduled replicas or Replica Scheduling Failure, do not treat it as a migration problem first. Confirm whether the issue is disk pressure, scheduled-capacity overcommitment, orphaned replica data, or unused PVCs. See Longhorn No Scheduled Replicas Under Disk Pressure.\nFor repeatable review, the public ops-toolbox Longhorn utilities provide read-only checks for scheduler pressure, orphaned replica data, and PVC ownership signals.\n","permalink":"https://trinidadmarroquin.com/field-notes/longhorn-enterprise-storage-migration/","section":"field-notes","summary":"Migration Strategies Attach-And-Clone (Lowest Downtime) For platforms that support concurrent attachment:\n1. Deploy new StorageClass targeting enterprise storage. 2. Create a clone of the existing PVC on the new StorageClass. 3. Attach the clone to a verification pod and validate data integrity. 4. Scale down the workload, detach old PVC, attach new PVC. 5. Scale up the workload. Downtime is limited to the time between scale-down and attach. Data integrity is verified before the cutover.\n","tags":["storage","kubernetes","migration","longhorn","operations"],"title":"Longhorn To Enterprise Storage Migration Patterns"},{"categories":["notes"],"content":"Running RKE2 across multiple data centers reveals problems that single-cluster operations never expose. The differences between sites — hardware generations, network latency, storage backend versions, DNS configurations, operating system versions — become the primary operational concern.\nThis post captures the lessons that held up across sites.\nInventory Design Predicts Operational Pain The inventory is the single source of truth for what runs where. If the inventory is incomplete or inconsistent, every automation step requires manual verification.\nA useful inventory groups nodes by site, role, and cluster in a way that supports both ad-hoc commands and playbook targeting:\nall: children: site-a_prod_rke2: hosts: site-a-etcd-1-rke2: site-a-etcd-2-rke2: site-a-mstr-1-rke2: site-a-wrkr-1-rke2: children: site-a_prod_rke2_etcd: site-a_prod_rke2_mstr: site-a_prod_rke2_wrkr: site-b_prod_rke2: # same structure The pattern is \u0026lt;site\u0026gt;-\u0026lt;environment\u0026gt;_rke2 with subgroups per role. This lets you target by site (site-a_prod_rke2), by role across all sites (_wrkr), or by specific combinations (site-a_prod_rke2:!site-a_prod_rke2_wrkr to exclude workers).\nThe Naming Convention Trap Node hostnames should encode site, role, and sequence number. A name like site-a-etcd-3-rke2 tells you where it is, what it does, which one in the sequence, and which distribution it belongs to.\nA name like ip-10-0-1-45 tells you nothing.\nInconsistent naming across sites is more expensive than bad naming. If one site uses site-a-etcd-1 and another uses site-b-etcd1-rke2, automation that parses node names will have divergent logic paths for each site.\nUpgrade Sequencing The pattern that held up:\netcd members (one at a time, verify quorum after each) → control plane nodes (one at a time, verify API after each) → workers (batched by group, verify workloads after each batch) Across Sites non-production site (full upgrade) → production site A (canary) → production site B → remaining production sites A minimum two-week gap between non-production and production upgrades allows time to observe issues. A one-week gap between production sites allows time to abort before the next site.\nPre-Upgrade Checks Before upgrading any node in any site:\n# Node health kubectl get nodes -o wide kubectl describe node \u0026lt;node\u0026gt; | grep -A5 Conditions # etcd health (from a control plane node) etcdctl endpoint health -w table --cluster # Workload health kubectl get pods --all-namespaces | grep -v Running | grep -v Completed # Storage health kubectl get pvc --all-namespaces | grep -v Bound # Pre-upgrade snapshot rke2 etcd-snapshot save What Breaks At Scale DNS Configuration Drift Every site in a multi-cluster fleet has a DNS configuration story. Some use netplan with static DNS, some use DHCP-injected domains, some have custom resolvers. The same Kubernetes manifest works differently depending on how the node resolver behaves.\nSee DNS Search Domain Debugging for the toolkit.\nStorage Backend Version Skew Different sites may have different vSphere versions, different storage appliance firmware, or different CSI driver versions. A storage class that works in one site may silently fail in another. Validate storage operations per site, not once globally.\nImage Registry Latency A container image that pulls in 5 seconds in one site may take 2 minutes in another. This affects startup time, rolling update windows, and node autoscaler responsiveness. Cache images in a local registry per site or use a CDN-backed registry proxy.\nNode Image Version Spread If each site has a slightly different node operating system version or kernel, the same workload may behave differently. The et al. pattern: standardize the node image across all sites and test updates in a non-production site first.\nUpgrade Velocity The time to upgrade a fleet is bounded by the slowest site and the serialization constraints. Parallelizing across sites while serializing within sites is the correct model, but it requires confidence that the non-production site completed successfully.\nDocument the upgrade timeline expectations before starting. If the fleet takes three weeks to upgrade, the team needs to know that before the first node is drained.\n","permalink":"https://trinidadmarroquin.com/posts/multi-site-rke2-lessons/","section":"posts","summary":"Running RKE2 across multiple data centers reveals problems that single-cluster operations never expose. The differences between sites — hardware generations, network latency, storage backend versions, DNS configurations, operating system versions — become the primary operational concern.\nThis post captures the lessons that held up across sites.\nInventory Design Predicts Operational Pain The inventory is the single source of truth for what runs where. If the inventory is incomplete or inconsistent, every automation step requires manual verification.\n","tags":["rke2","kubernetes","rancher","operations","sre"],"title":"Multi-Site RKE2 Operations: Lessons From The Fleet"},{"categories":["field-notes"],"content":"AI-assisted operations tools are entering the market rapidly. Their value depends entirely on the quality of the observability data they consume. If the data is noisy, incomplete, or unstructured, AI tools amplify the noise instead of reducing it.\nThis field note covers the foundations every operator should have in place before adding AI to the observability stack.\nRequired Foundations Structured Metrics Metrics must have consistent labels across services. The same label (service, environment, region) should mean the same thing in every metric. If one service uses env and another uses environment, any tool consuming both will produce unreliable correlations.\n# Good http_requests_total{service=\u0026#34;api\u0026#34;, environment=\u0026#34;prod\u0026#34;, region=\u0026#34;us-east-1\u0026#34;} 1024 # Bad (consumers cannot reliably aggregate) http_requests_total{svc=\u0026#34;api\u0026#34;, env=\u0026#34;prod\u0026#34;, az=\u0026#34;us-east-1a\u0026#34;} 1024 requests{service=\u0026#34;api\u0026#34;, region=\u0026#34;us-east-1\u0026#34;} 2048 Actionable Dashboards Every dashboard should answer a specific question. Questions like \u0026ldquo;is the service healthy?\u0026rdquo; and \u0026ldquo;is the deployment progressing?\u0026rdquo; are valid. Questions like \u0026ldquo;what do all these charts mean?\u0026rdquo; are not.\nA dashboard that takes more than 30 seconds to read during an incident will not be read during an incident.\nStructured dashboards — SLO at the top, symptom metrics in the middle, cause metrics below — give operators a consistent mental model across services. AI tools that consume dashboard state as context benefit from this structure because the relationship between metrics is already defined.\nCoherent Alert Routing Alert routing that mirrors the team structure makes AI-assisted triage more effective. If an AI tool receives an alert about a service, it should be able to determine which team owns it, which runbook applies, and where to route the notification.\nWithout structured routing, the AI tool either guesses or sends everything to a single channel, replicating the same problem the team already has.\nConsistent Incident Response Observability AI tools learn from incident response patterns. If every incident is handled differently — different communication channels, different severity definitions, different escalation paths — the AI cannot build a useful response model.\nStandardize the incident response process before expecting AI to augment it.\nWhat AI Can Add (After The Foundations) Once the foundations are in place, AI-assisted operations can help with:\nAnomaly correlation. Identifying that a latency spike and an error rate increase in different services share a root cause (e.g., a shared database or network link). Runbook suggestion. Recommending the relevant runbook based on the alert context and historical incidents. Incident timeline generation. Building a timeline from alert history, deployment events, and change records. Post-incident pattern analysis. Identifying recurring incident types across services that share a common vulnerability. None of these work if the underlying data is unstructured, inconsistent, or noisy.\nWhat AI Cannot Fix Missing metrics. If the service emits no metrics, no AI can diagnose it. Undefined SLOs. If the team has not defined acceptable reliability, no AI can tell them whether the system is healthy. Broken alert routing. If alerts go to the wrong team, AI cannot route them correctly without structured ownership data. No incident response process. AI can suggest actions, but if there is no team to execute them, the suggestions have no operational effect. The Order Of Operations 1. Define SLOs for each service. 2. Instrument services with structured metrics. 3. Build dashboards that answer specific questions. 4. Configure alert routing that matches team ownership. 5. Establish a consistent incident response process. 6. Automate runbook steps where possible. 7. THEN evaluate AI-assisted operations tools. Skipping any step before step 7 means the AI tool will be limited by the quality of the foundation it runs on.\n","permalink":"https://trinidadmarroquin.com/field-notes/observability-before-ai/","section":"field-notes","summary":"AI-assisted operations tools are entering the market rapidly. Their value depends entirely on the quality of the observability data they consume. If the data is noisy, incomplete, or unstructured, AI tools amplify the noise instead of reducing it.\nThis field note covers the foundations every operator should have in place before adding AI to the observability stack.\nRequired Foundations Structured Metrics Metrics must have consistent labels across services. The same label (service, environment, region) should mean the same thing in every metric. If one service uses env and another uses environment, any tool consuming both will produce unreliable correlations.\n","tags":["observability","monitoring","alerting","sre","incidents"],"title":"Observability Before AI: What Every Operator Needs First"},{"categories":["notes"],"content":"GitOps adoption usually starts with the tooling and the pipeline mechanics. The harder part is the operating discipline: how a team of humans uses Git to communicate intent, manage risk, and recover from mistakes without bypassing the process.\nThis post covers the practices that matter most once the pipeline is running.\nCommit Messages Are Deployment Descriptions In a GitOps model, the commit message is the primary documentation for every change. It is what operators read during incident review, what the rollback decision is based on, and what appears in the deployment log.\nA useful commit message for an infrastructure change:\nplatform/phx: bump K8s node template to ubuntu-2204-v2026.06.01 TICKET-4172 This image includes the kernel fix for CVE-2026-1234 and updates containerd to v2.0.4. The previous template (ubuntu-2204-v2026.04.15) is retained in vSphere and can be reinstated by reverting this commit. Pre-apply validation: netplan generate passed on site-a-etcd-1. Post-apply verification: kubelet version, node ready, and pod CIDR all confirmed on the canary node before fleet rollout. The format is:\nA subject line that identifies the target and the change. A ticket reference. A body that explains why, what the rollback path is, and what verification was done. Commit messages that say \u0026ldquo;fix bug\u0026rdquo; or \u0026ldquo;update config\u0026rdquo; are not useful in a GitOps model because the commit is the deployment instruction. Every operator should be able to read git log and understand what was deployed and why.\nTagging For Deployment Not every commit should trigger a deployment. Branch-based triggers (every push to main) work for application code with fast feedback loops. Infrastructure changes benefit from explicit tagging:\ngit tag deploy/platform/prod/v2026.06.10-01 The tag is the deployment request. The pipeline picks up the tag, validates the tree, and applies. This separates \u0026ldquo;merged\u0026rdquo; from \u0026ldquo;deployed\u0026rdquo; and gives operators control over timing.\nSemver for infrastructure is useful when the artifact has a meaningful version boundary:\nvmware-template/v2026.06.01 # date-based, not semantic terraform-module/storage/v1.4.2 # semantic, module consumers need compatibility pipeline-config/v2 # major version indicates breaking change in pipeline behavior Date-based versioning works better for infrastructure artifacts that do not have consumer compatibility contracts. Semantic versioning works for shared modules and APIs where consumers need to know the impact of an upgrade.\nCI Triggers CD: The Tradeoffs The most natural GitOps pattern is CI discovers a change and triggers CD. This works well when:\nThe change is small and scoped (single module version bump, one config key change). The target environment has good test coverage and fast feedback. The team is small and changes are infrequent. It creates problems when:\nA documentation PR or comment-only change triggers a full deployment pipeline. Multiple commits land within minutes and the pipeline cannot keep up. An urgent hotfix must skip validation but the trigger does not support it. The CI system is down and no changes can be deployed even if they are safe. These are not theoretical problems. Every team I have seen hit production scale with an automatic CI-to-CD trigger has added an approval gate or a manual release step within a year.\nPractical Middle Ground Separate the CI trigger from the CD decision:\nAction CI CD Trigger Every push to any branch Explicit tag or manual release Validation Lint, validate, plan Same, plus approval gate Apply Never Only from CD trigger Rollback No Git revert + CD trigger The CI pipeline runs on every push and publishes artifacts (plans, validation results, image hashes). The CD pipeline only runs when an operator creates a tag or clicks approve. This gives CI the speed of automation and CD the control of human judgment.\nWebhook Implementations Webhooks are the most common trigger mechanism for GitOps pipelines. The implementation choice matters for reliability.\nPolling Vs. Push Polling (checking Git on a cron schedule) is simpler to debug but introduces delay and wastes pipeline worker cycles. Push (webhook from the Git provider to the pipeline) is responsive but adds a failure point.\nFor infrastructure, a hybrid approach works:\nWebhook triggers the pipeline for normal operations (fast feedback). A periodic polling job catches missed webhooks and reconciles state (safety net). Webhook Reliability Patterns A webhook that fires but does not reach the pipeline is an undelivered deployment request. To handle this:\nMake webhooks idempotent. The pipeline should check whether the commit or tag has already been processed. If it has, skip. Log every webhook payload. Store the raw payload in a durable location so missed webhooks can be replayed. Monitor webhook delivery. If the Git provider reports failed deliveries, the platform team should get an alert. Use a webhook proxy or relay. A lightweight service that accepts webhooks, stores them, and forwards to the pipeline removes the pipeline\u0026rsquo;s uptime from the webhook delivery path. Avoiding The Webhook Stampede A force push that updates 20 commits in one event should not create 20 pipeline runs. The webhook handler should:\nBuffer events for a short window (5-30 seconds). Collapse multiple events on the same ref into a single trigger. Trigger the pipeline once with the latest commit. Without this, a rebase or force push can overwhelm the pipeline workers with redundant runs.\nOperating Cadence A GitOps team\u0026rsquo;s operating cadence should reflect the fact that Git is the control plane:\nActivity Cadence Tool Merge minor changes Daily as ready PR + CI validation Deploy to non-production After merge Tag or approval Deploy to production Scheduled or on demand Tag + CD Audit deployed state Weekly Pipeline verification job Rehearse rollback Monthly Pipeline rollback job Review Git log as deployment history Per incident git log --oneline The key principle: if it is not in Git, it is not deployed. If it is not in Git, it cannot be audited. If it is not in Git, it cannot be rolled back.\nWhat Breaks First New GitOps teams hit these failure modes early:\nThe merge-to-deploy pipeline. Every merge triggers a deployment, including documentation-only changes and WIP branches. Solution: tag-based or approval-gated CD. The credential sprawl. Pipeline credentials are shared across environments because it is easier to set up. Then a dev pipeline compromise becomes a prod incident. Solution: per-environment credentials scoped in the pipeline config. The ad-hoc exception. An urgent change is made via SSH because \u0026ldquo;the pipeline would take too long.\u0026rdquo; The change is not documented, not in Git, and not rolled back when the pipeline runs the next normal deployment. Solution: create a hotfix branch and expedited review path, still through the pipeline. The blind apply. The pipeline applies without verification. A failed apply is not detected until the next alert. Solution: verification jobs after every apply that check the declared state against the target. Documentation As Code A GitOps repo is not just configuration. It is the authoritative source for how the infrastructure works. If the documentation lives outside the repo (wiki, shared drive, chat history), it will drift from the configuration and operators will stop trusting it.\nMermaid Diagrams In Repo Mermaid lets you write diagrams as markdown code blocks. This is valuable in a GitOps repo because the diagram lives alongside the config it describes and changes in the same pull request.\nA pipeline flow diagram in a repo README:\ngraph LR PR[Pull Request] --\u0026gt; CI[CI: Lint \u0026#43; Validate \u0026#43; Plan] CI --\u0026gt; Review[Plan Review] Review --\u0026gt; Tag[Create Deployment Tag] Tag --\u0026gt; CD[CD: Apply Non-Prod] CD --\u0026gt; Approve[Approve Prod] Approve --\u0026gt; Prod[CD: Apply Prod] Prod --\u0026gt; Verify[Verify State] Verify --\u0026gt; Done[Done] style Tag fill:#e6ccff,stroke:#333 style Approve fill:#ffe6cc,stroke:#333 Show Mermaid source graph LR PR[Pull Request] --\u0026gt; CI[CI: Lint \u0026#43; Validate \u0026#43; Plan] CI --\u0026gt; Review[Plan Review] Review --\u0026gt; Tag[Create Deployment Tag] Tag --\u0026gt; CD[CD: Apply Non-Prod] CD --\u0026gt; Approve[Approve Prod] Approve --\u0026gt; Prod[CD: Apply Prod] Prod --\u0026gt; Verify[Verify State] Verify --\u0026gt; Done[Done] style Tag fill:#e6ccff,stroke:#333 style Approve fill:#ffe6cc,stroke:#333 An architecture diagram for a multi-site cluster fleet:\ngraph TB subgraph \u0026#34;Git Repo\u0026#34; CONFIG[Infra Config] PIPELINES[Pipeline Definitions] DOCS[Documentation] end CONFIG --\u0026gt; CI PIPELINES --\u0026gt; CI CI{CI Pipeline} --\u0026gt; VALIDATE[Validate] VALIDATE --\u0026gt; APPLY[Apply] subgraph \u0026#34;Site A\u0026#34; APPLY --\u0026gt; A_ETCD[etcd-1..5] APPLY --\u0026gt; A_MSTR[master-1..3] APPLY --\u0026gt; A_WRKR[worker-1..10] end subgraph \u0026#34;Site B\u0026#34; APPLY --\u0026gt; B_ETCD[etcd-1..5] APPLY --\u0026gt; B_MSTR[master-1..3] APPLY --\u0026gt; B_WRKR[worker-1..10] end subgraph \u0026#34;Site C\u0026#34; APPLY --\u0026gt; C_ETCD[etcd-1..3] APPLY --\u0026gt; C_MSTR[master-1..3] APPLY --\u0026gt; C_WRKR[worker-1..5] end Show Mermaid source graph TB subgraph \u0026#34;Git Repo\u0026#34; CONFIG[Infra Config] PIPELINES[Pipeline Definitions] DOCS[Documentation] end CONFIG --\u0026gt; CI PIPELINES --\u0026gt; CI CI{CI Pipeline} --\u0026gt; VALIDATE[Validate] VALIDATE --\u0026gt; APPLY[Apply] subgraph \u0026#34;Site A\u0026#34; APPLY --\u0026gt; A_ETCD[etcd-1..5] APPLY --\u0026gt; A_MSTR[master-1..3] APPLY --\u0026gt; A_WRKR[worker-1..10] end subgraph \u0026#34;Site B\u0026#34; APPLY --\u0026gt; B_ETCD[etcd-1..5] APPLY --\u0026gt; B_MSTR[master-1..3] APPLY --\u0026gt; B_WRKR[worker-1..10] end subgraph \u0026#34;Site C\u0026#34; APPLY --\u0026gt; C_ETCD[etcd-1..3] APPLY --\u0026gt; C_MSTR[master-1..3] APPLY --\u0026gt; C_WRKR[worker-1..5] end A state machine for a deployment strategy:\nstateDiagram-v2 [*] --\u0026gt; Staged Staged --\u0026gt; Validated: netplan generate Validated --\u0026gt; Applied: maintenance window Applied --\u0026gt; Verified: resolvectl verify Verified --\u0026gt; [*] Validated --\u0026gt; Staged: validation fails Applied --\u0026gt; Staged: rollback Show Mermaid source stateDiagram-v2 [*] --\u0026gt; Staged Staged --\u0026gt; Validated: netplan generate Validated --\u0026gt; Applied: maintenance window Applied --\u0026gt; Verified: resolvectl verify Verified --\u0026gt; [*] Validated --\u0026gt; Staged: validation fails Applied --\u0026gt; Staged: rollback These render automatically on GitHub, GitLab, and any markdown renderer that supports Mermaid. No screenshot tool, no image upload, no out-of-date asset.\nKeeping Diagrams Honest The risk with any documentation is that it drifts from reality. Mermaid reduces the friction to update (edit the text, rerender) but does not eliminate drift. Two patterns help:\nInline in README. For simple flows that change infrequently, write the Mermaid block directly in the markdown file. It is reviewed with every PR that touches the related config.\nGenerated from source. For diagrams that should always match the infrastructure, generate them in CI. A pipeline job can parse Terraform state, Ansible inventory, or a cluster registry and produce a Mermaid diagram:\n# Example: generate a cluster topology diagram from inventory ansible-inventory -i inventory/prod.yaml --list \\ | jq -r \u0026#39; ._meta.hostvars | to_entries[] | select(.value.group_names[] | contains(\u0026#34;rke2\u0026#34;)) | \u0026#34;\\(.key)[\\(.value.group_names | join(\u0026#34;,\u0026#34;))]\u0026#34; \u0026#39; \\ | mermaid-cli-input-generator \u0026gt; topology.mmd The generated diagram is published as a pipeline artifact and linked from the README. If the inventory changes, the diagram changes in the next pipeline run. No manual update step.\nTools like mermaid-cli (mmdc) can render .mmd files to SVG or PNG in CI, which is useful for platforms that do not support native Mermaid rendering:\nnpx @mermaid-js/mermaid-cli mmdc -i topology.mmd -o topology.svg -w 1200 Breaking READMEs Into Meaningful Places A single monolithic README works for a small repo with one audience. Most infrastructure repos have multiple audiences with different questions:\nAudience Wants To Know Where To Put It New team member What does this repo own? How do I set up locally? Top-level README.md Operator How do I deploy? How do I rollback? docs/deploy.md, docs/rollback.md Reviewer What does the pipeline do? What are the gates? docs/pipeline.md Auditor What changed, when, and who approved? Link to pipeline runs + CHANGELOG.md Consumer What version of the module is current? What is the API? docs/consumer.md or module-level README.md A structure that holds up:\ninfra-repo/ ├── README.md # repo overview, local setup, links to docs/ ├── docs/ │ ├── architecture.md # Mermaid diagrams for system architecture │ ├── pipeline.md # pipeline stages, gates, rollback procedure │ ├── deploy.md # deployment walkthrough for operators │ └── tenant-onboarding.md # how to add a new cluster or environment ├── CONTRIBUTING.md # PR workflow, commit message format, review expectations └── CHANGELOG.md # version history, notable changes, breaking changes The top-level README should be short enough to read in one minute. Everything else goes in docs/. This is the same principle as small functions in code: single responsibility makes each file reviewable and maintainable.\nPipeline-Friendly Documentation Documentation in a GitOps repo should be CI-friendly. This means:\nMarkdown, not PDF or Word. Markdown is reviewable in a PR diff. PDFs and Word docs are binary blobs that hide changes. Mermaid blocks, not screenshots. Screenshots cannot be diffed. A Mermaid block change shows up clearly in a PR diff as added and removed lines. Generated content has a CI step. If a diagram or table is generated from infrastructure data, the generation script runs in CI and fails if the output does not match what is committed. This is the same pattern as terraform validate or go generate. Links are relative. Absolute links to external wikis rot. Relative links within the repo are validated by the pipeline and break visibly if a file is moved. Summary Of Practices Practice Why It Matters Commit messages describe the deployment The commit log is the deployment history Tags trigger CD, not branches Separates merged from deployed CI runs on every push, CD only on request Speed with control Webhooks are idempotent and monitored Reliable delivery without duplicates Per-environment credentials Limits blast radius of credential leaks Verification after apply Detects failed or partial applies Rollback is a practiced procedure Not all changes revert cleanly GitOps is not just about putting YAML in a repo and syncing it. The operating model around the tooling is what determines whether the team trusts the pipeline or works around it.\n","permalink":"https://trinidadmarroquin.com/posts/operating-in-gitops/","section":"posts","summary":"GitOps adoption usually starts with the tooling and the pipeline mechanics. The harder part is the operating discipline: how a team of humans uses Git to communicate intent, manage risk, and recover from mistakes without bypassing the process.\nThis post covers the practices that matter most once the pipeline is running.\nCommit Messages Are Deployment Descriptions In a GitOps model, the commit message is the primary documentation for every change. It is what operators read during incident review, what the rollback decision is based on, and what appears in the deployment log.\n","tags":["gitops","cicd","devops","sre","automation"],"title":"Operating In A GitOps Environment: Practices That Hold Up"},{"categories":["projects"],"content":"Base images are production dependencies. Treat them like versioned platform artifacts, not disposable installer output.\nPacker should create the repeatable baseline. Per-environment configuration should happen later through Terraform, cloud-init, configuration management, or bootstrap scripts.\nBaseline Contents Images should include:\nOS updates at build time. required agents. VMware tools or cloud guest agents. logging and monitoring prerequisites. bootstrap entrypoint. security baseline packages. cleanup of machine identity before sealing. Images should not include environment-specific secrets, static hostnames, or one-off workload configuration.\nPatch Cadence Define how often images are rebuilt even when no feature changes are requested.\nRecommended triggers:\nmonthly patch cycle. critical CVE. guest agent update. bootstrap framework change. cloud or vSphere template requirement change. Retirement Template retirement is part of image hygiene.\nTrack:\ncurrent recommended image. previous rollback image. deprecated images. deletion date. dependent environments. Acceptance Criteria Images are reproducible from source. Build date and version are visible. Secrets are not baked into templates. Old images are retired intentionally. Downstream consumers know the supported image set. References Packer documentation: HCL Templates. Packer documentation: Build Block. ","permalink":"https://trinidadmarroquin.com/projects/packer-image-pipelines/base-image-hardening/","section":"projects","summary":"Base images are production dependencies. Treat them like versioned platform artifacts, not disposable installer output.\nPacker should create the repeatable baseline. Per-environment configuration should happen later through Terraform, cloud-init, configuration management, or bootstrap scripts.\nBaseline Contents Images should include:\nOS updates at build time. required agents. VMware tools or cloud guest agents. logging and monitoring prerequisites. bootstrap entrypoint. security baseline packages. cleanup of machine identity before sealing. Images should not include environment-specific secrets, static hostnames, or one-off workload configuration.\n","tags":["packer","images","security"],"title":"Packer Base Image Hardening"},{"categories":["projects"],"content":"A Packer build that completes is not necessarily a usable image.\nValidation gates should prove that downstream provisioning can consume the image without rediscovering template defects.\nBuild-Time Validation During the build, validate:\npackage installation. service enablement. guest agent status. SSH or WinRM access. cleanup scripts. cloud-init or customization readiness. Post-Build Validation After the template is created, clone it in a test environment.\nCheck:\nVM boots cleanly. hostname customization works. primary network config works. SSH access works through expected users. guest tools report healthy. bootstrap logs are present. storage prerequisites are installed. For Kubernetes node images, also validate container runtime, kubelet prerequisites, CSI prerequisites, and any required kernel modules or packages.\nFailure Handling Failed validation should prevent promotion.\nDo not fix a bad template with per-clone Terraform hacks. If every clone needs the same repair, the image is wrong.\nAcceptance Criteria Build succeeds. Clone test succeeds. Network and access are verified. Guest agents are healthy. Bootstrap can run idempotently. The image is promoted only after validation. References Packer documentation: Build Block. Packer documentation: Provisioners. ","permalink":"https://trinidadmarroquin.com/projects/packer-image-pipelines/image-validation-gates/","section":"projects","summary":"A Packer build that completes is not necessarily a usable image.\nValidation gates should prove that downstream provisioning can consume the image without rediscovering template defects.\nBuild-Time Validation During the build, validate:\npackage installation. service enablement. guest agent status. SSH or WinRM access. cleanup scripts. cloud-init or customization readiness. Post-Build Validation After the template is created, clone it in a test environment.\nCheck:\nVM boots cleanly. hostname customization works. primary network config works. SSH access works through expected users. guest tools report healthy. bootstrap logs are present. storage prerequisites are installed. For Kubernetes node images, also validate container runtime, kubelet prerequisites, CSI prerequisites, and any required kernel modules or packages.\n","tags":["packer","images","validation"],"title":"Packer Image Validation Gates"},{"categories":["projects"],"content":"Image versioning should make rollback boring.\nIf operators cannot tell which template built a VM, what changed in that template, and whether it is safe to roll back, the image pipeline is incomplete.\nVersion Format Use a version format that carries time and intent:\nubuntu-22.04-k8s-node-2026.06.10-1 ubuntu-22.04-base-2026.06.10-1 Include:\nOS family and version. purpose. build date. build sequence or semantic version. Metadata Store metadata with the image:\nGit commit. Packer version. builder plugin versions. package manifest or important package versions. validation result. deprecation state. In vSphere, use template names, notes, tags, or custom attributes consistently.\nPromotion Separate build from promotion:\nbuild -\u0026gt; validate -\u0026gt; mark candidate -\u0026gt; promote -\u0026gt; consume Terraform should consume promoted images, not whatever Packer happened to build most recently.\nAcceptance Criteria Every VM can be traced to an image version. Rollback image is known. Image metadata links to source. Promotion is explicit. Deprecated images are not silently consumed. References Packer documentation: HCL Templates. HashiCorp guidance on artifact and image workflows. ","permalink":"https://trinidadmarroquin.com/projects/packer-image-pipelines/image-versioning/","section":"projects","summary":"Image versioning should make rollback boring.\nIf operators cannot tell which template built a VM, what changed in that template, and whether it is safe to roll back, the image pipeline is incomplete.\nVersion Format Use a version format that carries time and intent:\nubuntu-22.04-k8s-node-2026.06.10-1 ubuntu-22.04-base-2026.06.10-1 Include:\nOS family and version. purpose. build date. build sequence or semantic version. Metadata Store metadata with the image:\nGit commit. Packer version. builder plugin versions. package manifest or important package versions. validation result. deprecation state. In vSphere, use template names, notes, tags, or custom attributes consistently.\n","tags":["packer","images","automation"],"title":"Packer Image Versioning"},{"categories":["projects"],"content":"Pipelines are privileged automation. Treat their credentials like production access, because that is what they are.\nCredential design should match the environment and action being performed.\nCredential Handling Use separate credentials for:\nread-only validation. non-production deploys. production plans. production applies. artifact publication. Prefer short-lived credentials from Vault, cloud identity federation, or platform-native workload identity. Avoid storing long-lived credentials in pipeline definitions.\nResource Design Pipeline resources should make flow visible:\nsource repository. versioned artifact. environment configuration. approval gate. deployment target. Do not let a pipeline secretly fetch mutable inputs without recording what it used.\nPromotion Promotion should move the same artifact or reviewed configuration forward.\nRecommended flow:\ndev -\u0026gt; integration -\u0026gt; staging -\u0026gt; production Promotion should not mean rebuilding a different artifact with the same name.\nAcceptance Criteria Production credentials are isolated from non-production jobs. Secret access is audited. Promotion uses immutable artifacts or reviewed commits. Branch rules match environment risk. Emergency bypass is documented and reviewed after use. ","permalink":"https://trinidadmarroquin.com/projects/cicd-pipeline-design/credential-handling-promotion/","section":"projects","summary":"Pipelines are privileged automation. Treat their credentials like production access, because that is what they are.\nCredential design should match the environment and action being performed.\nCredential Handling Use separate credentials for:\nread-only validation. non-production deploys. production plans. production applies. artifact publication. Prefer short-lived credentials from Vault, cloud identity federation, or platform-native workload identity. Avoid storing long-lived credentials in pipeline definitions.\nResource Design Pipeline resources should make flow visible:\nsource repository. versioned artifact. environment configuration. approval gate. deployment target. Do not let a pipeline secretly fetch mutable inputs without recording what it used.\n","tags":["cicd","secrets","automation"],"title":"Pipeline Credential Handling And Promotion"},{"categories":["projects"],"content":"Infrastructure pipelines should make change intent visible before they mutate anything.\nThe stage design should separate fast feedback from approval and execution.\nStage Model A practical infrastructure pipeline usually has:\nformat -\u0026gt; validate -\u0026gt; security checks -\u0026gt; plan -\u0026gt; review -\u0026gt; apply -\u0026gt; verify Each stage should produce evidence that the next stage can trust.\nValidation Validation should be fast and safe:\nformatting. static syntax validation. provider initialization without touching production state when possible. unit-style checks for generated manifests. policy checks for obvious violations. Planning Plan output is the core review artifact.\nFor Terraform, capture:\ncommand used. workspace or root module. variables used. provider lock file state. human-readable plan. JSON plan for automation. Apply Apply should be controlled by branch, environment, approval, or change window.\nAvoid automatic production applies from unreviewed commits. Automation should reduce toil, not remove accountability.\nPost-Deploy Verification The pipeline should verify the result:\nresource exists. service is reachable. health checks pass. monitoring sees the new state. drift does not immediately reappear. Acceptance Criteria Failed validation blocks planning. Plan review happens before production apply. Apply output is retained. Verification is explicit. Rollback or recovery notes are linked to the change. ","permalink":"https://trinidadmarroquin.com/projects/cicd-pipeline-design/validation-planning-apply-stages/","section":"projects","summary":"Infrastructure pipelines should make change intent visible before they mutate anything.\nThe stage design should separate fast feedback from approval and execution.\nStage Model A practical infrastructure pipeline usually has:\nformat -\u0026gt; validate -\u0026gt; security checks -\u0026gt; plan -\u0026gt; review -\u0026gt; apply -\u0026gt; verify Each stage should produce evidence that the next stage can trust.\nValidation Validation should be fast and safe:\nformatting. static syntax validation. provider initialization without touching production state when possible. unit-style checks for generated manifests. policy checks for obvious violations. Planning Plan output is the core review artifact.\n","tags":["cicd","terraform","automation"],"title":"Pipeline Validation Planning And Apply Stages"},{"categories":["projects"],"content":"Post-incident follow-up should improve the system, not just produce a document.\nThe useful output is a small set of changes that reduce recurrence, shorten detection, or make recovery safer.\nTimeline Build a factual timeline:\nfirst signal. detection time. acknowledgement time. mitigation actions. recovery time. customer or user impact. follow-up decisions. Avoid filling gaps with guesses. Mark unknowns explicitly.\nAnalysis Separate:\ntrigger. contributing factors. detection gap. mitigation gap. recovery gap. durable fix. Do not stop at the first human mistake. Ask what system condition made that mistake possible or likely.\nCorrective Actions Good actions are specific:\nadd alert for symptom X. remove noisy alert Y. add runbook step Z. test restore path monthly. enforce policy in CI. change ownership or escalation path. Bad actions are vague:\nbe more careful improve monitoring document better Acceptance Criteria Incident has a factual timeline. Customer or system impact is described clearly. Follow-up actions have owners and due dates. At least one action improves detection, mitigation, or prevention. Lessons are fed back into runbooks, alerts, or platform standards. References Google SRE Book: Managing Incidents. Google SRE Book: Postmortem Culture. ","permalink":"https://trinidadmarroquin.com/projects/observability-incident-response/post-incident-follow-up/","section":"projects","summary":"Post-incident follow-up should improve the system, not just produce a document.\nThe useful output is a small set of changes that reduce recurrence, shorten detection, or make recovery safer.\nTimeline Build a factual timeline:\nfirst signal. detection time. acknowledgement time. mitigation actions. recovery time. customer or user impact. follow-up decisions. Avoid filling gaps with guesses. Mark unknowns explicitly.\nAnalysis Separate:\ntrigger. contributing factors. detection gap. mitigation gap. recovery gap. durable fix. Do not stop at the first human mistake. Ask what system condition made that mistake possible or likely.\n","tags":["incidents","sre","operations"],"title":"Post-Incident Follow-Up"},{"categories":["projects"],"content":"Rancher is useful because it gives operators a single control plane for many Kubernetes clusters. That centralization is also the risk: if access, workspace boundaries, cluster ownership, and GitOps conventions are loose, the management plane becomes another source of drift.\nThe operating standard should make the boring parts explicit.\nAccess Model Rancher should authenticate normal users through Active Directory. Roles should be assigned to AD groups, not individual users, wherever possible.\nThis gives the platform team a single place to manage onboarding, offboarding, role changes, and audit expectations. Rancher supports external authentication providers, including Active Directory, and uses users and groups from that provider for authorization decisions.\nThe default posture should be:\nAD is the source of truth for human access. Rancher global admin access is limited to a small platform operations group. Cluster and project access is granted through AD groups mapped to Rancher roles. Individual user grants are temporary exceptions with an owner and expiration date. Access to the Rancher local cluster is restricted to trusted administrators only. Avoid broad site access. Rancher documents an option to allow any valid user from the external provider, but that is a poor default for production. Use a restricted or authorized-user model so only approved AD groups can log in.\nBreak-Glass Access Keep a local Rancher break-glass account for emergencies.\nThis is not a convenience account. It exists for cases where AD, SSO, DNS, certificate trust, or the external identity path is unavailable and operators must still recover the platform.\nBreak-glass expectations:\none or two local accounts maximum. unique long random password stored in the approved emergency secret store. MFA if supported by the chosen access path. no day-to-day use. every login triggers review or an incident note. password is rotated after use and on a scheduled cadence. account ownership is documented with the platform operations team. Rancher documentation explicitly notes that local users may be useful for rare circumstances such as external authentication provider outages or maintenance. That matches this model: AD for normal access, local account only for emergency recovery.\nRBAC Shape Rancher authorization is layered on top of Kubernetes RBAC. Treat Rancher roles as production access controls, not UI preferences.\nUse a small role model:\nplatform-admin: Rancher global administration and local cluster administration. cluster-admin: administrative access to specific downstream clusters. project-admin: management of namespaces and applications inside assigned Rancher projects. developer: workload read/write inside assigned projects, without cluster-level administration. read-only: observability, troubleshooting, and audit access without mutation. Prefer group-to-role mapping:\nAD group -\u0026gt; Rancher global role, cluster role, or project role -\u0026gt; Kubernetes RBAC enforcement Avoid assigning cluster-admin because it is easy. Kubernetes RBAC best practices emphasize least privilege, minimizing wildcard permissions, limiting access to privileged verbs, and treating impersonation, secret access, and workload creation as sensitive capabilities.\nFleet Workspace Organization Fleet uses namespaced GitRepo resources. Rancher-created Fleet workspaces commonly include fleet-local for the local cluster and fleet-default for registered downstream clusters.\nUse workspace boundaries intentionally:\nfleet-local is for Rancher management-plane resources. downstream cluster configuration belongs in workspaces that map cleanly to environment, tenant, or platform ownership. GitRepos should have narrow paths and explicit targets. credentials for private repositories belong in Kubernetes secrets in the same namespace as the GitRepo. Git credentials, Helm credentials, and sensitive values should not be stored in plain text Git. Fleet supports GitRepo targeting and namespace behavior, but that flexibility can cause damage if the repository scope is too broad. A GitRepo that points at too much of a monorepo or too many clusters makes blast radius hard to reason about.\nRepository Standards Each Fleet-managed repository should answer five questions quickly:\nWhich clusters does this apply to? Which team owns the change? Which environment receives it first? What validates the rendered manifests before merge? What is the rollback path? Recommended repository layout:\nclusters/ prod/ nonprod/ platform/ ingress/ storage/ monitoring/ apps/ team-a/ team-b/ Use labels and naming conventions consistently:\nenvironment=prod|nonprod|dev region=\u0026lt;region\u0026gt; cluster=\u0026lt;cluster-name\u0026gt; owner=\u0026lt;team\u0026gt; tier=platform|application The exact taxonomy can change, but it should be consistent enough that operators can select clusters safely and answer what a bundle affects.\nCluster Standards Every Rancher-managed cluster should have a small baseline before application teams depend on it:\nnamed owner and escalation path. Kubernetes version and upgrade policy. node pool roles and labels. ingress controller expectation. default StorageClass expectation. backup and restore expectation. monitoring and alert routing. namespace and project ownership model. Pod Security Admission posture. NetworkPolicy provider and default stance. This baseline should be visible in Git or a platform inventory, not only in someone’s memory.\nOperational Checks Useful periodic checks:\nkubectl get clusters.management.cattle.io kubectl get gitrepos -A kubectl get bundles -A kubectl get bundledeployments -A kubectl get users.management.cattle.io kubectl get globalrolebindings.management.cattle.io Review for:\nstale local users. direct user role bindings that should be AD group bindings. GitRepos without clear ownership. bundles stuck in modified or error state. clusters missing labels needed for targeting. credentials that are not covered by backup or rotation policy. Acceptance Criteria A Rancher-managed fleet is healthy when:\nnormal access comes from AD groups. the break-glass account exists, is tested, and is not used casually. local cluster access is restricted to trusted platform administrators. cluster and project roles are understandable without reverse engineering. GitRepos have clear ownership, scope, and target clusters. sensitive Git or Helm credentials are stored in Kubernetes secrets, not plain text repositories. every cluster has a documented baseline and owner. Fleet drift and deployment failures are visible to the platform team. References Rancher documentation: Configuring Authentication. Rancher documentation: Managing Role-Based Access Control. Fleet documentation: Create a GitRepo Resource. Kubernetes documentation: RBAC Good Practices. ","permalink":"https://trinidadmarroquin.com/projects/kubernetes-platform-operations/rancher-fleet-standards/","section":"projects","summary":"Rancher is useful because it gives operators a single control plane for many Kubernetes clusters. That centralization is also the risk: if access, workspace boundaries, cluster ownership, and GitOps conventions are loose, the management plane becomes another source of drift.\nThe operating standard should make the boring parts explicit.\nAccess Model Rancher should authenticate normal users through Active Directory. Roles should be assigned to AD groups, not individual users, wherever possible.\n","tags":["kubernetes","rancher","fleet","rbac","active-directory"],"title":"Rancher Fleet Standards"},{"categories":["notes"],"content":"RKE2 clusters across five sites had DNS search domains leaking into Kubernetes name resolution. Hostnames that should have resolved as-is were getting suffix expansion, and the behavior was inconsistent across data centers.\nThe fix required understanding which layer was injecting the search domain — and it was not always the same layer.\nThis post covers the audit, the diagnostics, the remediation script, and the edge cases that made it interesting.\nThe Audit The desired state was simple: netplan should have search: [], and the active resolver should show . (which means no search domain in systemd-resolved).\nThe first step was to audit every node across all sites, comparing the netplan config against the active resolver state:\nOUT=\u0026#34;dns-search-audit-$(date +%Y%m%d-%H%M%S).csv\u0026#34; echo \u0026#39;inventory_host,actual_hostname,resolv_conf_search,netplan_search\u0026#39; \u0026gt; \u0026#34;$OUT\u0026#34; ANSIBLE_NOCOLOR=1 \\ ANSIBLE_STDOUT_CALLBACK=default \\ ANSIBLE_HOST_KEY_CHECKING=False \\ ansible all \\ -i inventory/prod.yaml \\ -u operator -b -kK \\ -m shell -a \u0026#39; ACTUAL_HOSTNAME=$(hostname -s) RESOLV_SEARCH=$(awk \u0026#34;/^search/{\\$1=\\\u0026#34;\\\u0026#34;; sub(/^ /,\\\u0026#34;\\\u0026#34;); print; exit} /^domain/{print \\$2; exit}\u0026#34; /etc/resolv.conf) [ -z \u0026#34;$RESOLV_SEARCH\u0026#34; ] \u0026amp;\u0026amp; RESOLV_SEARCH=\u0026#34;NONE\u0026#34; NETPLAN_SEARCH=$(grep -R \u0026#34;search:\u0026#34; /etc/netplan/*.yaml /etc/netplan/*.yml 2\u0026gt;/dev/null | sed \u0026#34;s/.*search:[[:space:]]*//\u0026#34; | paste -sd \u0026#34;;\u0026#34; -) [ -z \u0026#34;$NETPLAN_SEARCH\u0026#34; ] \u0026amp;\u0026amp; NETPLAN_SEARCH=\u0026#34;NONE\u0026#34; printf \u0026#34;CSV|{{ inventory_hostname }}|%s|%s|%s\\n\u0026#34; \u0026#34;$ACTUAL_HOSTNAME\u0026#34; \u0026#34;$RESOLV_SEARCH\u0026#34; \u0026#34;$NETPLAN_SEARCH\u0026#34; \u0026#39; \\ | sed -n \u0026#39;s/.*CSV|//p\u0026#39; \\ | awk -F\u0026#39;|\u0026#39; \u0026#39;{print $1 \u0026#34;,\u0026#34; $2 \u0026#34;,\u0026#34; $3 \u0026#34;,\u0026#34; $4}\u0026#39; \\ | sort -t, -k1,1 \\ \u0026gt;\u0026gt; \u0026#34;$OUT\u0026#34; column -s, -t \u0026#34;$OUT\u0026#34; The output revealed three distinct patterns:\nPattern Sites resolv_conf_search netplan_search Already clean Site-A . [] Staged but not applied Site-B, Site-D internal.corp.example [] Dynamic link-level injection Site-C dc.corp.example NONE Sites B and D had already been \u0026ldquo;fixed\u0026rdquo; — the netplan files had search: [] — but the change had never been applied. The active resolver was still serving the old search domain.\nSite C was different. netplan_search showed NONE, meaning netplan was not the source. The domain was being injected somewhere else.\nThe Netplan Remediation Playbook For sites where netplan was the authority, the fix was straightforward. The playbook backed up existing netplan files, ensured search: [] was present, and staged the config:\n- name: Back up existing netplan files copy: src: \u0026#34;{{ item }}\u0026#34; dest: \u0026#34;{{ backup_dir }}/\u0026#34; remote_src: yes loop: \u0026#34;{{ query(\u0026#39;file_glob\u0026#39;, \u0026#39;/etc/netplan/*.yaml\u0026#39;) + query(\u0026#39;file_glob\u0026#39;, \u0026#39;/etc/netplan/*.yml\u0026#39;) }}\u0026#34; - name: Set DNS search to empty in netplan ansible.builtin.lineinfile: path: \u0026#34;{{ item }}\u0026#34; regexp: \u0026#39;^\\s+search:\u0026#39; line: \u0026#39; search: []\u0026#39; loop: \u0026#34;{{ query(\u0026#39;file_glob\u0026#39;, \u0026#39;/etc/netplan/*.yaml\u0026#39;) + query(\u0026#39;file_glob\u0026#39;, \u0026#39;/etc/netplan/*.yml\u0026#39;) }}\u0026#34; Key production decision: the playbook ran netplan generate to validate, but did not run netplan apply. Applying was deferred to a maintenance window, keeping the runtime resolver unchanged until a controlled rollout.\nDrift Detection After staging, a quick awk filter identified remaining drift:\nawk -F, \u0026#39;NR==1 || $3!=\u0026#34;.\u0026#34; || $4!=\u0026#34;[]\u0026#34;\u0026#39; \u0026#34;$OUT\u0026#34; | column -s, -t Drift showing internal.corp.example / [] was expected — netplan was clean, runtime was stale. That was the staging signal.\nThe Site-C Puzzle: Link-Level DNS Injection Site C did not respond to any netplan changes because netplan was never the source. The audit showed:\nnetplan_search = NONE resolv_conf_search = dc.corp.example Initial attempts to clear the domain at runtime failed:\n# This did NOT work resolvectl domain \u0026#34;\u0026#34; systemctl restart systemd-resolved The search domain came back every time. Restarting systemd-resolved re-read the link-level config and reapplied the domain.\nThe Diagnosis Inspecting the resolver state revealed the source:\nresolvectl domain Output:\nGlobal: Link 2 (eth0): dc.corp.example The domain was on Link 2 (eth0), not in the global scope. It was being injected at the interface level, probably via DHCP or systemd-networkd.\nThe fix was per-interface:\nresolvectl domain eth0 \u0026#34;\u0026#34; And crucially: do not restart systemd-resolved afterward. Restarting re-reads the link config and reapplies the domain.\nThe Production Run The runtime fix for Site C, scoped to exclude worker nodes:\nANSIBLE_NOCOLOR=1 \\ ANSIBLE_STDOUT_CALLBACK=default \\ ansible \u0026#39;site-c_prod_rke2:!site-c_prod_rke2_wrkr\u0026#39; \\ -i inventory/prod.yaml \\ -u operator -b -kK \\ -m shell -a \u0026#39; set -e echo \u0026#34;Before:\u0026#34; resolvectl domain grep \u0026#34;^search\u0026#34; /etc/resolv.conf || echo \u0026#34;no search line\u0026#34; resolvectl domain eth0 \u0026#34;\u0026#34; sleep 1 echo \u0026#34;After:\u0026#34; resolvectl domain grep \u0026#34;^search\u0026#34; /etc/resolv.conf || echo \u0026#34;no search line\u0026#34; \u0026#39; The critical detail: set -e and set -u are safe in Ansible\u0026rsquo;s /bin/sh, but set -o pipefail is not — /bin/sh on Ubuntu rejects it silently.\nThe Production Maintenance Window Site D was remediated during a fifteen-minute maintenance window. The script had a bug: it used set -o pipefail at the top, which worked fine in local Bash but failed when Ansible executed it remotely via /bin/sh.\nThe error was cryptic:\n/bin/sh: 2: set: Illegal option -o pipefail Fix: change set -euo pipefail to set -eu inside Ansible shell tasks. Keep set -euo pipefail only in the local script wrapper.\nAfter the fix, the remediation ran cleanly:\n== Drift remaining == inventory_host actual_hostname resolv_conf_search netplan_search resolv_conf_type Header only — no drift. All nodes across Site D now showed . / [].\nThe Subiquity YAML Trap During the Site-E audit, a precheck failed on all Longhorn nodes:\nsudo netplan generate Failed with a YAML error. The offending file:\nnetwork: ethernets: ens160: addresses: - 192.0.2.15/24 gateway4: 192.0.2.1 nameservers: addresses: - 198.51.100.2 - 198.51.100.3 search: [] version: 2 The search: [] line was misaligned — indented at the nameservers level but not nested under it. This is a known behavior of Subiquity (the Ubuntu Server installer): it sometimes writes search: with inconsistent indentation.\nThe fix:\ncp -a /etc/netplan/00-installer-config.yaml /etc/netplan/00-installer-config.yaml.bak.$(date +%s) python3 - \u0026lt;\u0026lt;PY from pathlib import Path p = Path(\u0026#34;/etc/netplan/00-installer-config.yaml\u0026#34;) text = p.read_text() text = text.replace(\u0026#34;\\nsearch:\u0026#34;, \u0026#34;\\n search:\u0026#34;) p.write_text(text) PY netplan generate The same indentation bug had appeared earlier on other sites — it is worth adding a YAML normalization step to any netplan automation that runs across Subiquity-provisioned nodes.\nThe Three-Layer Model The most useful mental model from this work is that DNS search domains on Ubuntu with systemd-resolved come from three independent layers:\nLayer Inspect With Source Netplan (static) netplan get, grep search: /etc/netplan/*.yaml Static config systemd-resolved (global) resolvectl domain (Global section) resolvectl domain \u0026quot;\u0026quot; systemd-resolved (per-link) resolvectl domain (Link sections) DHCP, systemd-networkd A fix that clears the wrong layer will appear to work but will not survive a reboot or service restart.\nKey Takeaways Audit before acting. Without the CSV audit, we would have assumed all sites had the same root cause. They did not.\nsearch . means no search domain. This is systemd-resolved behavior, not a bug. Do not try to \u0026ldquo;fix\u0026rdquo; it.\nDo not manually edit /etc/resolv.conf when it is a symlink. On modern Ubuntu, it points to /run/systemd/resolve/stub-resolv.conf. Changes will be overwritten.\nset -o pipefail breaks Ansible shell tasks. /bin/sh on Ubuntu does not support it. Use set -eu inside remote commands.\nSubiquity writes inconsistent YAML. If a netplan generate precheck fails after a fresh Ubuntu install, check the indentation of search:.\nStage configs, defer application. Running netplan generate validates the config without disrupting the runtime resolver. Apply during a maintenance window.\nCount patterns, not nodes. The awk command awk -F, 'NR\u0026gt;1 {print $3 \u0026quot;|\u0026quot; $4}' \u0026quot;$OUT\u0026quot; | sort | uniq -c | sort -nr quickly shows how many distinct resolver states exist in the fleet.\n","permalink":"https://trinidadmarroquin.com/posts/dns-search-domain-remediation/","section":"posts","summary":"RKE2 clusters across five sites had DNS search domains leaking into Kubernetes name resolution. Hostnames that should have resolved as-is were getting suffix expansion, and the behavior was inconsistent across data centers.\nThe fix required understanding which layer was injecting the search domain — and it was not always the same layer.\nThis post covers the audit, the diagnostics, the remediation script, and the edge cases that made it interesting.\n","tags":["dns","netplan","systemd-resolved","ansible","rke2","kubernetes","sre"],"title":"Removing DNS Search Domain Drift Across Multi-Site RKE2 Clusters"},{"categories":["field-notes"],"content":"Rollback is a deployment strategy that gets rehearsed less often than it should. A rollback plan that has never been tested is not a rollback plan.\nFor a runnable lab, see the rollback-deployment directory in the IaC repository. It uses a sentinel file to trigger Puppet-driven dpkg rollback.\nThe Sentinel File Pattern A sentinel file marks a failure condition. When it exists, automation triggers a rollback. In a Puppet-based lab:\nexec { \u0026#39;rollback-to-v1\u0026#39;: command =\u0026gt; \u0026#39;dpkg --force-depends -i /opt/v1/c-app.deb\u0026#39;, onlyif =\u0026gt; \u0026#39;test -f /tmp/simulate_failure\u0026#39;, } The sentinel file is a teaching proxy. In production, the sentinel would be a health check failure, a metrics threshold breach, or a monitoring alert.\nConditional Execution Puppet\u0026rsquo;s onlyif and unless control when rollback resources execute:\nonlyif runs the command only if the condition is true. unless runs the command only if the condition is false. exec { \u0026#39;check-not-rolled-back\u0026#39;: command =\u0026gt; \u0026#39;echo already rolled back\u0026#39;, unless =\u0026gt; \u0026#39;dpkg -l c-app | grep -q v2\u0026#39;, } The rollback should be idempotent. Running it twice should not cause a second rollback attempt.\nPackage-Level Rollback dpkg --force-depends -i installs a .deb file and replaces the current version. This is fast and works at the package level, but it has significant limitations:\nno database migration rollback. no cache invalidation. no session drain. no downstream dependency coordination. no state rollback. Package rollback is appropriate for stateless services. Stateful services require schema versioning, data recovery plans, and coordinated service restarts.\nDependency Chaining exec { \u0026#39;cleanup-v2-artifacts\u0026#39;: command =\u0026gt; \u0026#39;rm -rf /opt/v2\u0026#39;, require =\u0026gt; Exec[\u0026#39;rollback-to-v1\u0026#39;], } The cleanup only runs after the rollback completes. This prevents orphaned artifacts from causing confusion during the next deployment.\nProduction Rollback Checklist Before relying on an automated rollback:\nCan the previous version be restored within the recovery SLO? Are database schema changes reversible? Is there a way to invalidate cached data from the failed version? Are downstream services aware of the rollback? Has the rollback been tested in a non-production environment? Is there a communication plan for the rollback event? Acceptance Criteria Rollback triggers on sentinel file or health check failure. Rollback is idempotent. Previous version is restored without manual intervention. Post-rollback cleanup removes artifacts from the failed version. Rollback duration is measured and documented. ","permalink":"https://trinidadmarroquin.com/field-notes/rollback-sentinel-package-management/","section":"field-notes","summary":"Rollback is a deployment strategy that gets rehearsed less often than it should. A rollback plan that has never been tested is not a rollback plan.\nFor a runnable lab, see the rollback-deployment directory in the IaC repository. It uses a sentinel file to trigger Puppet-driven dpkg rollback.\nThe Sentinel File Pattern A sentinel file marks a failure condition. When it exists, automation triggers a rollback. In a Puppet-based lab:\nexec { \u0026#39;rollback-to-v1\u0026#39;: command =\u0026gt; \u0026#39;dpkg --force-depends -i /opt/v1/c-app.deb\u0026#39;, onlyif =\u0026gt; \u0026#39;test -f /tmp/simulate_failure\u0026#39;, } The sentinel file is a teaching proxy. In production, the sentinel would be a health check failure, a metrics threshold breach, or a monitoring alert.\n","tags":["deployment","cicd","puppet","automation"],"title":"Rollback Strategies With Sentinel Files And Package Management"},{"categories":["field-notes"],"content":"Local infrastructure labs often start with hardcoded passwords, localhost endpoints, and convenience tokens. That is normal for learning, but dangerous when the lab pattern becomes a production pattern without review.\nThe useful distinction is not \u0026ldquo;lab bad, production good.\u0026rdquo; The useful distinction is knowing which shortcuts are temporary and what must change before the pattern is reused.\nCommon Lab Shortcuts Terraform-managed Docker labs often include:\nGrafana admin credentials in container environment variables. Concourse local users such as admin:admin. Vault dev server tokens in shell environment files. database passwords pulled into Terraform state. generated private keys written to local files. privileged containers for CI workers or system exporters. localhost endpoints that assume a single operator workstation. Each shortcut may be acceptable in a disposable lab. None should cross into shared infrastructure by accident.\nMain Risk Terraform state can retain sensitive values even when the original source is Vault or an environment variable.\nIf Terraform reads a secret and uses it in a resource argument, assume the value may be present in state unless the provider and resource explicitly avoid storing it.\nCheck state handling before treating the workflow as safe:\nterraform state list terraform state show \u0026lt;resource-address\u0026gt; Do not paste state output into tickets, chat, or public examples without reviewing it first.\nBetter Lab Pattern For local examples, prefer obvious placeholders:\nexport TF_VAR_grafana_admin_password=\u0026#39;change-me-for-local-lab\u0026#39; export TF_VAR_vault_addr=\u0026#39;http://127.0.0.1:8200\u0026#39; Keep real local values in ignored files:\n*.auto.tfvars terraform.tfvars .env *.env keys/ Commit example files only:\nterraform.tfvars.example vault.env.example grafana.env.example The example should teach the required inputs without carrying working secrets.\nVault Integration Checks When Vault is part of the lab, verify:\nthe workflow does not require committing a Vault token. the token is short-lived or clearly marked as a dev token. secret paths are documented. policies are narrower than full administrative access. generated files containing keys are ignored. cleanup steps revoke or rotate credentials when the lab is done. For shared environments, prefer identity-based authentication over passing a reusable root-like token into Terraform or a container.\nContainer Credential Checks For Grafana, Concourse, Prometheus, or similar containers, review:\nwhich ports are exposed to the host. whether default credentials are still active. whether admin credentials are rotated after first start. whether service-account tokens are stored in state. whether the container needs privileged mode. whether mounted host paths expose Docker, system files, or secrets. A monitoring lab that mounts /var/run/docker.sock or /var/lib/docker may be useful, but it should be treated as privileged access to the host.\nPromotion Checklist Before reusing a local lab pattern in a team environment:\nreplace hardcoded credentials with an approved secret source. review Terraform state for sensitive values. add .gitignore rules for state, plans, keys, and env files. remove default admin users or force password rotation. restrict exposed ports and network reachability. document cleanup and credential revocation. separate lab names from production names. Operating Rule A lab should make shortcuts visible.\nIf a secret is hardcoded for teaching, name it like a placeholder. If a key is generated locally, ignore it. If Terraform touches the value, assume state must be protected.\nThe goal is to preserve the speed of a lab without accidentally teaching unsafe production defaults.\n","permalink":"https://trinidadmarroquin.com/field-notes/terraform-lab-secret-handling/","section":"field-notes","summary":"Local infrastructure labs often start with hardcoded passwords, localhost endpoints, and convenience tokens. That is normal for learning, but dangerous when the lab pattern becomes a production pattern without review.\nThe useful distinction is not \u0026ldquo;lab bad, production good.\u0026rdquo; The useful distinction is knowing which shortcuts are temporary and what must change before the pattern is reused.\nCommon Lab Shortcuts Terraform-managed Docker labs often include:\nGrafana admin credentials in container environment variables. Concourse local users such as admin:admin. Vault dev server tokens in shell environment files. database passwords pulled into Terraform state. generated private keys written to local files. privileged containers for CI workers or system exporters. localhost endpoints that assume a single operator workstation. Each shortcut may be acceptable in a disposable lab. None should cross into shared infrastructure by accident.\n","tags":["terraform","vault","secrets","cicd","observability"],"title":"Secret Handling In Terraform Managed Labs"},{"categories":["field-notes"],"content":"SRE Is Not A Role, It Is A Decision Framework The tools matter less than the judgment about when to use them. SRE is the discipline of converting operational data into decisions about system reliability.\nThe Essential Questions Every operational decision reduces to three questions:\nWhat is the acceptable failure rate? (SLO target) How much failure budget is left? (Error budget tracking) What are we doing about it? (Error budget policy) If the team cannot answer these three questions for a service, the SRE practice has not arrived yet.\nError Budget Policy An error budget without a consumption policy is a metric, not a management tool. The policy answers:\nWho decides when the budget is consumed? What happens when it is consumed? (Freeze features? Require second approval? Auto-rollback?) Who decides the budget is replenished? What counts as budget consumption? (All 5xx responses? Only customer-facing errors? Timeouts only?) Toil Threshold If a task requires human intervention, is repetitive, can be automated, and has no enduring value, it is toil. The SRE mindset says: track time spent on toil, set a target (under 50% of time), and when toil exceeds the target, pause feature work to reduce it.\nToil that is not tracked will expand to fill available time.\nIncident Response Priorities 1. Mitigate (stop the bleeding) 2. Communicate (status, ETA, affected scope) 3. Investigate (find root cause) 4. Document (timeline, actions, lessons) 5. Follow up (action items, verification) Reverse this order during an incident and the mitigation is delayed.\nRunbook Utility A runbook is valuable if it answers:\nWhat symptom triggers this runbook? What is the expected outcome of following it? How do I verify the outcome? What do I do if verification fails? If the runbook requires interpretation on any of these points during an incident, it needs revision.\nMonitoring Philosophy Monitor what breaks, not what is easy to monitor. Common operational data (CPU, memory, disk) is table stakes. The metrics that matter are the ones that predict failure before it happens: certificate expiry, queue depth, error rate trends, deployment frequency, restart count.\nThe Automation Test Before automating a task, answer:\nHow often is this task performed? What is the cost of the automation failing? Can the automation verify its own output? What is the rollback plan for the automation? If the answer to any of these is unclear, the task is not ready to automate.\nOperational Review Cadence Activity Cadence Purpose error budget check Weekly Confirm remaining budget, adjust risk tolerance toil assessment Monthly Track time spent, identify automation candidates incident review Per incident Extract action items, update runbooks SLO review Quarterly Are the right things being measured? disaster recovery drill Quarterly Does the procedure actually work? capacity review Per growth signal Will the system survive the next spike? The SRE Trap The most common SRE failure is building sophisticated observability and automation while neglecting the decision framework. A team with perfect dashboards, comprehensive alerting, and full automation but no SLO targets or error budget policy has invested in tooling without building the judgment to use it.\nThe tooling is valuable. The judgment is essential.\n","permalink":"https://trinidadmarroquin.com/field-notes/sre-mindset/","section":"field-notes","summary":"SRE Is Not A Role, It Is A Decision Framework The tools matter less than the judgment about when to use them. SRE is the discipline of converting operational data into decisions about system reliability.\nThe Essential Questions Every operational decision reduces to three questions:\nWhat is the acceptable failure rate? (SLO target) How much failure budget is left? (Error budget tracking) What are we doing about it? (Error budget policy) If the team cannot answer these three questions for a service, the SRE practice has not arrived yet.\n","tags":["sre","operations","incidents","platform"],"title":"SRE Mindset: Operational Judgment Over Tooling"},{"categories":["field-notes"],"content":"Remote state has a chicken-and-egg problem: Terraform needs somewhere to store state, but the storage account and container may also be managed by Terraform.\nThe clean approach is to treat backend bootstrapping as a short, explicit phase. Create the state storage resources first, migrate state intentionally, then use the remote backend for the rest of the infrastructure.\nTarget Shape For Azure Blob Storage, the backend needs:\na resource group. a storage account. a private blob container. a stable state key per root module or environment. Example backend shape:\nterraform { backend \u0026#34;azurerm\u0026#34; { resource_group_name = \u0026#34;platform-state-rg\u0026#34; storage_account_name = \u0026#34;platformstateacct\u0026#34; container_name = \u0026#34;terraform-state\u0026#34; key = \u0026#34;prod/network/terraform.tfstate\u0026#34; } } The exact names should match the environment and ownership model. Avoid reusing one vague state key for unrelated infrastructure.\nBootstrap Sequence Start with the backend block commented out or absent:\nterraform init terraform validate terraform plan terraform apply After the resource group, storage account, and container exist, add the backend block and migrate:\nterraform init -migrate-state Verify that Terraform now reads and writes remote state:\nterraform state list terraform plan The plan should be clean unless other intentional changes exist.\nState Key Rules Use state keys that express blast radius:\nprod/network/terraform.tfstate prod/aks/terraform.tfstate nonprod/network/terraform.tfstate shared/observability/terraform.tfstate Avoid:\nterraform.tfstate main.tfstate test.tfstate Those names hide ownership and make recovery harder when multiple roots exist.\nAccess And Safety Checks Before using the backend for shared work, confirm:\nthe container is private. only the automation and operators that need state access can read it. state access is logged through Azure activity logs or storage diagnostics where required. the storage account is protected by the expected network and identity controls. state files are excluded from Git. At minimum, keep this out of source control:\n**/*.tfstate **/*.tfstate.* **/*.tfplan Recovery Notes If backend migration fails, do not delete local state casually. First identify where the current authoritative state lives:\nterraform state list Then inspect the backend configuration and rerun initialization only after the storage account, container, and key are correct:\nterraform init -reconfigure Use -reconfigure when changing backend settings without migrating existing state. Use -migrate-state when intentionally moving state from one backend to another.\nOperating Rule Backend bootstrapping is infrastructure work, not a throwaway setup step.\nWrite down:\nwho owns the state storage. which root module owns each key. how access is granted. how state is recovered. how old local state files are removed after migration. The goal is not just remote state. The goal is state that an operator can find, protect, and recover under pressure.\n","permalink":"https://trinidadmarroquin.com/field-notes/terraform-azure-backend-bootstrap/","section":"field-notes","summary":"Remote state has a chicken-and-egg problem: Terraform needs somewhere to store state, but the storage account and container may also be managed by Terraform.\nThe clean approach is to treat backend bootstrapping as a short, explicit phase. Create the state storage resources first, migrate state intentionally, then use the remote backend for the rest of the infrastructure.\nTarget Shape For Azure Blob Storage, the backend needs:\na resource group. a storage account. a private blob container. a stable state key per root module or environment. Example backend shape:\n","tags":["terraform","azure","state","cloud"],"title":"Terraform Azure Backend Bootstrap"},{"categories":["projects"],"content":"Terraform modules should represent operational concepts, not just folders around resources.\nHashiCorp guidance recommends moderation: modules are useful when they raise the abstraction level, but thin wrappers around single resources often add complexity without improving ownership.\nBoundary Principles Good module boundaries answer:\nWho owns this infrastructure after apply? What lifecycle does it follow? What inputs must vary by environment? What outputs does another layer consume? What failure modes does the module hide or expose? Avoid modules that combine unrelated ownership domains. A Kubernetes node module, network module, and storage module may interact, but they are not necessarily owned by the same team or changed on the same cadence.\nRoot Modules Root modules should map to deployable units:\nenvironments/prod/network environments/prod/compute environments/nonprod/kubernetes This makes plan review safer because the blast radius is visible from the working directory.\nReusable Modules Reusable modules should have:\nclear inputs. stable outputs. examples. versioning. provider requirements. assumptions documented near the code. Keep module trees relatively flat. Deep nesting makes ownership and state movement harder to understand.\nAcceptance Criteria Each module has a named operational owner. Root modules match environment and blast-radius boundaries. Reusable modules expose architecture-level concepts. Module outputs are intentional contracts. Refactors use moved blocks or explicit state moves. References Terraform documentation: Creating Modules. Terraform documentation: Best Practices for Composing Modules. Terraform documentation: Refactoring Modules. ","permalink":"https://trinidadmarroquin.com/projects/terraform-infrastructure-modules/operational-module-boundaries/","section":"projects","summary":"Terraform modules should represent operational concepts, not just folders around resources.\nHashiCorp guidance recommends moderation: modules are useful when they raise the abstraction level, but thin wrappers around single resources often add complexity without improving ownership.\nBoundary Principles Good module boundaries answer:\nWho owns this infrastructure after apply? What lifecycle does it follow? What inputs must vary by environment? What outputs does another layer consume? What failure modes does the module hide or expose? Avoid modules that combine unrelated ownership domains. A Kubernetes node module, network module, and storage module may interact, but they are not necessarily owned by the same team or changed on the same cadence.\n","tags":["terraform","modules","operations"],"title":"Terraform Operational Module Boundaries"},{"categories":["projects"],"content":"Terraform plan review is where infrastructure intent becomes operational risk.\nA good review does not ask only whether the syntax is valid. It asks whether the planned changes match the expected blast radius.\nReview Inputs Every plan review should include:\nchanged files. selected workspace or root module. variable files used. provider versions. plan output. expected changes. rollback or recovery notes. Do not review a plan if the command that generated it is unclear.\nState Safety Terraform state maps configuration addresses to real infrastructure. Treat state as production data.\nExpectations:\nuse remote state with locking for shared work. do not store state in Git. do not manually edit state files. use terraform state commands or moved blocks for refactors. back up state before risky migrations. Plan Signals Review carefully when the plan includes:\nresource replacement. deletion of stateful resources. changes to network paths. changes to IAM or secrets. provider default changes. drift corrections the operator did not expect. Automation Gates Useful gates:\nterraform fmt -check terraform init -backend=false terraform validate terraform plan -out=tfplan terraform show -json tfplan Policy checks can help, but human review is still needed for ownership, timing, and operational risk.\nAcceptance Criteria The reviewer can explain every create, update, delete, and replacement. State movement is explicit. Secrets are not exposed in plan artifacts. The apply target and variable inputs are known. Rollback or recovery is realistic. References Terraform documentation: State. Terraform documentation: State Locking. Terraform documentation: Plan and Apply. ","permalink":"https://trinidadmarroquin.com/projects/terraform-infrastructure-modules/plan-review-practices/","section":"projects","summary":"Terraform plan review is where infrastructure intent becomes operational risk.\nA good review does not ask only whether the syntax is valid. It asks whether the planned changes match the expected blast radius.\nReview Inputs Every plan review should include:\nchanged files. selected workspace or root module. variable files used. provider versions. plan output. expected changes. rollback or recovery notes. Do not review a plan if the command that generated it is unclear.\n","tags":["terraform","review","infrastructure","state"],"title":"Terraform Plan Review Practices"},{"categories":["notes"],"content":"The DevOps Dirty Dozen covered anti-patterns in DevOps culture. SRE has its own set of traps — practices that look like reliability work but do not improve reliability.\n1. Alert Frequency As A Paging Criterion \u0026ldquo;If it pages, it matters\u0026rdquo; sounds correct. The reality is that teams habituate to frequent pages within weeks. An alert that fires every night at 3 AM for a non-critical condition trains operators to silence notifications, miss the genuine page, and burn out.\nThe correct criterion is not frequency. It is whether the condition requires a human to act within minutes. Everything else is a ticket, a dashboard tile, or a log entry.\n2. Dashboards Without Decisions A dashboard with 50 charts and no annotation of what the operator should look for is a screensaver. Every dashboard should answer a question. If the question is not defined before the charts are added, the dashboard will accumulate visual noise until it is useless.\nThe pattern: title each dashboard as a question (\u0026ldquo;Is the API healthy?\u0026rdquo;) and add only the metrics that answer it. Auxiliary data goes in a drill-down link.\n3. SLI Collection Without SLO Targets Collecting latency and error rate is not SRE. SLI data without SLO targets tells operators what is happening but not whether it is acceptable. The SLO is the decision rule that converts a metric into an operational judgment.\nWithout SLO targets, every latency spike is a potential incident and no one can agree on whether the system is healthy.\n4. Error Budgets Nobody Uses Defining an error budget and then ignoring it when making release decisions is a paperwork exercise. The error budget only matters if it changes behavior.\nWhen the budget is exhausted, the team stops shipping features and dedicates capacity to reliability work. If that never happens, the error budget is decoration.\n5. Incident Reviews Without Action Items An incident review that produces a narrative but no action items is a book club. The output of a review is a set of validated actions that reduce the probability or impact of a similar incident.\nThe pattern: each action item must have an owner, a deadline, and a verification step. If it cannot be verified, it is not an action item.\n6. Toil Automation That Creates More Toil Automating a manual task by building a script that requires constant maintenance, manual input files, and debugging mid-run is not a reduction in toil. It is replacing one form of toil with another.\nThe test: if the automation needs an operator to intervene more than once per month, the task is not automated. It is partially delegated to a fragile script.\n7. Pager Duty Without OODA Loops Paging an operator without giving them the information to make a decision is ineffective. The alert should include the symptom, the affected component, a link to the relevant dashboard, and the runbook.\nAn alert that says \u0026ldquo;CPU is high on server X\u0026rdquo; sends the operator into a discovery loop that should have been completed before the alert fired.\n8. Platform Teams Building What Operators Did Not Ask For A platform team that builds abstractions without consulting the operators who will use them produces tools that do not match the operational model. The result is shadow infrastructure and workarounds.\nThe most reliable platform features are the ones operators contributed to the design of.\n9. Monitoring The Symptoms, Not The Causes Monitoring node CPU and memory is table stakes. Monitoring the causes of node CPU and memory spikes — deployment patterns, traffic shifts, certificate expiry — is where the operational leverage is.\nCause-level alerts are actionable. Symptom-level alerts are informational.\n10. Perfect System Design Over Practical Incident Response Teams that spend months designing the perfect system architecture while neglecting incident response practice will fail faster during an incident than a team with imperfect architecture and well-rehearsed response procedures.\nRecovery speed is an architectural property, but it is also a practiced skill.\n11. Reliability As The SRE Team\u0026rsquo;s Problem If only the SRE team cares about reliability, the SRE team will be the bottleneck for every change and the scapegoat for every incident. Reliability must be a shared concern embedded in how development, QA, and operations teams evaluate their work.\nThe SRE team\u0026rsquo;s job is to enable and verify reliability practices, not to be the sole owner of them.\n12. Treating The Runbook As The Solution A runbook is documentation of a known failure mode. It is not the solution to the failure mode. If the same runbook is executed more than a few times, the condition should be automated.\nThe pattern: every time a runbook is used, evaluate whether the steps can be automated. If they cannot be automated, evaluate whether the system can be changed to eliminate the failure mode.\n","permalink":"https://trinidadmarroquin.com/posts/sre-dirty-dozen/","section":"posts","summary":"The DevOps Dirty Dozen covered anti-patterns in DevOps culture. SRE has its own set of traps — practices that look like reliability work but do not improve reliability.\n1. Alert Frequency As A Paging Criterion \u0026ldquo;If it pages, it matters\u0026rdquo; sounds correct. The reality is that teams habituate to frequent pages within weeks. An alert that fires every night at 3 AM for a non-critical condition trains operators to silence notifications, miss the genuine page, and burn out.\n","tags":["sre","anti-patterns","incidents","observability","platform"],"title":"The SRE Dirty Dozen: Common Anti-Patterns In Site Reliability Engineering"},{"categories":["notes"],"content":"Storage incidents in Kubernetes are rarely just storage incidents.\nIn one troubleshooting session, a monitoring namespace looked broken in several different ways at once. Alertmanager was stuck pending. Prometheus was stuck in init. Node exporter pods were being rejected by Pod Security. The system upgrade controller had a long tail of evicted pods. Several control-plane nodes were under disk pressure.\nThe useful move was not to fix the first scary event. It was to separate the failures by ownership boundary: scheduler, storage class, CSI driver, VM configuration, and GitOps drift.\nStart With The Events The cluster events showed multiple unrelated-looking warnings:\nFailedCreate daemonset/kube-prometheus-stack-prometheus-node-exporter EvictionThresholdMet node/cluster-a-cp-01 FailedScheduling pod/system-upgrade-controller-... The upgrade controller was blocked because it had required affinity for control-plane nodes, while all eligible control-plane nodes had DiskPressure taints:\n0/6 nodes are available: 3 node(s) didn\u0026#39;t match Pod\u0026#39;s node affinity/selector, 3 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. That mattered, but it was not the same problem as monitoring storage. It was a cluster health issue that could obscure the storage issue if treated as the root cause for everything.\nFor monitoring, the important evidence was in the namespace itself:\nkubectl get pods -n vcobserve -o wide The relevant state was:\nalertmanager-kube-prometheus-stack-alertmanager-0 0/2 Pending prometheus-kube-prometheus-stack-prometheus-0 0/2 Init:0/1 The PVCs told the real story:\nkubectl get pvc -n vcobserve -o wide NAME STATUS STORAGECLASS alertmanager-... Pending vsphere-storage-class prometheus-... Bound hpe-alletra-ssd Two monitoring components were failing for different storage reasons:\nAlertmanager referenced an old or missing vSphere StorageClass name. Prometheus still had an older HPE CSI-backed PVC. The current target storage path was vSphere CSI through vsphere-csi-sc. Check Storage Classes Before Recreating Workloads The cluster had more than one default-looking storage path over its lifetime:\nkubectl get storageclass NAME PROVISIONER hpe-alletra-ssd (default) csi.hpe.com vsphere-csi-sc (default) csi.vsphere.vmware.com That is a red flag. Even if only one default is effective at a time, workloads can carry explicit storageClassName values from older Helm values, Prometheus Operator CRs, or historical PVCs.\nThe storage question became concrete:\nWhich PVCs and PVs still depend on the old HPE path? Which monitoring CRs will recreate PVCs with the wrong class if ArgoCD syncs? Which nodes can actually attach and mount vSphere CSI volumes? For stale CSI references, the cluster-wide checks were simple:\nkubectl get pvc -A -o json | jq -r \u0026#39;.items[] | select(.spec.storageClassName == \u0026#34;hpe-alletra-ssd\u0026#34;) | [.metadata.namespace,.metadata.name,.status.phase,.spec.volumeName] | @tsv\u0026#39; kubectl get pv -o json | jq -r \u0026#39;.items[] | select(.spec.storageClassName == \u0026#34;hpe-alletra-ssd\u0026#34;) | [.metadata.name,.status.phase,(.spec.claimRef.namespace // \u0026#34;\u0026#34;),(.spec.claimRef.name // \u0026#34;\u0026#34;)] | @tsv\u0026#39; After remediation, both commands returned no rows.\nCSI Pods Running Does Not Prove Mounts Work Both CSI stacks had running pods:\nkubectl get pods -n hpe-storage -o wide kubectl get pods -n vmware-system-csi -o wide That confirmed driver availability, not workload success.\nThe old Prometheus PV showed the historical dependency:\npv.kubernetes.io/provisioned-by: csi.hpe.com external-attacher/csi-hpe-com The desired end state was not \u0026ldquo;CSI pods are running.\u0026rdquo; The desired end state was more specific:\nPrometheus and Alertmanager PVCs are bound to vsphere-csi-sc. Pods can schedule without a permanent node pin. VolumeAttachment objects show attached volumes on the selected worker. Containers move past init and report ready. The vSphere-Specific Trap: Disk UUID Visibility After moving the monitoring PVC intent to vsphere-csi-sc, provisioning worked, but pod mounts still failed on workers that were missing the vSphere disk UUID setting.\nFor vSphere CSI, worker VMs need disk UUID visibility enabled:\ndisk.EnableUUID = \u0026#34;TRUE\u0026#34; Without that, a volume can appear provisioned and even reach the attach path, while kubelet and the CSI node plugin cannot reliably discover the attached virtual disk inside the guest.\nThe remediation pattern was deliberately operational:\nkubectl cordon \u0026lt;worker\u0026gt; kubectl drain \u0026lt;worker\u0026gt; --ignore-daemonsets --delete-emptydir-data Then, outside Kubernetes:\nPower off VM Set disk.EnableUUID = \u0026#34;TRUE\u0026#34; Power on VM Then return the node:\nkubectl uncordon \u0026lt;worker\u0026gt; kubectl get nodes This was repeated one worker at a time. The point was not just to set a VM flag. The point was to preserve workload availability while changing a prerequisite underneath the CSI node path.\nTemporary Pins Are Fine, But Remove Them During the incident, pinning Prometheus and Alertmanager to a known-good worker helped prove the storage path without constantly fighting scheduler movement.\nThat pin had to be temporary.\nOnce the workers were fixed, the Prometheus Operator CRs were patched to remove the node selector:\nkubectl patch prometheus -n vcobserve kube-prometheus-stack-prometheus --type=merge -p \u0026#39;{\u0026#34;spec\u0026#34;:{\u0026#34;nodeSelector\u0026#34;:null}}\u0026#39; kubectl patch alertmanager -n vcobserve kube-prometheus-stack-alertmanager --type=merge -p \u0026#39;{\u0026#34;spec\u0026#34;:{\u0026#34;nodeSelector\u0026#34;:null}}\u0026#39; Then verify both the CRs and rendered StatefulSets:\nkubectl get prometheus -n vcobserve kube-prometheus-stack-prometheus -o jsonpath=\u0026#39;{.spec.nodeSelector}{\u0026#34;\\n\u0026#34;}\u0026#39; kubectl get alertmanager -n vcobserve kube-prometheus-stack-alertmanager -o jsonpath=\u0026#39;{.spec.nodeSelector}{\u0026#34;\\n\u0026#34;}\u0026#39; kubectl get statefulset -n vcobserve prometheus-kube-prometheus-stack-prometheus -o jsonpath=\u0026#39;{.spec.template.spec.nodeSelector}{\u0026#34;\\n\u0026#34;}\u0026#39; kubectl get statefulset -n vcobserve alertmanager-kube-prometheus-stack-alertmanager -o jsonpath=\u0026#39;{.spec.template.spec.nodeSelector}{\u0026#34;\\n\u0026#34;}\u0026#39; Empty output was the expected result.\nProve Mobility, Not Just Recovery The strongest validation was that monitoring could move to another worker and keep its vSphere CSI volumes healthy.\nAfter removing the temporary pin, the operator rolled the StatefulSet pods and they scheduled on another worker:\nalertmanager-kube-prometheus-stack-alertmanager-0 2/2 Running cluster-a-worker-03 prometheus-kube-prometheus-stack-prometheus-0 2/2 Running cluster-a-worker-03 The PVCs were bound to the intended StorageClass:\nNAME STATUS STORAGECLASS alertmanager-... Bound vsphere-csi-sc prometheus-... Bound vsphere-csi-sc And the VolumeAttachment objects moved with the pods:\nATTACHER PV NODE ATTACHED csi.vsphere.vmware.com pvc-15299db7-4fca-47c4-92f2-5fdbba02a43d cluster-a-worker-03 true csi.vsphere.vmware.com pvc-c92e2898-b016-4a2a-a2b1-591c3b423948 cluster-a-worker-03 true Finally, the monitoring CRs reported healthy reconciliation:\nprometheus.monitoring.coreos.com/... READY 1 RECONCILED True AVAILABLE True alertmanager.monitoring.coreos.com/... READY 1 RECONCILED True AVAILABLE True That proved more than recovery. It proved the storage class, CSI controller, CSI node plugin, vSphere VM settings, scheduler placement, and Prometheus Operator reconciliation all agreed.\nGitOps Follow-Up Live fixes are not finished until GitOps will preserve them.\nThe follow-up was to update the ArgoCD-managed kube-prometheus-stack values so the next sync would not reintroduce the incident:\nSet Prometheus storage explicitly to vsphere-csi-sc. Set Alertmanager storage explicitly to vsphere-csi-sc. Remove references to hpe-alletra-ssd and the old vsphere-storage-class name. Do not permanently pin monitoring pods to one worker. Document disk.EnableUUID = \u0026quot;TRUE\u0026quot; as a worker VM requirement for vSphere CSI. Validate rendered Prometheus and Alertmanager CRs before sync. The operational fix was Kubernetes work. The durable fix was configuration ownership.\nLessons The incident had several overlapping symptoms, but the useful model was simple:\nEvents show pressure points, not always root cause. PVC and PV metadata reveal historical storage dependencies. CSI controller health does not prove guest-level volume discovery. vSphere CSI depends on VM configuration, not just Kubernetes objects. Temporary scheduling pins are diagnostic tools, not final architecture. Recovery should prove workload mobility, not only that one pod is running now. GitOps must be updated after live remediation or the cluster will drift back. The key storage lesson: when migrating Kubernetes workloads to vSphere CSI, do not stop at \u0026ldquo;PVC is bound.\u0026rdquo; Follow the volume all the way through provisioning, attachment, guest discovery, mount, pod readiness, rescheduling, and GitOps reconciliation.\n","permalink":"https://trinidadmarroquin.com/posts/kubernetes-vsphere-csi-troubleshooting/","section":"posts","summary":"Storage incidents in Kubernetes are rarely just storage incidents.\nIn one troubleshooting session, a monitoring namespace looked broken in several different ways at once. Alertmanager was stuck pending. Prometheus was stuck in init. Node exporter pods were being rejected by Pod Security. The system upgrade controller had a long tail of evicted pods. Several control-plane nodes were under disk pressure.\nThe useful move was not to fix the first scary event. It was to separate the failures by ownership boundary: scheduler, storage class, CSI driver, VM configuration, and GitOps drift.\n","tags":["kubernetes","vsphere","vmware","storage","troubleshooting","sre"],"title":"Troubleshooting Kubernetes Storage Migration To vSphere CSI"},{"categories":["projects"],"content":"Vault deployment work should be treated as security infrastructure, not just another stateful service.\nThe runbooks need to cover normal operation and the uncomfortable moments: initialization, sealing, unsealing, leader changes, backup, and recovery.\nDeployment Baseline Document:\nstorage backend. HA topology. TLS certificates. seal mechanism. audit devices. auth methods. backup and restore path. monitoring and alerting expectations. Vault should not run in production without audit logging and a tested recovery path.\nInitialization And Unseal Initialization creates the root token and unseal or recovery material. That ceremony needs named participants, secure storage, and evidence that the root token was revoked or locked away according to policy.\nUnseal runbooks should include:\nwho can participate. where recovery material is stored. how quorum is reached. how the active node is identified. how success is verified. Operational Checks Useful checks:\nvault status vault operator raft list-peers vault audit list vault auth list vault secrets list Acceptance Criteria HA is documented and tested. audit logging is enabled. unseal or recovery process has named owners. root token handling is documented. backups are encrypted and restore-tested. References Vault documentation: Seal/Unseal. Vault documentation: High Availability. Vault documentation: Production Hardening. ","permalink":"https://trinidadmarroquin.com/projects/secrets-management-with-vault/deployment-and-unseal-runbooks/","section":"projects","summary":"Vault deployment work should be treated as security infrastructure, not just another stateful service.\nThe runbooks need to cover normal operation and the uncomfortable moments: initialization, sealing, unsealing, leader changes, backup, and recovery.\nDeployment Baseline Document:\nstorage backend. HA topology. TLS certificates. seal mechanism. audit devices. auth methods. backup and restore path. monitoring and alerting expectations. Vault should not run in production without audit logging and a tested recovery path.\n","tags":["vault","secrets","operations"],"title":"Vault Deployment And Unseal Runbooks"},{"categories":["projects"],"content":"Vault policies are path-based and deny by default. That makes policy design powerful, but it also makes messy path design painful.\nStart by organizing secrets engines and paths around ownership and consumption patterns.\nPolicy Design Policies should be small, named clearly, and mapped to roles or groups.\nGood policy names describe intent:\nteam-a-kv-read platform-transit-admin ci-prod-db-read Avoid broad wildcard policies unless the path is already tightly scoped.\nAuth Methods Use auth methods that match the consumer:\nLDAP or OIDC for humans. Kubernetes auth for pods. AppRole for controlled machine workflows. cloud auth methods for provider-native workloads. Map external identity groups to Vault policies rather than assigning policies one user at a time.\nSecrets Engines Organize mounts by lifecycle and ownership:\nkv/ for static secrets with clear owners. database/ for dynamic database credentials. pki/ for certificate issuance. transit/ for cryptographic operations. Do not mix unrelated teams or environments in the same path if policy boundaries will become confusing.\nReview Checklist Does the policy grant only needed capabilities? Is list safe, or would key names leak sensitive information? Are dynamic credentials preferred where practical? Are human and machine auth paths separated? Is there a revocation path? References Vault documentation: Policies. Vault documentation: Authentication. Vault documentation: Secrets Engines. ","permalink":"https://trinidadmarroquin.com/projects/secrets-management-with-vault/policy-auth-secrets-engines/","section":"projects","summary":"Vault policies are path-based and deny by default. That makes policy design powerful, but it also makes messy path design painful.\nStart by organizing secrets engines and paths around ownership and consumption patterns.\nPolicy Design Policies should be small, named clearly, and mapped to roles or groups.\nGood policy names describe intent:\nteam-a-kv-read platform-transit-admin ci-prod-db-read Avoid broad wildcard policies unless the path is already tightly scoped.\nAuth Methods Use auth methods that match the consumer:\n","tags":["vault","secrets","security"],"title":"Vault Policy Auth And Secrets Engines"},{"categories":["projects"],"content":"Vault operations are mostly about lifecycle control: who received a secret, how long it is valid, when it renews, and how quickly it can be revoked.\nTokens and leases are not implementation details. They are the operational handles for incident response.\nToken Practices Use short-lived and renewable tokens where possible. Avoid long-lived service tokens unless there is a documented reason.\nToken expectations:\nroot tokens are not used for normal administration. human tokens come from an auth method. automation tokens are scoped and rotated. orphan and periodic tokens are reviewed carefully. token accessors are treated as sensitive operational data. Lease Practices Dynamic secrets have leases. Consumers must renew or replace them before expiry.\nUseful commands:\nvault lease lookup \u0026lt;lease-id\u0026gt; vault lease renew \u0026lt;lease-id\u0026gt; vault lease revoke \u0026lt;lease-id\u0026gt; vault lease revoke -prefix \u0026lt;path-prefix\u0026gt; Prefix revocation is especially useful when a system or path is compromised.\nAudit And Recovery Audit logs should be enabled before production use and routed to protected storage.\nRecovery expectations:\nencrypted backups. restore testing. documented unseal or recovery-key process. revocation runbook for compromised paths. incident process for suspicious token activity. Acceptance Criteria root token is controlled and not used casually. dynamic secret TTLs match workload needs. audit logs are protected from application teams. revocation is tested. backups can be restored by named operators. References Vault documentation: Tokens. Vault documentation: Lease, Renew, and Revoke. Vault documentation: Audit Devices. ","permalink":"https://trinidadmarroquin.com/projects/secrets-management-with-vault/token-lease-audit-recovery/","section":"projects","summary":"Vault operations are mostly about lifecycle control: who received a secret, how long it is valid, when it renews, and how quickly it can be revoked.\nTokens and leases are not implementation details. They are the operational handles for incident response.\nToken Practices Use short-lived and renewable tokens where possible. Avoid long-lived service tokens unless there is a documented reason.\nToken expectations:\nroot tokens are not used for normal administration. human tokens come from an auth method. automation tokens are scoped and rotated. orphan and periodic tokens are reviewed carefully. token accessors are treated as sensitive operational data. Lease Practices Dynamic secrets have leases. Consumers must renew or replace them before expiry.\n","tags":["vault","secrets","recovery"],"title":"Vault Token Lease Audit And Recovery Practices"},{"categories":["projects"],"content":"Automation works best when vCenter exposes stable contracts.\nTerraform, Packer, Kubernetes, CSI drivers, and backup tools all depend on vCenter inventory behaving predictably.\nTerraform Contracts Terraform needs stable references for:\ndatacenters. clusters and resource pools. datastores and datastore clusters. folders. port groups. templates. tags and custom attributes. If these names change casually, infrastructure code becomes fragile.\nPacker Contracts Packer needs a reliable build path:\nbuild network. ISO or content library access. temporary VM folder. template destination. credentials with limited build permissions. cleanup behavior for failed builds. Template promotion should include validation before downstream Terraform consumes the image.\nKubernetes And CSI Contracts Kubernetes nodes rely on vCenter for VM placement, disks, networking, and CSI integration.\nDocument node requirements such as:\nVM hardware version. VMware tools health. disk.EnableUUID = \u0026quot;TRUE\u0026quot; for vSphere CSI. port group assignment. datastore policy. backup exclusions or inclusions. Backup Workflows Backup tooling needs clear policy:\nwhich VMs are protected. which disks are excluded. snapshot coordination expectations. restore test cadence. application-consistent versus crash-consistent behavior. References VMware vSphere documentation: vCenter Server and Host Management. Terraform vSphere provider documentation. HashiCorp Packer documentation. ","permalink":"https://trinidadmarroquin.com/projects/vcenter-platform-administration/automation-integration-points/","section":"projects","summary":"Automation works best when vCenter exposes stable contracts.\nTerraform, Packer, Kubernetes, CSI drivers, and backup tools all depend on vCenter inventory behaving predictably.\nTerraform Contracts Terraform needs stable references for:\ndatacenters. clusters and resource pools. datastores and datastore clusters. folders. port groups. templates. tags and custom attributes. If these names change casually, infrastructure code becomes fragile.\nPacker Contracts Packer needs a reliable build path:\nbuild network. ISO or content library access. temporary VM folder. template destination. credentials with limited build permissions. cleanup behavior for failed builds. Template promotion should include validation before downstream Terraform consumes the image.\n","tags":["vsphere","terraform","packer","kubernetes"],"title":"vCenter Automation Integration Points"},{"categories":["projects"],"content":"vCenter hygiene is the difference between a virtualization platform and a pile of VMs.\nThe goal is to keep inventory, networks, storage, permissions, and templates understandable enough that automation can safely depend on them.\nInventory Organization Folders, tags, and naming should expose ownership and lifecycle.\nRecommended metadata:\nenvironment. application or platform owner. lifecycle state. backup policy. automation owner. cost or capacity group. Use vSphere tags and custom attributes for metadata that operators need to search, report, or automate against.\nDatastores And Networks Datastore and port group names should be stable contracts for automation.\nReview regularly:\ndatastore free space and overcommitment. orphaned disks and snapshots. port groups no longer used. VLAN or distributed switch drift. datastore clusters and placement rules. Terraform and Packer depend on consistent names. Rename operations should be treated as platform changes.\nTemplate Hygiene Templates should be versioned, patched, and retired intentionally.\nBaseline expectations:\ncurrent OS patches. VMware tools installed and healthy. cloud-init or guest customization readiness documented. stale machine identity removed before sealing. bootstrap prerequisites installed. template version and build date visible. Permissions Use least privilege roles for automation accounts. Avoid giving broad administrator access to every tool that touches vCenter.\nSeparate roles for:\ntemplate build. VM provisioning. read-only inventory discovery. backup operations. break-glass administration. References VMware vSphere documentation: vCenter Server and Host Management. VMware vSphere documentation: Tags and Custom Attributes. VMware vSphere documentation: Organizing Your vSphere Inventory. ","permalink":"https://trinidadmarroquin.com/projects/vcenter-platform-administration/platform-hygiene/","section":"projects","summary":"vCenter hygiene is the difference between a virtualization platform and a pile of VMs.\nThe goal is to keep inventory, networks, storage, permissions, and templates understandable enough that automation can safely depend on them.\nInventory Organization Folders, tags, and naming should expose ownership and lifecycle.\nRecommended metadata:\nenvironment. application or platform owner. lifecycle state. backup policy. automation owner. cost or capacity group. Use vSphere tags and custom attributes for metadata that operators need to search, report, or automate against.\n","tags":["vsphere","vmware","vcenter"],"title":"vCenter Platform Hygiene"},{"categories":["projects"],"content":"VM lifecycle runbooks should make common operations repeatable without hiding the risks.\nThe same lifecycle pattern applies whether the VM is a Kubernetes node, platform appliance, or application server.\nLifecycle States Track each VM through clear states:\nrequested. provisioned. configured. in service. maintenance. retired. deleted. Each state should have an owner and exit criteria.\nStandard Runbooks Useful runbooks include:\nprovision a VM from template. resize CPU, memory, or disk. add or change network adapters. audit or remove unused virtual CD-ROM devices. snapshot before risky maintenance. restore from backup or snapshot. retire and delete a VM. investigate guest customization failure. Each runbook should list read-only verification commands first, then mutation steps.\nCapacity Review Capacity review should include:\ncluster CPU and memory headroom. datastore utilization and growth rate. snapshot age and size. VM sprawl and powered-off inventory. resource pool constraints. HA and maintenance mode headroom. Capacity is not just percent used. It is whether the platform can tolerate failure and maintenance.\nIncident Response During an incident, preserve evidence:\nvCenter task history. VM events. datastore alarms. host health. recent automation runs. guest logs when accessible. Avoid making multiple speculative changes at once. vCenter incidents often cross host, storage, network, and guest boundaries. For a specific virtual media cleanup pattern, see vSphere CD-ROM Host Device Cleanup With govc.\nReferences VMware vSphere documentation: vCenter Server and Host Management. VMware vSphere documentation: Working with vSphere Tasks. VMware vSphere documentation: Troubleshooting Overview. ","permalink":"https://trinidadmarroquin.com/projects/vcenter-platform-administration/vm-lifecycle-runbooks/","section":"projects","summary":"VM lifecycle runbooks should make common operations repeatable without hiding the risks.\nThe same lifecycle pattern applies whether the VM is a Kubernetes node, platform appliance, or application server.\nLifecycle States Track each VM through clear states:\nrequested. provisioned. configured. in service. maintenance. retired. deleted. Each state should have an owner and exit criteria.\nStandard Runbooks Useful runbooks include:\nprovision a VM from template. resize CPU, memory, or disk. add or change network adapters. audit or remove unused virtual CD-ROM devices. snapshot before risky maintenance. restore from backup or snapshot. retire and delete a VM. investigate guest customization failure. Each runbook should list read-only verification commands first, then mutation steps.\n","tags":["vsphere","vmware","operations"],"title":"VM Lifecycle Runbooks"},{"categories":["field-notes"],"content":"Use this checklist when a Kubernetes workload has a PVC that should use vSphere CSI, but the pod is pending, stuck in init, or failing to mount the volume. For resize-specific CNS task triage, see vSphere CSI CNS ExtendVolume Triage. If GitOps keeps restarting CSI controllers during a storage freeze, see GitOps-Owned vSphere CSI Maintenance Pauses.\nThe goal is to follow the volume from intent to pod readiness:\nStorageClass -\u0026gt; PVC -\u0026gt; PV -\u0026gt; VolumeAttachment -\u0026gt; worker VM -\u0026gt; kubelet mount -\u0026gt; pod ready Check The Workload State Start with pods and PVCs in the affected namespace:\nkubectl get pods -n \u0026lt;namespace\u0026gt; -o wide kubectl get pvc -n \u0026lt;namespace\u0026gt; -o wide Look for:\npods stuck in Pending, ContainerCreating, or Init:*. PVCs stuck in Pending. PVCs bound to an unexpected STORAGECLASS. pods pinned to one worker because of a temporary nodeSelector. Describe the blocked pod and PVC:\nkubectl describe pod -n \u0026lt;namespace\u0026gt; \u0026lt;pod-name\u0026gt; kubectl describe pvc -n \u0026lt;namespace\u0026gt; \u0026lt;pvc-name\u0026gt; Useful event reasons include:\nFailedScheduling. FailedAttachVolume. FailedMount. ProvisioningFailed. Confirm The StorageClass List available StorageClasses:\nkubectl get storageclass For vSphere CSI, the provisioner should be:\ncsi.vsphere.vmware.com If a workload is expected to use a class such as vsphere-csi-sc, verify the PVC actually references it:\nkubectl get pvc -n \u0026lt;namespace\u0026gt; \u0026lt;pvc-name\u0026gt; -o jsonpath=\u0026#39;{.spec.storageClassName}{\u0026#34;\\n\u0026#34;}\u0026#39; If migrating from an older storage backend, search for stale class references:\nkubectl get pvc -A -o json \\ | jq -r \u0026#39;.items[] | select(.spec.storageClassName == \u0026#34;hpe-alletra-ssd\u0026#34;) | [.metadata.namespace,.metadata.name,.status.phase,.spec.volumeName] | @tsv\u0026#39; kubectl get pv -o json \\ | jq -r \u0026#39;.items[] | select(.spec.storageClassName == \u0026#34;hpe-alletra-ssd\u0026#34;) | [.metadata.name,.status.phase,(.spec.claimRef.namespace // \u0026#34;\u0026#34;),(.spec.claimRef.name // \u0026#34;\u0026#34;)] | @tsv\u0026#39; No output is expected after the old storage class is fully removed.\nCheck vSphere CSI Components Verify the vSphere CSI controller and node pods are running:\nkubectl get pods -n vmware-system-csi -o wide This only proves the driver pods are up. It does not prove workload mounts can succeed.\nCheck recent CSI-related events:\nkubectl get events -A --sort-by=.lastTimestamp \\ | grep -i \u0026#39;vsphere\\|csi\\|volume\\|mount\\|attach\u0026#39; If the issue is namespace-specific, narrow the event query:\nkubectl get events -n \u0026lt;namespace\u0026gt; --sort-by=.lastTimestamp Follow The PV Get the PV backing the PVC:\nkubectl get pvc -n \u0026lt;namespace\u0026gt; \u0026lt;pvc-name\u0026gt; -o jsonpath=\u0026#39;{.spec.volumeName}{\u0026#34;\\n\u0026#34;}\u0026#39; Inspect the PV:\nkubectl get pv \u0026lt;pv-name\u0026gt; -o yaml Confirm:\nspec.storageClassName is the expected vSphere CSI class. pv.kubernetes.io/provisioned-by is csi.vsphere.vmware.com. there are no finalizers or annotations from an older CSI provider. Check VolumeAttachments List attachments:\nkubectl get volumeattachments -o wide Expected healthy signal:\nATTACHER PV NODE ATTACHED csi.vsphere.vmware.com pvc-... worker-name true If attachment exists but the pod cannot mount, inspect it:\nkubectl describe volumeattachment \u0026lt;volumeattachment-name\u0026gt; This helps separate attach failures from guest-level mount or device discovery failures.\nVerify Worker VM Requirement For vSphere CSI, worker VMs need disk UUID visibility enabled:\ndisk.EnableUUID = \u0026#34;TRUE\u0026#34; Without this setting, volumes may provision and attach, but kubelet and the CSI node plugin may fail to discover or mount the disk inside the guest.\nWith govc, inspect the VM extra config:\ngovc vm.info -e /path/to/worker-vm | grep -i \u0026#39;disk.enableUUID\u0026#39; Expected:\ndisk.EnableUUID = TRUE If it is missing, remediate one worker at a time:\nkubectl cordon \u0026lt;worker\u0026gt; kubectl drain \u0026lt;worker\u0026gt; --ignore-daemonsets --delete-emptydir-data Then in vSphere:\nPower off VM Set disk.EnableUUID = \u0026#34;TRUE\u0026#34; Power on VM Return the node:\nkubectl uncordon \u0026lt;worker\u0026gt; kubectl get nodes Remove Temporary Scheduling Pins Temporary nodeSelector pins are useful during troubleshooting, but they should not remain as the final fix.\nFor Prometheus Operator resources, check selectors on the CRs:\nkubectl get prometheus -n \u0026lt;namespace\u0026gt; \u0026lt;name\u0026gt; -o jsonpath=\u0026#39;{.spec.nodeSelector}{\u0026#34;\\n\u0026#34;}\u0026#39; kubectl get alertmanager -n \u0026lt;namespace\u0026gt; \u0026lt;name\u0026gt; -o jsonpath=\u0026#39;{.spec.nodeSelector}{\u0026#34;\\n\u0026#34;}\u0026#39; Check the rendered StatefulSets too:\nkubectl get statefulset -n \u0026lt;namespace\u0026gt; \u0026lt;prometheus-statefulset\u0026gt; -o jsonpath=\u0026#39;{.spec.template.spec.nodeSelector}{\u0026#34;\\n\u0026#34;}\u0026#39; kubectl get statefulset -n \u0026lt;namespace\u0026gt; \u0026lt;alertmanager-statefulset\u0026gt; -o jsonpath=\u0026#39;{.spec.template.spec.nodeSelector}{\u0026#34;\\n\u0026#34;}\u0026#39; Empty output means no node selector is set.\nIf a temporary selector must be removed live:\nkubectl patch prometheus -n \u0026lt;namespace\u0026gt; \u0026lt;name\u0026gt; --type=merge -p \u0026#39;{\u0026#34;spec\u0026#34;:{\u0026#34;nodeSelector\u0026#34;:null}}\u0026#39; kubectl patch alertmanager -n \u0026lt;namespace\u0026gt; \u0026lt;name\u0026gt; --type=merge -p \u0026#39;{\u0026#34;spec\u0026#34;:{\u0026#34;nodeSelector\u0026#34;:null}}\u0026#39; Make the same change in GitOps-managed values after the live fix.\nProve Mobility Recovery is not complete just because the pod runs once.\nValidate that the workload can run on a worker with vSphere CSI volumes attached:\nkubectl get pods -n \u0026lt;namespace\u0026gt; -o wide kubectl get pvc -n \u0026lt;namespace\u0026gt; -o wide kubectl get volumeattachments -o wide Expected signals:\npods are Running and ready. PVCs are Bound to the intended vSphere CSI StorageClass. VolumeAttachment entries are ATTACHED=true on the worker hosting the pod. workload CRs report reconciled and available, if using an operator. For Prometheus Operator resources:\nkubectl get prometheus,alertmanager -n \u0026lt;namespace\u0026gt; -o wide Expected:\nRECONCILED AVAILABLE True True GitOps Follow-Up After live remediation, update the desired state:\nset the workload storage class explicitly to the vSphere CSI class. remove old storage class names from Helm values or manifests. remove temporary node pins. document disk.EnableUUID = \u0026quot;TRUE\u0026quot; in VM template or worker provisioning requirements. render the Prometheus and Alertmanager CRs before syncing. The durable fix is not a successful manual mount. It is a desired state that recreates the same healthy storage path after the next sync, rollout, or node replacement.\n","permalink":"https://trinidadmarroquin.com/field-notes/vsphere-csi-attach-mount-checklist/","section":"field-notes","summary":"Use this checklist when a Kubernetes workload has a PVC that should use vSphere CSI, but the pod is pending, stuck in init, or failing to mount the volume. For resize-specific CNS task triage, see vSphere CSI CNS ExtendVolume Triage. If GitOps keeps restarting CSI controllers during a storage freeze, see GitOps-Owned vSphere CSI Maintenance Pauses.\nThe goal is to follow the volume from intent to pod readiness:\nStorageClass -\u0026gt; PVC -\u0026gt; PV -\u0026gt; VolumeAttachment -\u0026gt; worker VM -\u0026gt; kubelet mount -\u0026gt; pod ready Check The Workload State Start with pods and PVCs in the affected namespace:\n","tags":["kubernetes","vsphere","vmware","storage","troubleshooting"],"title":"vSphere CSI Attach And Mount Checklist"},{"categories":["field-notes"],"content":"First-boot bootstrap scripts should be easy to debug after the VM is already running. For operational bootstrap, fail-fast is not always the best default.\nIf a bootstrap framework configures hostname, SSH, disks, iSCSI, and node labels, stopping at the first failure may hide useful evidence from later checks. A better pattern is to run each stage, log the result, continue, and summarize failures at the end.\nSymptoms Bootstrap log stops after the first or second script. Later configuration steps never run. cloud-init-output.log shows only partial output. Re-running the bootstrap manually succeeds, but first boot did not. Log lines appear duplicated or interleaved. Use A Stage Runner Wrap each bootstrap step so failures are recorded but do not prevent later diagnostics:\n#!/usr/bin/env bash set -uo pipefail LOG_FILE=\u0026#34;/var/log/platform-bootstrap.log\u0026#34; FAILED_STEPS=() log() { echo \u0026#34;[$(date -Is)] $*\u0026#34; | tee -a \u0026#34;$LOG_FILE\u0026#34; } run_step() { local label=\u0026#34;$1\u0026#34; shift log \u0026#34;========== START ${label} ==========\u0026#34; if \u0026#34;$@\u0026#34; 2\u0026gt;\u0026amp;1 | tee -a \u0026#34;$LOG_FILE\u0026#34;; then log \u0026#34;========== PASS ${label} ==========\u0026#34; else local rc=${PIPESTATUS[0]} log \u0026#34;========== FAIL ${label} rc=${rc} ==========\u0026#34; FAILED_STEPS+=(\u0026#34;${label}:rc=${rc}\u0026#34;) fi } Then call each stage explicitly:\nrun_step \u0026#34;01_set_hostname\u0026#34; sudo \u0026#34;${SCRIPT_DIR}/01_set_hostname.sh\u0026#34; \u0026#34;${VARS_FILE}\u0026#34; \u0026#34;${HOSTNAME_VALUE}\u0026#34; run_step \u0026#34;02_base_and_ssh\u0026#34; sudo \u0026#34;${SCRIPT_DIR}/02_base_and_ssh.sh\u0026#34; \u0026#34;${VARS_FILE}\u0026#34; run_step \u0026#34;05_iscsi\u0026#34; sudo \u0026#34;${SCRIPT_DIR}/05_iscsi.sh\u0026#34; \u0026#34;${VARS_FILE}\u0026#34; run_step \u0026#34;03_data_disk_lvm\u0026#34; sudo \u0026#34;${SCRIPT_DIR}/03_data_disk_lvm.sh\u0026#34; \u0026#34;${VARS_FILE}\u0026#34; run_step \u0026#34;04_setup_rancher_disk_label\u0026#34; sudo \u0026#34;${SCRIPT_DIR}/04_setup_rancher_dik_lable.sh\u0026#34; \u0026#34;${VARS_FILE}\u0026#34; Add A Summary End every run with a summary:\nlog \u0026#34;========== BOOTSTRAP SUMMARY ==========\u0026#34; if (( ${#FAILED_STEPS[@]} \u0026gt; 0 )); then log \u0026#34;Completed with failed steps:\u0026#34; for failed in \u0026#34;${FAILED_STEPS[@]}\u0026#34;; do log \u0026#34;FAILED: ${failed}\u0026#34; done else log \u0026#34;Completed successfully.\u0026#34; fi exit 0 Exit 0 when the goal is evidence collection and the host should stay reachable. Use a non-zero exit only when another system must treat the bootstrap as failed.\nPrevent Concurrent Runs If cloud-init, a systemd unit, or a manual command can trigger the same script, add a lock:\nLOCK_FILE=\u0026#34;/run/platform-bootstrap.lock\u0026#34; exec 9\u0026gt;\u0026#34;${LOCK_FILE}\u0026#34; if ! flock -n 9; then echo \u0026#34;[$(date -Is)] Another platform-bootstrap process is already running; exiting.\u0026#34; | tee -a \u0026#34;$LOG_FILE\u0026#34; exit 0 fi Duplicate log entries often mean the script was run twice or that an outer tee is also writing the same output.\nAvoid Double Logging If the bootstrap script writes its own log, keep cloud-init simple:\n#cloud-config runcmd: - [ bash, -lc, \u0026#39;/usr/local/bin/platform-bootstrap\u0026#39; ] Avoid wrapping it like this unless you intentionally want a second wrapper log:\nruncmd: - [ bash, -lc, \u0026#39;/usr/local/bin/platform-bootstrap 2\u0026gt;\u0026amp;1 | tee -a /var/log/platform-bootstrap.log\u0026#39; ] Operating Rule Bootstrap should produce enough evidence for the next operator:\nwhat ran. what passed. what failed. what command returned the failure. whether a second bootstrap process tried to run. The goal is not to hide failures. The goal is to preserve enough context to fix them without guessing.\n","permalink":"https://trinidadmarroquin.com/field-notes/bootstrap-logging-idempotency/","section":"field-notes","summary":"First-boot bootstrap scripts should be easy to debug after the VM is already running. For operational bootstrap, fail-fast is not always the best default.\nIf a bootstrap framework configures hostname, SSH, disks, iSCSI, and node labels, stopping at the first failure may hide useful evidence from later checks. A better pattern is to run each stage, log the result, continue, and summarize failures at the end.\nSymptoms Bootstrap log stops after the first or second script. Later configuration steps never run. cloud-init-output.log shows only partial output. Re-running the bootstrap manually succeeds, but first boot did not. Log lines appear duplicated or interleaved. Use A Stage Runner Wrap each bootstrap step so failures are recorded but do not prevent later diagnostics:\n","tags":["bootstrap","linux","cloud-init","operations"],"title":"Bootstrap Logging, Locks, And Continue-On-Failure Behavior"},{"categories":["notes"],"content":"The hardest part of automating VM provisioning is not always creating the VM. It is deciding which layer owns first boot.\nIn one troubleshooting session, a Terraform refactor exposed a chain of hidden coupling between Packer, Terraform, vSphere customization, cloud-init, bootstrap scripts, netplan, SSH, and iSCSI. Each tool was doing something useful. The problem was that several of them were trying to own the same parts of the machine lifecycle.\nThat is where first-boot ownership matters.\nThe Layers The system had several layers:\nPacker built the Ubuntu template. Terraform cloned the VM through the vSphere provider. vSphere guest customization set hostname, primary network, gateway, and DNS. Terraform guestinfo injected cloud-init metadata and userdata. cloud-init ran first-boot commands. bootstrap scripts configured users, SSH, disks, iSCSI, and node-specific behavior. Kubernetes CSI handled persistent volume lifecycle later. None of those layers is wrong. The failure mode appears when the boundaries are unclear.\nPacker Owns The Template Baseline Packer should produce a template that is ready to be cloned. That means the image has the right packages and services installed, but it should not contain stale first-boot state.\nPacker should own:\nOS package baseline. open-vm-tools. cloud-init installation and service enablement. util-linux-extra when VMware customization requires hwclock. bootstrap framework placement under a known path. cleaning cloud-init instance state before sealing. Packer should not own per-clone identity. It should not leave a template thinking it has already completed the first boot of a real machine.\nA practical pattern is to install only a static bootstrap entrypoint and unit into the image:\n/usr/local/bin/platform-bootstrap /etc/systemd/system/platform-bootstrap.service /var/lib/platform-bootstrap/ /var/log/platform-bootstrap.log The entrypoint should be idempotent, expose --version and --check, log to a stable file, write machine-readable status, and create a completion marker only after success. It should not contain site values, passwords, tokens, server URLs, DNS settings, SSH keys, or cluster join config.\nThat keeps Packer responsible for placing the mechanism, while Terraform and cloud-init decide when to run it and what runtime values to provide.\nAlso separate Packer build success from bootstrap placement success. If a Packer build creates a VM but never reaches SSH, none of the file or shell provisioners ran. An empty /opt, missing /usr/local/bin/platform-bootstrap, or absent service unit usually means the build stopped at communicator reachability, not that cloud-init or Terraform failed later.\nTerraform Owns VM Intent Terraform should describe the VM and the vSphere resources around it: folder, resource pool, datastore, network, CPU, memory, disks, template, and customization settings.\nTerraform should own:\nVM inventory. site and environment input values. vSphere resource placement. module boundaries for repeated clone behavior. guestinfo keys when cloud-init is intentionally part of the clone flow. Terraform should not be used to repair template defects during every clone. If a package is missing from the template, fix Packer. If a guest setting is environment-specific, pass it intentionally.\nvSphere Customization Owns Primary Network When Chosen In this case, vSphere customization was already generating /etc/netplan/99-netcfg-vmware.yaml and setting the primary NIC, hostname, gateway, and DNS.\nThat is a valid model. The mistake is letting cloud-init or a template-era netplan file also own the same primary route.\nThe operating rule became:\nvSphere customization owns primary network bootstrap owns additional storage networking That eliminated conflicts between 99-netcfg-vmware.yaml, old template netplan files, and cloud-init network config.\nCloud-Init Should Trigger, Not Compete Cloud-init is useful as the first-boot trigger. It can consume VMware guestinfo userdata and call a bootstrap entrypoint.\nThe useful pattern was minimal:\n#cloud-config ssh_pwauth: true disable_root: true runcmd: - [ bash, -lc, \u0026#39;mkdir -p /run/sshd\u0026#39; ] - [ bash, -lc, \u0026#39;passwd -u ubuntu || true\u0026#39; ] - [ bash, -lc, \u0026#39;systemctl enable --now ssh\u0026#39; ] - [ bash, -lc, \u0026#39;/usr/local/bin/platform-bootstrap\u0026#39; ] The point is not that every site should use exactly that YAML. The point is that cloud-init should have a clear job. In this model, its job is to trigger bootstrap and avoid breaking the template user.\nBootstrap Owns Guest Configuration Bootstrap is where environment-specific guest configuration belongs when it cannot be cleanly represented as vSphere customization.\nBootstrap should own:\nfinal SSH policy. local users and sudo rules. data disk preparation. iSCSI networking and discovery when enabled for a data center. storage readiness checks. logging that continues even when a step fails. The logging behavior matters. A bootstrap that exits on the first failure can hide the rest of the system state. For operations, it is often better to log failed steps and continue to a summary, especially when later checks provide useful evidence.\nCSI Owns Volume Lifecycle iSCSI sessions can be active without new disks showing in lsblk. That is expected when Kubernetes CSI is responsible for creating and mapping volumes.\nThe storage model is:\nnode is storage-ready CSI creates volume array maps LUN to node IQN node sees disk kubelet mounts volume That distinction avoids chasing a false failure. No disk in lsblk does not necessarily mean iSCSI is broken. It may mean no LUN has been mapped yet.\nThe Lesson First boot needs an ownership model. Without one, every tool tries to be helpful and the result is fragile.\nA good model is boring:\nPacker builds the baseline. Terraform declares the clone. vSphere customization owns primary network if you choose that path. cloud-init triggers bootstrap. bootstrap owns guest-specific operating system setup. CSI owns persistent storage lifecycle. Once the boundaries are explicit, troubleshooting gets easier. Instead of asking why the VM is broken, you can ask a sharper question: which layer owns the behavior that failed?\n","permalink":"https://trinidadmarroquin.com/posts/first-boot-ownership-packer-terraform-cloud-init/","section":"posts","summary":"The hardest part of automating VM provisioning is not always creating the VM. It is deciding which layer owns first boot.\nIn one troubleshooting session, a Terraform refactor exposed a chain of hidden coupling between Packer, Terraform, vSphere customization, cloud-init, bootstrap scripts, netplan, SSH, and iSCSI. Each tool was doing something useful. The problem was that several of them were trying to own the same parts of the machine lifecycle.\n","tags":["packer","terraform","vsphere","cloud-init","systems-thinking"],"title":"First-Boot Ownership Between Packer, Terraform, vSphere, Cloud-Init, And Bootstrap"},{"categories":["field-notes"],"content":"govc is useful when Terraform says one thing and the VM behaves like something else. It gives a direct view into vSphere inventory, VM paths, NIC backing, and extra config. For virtual CD-ROM device cleanup, see vSphere CD-ROM Host Device Cleanup With govc.\nLoad Environment Use a clearly named helper file, not vars.env if it is only for govc:\nsource govc.env govc about Typical values:\nexport GOVC_URL=\u0026#39;vcenter.example.com\u0026#39; export GOVC_USERNAME=\u0026#39;administrator@vsphere.local\u0026#39; export GOVC_PASSWORD=\u0026#39;...\u0026#39; export GOVC_INSECURE=1 export GOVC_DATACENTER=\u0026#39;DC-Site-A\u0026#39; Find A VM Path Search by VM name:\ngovc find / -type m -name \u0026#39;cluster-a-worker-02\u0026#39; Example result:\n/DC-Site-A/vm/K8s-Cluster/NonProd/cluster-a-worker-02 Use that full path for later commands.\nCheck NIC Backing govc device.ls -vm /DC-Site-A/vm/K8s-Cluster/NonProd/cluster-a-worker-02 Example useful output:\nethernet-0 VirtualVmxnet3 VLAN 232 k8s-nonprod ethernet-1 VirtualVmxnet3 iscsi test 1 ethernet-2 VirtualVmxnet3 iscsi test 2 Compare with a known-good neighbor:\ngovc device.ls -vm /DC-Site-A/vm/K8s-Cluster/NonProd/cluster-a-worker-01 govc device.ls -vm /DC-Site-A/vm/K8s-Cluster/NonProd/cluster-a-worker-02 This catches a common failure: the VM has a nonproduction IP address but is attached to the wrong port group.\nInspect Extra Config govc vm.info -e /DC-Site-A/vm/K8s-Cluster/NonProd/cluster-a-worker-02 \\ | grep -i \u0026#39;guestinfo\\|ethernet\\|disk.enableUUID\u0026#39; Useful keys:\nguestinfo.userdata guestinfo.userdata.encoding guestinfo.metadata guestinfo.network-config disk.enableUUID ethernet0.pciSlotNumber Connect A NIC Manually For one-off repair while debugging:\ngovc device.connect -vm /DC-Site-A/vm/K8s-Cluster/NonProd/cluster-a-worker-02 ethernet-0 Prefer fixing the template or Terraform config after confirming the cause.\nWhen To Use This Use govc when:\nthe guest cannot reach its gateway. Terraform vm_network looks correct but the VM is on the wrong port group. guestinfo userdata does not appear to run. you need to compare a broken clone against a working neighbor. you need the canonical vSphere VM path for troubleshooting commands. govc is especially useful because it verifies what vCenter actually did, not what Terraform intended.\n","permalink":"https://trinidadmarroquin.com/field-notes/govc-vsphere-verification/","section":"field-notes","summary":"govc is useful when Terraform says one thing and the VM behaves like something else. It gives a direct view into vSphere inventory, VM paths, NIC backing, and extra config. For virtual CD-ROM device cleanup, see vSphere CD-ROM Host Device Cleanup With govc.\nLoad Environment Use a clearly named helper file, not vars.env if it is only for govc:\nsource govc.env govc about Typical values:\nexport GOVC_URL=\u0026#39;vcenter.example.com\u0026#39; export GOVC_USERNAME=\u0026#39;administrator@vsphere.local\u0026#39; export GOVC_PASSWORD=\u0026#39;...\u0026#39; export GOVC_INSECURE=1 export GOVC_DATACENTER=\u0026#39;DC-Site-A\u0026#39; Find A VM Path Search by VM name:\n","tags":["govc","vsphere","vmware","terraform"],"title":"govc Commands For vSphere VM Verification"},{"categories":["field-notes"],"content":"A Kubernetes node can be storage-ready before any new disk appears in lsblk. With CSI-backed storage, the disk appears only after the array maps a LUN to the node.\nTarget State For a node with primary and iSCSI networks:\nens192 -\u0026gt; service network, default route, DNS iscsi01 -\u0026gt; 169.253.0.x/24 iscsi02 -\u0026gt; 169.253.1.x/24 The node should have:\ndeterministic netplan. reachable storage portals. open-iscsi installed and enabled. an initiator name. successful target discovery. persistent iSCSI node startup. Verify Network First ip -br addr ip route ping -c 3 169.253.0.11 ping -c 3 169.253.1.11 If portal pings fail, stop and fix networking before touching iSCSI.\nInstall And Enable iSCSI sudo apt-get update sudo apt-get install -y open-iscsi sudo systemctl enable --now iscsid sudo systemctl enable --now open-iscsi Check the service:\nsystemctl status iscsid --no-pager Initiator Name Check the current initiator:\ncat /etc/iscsi/initiatorname.iscsi Regenerate if needed:\nsudo cp -a /etc/iscsi/initiatorname.iscsi /var/tmp/initiatorname.iscsi.backup.$(date +%F-%H%M%S) 2\u0026gt;/dev/null || true echo \u0026#34;InitiatorName=$(sudo /sbin/iscsi-iname)\u0026#34; | sudo tee /etc/iscsi/initiatorname.iscsi sudo systemctl restart iscsid Discover And Login Discover targets:\nsudo iscsiadm -m discovery -t sendtargets -p 169.253.0.11 sudo iscsiadm -m discovery -t sendtargets -p 169.253.1.11 Login to all discovered nodes:\nsudo iscsiadm -m node --login Make sessions persistent:\nsudo iscsiadm -m node -o update -n node.startup -v automatic Verify:\nsudo iscsiadm -m session sudo iscsiadm -m session -P 3 | grep -i \u0026#39;Attached scsi disk\u0026#39; || true lsblk Why No Disk May Appear Yet If sessions exist but lsblk shows no new disk, that can be correct.\nCSI flow:\nPVC created CSI asks array to create volume array maps LUN to node IQN node sees disk kubelet mounts volume for pod Before the array maps a LUN, the node can be connected to the array but have no new block device.\nBootstrap Pattern For data centers with iSCSI arrays, make this opt-in through environment values:\nENABLE_ISCSI=true ISCSI_PORTALS=\u0026#34;169.253.0.11 169.253.1.11\u0026#34; ISCSI_TARGET_IQN=\u0026#34;iqn.2007-11.com.nimblestorage:austin-g5f6a0959b70c5587\u0026#34; ENABLE_MULTIPATH=false Then run the storage stage before any bootstrap step that expects attached storage devices.\nOperating Model Bootstrap prepares the node for storage. CSI owns volume creation and LUN mapping. The array remains the authority for volume lifecycle. Kubernetes consumes the resulting devices through the CSI workflow. ","permalink":"https://trinidadmarroquin.com/field-notes/iscsi-bootstrap-kubernetes-nodes/","section":"field-notes","summary":"A Kubernetes node can be storage-ready before any new disk appears in lsblk. With CSI-backed storage, the disk appears only after the array maps a LUN to the node.\nTarget State For a node with primary and iSCSI networks:\nens192 -\u0026gt; service network, default route, DNS iscsi01 -\u0026gt; 169.253.0.x/24 iscsi02 -\u0026gt; 169.253.1.x/24 The node should have:\ndeterministic netplan. reachable storage portals. open-iscsi installed and enabled. an initiator name. successful target discovery. persistent iSCSI node startup. Verify Network First ip -br addr ip route ping -c 3 169.253.0.11 ping -c 3 169.253.1.11 If portal pings fail, stop and fix networking before touching iSCSI.\n","tags":["kubernetes","iscsi","storage","netplan","vsphere"],"title":"iSCSI Bootstrap Readiness For Kubernetes Nodes"},{"categories":["field-notes"],"content":"VMware guest customization can write /etc/netplan/99-netcfg-vmware.yaml. If the template already has another active netplan file, the host can end up with multiple network authorities.\nThat can break routing, DNS, bootstrap scripts, Kubernetes node communication, and storage initialization.\nSymptoms Common signs:\nError: Conflicting default route declarations for IPv4 first declared in nic0 but also in ens192 or:\nDestination Host Unreachable or DNS search drift like:\nresolv_conf_search = . netplan_search = [] Inspect Netplan Ownership ls -l /etc/netplan sudo grep -R \u0026#34;routes:\\|gateway\\|search:\u0026#34; /etc/netplan -n If both a template file and VMware file are present, decide which one owns networking.\nExample conflict:\n/etc/netplan/01-template-static.yaml /etc/netplan/99-netcfg-vmware.yaml Verify Active Routing ip -br addr ip route networkctl status ens192 --no-pager If the default route is missing, test the expected route manually:\nsudo ip route replace default via 192.0.2.1 dev ens192 ping -c 3 192.0.2.1 If ARP fails, compare the vSphere port group against a working VM:\ngovc device.ls -vm /DC-Site-A/vm/K8s-Cluster/NonProd/cluster-a-worker-01 govc device.ls -vm /DC-Site-A/vm/K8s-Cluster/NonProd/cluster-a-worker-02 Normalize To One Netplan File If VMware customization is the current network authority, keep only the VMware-generated file:\nif [ -f /etc/netplan/99-netcfg-vmware.yaml ]; then for f in /etc/netplan/*.yaml /etc/netplan/*.yml; do [ -f \u0026#34;$f\u0026#34; ] || continue case \u0026#34;$f\u0026#34; in /etc/netplan/99-netcfg-vmware.yaml) echo \u0026#34;keeping $f\u0026#34; ;; *) echo \u0026#34;removing $f\u0026#34; rm -f \u0026#34;$f\u0026#34; ;; esac done fi Then validate and apply:\nsudo netplan generate sudo netplan apply ip route resolvectl status Important Shell Detail If this is run through Ansible shell, remember that /bin/sh may execute the script. Bash-only syntax such as [[ ... ]] can fail:\n/bin/sh: 47: [[: not found Use POSIX [ ... ] syntax or explicitly run the script with Bash.\nOperating Model Pick one owner:\nVMware customization owns primary NIC, hostname, gateway, and DNS. Bootstrap owns additional storage NICs and iSCSI-specific netplan. Cloud-init userdata runs bootstrap and avoids competing with network ownership. The failure pattern is usually not one bad command. It is multiple systems trying to own the same network configuration.\nIf VMware customization owns DNS, make sure Terraform separates the VM identity domain from resolver search suffixes. See Terraform vSphere DNS Search Suffix Ownership.\n","permalink":"https://trinidadmarroquin.com/field-notes/netplan-vmware-customization-conflicts/","section":"field-notes","summary":"VMware guest customization can write /etc/netplan/99-netcfg-vmware.yaml. If the template already has another active netplan file, the host can end up with multiple network authorities.\nThat can break routing, DNS, bootstrap scripts, Kubernetes node communication, and storage initialization.\nSymptoms Common signs:\nError: Conflicting default route declarations for IPv4 first declared in nic0 but also in ens192 or:\nDestination Host Unreachable or DNS search drift like:\nresolv_conf_search = . netplan_search = [] Inspect Netplan Ownership ls -l /etc/netplan sudo grep -R \u0026#34;routes:\\|gateway\\|search:\u0026#34; /etc/netplan -n If both a template file and VMware file are present, decide which one owns networking.\n","tags":["vsphere","netplan","cloud-init","linux","terraform"],"title":"Netplan Conflicts After VMware Guest Customization"},{"categories":["projects"],"content":"This project started as a repository cleanup and turned into a clearer operating model for vSphere infrastructure.\nThe original Terraform repo had grown into a collection of copied root modules. Each site and environment had its own directory with a familiar set of files: main.tf, variables.tf, output.tf, terraform.tfvars, vars.auto.tfvars, vars.env, local state files, and copied templates. That worked while the repo was small, but it made every change harder to review.\nThe target was not to make the repo clever. The target was to make it understandable.\nGoals Group active infrastructure by site and environment. Move repeated VM creation logic into a shared module. Preserve historical state and templates without letting them clutter active work. Separate Terraform inputs from operator helper files. Make future changes easier to validate with terraform plan. Target Layout The refactor moved toward this structure:\nterraform-vsphere-platform/ environments/ site-a/prod/ site-a/nonprod/ site-b/nonprod/ site-c/tools/ modules/ vsphere-vm-group/ templates/ archive/ legacy-roots/ legacy-state/ legacy-templates/ The important design choice was environments/\u0026lt;site\u0026gt;/\u0026lt;env\u0026gt;. That keeps production and nonproduction roots near each other, makes site ownership visible, and avoids long historical directory names like Terraform-Site-A-prod.\nModule Boundary The first reusable module was not a single VM module. The existing Terraform already created groups of VMs from a map, so the correct initial module boundary was a VM group:\nmodules/vsphere-vm-group/ main.tf variables.tf outputs.tf versions.tf That module owns repeated vSphere behavior: datacenter, datastore, cluster, network, template lookups, VM cloning, disks, guest customization, and outputs. Environment roots own inventory: VM names, IPs, site settings, gateway, DNS, resource pool, template choice, and operational flags.\nMigration Approach The repo move was done with a dry-run migration script before moving files for real. That mattered because the repo contained state files, old templates, nested .terraform directories, one-off rebuilds, and historical folders.\nThe script created the new layout, moved active environment files, archived local state, and copied templates conservatively. A small directory-creation cache made the dry-run output readable instead of repeating the same mkdir -p lines hundreds of times.\nAfter the move, empty legacy directories were removed only after inspection. That kept the Git history clean and made the refactor reviewable.\nState Safety Moving a Terraform resource into a module changes its address. For example:\nvsphere_virtual_machine.vm[\u0026#34;lb1\u0026#34;] becomes:\nmodule.vm_group.vsphere_virtual_machine.vm[\u0026#34;lb1\u0026#34;] That requires explicit state movement before apply:\nterraform state mv \\ \u0026#39;vsphere_virtual_machine.vm[\u0026#34;lb1\u0026#34;]\u0026#39; \\ \u0026#39;module.vm_group.vsphere_virtual_machine.vm[\u0026#34;lb1\u0026#34;]\u0026#39; The validation target was simple: after state movement, terraform plan should show no unexpected destroy/recreate behavior.\nOperational Lessons The refactor surfaced more than file duplication. It exposed hidden ownership boundaries between Terraform, vSphere customization, Packer-built templates, cloud-init, bootstrap scripts, netplan, and iSCSI configuration.\nThe most useful outcome was a clearer model:\nTerraform owns VM intent and vSphere resource configuration. Environment roots own site-specific inventory and values. Shared modules own repeated provisioning behavior. Packer owns the template baseline. Bootstrap owns first-boot operating system configuration that must vary by environment. CSI owns volume lifecycle after the node joins Kubernetes. Result The repo became easier to scan, safer to change, and better aligned with how the infrastructure is operated. The refactor also created a reusable pattern for future site migrations: move layout first, preserve behavior, validate plans, then improve module behavior one environment at a time.\nThat is the useful version of infrastructure modernization: fewer surprises, clearer ownership, and smaller changes that operators can reason about under pressure.\n","permalink":"https://trinidadmarroquin.com/projects/terraform-vsphere-refactor/","section":"projects","summary":"This project started as a repository cleanup and turned into a clearer operating model for vSphere infrastructure.\nThe original Terraform repo had grown into a collection of copied root modules. Each site and environment had its own directory with a familiar set of files: main.tf, variables.tf, output.tf, terraform.tfvars, vars.auto.tfvars, vars.env, local state files, and copied templates. That worked while the repo was small, but it made every change harder to review.\n","tags":["terraform","vsphere","infrastructure","sre"],"title":"Refactoring A vSphere Terraform Repo Into Environment Roots And Shared Modules"},{"categories":["notes"],"content":"A Terraform refactor can look like a file organization problem until the first plan, clone, and bootstrap run.\nThe repo I was working on had the usual signs of age: copied environment roots, local state files, duplicated templates, inconsistent variable files, and old one-off directories that were still sitting beside active infrastructure. The obvious fix was to move toward a cleaner layout:\nenvironments/\u0026lt;site\u0026gt;/\u0026lt;env\u0026gt;/ modules/vsphere-vm-group/ templates/ archive/ That was the right direction. It was also only the beginning.\nThe Refactor Was The Easy Part The first pass was structural. Move active roots under environments/, preserve old state under archive/, keep templates available, and create a shared vsphere-vm-group module.\nThe design decision that mattered most was matching the module boundary to the real Terraform shape. The existing roots created VM groups from a map, so a VM group module made more sense than forcing everything through a single-VM abstraction.\nThat gave the repo a better operating shape:\nSite and environment roots became easier to find. Module behavior became reusable. Historical artifacts stopped cluttering active work. terraform plan became easier to reason about. But then the refactor exposed the actual system.\nThe Module Changed First-Boot Behavior Moving VM creation into a module changed more than the address of the resource. The module also started injecting VMware guestinfo data for cloud-init:\nguestinfo.metadata guestinfo.userdata guestinfo.network-config That meant the VM was no longer just a vSphere clone with guest customization. It was now a vSphere clone plus cloud-init plus bootstrap execution.\nThat matters because first boot was already being influenced by multiple systems:\nPacker built the template. vSphere customization set hostname, network, and DNS. Terraform provided the clone configuration. cloud-init consumed guestinfo data. Bootstrap scripts configured SSH, users, disks, and storage networking. Once cloud-init was enabled and guestinfo userdata was injected, login behavior changed. SSH was reachable, but authentication failed. That was not a Terraform syntax problem. It was an ownership problem.\nPreserve Behavior Before Improving It One lesson from this session: refactors should preserve behavior first.\nIf an environment root has special behavior, the module needs to support it explicitly before that root is converted. In this case, important behavior included:\nwait_for_guest_ip_timeout = 0 disk UUID support for Kubernetes nodes two disks vSphere guest customization post-create power behavior cloud-init userdata bootstrap execution The module could support those things, but they had to be treated as intentional inputs, not incidental details copied from an old main.tf.\nNetwork Ownership Was The Real Problem The most useful discovery was that several systems were trying to influence networking.\nVMware customization created:\n/etc/netplan/99-netcfg-vmware.yaml The template also had older netplan files. In some cases both sets of files declared default routes. That produced errors like:\nError: Conflicting default route declarations for IPv4 first declared in nic0 but also in ens192 That kind of conflict can disturb Kubernetes node communication, DNS resolution, kubelet behavior, storage paths, and bootstrap scripts. It is not just cosmetic drift.\nThe fix was to choose one authority. For the immediate path, VMware customization owned the primary NIC, hostname, gateway, and DNS. Bootstrap owned additional storage NICs and iSCSI setup. Cloud-init ran bootstrap but did not try to own the same network settings.\nThe useful rule became simple:\none network authority per layer The Storage Lesson The iSCSI work exposed another boundary.\nThe node needed to be storage-ready, but it did not need to see every disk immediately. For Kubernetes with CSI-backed storage, the sequence is:\nPVC -\u0026gt; CSI -\u0026gt; array creates volume -\u0026gt; array maps LUN -\u0026gt; node sees disk -\u0026gt; kubelet mounts volume So a new node can have working iSCSI sessions and still not show a new disk in lsblk until the array maps a LUN to that node\u0026rsquo;s initiator.\nThat distinction matters during troubleshooting. Connected to the array does not mean a volume is mapped. It only means the node is ready for the storage lifecycle to happen.\nWhat Changed Operationally The end state was not just a cleaner repo. The real improvement was clearer ownership:\nTerraform owns VM intent and vSphere resources. Environment roots own site-specific values. Shared modules own repeated provisioning behavior. Packer owns the template baseline. cloud-init triggers first-boot behavior. Bootstrap owns environment-specific OS configuration. CSI owns persistent volume lifecycle. That model is easier to operate because each layer has a job. When something fails, the question becomes sharper: which layer owns this behavior?\nTakeaways A Terraform refactor can reveal hidden coupling between infrastructure layers. Module boundaries should match the real resource shape before they are made more abstract. State moves are part of module refactors, not an afterthought. Do not let vSphere customization, cloud-init, and bootstrap all own the same network settings. Guest logs and generated OS config often explain more than Terraform plan output. A clean repo layout helps, but clear ownership is what makes the system operable. The refactor succeeded because it forced the infrastructure model to become explicit. That is the real value of the work.\n","permalink":"https://trinidadmarroquin.com/posts/terraform-refactor-hidden-infrastructure-coupling/","section":"posts","summary":"A Terraform refactor can look like a file organization problem until the first plan, clone, and bootstrap run.\nThe repo I was working on had the usual signs of age: copied environment roots, local state files, duplicated templates, inconsistent variable files, and old one-off directories that were still sitting beside active infrastructure. The obvious fix was to move toward a cleaner layout:\nenvironments/\u0026lt;site\u0026gt;/\u0026lt;env\u0026gt;/ modules/vsphere-vm-group/ templates/ archive/ That was the right direction. It was also only the beginning.\n","tags":["terraform","vsphere","packer","cloud-init","systems-thinking"],"title":"Terraform Refactor Notes: When Repo Cleanup Exposes Infrastructure Coupling"},{"categories":["field-notes"],"content":"When an existing Terraform resource is moved into a module, Terraform sees a new resource address. If state is not moved first, the plan may try to create the module resource and destroy the old root resource.\nSymptom After converting a root resource to a module call, the plan shows addresses like this:\n# module.vm_group.vsphere_virtual_machine.vm[\u0026#34;lb1\u0026#34;] will be created while the existing state still has:\nvsphere_virtual_machine.vm[\u0026#34;lb1\u0026#34;] Check Current State Run from the environment root:\nterraform state list For vSphere VMs, filter the list:\nterraform state list | grep vsphere_virtual_machine Move State Move the existing resource address to the module address:\nterraform state mv \\ \u0026#39;vsphere_virtual_machine.vm[\u0026#34;lb1\u0026#34;]\u0026#39; \\ \u0026#39;module.vm_group.vsphere_virtual_machine.vm[\u0026#34;lb1\u0026#34;]\u0026#39; Repeat for each for_each key:\nterraform state mv \\ \u0026#39;vsphere_virtual_machine.vm[\u0026#34;wrkr1\u0026#34;]\u0026#39; \\ \u0026#39;module.vm_group.vsphere_virtual_machine.vm[\u0026#34;wrkr1\u0026#34;]\u0026#39; Validate Run:\nterraform validate terraform plan The target result is no unexpected replacement. If the plan still shows replacement, inspect the forced replacement field before applying:\nterraform plan -out=tfplan terraform show -no-color tfplan \u0026gt; plan.txt grep -n \u0026#34;must be replaced\\|forces replacement\u0026#34; plan.txt Notes Do not use terraform state rm for this case. The resource still exists; only its Terraform address changed. Move state before applying the module conversion. Preserve behavior first. Improve module defaults after the plan is stable. If the environment uses for_each, the keys must stay stable or Terraform will treat the instances as different resources. ","permalink":"https://trinidadmarroquin.com/field-notes/terraform-state-move-module-refactor/","section":"field-notes","summary":"When an existing Terraform resource is moved into a module, Terraform sees a new resource address. If state is not moved first, the plan may try to create the module resource and destroy the old root resource.\nSymptom After converting a root resource to a module call, the plan shows addresses like this:\n# module.vm_group.vsphere_virtual_machine.vm[\u0026#34;lb1\u0026#34;] will be created while the existing state still has:\nvsphere_virtual_machine.vm[\u0026#34;lb1\u0026#34;] Check Current State Run from the environment root:\n","tags":["terraform","state","modules","vsphere"],"title":"Terraform State Moves During A Module Refactor"},{"categories":["field-notes"],"content":"Terraform repositories often accumulate terraform.tfvars, *.auto.tfvars, shell env files, examples, and helper scripts. That works until nobody remembers which file is authoritative.\nProblem A single environment directory may contain:\nterraform.tfvars vars.auto.tfvars vars.env terraform.tfvars.example vars.auto.tfvars.example vars.env.example That creates three problems:\nTerraform may load values from multiple files. shell helper files look like Terraform configuration. secrets may drift into files that should be committed safely. How Terraform Loads Values Terraform automatically loads:\nterraform.tfvars *.auto.tfvars If the same variable appears in both, the result can be confusing during plan review.\nvars.env is not special to Terraform. It only matters if a person or script sources it.\nRecommended Model Use a small, explicit model:\nvariables.tf # defines inputs terraform.tfvars.example # checked-in sample values terraform.tfvars # real non-secret values, usually ignored if sensitive govc.env.example # checked-in govc helper sample govc.env # real govc helper values, ignored Secrets should come from environment variables, a secret manager, or another approved workflow:\nexport TF_VAR_vsphere_user=\u0026#39;...\u0026#39; export TF_VAR_vsphere_password=\u0026#39;...\u0026#39; export TF_VAR_admin_password=\u0026#39;...\u0026#39; Rename Helper Files If vars.env exists only to run govc, rename it:\nmv vars.env govc.env That makes the purpose obvious:\nsource govc.env govc about govc device.ls -vm /DC-Site-A/vm/K8s-Cluster/NonProd/cluster-a-worker-02 Classify Existing Values Inspect what is defined across files:\ngrep -h \u0026#39;^[a-zA-Z_][a-zA-Z0-9_]*[[:space:]]*=\u0026#39; terraform.tfvars vars.auto.tfvars 2\u0026gt;/dev/null List variables Terraform expects:\ngrep \u0026#39;^variable \u0026#34;\u0026#39; variables.tf Then classify each value:\nTerraform input: move to terraform.tfvars or the example file. Secret: move to TF_VAR_..., Vault, or another secret workflow. Operator helper value: move to govc.env or another clearly named tool file. Obsolete value: remove after a clean plan confirms it is not needed. Check .gitignore At minimum:\n**/.terraform/ **/*.tfstate **/*.tfstate.* **/*.tfplan **/terraform.tfvars **/govc.env Decide intentionally whether to commit .terraform.lock.hcl. Many teams commit it for provider consistency, but avoid unmanaged lock files scattered across old roots unless that is the chosen model.\nValidate After Cleanup Run from the environment root:\nterraform validate terraform plan The cleanup is successful when the plan is unchanged except for expected variable hygiene changes.\nOperating Rule Names should explain ownership:\nterraform.tfvars is for Terraform values. govc.env is for govc CLI context. TF_VAR_... or a secret manager is for sensitive Terraform inputs. If a file name does not explain who consumes it, it will eventually confuse an operator.\n","permalink":"https://trinidadmarroquin.com/field-notes/terraform-variable-file-hygiene/","section":"field-notes","summary":"Terraform repositories often accumulate terraform.tfvars, *.auto.tfvars, shell env files, examples, and helper scripts. That works until nobody remembers which file is authoritative.\nProblem A single environment directory may contain:\nterraform.tfvars vars.auto.tfvars vars.env terraform.tfvars.example vars.auto.tfvars.example vars.env.example That creates three problems:\nTerraform may load values from multiple files. shell helper files look like Terraform configuration. secrets may drift into files that should be committed safely. How Terraform Loads Values Terraform automatically loads:\n","tags":["terraform","govc","secrets","operations"],"title":"Terraform Variable File Hygiene"},{"categories":["field-notes"],"content":"Terraform can successfully ask vSphere to clone a VM and still fail during guest customization. In that case, the problem is usually inside the guest/template, not Terraform syntax.\nSymptom Terraform reports an error like:\nVirtual machine customization failed An error occurred while customizing VM ... For details reference the log file /var/log/vmware-imc/toolsDeployPkg.log in the guest OS. Terraform may leave the VM in place to help troubleshooting.\nFirst Checks From the VM console or SSH if available:\nsudo tail -n 200 /var/log/vmware-imc/toolsDeployPkg.log sudo tail -n 200 /var/log/cloud-init.log sudo tail -n 200 /var/log/cloud-init-output.log systemctl status open-vm-tools --no-pager Check VMware tools:\nwhich vmtoolsd vmware-toolbox-cmd -v systemctl status open-vm-tools --no-pager Common Template Issues open-vm-tools is missing or not running. hwclock is missing on Ubuntu/Debian templates. cloud-init state was not cleaned before templating. SSH or network services are disabled in the template. vSphere customization and cloud-init are competing for the same network settings. For Ubuntu templates, hwclock is provided by util-linux-extra:\nsudo apt-get update sudo apt-get install -y util-linux-extra open-vm-tools command -v hwclock Retry Safely For current Terraform versions, prefer -replace for a failed test VM:\nterraform apply -replace=\u0026#39;vsphere_virtual_machine.vm[\u0026#34;lb1\u0026#34;]\u0026#39; If the VM is managed inside a module:\nterraform apply -replace=\u0026#39;module.vm_group.vsphere_virtual_machine.vm[\u0026#34;lb1\u0026#34;]\u0026#39; Verify The Plan Before applying, save and inspect the plan:\nterraform plan -out=tfplan terraform show -no-color tfplan \u0026gt; plan.txt grep -n \u0026#34;must be replaced\\|forces replacement\u0026#34; plan.txt Do not assume a replacement is safe just because the VM is a worker. Check the resource key, name, folder, datastore, network, and disks.\nOperating Model Packer should fix template defects. Terraform should clone and customize from a known-good template. vSphere guest customization should own only the settings you intentionally use it for. Bootstrap should handle guest configuration that must vary by environment. If Terraform clone customization fails, get the guest logs before deleting the VM. They usually explain the real failure.\n","permalink":"https://trinidadmarroquin.com/field-notes/terraform-vsphere-clone-customization-failures/","section":"field-notes","summary":"Terraform can successfully ask vSphere to clone a VM and still fail during guest customization. In that case, the problem is usually inside the guest/template, not Terraform syntax.\nSymptom Terraform reports an error like:\nVirtual machine customization failed An error occurred while customizing VM ... For details reference the log file /var/log/vmware-imc/toolsDeployPkg.log in the guest OS. Terraform may leave the VM in place to help troubleshooting.\nFirst Checks From the VM console or SSH if available:\n","tags":["terraform","vsphere","vmware","packer"],"title":"Terraform vSphere Clone Customization Failures"},{"categories":["notes"],"content":"LLMs can speed up infrastructure troubleshooting, but only if the operator stays in control.\nThe useful pattern is not \u0026ldquo;ask the model what is wrong and do what it says.\u0026rdquo; That is risky. The better pattern is to treat the model like a second engineer who can help organize evidence, generate hypotheses, suggest verification commands, and turn the final diagnosis into reusable documentation.\nThat distinction matters.\nDuring a recent Terraform and vSphere refactor, an LLM helped connect several layers of the system: Terraform modules, vSphere guest customization, Packer-built templates, cloud-init, netplan, bootstrap scripts, govc, iSCSI, and Kubernetes CSI. That was useful. It also suggested at least one invalid vSphere provider argument. terraform validate caught it.\nThat is the right relationship: use the model to accelerate thinking, then verify everything with the system.\nStart With Evidence LLMs work better when the prompt includes real operational evidence instead of a vague symptom.\nUseful context includes:\nexact command output. error messages. recent changes. relevant file snippets. current assumptions. what has already been ruled out. For infrastructure work, a good prompt is often more like an incident handoff than a search query.\nInstead of:\nTerraform broke my VM. What happened? Use:\nTerraform created the VM, but vSphere guest customization failed. Here is the exact error, the relevant main.tf block, and the toolsDeployPkg.log excerpt. Please list likely causes ranked by evidence, and give read-only verification commands first. That gives the model something to reason from.\nAsk For Diagnostic Branches The most useful output is not a single answer. It is a short list of likely branches and how to test each one.\nExample structure:\nRank likely causes by evidence. For each cause, provide: - why it fits - how to verify safely - what would disprove it - what fix should wait until verification That keeps the operator from jumping straight to a fix.\nIn the Terraform/vSphere session, this helped separate several different failure modes:\nTerraform state/address changes during module conversion. vSphere guest customization failures inside the VM. cloud-init being disabled in the template. duplicate netplan files causing route conflicts. wrong vSphere port group backing. iSCSI sessions being present before CSI mapped a LUN. Those are different problems. Treating them as one big \u0026ldquo;VM provisioning is broken\u0026rdquo; issue would have wasted time.\nSeparate Safe Checks From Changes For SRE work, prompt the model to distinguish read-only diagnostics from actions that mutate infrastructure.\nGood read-only checks:\nterraform state list terraform plan -out=tfplan terraform show -no-color tfplan \u0026gt; plan.txt govc device.ls -vm /path/to/vm govc vm.info -e /path/to/vm ip route cloud-init status --long Potentially mutating actions:\nterraform apply terraform state mv terraform apply -replace=... netplan apply iscsiadm -m node --login The model can suggest both, but the operator should decide when to cross from observation to change.\nMake The Model Show Its Assumptions One useful prompt is:\nWhat assumptions are you making, and which command would verify each one? This is especially useful when a system has multiple layers. For example, when a VM could not reach its gateway, several explanations were plausible:\nwrong netmask. missing default route. wrong vSphere port group. firewall. cloud-init network conflict. The evidence eventually showed the VM had a UAT IP address but was attached to an Internal VLAN port group. That is not something to guess. It was verified by comparing govc device.ls output between a working VM and a broken VM.\nValidate Provider-Specific Advice LLMs can be wrong about provider details.\nIn this session, the model suggested adding unsupported vSphere network interface arguments:\nconnected = true start_connected = true Terraform rejected them:\nError: Unsupported argument That was a useful failure because validation caught it before apply.\nThe rule is simple:\nterraform fmt terraform validate terraform plan Do not trust provider syntax from memory, a blog post, or an LLM. Validate it against the provider you are actually using.\nUse The Model To Preserve The Investigation One of the highest-value uses of an LLM is after the issue is understood.\nAsk it to turn the session into reusable material:\nSummarize this as a field note with symptom, checks, root cause, fix, and operating model. That is how a messy troubleshooting session becomes documentation.\nFrom one Terraform/vSphere investigation, the useful outputs became separate notes:\nTerraform state moves during a module refactor. vSphere cloud-init guestinfo verification. Terraform clone customization failures. netplan conflicts after VMware customization. govc verification commands. iSCSI bootstrap readiness for Kubernetes nodes. first-boot ownership between Packer, Terraform, vSphere, cloud-init, and bootstrap. Each note answers a different operational question. That is better than one giant transcript and better than several posts repeating the same lesson.\nProtect Secrets And Context Operational prompts often include sensitive material by accident.\nBefore sharing content with any LLM, remove or replace:\npasswords. tokens. private keys. customer names if not appropriate. public IPs or internal hostnames if sensitive. real usernames when unnecessary. full state files. Use representative snippets. Keep enough context to debug the issue, but not enough to leak credentials or expose infrastructure unnecessarily.\nGood Prompts For Operators These prompts are useful during troubleshooting:\nList the likely causes ranked by evidence. Do not suggest fixes yet. Give me read-only verification commands first. Mark any command that changes state. What would disprove your current theory? Compare these two outputs and identify the operationally meaningful difference. Turn this diagnosis into a field note: symptom, checks, root cause, fix, and prevention. What advice here depends on provider-specific behavior that I should validate locally? The Operator Still Owns The System LLMs can compress a lot of analysis time. They can also confidently suggest the wrong thing.\nThe operator still owns:\njudgment. blast radius. command execution. validation. rollback. documentation quality. That is not a limitation. That is the right division of labor.\nUse the model to speed up investigation, organize possibilities, and produce clearer notes. Do not use it to skip understanding. Infrastructure work still rewards evidence, small changes, and careful verification.\nThe best outcome is not that the LLM solved the problem. The best outcome is that the operator solved the problem faster and left behind better documentation for the next person.\n","permalink":"https://trinidadmarroquin.com/posts/llm-assisted-infrastructure-troubleshooting/","section":"posts","summary":"LLMs can speed up infrastructure troubleshooting, but only if the operator stays in control.\nThe useful pattern is not \u0026ldquo;ask the model what is wrong and do what it says.\u0026rdquo; That is risky. The better pattern is to treat the model like a second engineer who can help organize evidence, generate hypotheses, suggest verification commands, and turn the final diagnosis into reusable documentation.\nThat distinction matters.\nDuring a recent Terraform and vSphere refactor, an LLM helped connect several layers of the system: Terraform modules, vSphere guest customization, Packer-built templates, cloud-init, netplan, bootstrap scripts, govc, iSCSI, and Kubernetes CSI. That was useful. It also suggested at least one invalid vSphere provider argument. terraform validate caught it.\n","tags":["llm","operations","troubleshooting","systems-thinking","sre"],"title":"Using LLMs During Infrastructure Troubleshooting Without Turning Off Your Brain"},{"categories":["field-notes"],"content":"Terraform can inject cloud-init data into vSphere clones through VMware guestinfo keys. That only works if the template is built to let cloud-init run on first boot.\nSymptom Terraform creates or replaces the VM, but the expected userdata.yaml behavior does not happen. Bootstrap does not run, SSH is not configured, or /var/log/platform-bootstrap.log is missing.\nCheck Guestinfo From vSphere Use govc to inspect the VM extra config:\nsource govc.env govc vm.info -e /DC-Site-A/vm/K8s-Cluster/OpsTools/cluster-a-api-lb-01 \\ | grep \u0026#39;guestinfo\\|disk.enableUUID\u0026#39; Look for keys like:\nguestinfo.metadata guestinfo.metadata.encoding guestinfo.userdata guestinfo.userdata.encoding guestinfo.network-config guestinfo.network-config.encoding If the keys are not present, Terraform did not inject them. Check whether the environment is still using a root-level vsphere_virtual_machine resource instead of the shared module.\nCheck Cloud-Init In The Guest From the VM console or SSH:\ncloud-init status --long sudo tail -n 200 /var/log/cloud-init.log sudo tail -n 200 /var/log/cloud-init-output.log If cloud-init is disabled, you may see:\nstatus: disabled detail: cloud-init disabled by cloud-init-generator In that state, guestinfo can be present but ignored.\nTemplate Requirements The Packer-built template should have:\ncloud-init installed. open-vm-tools installed. no /etc/cloud/cloud-init.disabled file. cloud-init services enabled for cloned VMs. instance state cleaned before templating. VMware datasource available. Useful template checks:\nls -l /etc/cloud/cloud-init.disabled systemctl is-enabled cloud-init-local cloud-init cloud-config cloud-final systemctl is-enabled open-vm-tools Before sealing the template:\nsudo rm -f /etc/cloud/cloud-init.disabled sudo cloud-init clean --logs --machine-id sudo truncate -s 0 /etc/machine-id sudo systemctl enable cloud-init-local cloud-init cloud-config cloud-final open-vm-tools sudo shutdown -h now Terraform Pattern In the module, guestinfo injection usually looks like this:\nextra_config = { \u0026#34;guestinfo.userdata\u0026#34; = base64encode( templatefile(\u0026#34;${var.template_path}/userdata.yaml\u0026#34;, { name = each.value.name ssh_username = var.ssh_username public_key = var.public_key admin_password = var.admin_password }) ) \u0026#34;guestinfo.userdata.encoding\u0026#34; = \u0026#34;base64\u0026#34; } Caution Do not test first-boot cloud-init behavior by enabling cloud-init on an already-customized clone unless the VM is disposable. Cloud-init may re-run identity, user, or network steps and break access.\nFix the template, replace the test VM, and verify from logs.\nIf both vSphere customization and guestinfo network config can set DNS search suffixes, keep them driven by one module input. See Terraform vSphere DNS Search Suffix Ownership.\n","permalink":"https://trinidadmarroquin.com/field-notes/vsphere-cloud-init-guestinfo/","section":"field-notes","summary":"Terraform can inject cloud-init data into vSphere clones through VMware guestinfo keys. That only works if the template is built to let cloud-init run on first boot.\nSymptom Terraform creates or replaces the VM, but the expected userdata.yaml behavior does not happen. Bootstrap does not run, SSH is not configured, or /var/log/platform-bootstrap.log is missing.\nCheck Guestinfo From vSphere Use govc to inspect the VM extra config:\nsource govc.env govc vm.info -e /DC-Site-A/vm/K8s-Cluster/OpsTools/cluster-a-api-lb-01 \\ | grep \u0026#39;guestinfo\\|disk.enableUUID\u0026#39; Look for keys like:\n","tags":["vsphere","cloud-init","terraform","packer"],"title":"vSphere Guestinfo And Cloud-Init On Cloned VMs"},{"categories":["field-notes"],"content":"After an RKE2/Rancher upgrade, several kube-proxy pods entered CrashLoopBackOff. The initial event stream pointed at liveness probe failures:\nWarning Unhealthy kubelet Liveness probe failed: HTTP probe failed with statuscode: 503 Warning BackOff kubelet Back-off restarting failed container kube-proxy At first glance, this looked like a Kubernetes 1.31 kube-proxy probe behavior change or an RKE2 manifest mismatch. The real cause was more operational: UFW was active on a subset of Kubernetes nodes and was interfering with kube-proxy\u0026rsquo;s iptables dataplane programming.\nThe important lesson: kube-proxy was not crashing because the binary was broken. Kubelet was killing it because the health endpoint reported an unhealthy dataplane, and the unhealthy dataplane was caused by host firewall drift.\nSymptom The cluster showed mixed kube-proxy status after upgrade:\nsudo /var/lib/rancher/rke2/bin/kubectl \\ --kubeconfig /etc/rancher/rke2/rke2.yaml \\ -n kube-system get pods -o wide | grep kube-proxy Example pattern:\nkube-proxy-cluster-a-etcd-1 0/1 CrashLoopBackOff kube-proxy-cluster-a-etcd-2 0/1 CrashLoopBackOff kube-proxy-cluster-a-etcd-3 1/1 Running kube-proxy-cluster-a-worker-2 0/1 CrashLoopBackOff kube-proxy-cluster-a-worker-4 0/1 CrashLoopBackOff That mixed state matters. If every kube-proxy pod fails, suspect a cluster-wide configuration issue. If only some nodes fail, suspect node-specific drift: host firewall state, kernel modules, sysctls, iptables backend, conntrack pressure, or node image differences.\nFirst Principle: Use The Node-Local Control Plane When cluster networking is suspect, avoid testing through paths that depend on cluster networking. On an RKE2 server node, use the local kubeconfig and local API endpoint path.\nsudo ss -lntp | egrep \u0026#39;:6443|:2379|:9345\u0026#39; || true sudo /var/lib/rancher/rke2/bin/kubectl \\ --kubeconfig /etc/rancher/rke2/rke2.yaml \\ get --raw=\u0026#39;/readyz?verbose\u0026#39; Healthy output should include checks like:\n[+]ping ok [+]etcd ok [+]etcd-readiness ok [+]informer-sync ok readyz check passed This separates API server and etcd health from service routing, DNS, kube-proxy, CNI, or load balancer issues.\nCheck kube-proxy Health From The Node kube-proxy health endpoints are commonly bound to localhost on the node, often 127.0.0.1:10256. Testing them remotely may fail even when kube-proxy is healthy.\nOn the affected node:\nsudo ss -lntp | egrep \u0026#39;10256|kube-proxy\u0026#39; || true curl -sS -o /dev/null -w \u0026#39;healthz=%{http_code}\\n\u0026#39; \\ http://127.0.0.1:10256/healthz || echo \u0026#39;healthz=connect-failed\u0026#39; curl -sS -o /dev/null -w \u0026#39;livez=%{http_code}\\n\u0026#39; \\ http://127.0.0.1:10256/livez || echo \u0026#39;livez=connect-failed\u0026#39; Interpretation:\nResult Meaning healthz=503, livez=200 Likely Kubernetes 1.31 health endpoint semantics or liveness probe mismatch both connect-failed kube-proxy is not binding the health port or is exiting early both 200 on one node but failures elsewhere node-specific issue; test a failing node directly Kubernetes 1.31 Probe Semantics Check Kubernetes 1.31 added /livez to kube-proxy to preserve liveness-style behavior, while /healthz may report dataplane readiness and can return non-200 when kube-proxy believes the dataplane is stale or unhealthy.\nCheck the probe paths:\nsudo /var/lib/rancher/rke2/bin/kubectl \\ --kubeconfig /etc/rancher/rke2/rke2.yaml \\ -n kube-system get ds kube-proxy \\ -o jsonpath=\u0026#39;{.spec.template.spec.containers[0].livenessProbe.httpGet.path}{\u0026#34;\\n\u0026#34;}\u0026#39; sudo /var/lib/rancher/rke2/bin/kubectl \\ --kubeconfig /etc/rancher/rke2/rke2.yaml \\ -n kube-system get ds kube-proxy \\ -o jsonpath=\u0026#39;{.spec.template.spec.containers[0].startupProbe.httpGet.path}{\u0026#34;\\n\u0026#34;}\u0026#39; A tactical mitigation, if /healthz is flapping while /livez is stable:\nsudo /var/lib/rancher/rke2/bin/kubectl \\ --kubeconfig /etc/rancher/rke2/rke2.yaml \\ -n kube-system patch ds kube-proxy --type=\u0026#39;json\u0026#39; -p=\u0026#39;[ {\u0026#34;op\u0026#34;:\u0026#34;replace\u0026#34;,\u0026#34;path\u0026#34;:\u0026#34;/spec/template/spec/containers/0/livenessProbe/httpGet/path\u0026#34;,\u0026#34;value\u0026#34;:\u0026#34;/livez\u0026#34;}, {\u0026#34;op\u0026#34;:\u0026#34;replace\u0026#34;,\u0026#34;path\u0026#34;:\u0026#34;/spec/template/spec/containers/0/startupProbe/httpGet/path\u0026#34;,\u0026#34;value\u0026#34;:\u0026#34;/livez\u0026#34;} ]\u0026#39; In this incident, probe semantics were worth checking, but they were not the durable root cause.\nPull kube-proxy Logs The logs showed kube-proxy starting cleanly:\nsudo /var/lib/rancher/rke2/bin/kubectl \\ --kubeconfig /etc/rancher/rke2/rke2.yaml \\ -n kube-system logs kube-proxy-\u0026lt;node-name\u0026gt; -c kube-proxy --tail=200 sudo /var/lib/rancher/rke2/bin/kubectl \\ --kubeconfig /etc/rancher/rke2/rke2.yaml \\ -n kube-system logs kube-proxy-\u0026lt;node-name\u0026gt; -c kube-proxy --previous --tail=200 Representative lines:\nSuccessfully retrieved node IP(s) kube-proxy running in dual-stack mode Using iptables Proxier Starting service config controller Starting endpoint slice config controller Starting node config controller Caches are synced That is a clue. If kube-proxy logs look normal and the container still restarts, kubelet may be killing it because the health check fails after startup. In that case, focus on why kube-proxy reports the dataplane unhealthy.\nCheck Node Networking Requirements On a failing node:\n# iptables backend iptables --version || true update-alternatives --display iptables 2\u0026gt;/dev/null | sed -n \u0026#39;1,120p\u0026#39; || true # required sysctls sysctl net.ipv4.ip_forward sysctl net.bridge.bridge-nf-call-iptables 2\u0026gt;/dev/null || true sysctl net.bridge.bridge-nf-call-ip6tables 2\u0026gt;/dev/null || true # common modules lsmod | egrep \u0026#39;br_netfilter|nf_conntrack|ip_tables|iptable_nat|x_tables|ip_vs\u0026#39; || true # conntrack pressure sysctl net.netfilter.nf_conntrack_count 2\u0026gt;/dev/null || true sysctl net.netfilter.nf_conntrack_max 2\u0026gt;/dev/null || true In this incident, the expected modules and sysctls were present, and conntrack pressure was low:\niptables v1.8.10 (nf_tables) net.ipv4.ip_forward = 1 net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 net.netfilter.nf_conntrack_count = 817 net.netfilter.nf_conntrack_max = 131072 That reduced suspicion on kernel module, sysctl, and conntrack exhaustion problems.\nThe Smoking Gun: UFW Was Active Check host firewall managers:\nsudo systemctl is-active firewalld 2\u0026gt;/dev/null || true sudo systemctl is-active ufw 2\u0026gt;/dev/null || true The failing node returned:\ninactive active That second line was the key: UFW was active on the Kubernetes node.\nkube-proxy dynamically programs iptables rules for Services and NodePorts. UFW also manages iptables policy and chains. When both operate on the same node, UFW can interfere with kube-proxy\u0026rsquo;s view of the dataplane through default forwarding policy, rule ordering, chain changes, or reload behavior.\nThe result can look like this:\nUFW changes host firewall policy kube-proxy cannot reliably maintain service rules kube-proxy reports /healthz as unhealthy kubelet kills kube-proxy due to failed liveness probe kube-proxy enters CrashLoopBackOff Confirm The Hypothesis On one failing node only:\nsudo ufw status verbose || true sudo systemctl stop ufw sudo systemctl disable ufw sudo ufw disable || true Then restart the node\u0026rsquo;s RKE2 service or delete the kube-proxy pod.\nFor an RKE2 server node:\nsudo systemctl restart rke2-server For an RKE2 agent node:\nsudo systemctl restart rke2-agent Or delete only the kube-proxy pod from a server node with local kubeconfig:\nsudo /var/lib/rancher/rke2/bin/kubectl \\ --kubeconfig /etc/rancher/rke2/rke2.yaml \\ -n kube-system delete pod kube-proxy-\u0026lt;node-name\u0026gt; Validate:\nsudo /var/lib/rancher/rke2/bin/kubectl \\ --kubeconfig /etc/rancher/rke2/rke2.yaml \\ -n kube-system get pod kube-proxy-\u0026lt;node-name\u0026gt; curl -sS -o /dev/null -w \u0026#39;healthz=%{http_code}\\n\u0026#39; \\ http://127.0.0.1:10256/healthz || echo \u0026#39;healthz=connect-failed\u0026#39; In this case, kube-proxy immediately returned to 1/1 Running with healthz=200 after UFW was disabled. That confirmed the root cause.\nRecovery Rollout Disable UFW across all Kubernetes nodes, one node class at a time.\nRecommended order for an HA RKE2 environment:\netcd nodes, one at a time control-plane/server nodes, one at a time monitor/infra nodes, one at a time worker nodes, failing nodes first On each node:\nsudo ufw status verbose || true sudo systemctl stop ufw sudo systemctl disable ufw sudo ufw disable || true Then restart the appropriate service:\n# RKE2 server node sudo systemctl restart rke2-server # RKE2 agent node sudo systemctl restart rke2-agent Validate after each node:\ncurl -sS -o /dev/null -w \u0026#39;healthz=%{http_code}\\n\u0026#39; \\ http://127.0.0.1:10256/healthz || echo \u0026#39;healthz=connect-failed\u0026#39; Cluster-wide validation:\nsudo /var/lib/rancher/rke2/bin/kubectl \\ --kubeconfig /etc/rancher/rke2/rke2.yaml \\ -n kube-system get pods -o wide | grep kube-proxy All kube-proxy pods should settle at 1/1 Running without climbing restarts.\nRBAC Red Herring One previous kube-proxy log showed a transient RBAC denial:\nFailed to retrieve node info: nodes \u0026#34;\u0026lt;node\u0026gt;\u0026#34; is forbidden: User \u0026#34;system:kube-proxy\u0026#34; cannot get resource \u0026#34;nodes\u0026#34; This looked concerning, but the effective permission check later passed:\nsudo /var/lib/rancher/rke2/bin/kubectl \\ --kubeconfig /etc/rancher/rke2/rke2.yaml \\ auth can-i get nodes --as=system:kube-proxy Expected output:\nyes The cluster had both relevant bindings:\nsystem:kube-proxy roleRef=system:kube-proxy system:node-proxier roleRef=system:node-proxier That made the RBAC denial likely transient during upgrade churn, not the persistent cause of kube-proxy CrashLoopBackOff.\nDurable Prevention Add an explicit control to the node baseline: Kubernetes nodes should not run UFW unless the firewall policy is intentionally designed and tested with the CNI and kube-proxy mode.\nFor most RKE2 nodes, the practical baseline is:\nsudo systemctl stop ufw || true sudo systemctl disable ufw || true sudo ufw disable || true Optional if your baseline allows package removal:\nsudo apt-get purge -y ufw Add this to the image build, cloud-init, Ansible role, or node bootstrap script. Do not rely on manual cleanup after an incident.\nPre-Upgrade Check Before future RKE2 upgrades, run a lightweight node drift check:\nfor node in $(kubectl get nodes -o name | cut -d/ -f2); do echo \u0026#34;=== $node ===\u0026#34; ssh \u0026#34;$node\u0026#34; \u0026#39;systemctl is-active ufw 2\u0026gt;/dev/null || true; systemctl is-active firewalld 2\u0026gt;/dev/null || true\u0026#39; done Flag any node where ufw or firewalld is active and validate whether that is intentional.\nLessons Learned Mixed kube-proxy failures usually point at node drift, not a universal cluster defect. kube-proxy logs can look normal when kubelet is killing the container due to health checks. Test kube-proxy health from the node, not through a path that depends on cluster networking. Kubernetes 1.31 /healthz vs /livez behavior is worth checking, but do not stop there. Host firewall managers like UFW can masquerade as upgrade regressions. Recovery is not complete until the node baseline prevents UFW from returning on reboot or rebuild. References RKE2 Documentation Kubernetes kube-proxy Kubernetes Debug Services Kubernetes System Logs ","permalink":"https://trinidadmarroquin.com/field-notes/rke2-kube-proxy-crashloop-ufw-after-upgrade/","section":"field-notes","summary":"After an RKE2/Rancher upgrade, several kube-proxy pods entered CrashLoopBackOff. The initial event stream pointed at liveness probe failures:\nWarning Unhealthy kubelet Liveness probe failed: HTTP probe failed with statuscode: 503 Warning BackOff kubelet Back-off restarting failed container kube-proxy At first glance, this looked like a Kubernetes 1.31 kube-proxy probe behavior change or an RKE2 manifest mismatch. The real cause was more operational: UFW was active on a subset of Kubernetes nodes and was interfering with kube-proxy\u0026rsquo;s iptables dataplane programming.\n","tags":["rke2","kubernetes","kube-proxy","ufw","iptables","troubleshooting","upgrade"],"title":"RKE2 kube-proxy CrashLoopBackOff After Upgrade Due To UFW"},{"categories":["DevOps Dirty Dozen"],"content":"Part 5 of the DevOps Dirty Dozen Series: Non est instrumentum quod sufficit — the tool alone is not enough.\nInsight: Stresses that tools without collaboration and culture are ineffective.\nDevOps is awash in tooling. CI/CD pipelines, container orchestrators, observability stacks, incident management platforms, infrastructure-as-code engines, security scanners, feature flag systems — the list is nearly infinite. And yet, organizations that invest heavily in the toolchain often find themselves wondering why their DevOps transformation has not delivered the promised results.\nThe culprit is not the tools themselves. It is the belief that tools alone are enough.\nIn this fifth article of the DevOps Dirty Dozen, we examine over-reliance on tools — what it looks like, why it is seductive, and how teams can rebalance their approach to treat tools as enablers rather than solutions.\nOriginally published on LinkedIn.\nTools amplify culture. They do not create it.\nThe Anatomy Of Tool Over-Reliance Over-reliance on tools is easy to spot once you know what to look for. It manifests in behaviors that prioritize tool adoption over the deeper work of building culture, refining process, and fostering collaboration:\nTool-Driven Transformation: A leader declares, \u0026ldquo;We are adopting DevOps,\u0026rdquo; and the first action is purchasing a tool. The assumption is that the tool will create the change, rather than the other way around.\nAutomation Without Understanding: Teams rush to automate workflows using a new tool before understanding whether the process is sound. The tool becomes a bandage over a wound that needs surgery.\nTool Hopping: When outcomes do not improve, the response is not to examine culture or process. It is to replace the tool. \u0026ldquo;If only we used X instead of Y, everything would work.\u0026rdquo;\nProcess by Configuration: Teams assume that configuring a tool to enforce a workflow is equivalent to building a healthy process. Configuration replaces conversation.\nMetrics Without Meaning: Tools generate dashboards and reports, but nobody acts on them. The data exists because the tool produces it, not because the team has a practice of using it.\nA tool cannot fix what only people and process can.\nThe Cost Of Over-Reliance On Tools The cost of treating tools as solutions rather than enablers is subtle but significant:\nCultural Debt: When tools are adopted without addressing underlying trust, collaboration, and safety issues, the cultural problems persist. The tool just masks them.\nFalse Confidence: A sophisticated monitoring stack can create the illusion of observability. The team has dashboards but still cannot answer basic questions about system behavior during an incident.\nWasted Investment: Tools that are not embedded in a healthy culture and clear process will be underutilized, misconfigured, or abandoned. The licensing cost is the smallest part of the loss.\nProcess Blindness: When a tool automates a bad process, the process becomes invisible. Teams stop questioning whether the workflow makes sense because \u0026ldquo;the tool handles it.\u0026rdquo;\nCollaboration Theater: Collaboration tools create channels, boards, and threads that simulate communication without producing shared understanding. The artifact of collaboration replaces the act.\nTools can make you feel like you have arrived. Culture is the next climb.\nA Real-World Example: The Incident Management Platform That Changed Nothing Consider a team that implemented a well-regarded incident management platform. It brought on-call scheduling, automated alert routing, status pages, and postmortem templates. The tooling was best in class.\nBut the organization still had a blame culture. Postmortems were exercises in identifying who made the error. The on-call rotation was a source of anxiety, not ownership. Alerts were tuned to avoid waking anyone rather than signaling real problems.\nThe platform itself was excellent. It did everything it was designed to do. The team had simply expected the tool to solve problems that were cultural and procedural, not technical. A year later, the platform was in use, but incident response had not improved. The tool was not the missing piece — psychological safety, clear escalation policies, and a learning-oriented postmortem practice were.\nWhy Over-Reliance On Tools Occurs Over-reliance on tools is not a sign of laziness. It is a pattern driven by systemic incentives:\nTools Are Tangible: Buying a tool is a visible, measurable action. It generates announcements, blog posts, and resume lines. Improving culture is invisible and slow.\nTool Procurement Is Easier Than Change Management: It is easier to get budget approval for a tool than to shift organizational behavior. Tools fit into existing procurement processes. Culture change does not.\nVendor Narratives Are Compelling: Every tool vendor promises transformation. The marketing speaks directly to the pain, and the pitch is tailored to make the tool seem like the missing link.\nFear Of Obsolescence: Teams worry that not adopting the latest tool will leave them behind. The fear of being perceived as outdated drives adoption without reflection.\nTool Fatigue Masks Deeper Issues: When teams are overwhelmed by tool choices, they default to the belief that they simply have not found the right tool yet. The search itself becomes a distraction from examining culture and process.\nThe search for the perfect tool is a symptom, not a strategy.\nRestoring Balance: Tools As Enablers, Not Solutions The goal is not to abandon tools. It is to restore them to their proper role as enablers of good culture and process.\nStart With Culture And Process: Before selecting a tool, define the behavior you want to enable. What does good collaboration look like? What does a healthy incident response feel like? The tool should reinforce behaviors that already exist or are being built.\nAdopt Tools Slowly, Retire Them Deliberately: Every tool adoption should include a hypothesis. What problem is this solving? How will we know if it is working? And when will we revisit that decision? Adopting a tool is not a terminal decision.\nMeasure What Matters, Not What Is Easy: The tool can generate data. The team must decide what is meaningful. Lead time, deployment frequency, mean time to recovery, and change failure rate matter more than dashboard count or alert volume.\nInvest In Practice, Not Just Platform: Training, runbooks, incident drills, and postmortem habits matter more than the tooling that supports them. A team with strong practices and mediocre tools will outperform a team with weak practices and excellent tools.\nTreat Tools As Accountable Infrastructure: Every tool should have an owner who is responsible not just for uptime, but for whether the tool is actually improving outcomes. If it is not, the tool should be removed.\nTools, culture, and process support the weight together. Remove one leg, and the whole thing tips.\nApplying The Scientific Method Over-reliance on tools can be countered by treating tool adoption as a hypothesis to be tested:\nAsk Questions: What specific behavior are we trying to change? Is this a tool problem, a process problem, or a culture problem?\nGather Data: Before adopting a tool, measure the current state. How long do incidents take to resolve? How often do deployments fail? What is the team\u0026rsquo;s sense of psychological safety?\nForm Hypotheses: \u0026ldquo;If we adopt X tool, we expect Y outcome within Z time.\u0026rdquo; Be specific. The hypothesis makes the tool accountable.\nTest And Observe: Implement the tool with a clear pilot scope. Measure the same metrics. Did the outcome improve? If not, was the problem the tool, or the assumption that a tool could fix it?\nIterate: Based on evidence, decide whether to expand, adjust, or sunset the tool. The question is never \u0026ldquo;Is this tool good?\u0026rdquo; It is \u0026ldquo;Is this tool making things better?\u0026rdquo;\nCarl Sagan\u0026rsquo;s Baloney Detection Kit Before adopting any tool, apply critical thinking to evaluate the claims:\nQuestion Assumptions: Is the tool being adopted because it solves a real problem, or because it is the current industry trend? Is the problem technical or cultural?\nSeek Evidence: What data supports the claim that this tool will improve outcomes? Are there case studies from similar organizations? Or is the evidence anecdotal?\nTest Before Scaling: Does the tool work in your context? A proof of concept in a controlled environment can reveal mismatches before widespread adoption.\nConsider Alternative Explanations: Could the same outcome be achieved with process changes, training, or better communication? Is the tool solving a symptom rather than a root cause?\nAvoid Bandwagon Fallacy: Adoption by peers and industry leaders does not mean the tool is right for your team. Every organization has unique constraints.\nThe best tool decision is the one you can articulate without buzzwords.\nMoving Forward Together Tools are an essential part of any DevOps practice. They automate, monitor, orchestrate, and inform. But they are not a substitute for the harder work of building a collaborative culture, designing clear processes, and fostering psychological safety.\nThe teams that succeed with DevOps are not the ones with the most sophisticated toolchains. They are the ones where the tools fade into the background, supporting the work without dominating the conversation. The tool is never the hero. The team is.\nHas your organization fallen into the trap of relying on tools to solve cultural or process problems? What helped you restore balance? Share your experiences as we continue through the DevOps Dirty Dozen.\nReferences The DevOps Handbook by Gene Kim, Patrick Debois, John Willis, and Jez Humble Accelerate: The Science of Lean Software and DevOps by Nicole Forsgren, Jez Humble, and Gene Kim DORA: Generative organizational culture Team Topologies by Matthew Skelton and Manuel Pais Conway\u0026rsquo;s Law — Melvin Conway HBR: Culture Eats Strategy For Breakfast The Three Ways: Principles Underpinning DevOps ","permalink":"https://trinidadmarroquin.com/posts/devops-dirty-dozen/over-reliance-on-tools/","section":"posts","summary":"Part 5 of the DevOps Dirty Dozen Series: Non est instrumentum quod sufficit — the tool alone is not enough.\nInsight: Stresses that tools without collaboration and culture are ineffective.\nDevOps is awash in tooling. CI/CD pipelines, container orchestrators, observability stacks, incident management platforms, infrastructure-as-code engines, security scanners, feature flag systems — the list is nearly infinite. And yet, organizations that invest heavily in the toolchain often find themselves wondering why their DevOps transformation has not delivered the promised results.\n","tags":["devops","sre","tools","culture","process","collaboration"],"title":"Tools Are Not Enough: The DevOps Over-Reliance on Tools Anti-Pattern"},{"categories":["DevOps Dirty Dozen"],"content":"Part 4 of the DevOps Dirty Dozen Series: Homo homini lupus — man is a wolf to man.\nInsight: Reflects the destructive nature of blame within teams.\nDevOps is supposed to create an atmosphere of learning from mistakes, free from finger-pointing and blame. After all, the promise of automation removes the blame on individuals. If something went wrong, the process and automation has a gap, and the team must work together to close it.\nCollaboration, accountability, and trust are the lifeblood of DevOps. Yet, when things go wrong, it is easy to fall into the blame trap. Blame culture arises when failure prompts finger-pointing rather than problem-solving, stalling progress and poisoning team dynamics.\nIn this fourth article of the DevOps Dirty Dozen, we delve into DevOps blame culture — its origin, its cost, and recommendations to focus on improvement, learning, collaboration, and innovation.\nOriginally published on LinkedIn.\nBlame culture grows deep. Pulling it out requires digging at the roots.\nThe Anatomy Of Blame Culture Blame culture does not manifest overnight. It creeps in subtly, often unnoticed, until it becomes ingrained in an organization\u0026rsquo;s DNA. Here is what it looks like in practice:\nFear-Based Leadership: Leaders focus on assigning fault during failures, fostering fear of reprimand over constructive feedback. This stifles open communication and learning. Silos of Defensiveness: Teams retreat into protective bubbles, reluctant to share progress or risks for fear of being blamed. Instead of collaboration, competition and mistrust thrive. Reactive Crisis Management: When incidents occur, the first instinct is to hunt for scapegoats rather than solutions. Time is wasted debating fault while the real issue lingers unresolved. Performance-Over-Process Focus: Teams are evaluated based on individual performance, making them more inclined to deflect responsibility to avoid being labeled as \u0026ldquo;the problem.\u0026rdquo; Blameless in Words, Not Actions: Organizations may say they support blameless postmortems, but when actions do not align, a blame culture persists under the surface. At its core, blame culture is a symptom of broken systems and poor leadership. By recognizing its anatomy, organizations can begin dismantling its foundations.\nBlame is not a productivity tool. It is a productivity killer.\nThe Cost Of Blame Culture Blame culture is not just a minor inconvenience. It is a productivity killer that eats away at innovation, collaboration, and morale. The ripple effects can impact everything from team dynamics to customer satisfaction.\nStifles Innovation: Fear of failure discourages experimentation and creativity. Breaks Collaboration: Finger-pointing fosters hostility and fractures team unity. Slows Incident Resolution: Energy is spent assigning fault rather than addressing root causes. Reduces Morale: Persistent blame culture leads to burnout and disengagement. Real-World Example: A retail company struggling with frequent outages conducted punitive postmortems, leading to mistrust among teams. By shifting to blameless retrospectives and focusing on systemic improvements, they fostered a culture of shared responsibility and drastically reduced incident frequency.\nThe cost of blame extends beyond the immediate; it hinders growth, breeds distrust, and leaves a trail of missed opportunities. Recognizing these consequences is the first step toward building a healthier, more collaborative culture.\nThe cycle repeats until someone chooses to break it with trust.\nThe Origins Of Blame Culture Blame culture does not materialize out of nowhere. It is often the result of systemic weaknesses and ingrained behaviors that go unaddressed for too long.\nLack of Psychological Safety: When teams fear retaliation for mistakes, they are less likely to take risks or share honest feedback. Overemphasis on Accountability: While accountability is essential, it can morph into blame if not paired with a culture of learning. Poor Incident Management: A reactive, punitive approach to incidents fosters mistrust and defensiveness. Historical Biases: Legacy behaviors or leadership styles can perpetuate blame cycles. By identifying where blame culture starts — whether in leadership practices, historical biases, or poor incident management — teams can shift their focus toward creating an environment where trust and accountability thrive.\nTransformation is not a policy change. It is a gear shift in how the team thinks.\nTransforming Blame Into Growth Blame does not have to be the end of the story. With intentional action, it can be the spark for transformation, turning finger-pointing into opportunities for learning and collaboration.\nBuild Psychological Safety: Foster an environment where mistakes are viewed as opportunities for learning, not punishment. Shift the Focus: Replace \u0026ldquo;Who caused this?\u0026rdquo; with \u0026ldquo;What caused this?\u0026rdquo; and \u0026ldquo;How can we prevent it?\u0026rdquo; Encourage Postmortems: Conduct blameless retrospectives to identify systemic issues and improve processes. Celebrate Collaboration: Reward teams for working together to solve problems rather than highlighting individual contributions. When teams replace blame with trust, curiosity, and systemic improvement, they create an environment where people are empowered to innovate, experiment, and grow. The key is to foster a culture that sees failure as a stepping stone, not a dead end.\nApplying The Scientific Method To Blame Culture Tackling blame culture requires more than good intention. It demands a structured, evidence-based approach. By applying the principles of scientific inquiry, teams can move from assumption-driven reactions to data-informed solutions.\nChallenge Biases: Are we attributing failure to individuals rather than systemic flaws? Encourage Diverse Perspectives: What can we learn from different team members about the issue? Test Hypotheses: How can we address root causes and measure improvement? With a scientific mindset, teams can break free from the blame loop, focusing on objective analysis and actionable improvements. It is not about finding fault — it is about finding a better way forward.\nCarl Sagan\u0026rsquo;s Baloney Detection Kit To escape the blame trap, teams must develop a mindset rooted in critical thinking. Carl Sagan\u0026rsquo;s famous Baloney Detection Kit can help teams evaluate how blame culture perpetuates and how to move beyond it:\nQuestion Assumptions: Are we assuming individual incompetence caused the issue, or are there systemic weaknesses at play? Is this really a \u0026ldquo;people problem,\u0026rdquo; or are workflows, tools, or expectations contributing? Follow the Evidence: What does the data tell us about the incident? Are we relying on hard facts or hearsay to assign responsibility? Have we explored historical patterns to identify recurring issues? Encourage Diverse Perspectives: Are all stakeholders involved in the discussion? Input from multiple perspectives often highlights overlooked root causes. Are we listening to quieter voices or just deferring to the loudest or most senior person? Test Hypotheses: Instead of assuming blame, can we test a hypothesis? For example, \u0026ldquo;If we adjust X process, will it eliminate Y issue?\u0026rdquo; Are we iterating and improving, or just reacting to problems on a surface level? Avoid Logical Fallacies: Are we falling for the ad hominem trap, blaming individuals instead of addressing the real cause? Is hindsight bias coloring our postmortem analysis? By applying these principles, teams can reframe failure as a learning opportunity rather than a blame game, fostering growth and collaboration.\n\u0026lsquo;Homo homini lupus\u0026rsquo; does not have to be the final word. Teams can choose a different way forward.\nMoving Forward Together Blame culture is one of the most insidious DevOps anti-patterns, but it is also one of the most addressable. The journey to transformation starts with a shared commitment to growth, accountability, and systemic improvement.\nBlame culture is a silent killer, eroding trust, stalling progress, and sapping the morale of even the most talented teams. But it does not have to be this way. With deliberate effort to build psychological safety, focus on systemic improvement, and embrace a culture of learning, organizations can replace the blame game with collaboration and innovation.\nHave you encountered blame culture in your organization? How did it affect your teams, and what strategies helped turn the tide? Share your experiences as we continue uncovering the DevOps Dirty Dozen — one anti-pattern at a time.\nReferences HBR: Blame Culture Is Toxic. Here\u0026rsquo;s How to Stop It. Killer Innovations: The Blame Culture and How It Kills Innovation Psychology Today: Do You Work in a Blame Culture? ACM Agile: Blame Culture The DevOps Handbook by Gene Kim, Patrick Debois, John Willis, and Jez Humble DORA: Generative organizational culture Etsy\u0026rsquo;s Debriefing Facilitation Guide ","permalink":"https://trinidadmarroquin.com/posts/devops-dirty-dozen/transforming-blame-culture/","section":"posts","summary":"Part 4 of the DevOps Dirty Dozen Series: Homo homini lupus — man is a wolf to man.\nInsight: Reflects the destructive nature of blame within teams.\nDevOps is supposed to create an atmosphere of learning from mistakes, free from finger-pointing and blame. After all, the promise of automation removes the blame on individuals. If something went wrong, the process and automation has a gap, and the team must work together to close it.\n","tags":["devops","sre","blame-culture","psychological-safety","incidents","systems-thinking"],"title":"Transforming Blame Culture: The DevOps Silent Productivity Killer Anti-Pattern"},{"categories":["DevOps Dirty Dozen"],"content":"Part 3 of the DevOps Dirty Dozen Series: Festina lente — make haste slowly.\nInsight: Encourages careful planning before rushing into automation.\nDevOps was supposed to guarantee seamless automation, but too often, it can bring forth disarray and disorder. Automation meant to usher in stable pipelines and repeatable processes can wreak havoc on deployments and undermine the very principles DevOps was built upon. Why does chaos seem to persist? What makes it so destructive? And how can we bring to bear essential tools to deliver on DevOps\u0026rsquo; promise of stable and repeatable deployments?\nOriginally published on LinkedIn.\nAutomation is powerful. Automating the wrong thing makes the problem worse, faster.\nThe Anatomy Of Automation Chaos Automation\u0026rsquo;s potential is immense, but its misuse opens a Pandora\u0026rsquo;s Box of unintended consequences. Poorly planned automation is like building on a shaky foundation — it magnifies inefficiencies, accelerates failure, and creates complexities that are difficult to unwind. This anti-pattern, often born from haste, reflects the classic mistake of addressing symptoms rather than root causes.\nKey drivers of automating chaos include:\nBand-Aid Solutions: Teams under pressure often rush to automate problematic workflows instead of redesigning them. While this may provide a short-term win, it locks in inefficiencies and perpetuates dysfunction. Lack of Process Clarity: Automation efforts that skip the crucial step of mapping out workflows risk embedding errors into the system, making them more difficult to identify and resolve later. Overconfidence in Tools: A shiny new automation tool can tempt teams to implement it prematurely, leading to poorly planned integrations. Tool Overload: Too many automation tools, implemented without a cohesive strategy, create overlapping processes and obscure accountability. Siloed Decisions: Automation strategies designed without cross-team input often ignore downstream impacts, creating bottlenecks and misalignment. Automating a flawed process amplifies the flaws. Fix the process first, then automate.\nThe Cost Of Automating Chaos The aftermath of automating chaos involves significant time and resources dedicated to troubleshooting, repairing, and reworking automated processes. This reactive approach means teams spend more time fixing what is broken rather than pushing forward on new features or improvements.\nWhen automation goes awry, the fallout can be significant:\nFaster Failures: Automation amplifies the speed of flawed processes, turning small inefficiencies into large-scale problems. Erosion of Confidence: When automation leads to disruptions rather than improvements, trust in DevOps practices and leadership diminishes. Increased Technical Debt: Quick fixes add layers of complexity, creating a brittle system that becomes harder to maintain over time. Lost Time and Resources: Efforts to fix poorly automated workflows divert attention from innovation and value-driven work. Speed is not the goal. Correctness at speed is.\nWhy Chaos Happens In The First Place In the fast-paced tech industry, there is often intense pressure to release products or updates swiftly. This urgency can lead teams to adopt automation solutions hastily, without thorough planning or consideration of long-term impacts.\nPressure to Deliver Quickly: Teams often rush into automation under tight deadlines, prioritizing speed over strategy. Leadership Blind Spots: Leaders may view automation as a universal fix, without understanding the underlying process issues. Inadequate Training: Teams may lack the expertise to evaluate processes critically before automating them. Tool Mismanagement: Adopting tools without clear ownership or alignment leads to fragmented automation efforts. Resistance to Change: Teams may automate outdated processes because redesigning them feels too disruptive. Breaking Down Chaos Before any automation tool is implemented, a comprehensive analysis of current workflows is necessary. This involves mapping out each step and understanding who is responsible for what, with the end goal of identifying bottlenecks or redundancies. Teams should use process mining techniques or workflow analysis to gather data on how tasks are currently performed. This step is crucial to ensure that automation targets the right areas for improvement and does not merely automate chaos. It is about asking, \u0026ldquo;Is this process even worth automating?\u0026rdquo; or \u0026ldquo;Can we streamline this before automation?\u0026rdquo;\nTo avoid automating chaos, teams must focus on discipline and collaboration:\nUnderstand Before Automating: Conduct thorough assessments of workflows to identify inefficiencies and unnecessary steps before introducing automation. Start Small and Scale Thoughtfully: Begin with automating a small, well-understood process and monitor its impact before expanding. Prioritize Cross-Team Collaboration: Ensure all relevant stakeholders are involved in automation decisions to prevent siloed thinking and unintended consequences. Invest in Observability: Implement monitoring and logging tools that provide insight into automated processes, helping to identify and address issues early. Treat automation as an experiment: understand the problem, test the solution, and refine based on evidence.\nApplying The Scientific Method The initial step is about setting the foundation for why automation is even being considered. Teams should ask critical questions like, \u0026ldquo;What specific problem are we trying to solve with this automation?\u0026rdquo; or \u0026ldquo;Is the process we are looking at actually a bottleneck, or does it just feel inefficient due to lack of understanding?\u0026rdquo; These questions help focus efforts and ensure that automation addresses real, not perceived, issues.\nThe scientific method can serve as a guide for effective automation:\nAsk Questions: What problem are we solving? Is this process worth automating? Gather Data: Analyze workflows and identify inefficiencies before automating. Form Hypotheses: Define the expected outcomes of automation. Test and Observe: Pilot automation on a small scale and evaluate its effectiveness. Iterate: Refine workflows and automation based on feedback and outcomes. Carl Sagan\u0026rsquo;s Baloney Detection Kit Before automating any process, it is crucial to challenge the fundamental assumptions about why automation is needed. Is the process actually inefficient, or does it only seem so because of lack of understanding or training? Is there a possibility that the current process is fundamentally flawed and needs redesign rather than automation? This step involves critically examining the status quo, questioning whether automation is the right response or if it is just a band-aid for deeper issues.\nAs Carl Sagan\u0026rsquo;s Baloney Detection Kit suggests, critical thinking is essential:\nQuestion Assumptions: Is this process worth automating, or is it fundamentally flawed? Seek Evidence: What data supports the decision to automate? Are the benefits measurable? Test Before Scaling: Are there simpler solutions that achieve the same result? Can automation be introduced incrementally? Process clarity first. Automation second. Reliability always.\nMoving Forward Together Automation in the context of DevOps holds transformative potential, promising to streamline operations, reduce human error, and drive innovation. However, to harness this potential without descending into chaos, a structured and thoughtful approach is imperative:\nUnderstand the Problem: Before any automation tool is implemented, it is crucial to have a deep understanding of the problem space. Is it due to poor process design, lack of training, or an over-reliance on manual processes where automation could genuinely improve matters? Build Stable Processes: Automation should not be the first step in process improvement. Instead, it should come after processes have been refined and stabilized. Map out workflows, eliminate unnecessary steps, and ensure the process is as lean as possible before automating it. Foster Collaboration: Automation decisions should not be made in isolation. Involvement from various teams — development, operations, quality assurance, security, and business stakeholders — ensures that automated solutions are holistic and consider the entire system\u0026rsquo;s health. Continuous Improvement: Automation should be part of a continuous improvement cycle. Post-implementation, teams should regularly revisit automated processes to tweak, optimize, or reconsider automation if the benefits are not as expected. This disciplined approach can help organizations avoid the pitfalls of automating chaos, leading to stable, repeatable, and efficient workflows that embody the true spirit of DevOps.\nWhat is your experience with automation gone awry? Have you witnessed the consequences of automating chaos, or found strategies to prevent it? Let us share insights and solutions as we continue breaking down the DevOps Dirty Dozen.\nReferences The DevOps Handbook by Gene Kim, Patrick Debois, John Willis, and Jez Humble Continuous Delivery by Jez Humble and David Farley The Three Ways: Principles Underpinning DevOps DevOps Institute Arrested DevOps Podcast ","permalink":"https://trinidadmarroquin.com/posts/devops-dirty-dozen/closing-pandoras-box-automating-chaos/","section":"posts","summary":"Part 3 of the DevOps Dirty Dozen Series: Festina lente — make haste slowly.\nInsight: Encourages careful planning before rushing into automation.\nDevOps was supposed to guarantee seamless automation, but too often, it can bring forth disarray and disorder. Automation meant to usher in stable pipelines and repeatable processes can wreak havoc on deployments and undermine the very principles DevOps was built upon. Why does chaos seem to persist? What makes it so destructive? And how can we bring to bear essential tools to deliver on DevOps\u0026rsquo; promise of stable and repeatable deployments?\n","tags":["devops","sre","automation","process","systems-thinking"],"title":"Closing Pandora's Box: The DevOps Automating Chaos Anti-Pattern"},{"categories":["DevOps Dirty Dozen"],"content":"Part 2 of the DevOps Dirty Dozen Series: Non multa, sed multum — not many, but much.\nInsight: An excess of tools fragments focus. Depth and coherence matter more than choice.\nDevOps was supposed to simplify delivery. Instead, it has delivered a paradox: the more tools we adopt, the less we seem to accomplish. Teams spend more time evaluating, configuring, and integrating tools than shipping value. The promise of \u0026ldquo;best in class\u0026rdquo; per category has produced stacks that no single person fully understands.\nIn this second article of the DevOps Dirty Dozen, we examine tool overload — how it happens, why it persists, and how to reclaim focus without sacrificing capability.\nOriginally published on LinkedIn.\nEach tool made sense at the time. Together, they become something nobody owns.\nThe Anatomy Of Tool Overload Tool overload is not simply having too many tools. It is what happens when tool diversity outpaces the team\u0026rsquo;s ability to maintain coherence. Here is what it looks like:\nDuplicated capability: Three monitoring tools, two CI systems, four secrets management approaches — each justified by a different team or season. Context fragmentation: Engineers context-switch between a dozen UIs and CLIs just to push a single change from commit to production. Integration debt: Every tool needs to talk to every other tool. The glue code, webhook wiring, and credential exchange become the real infrastructure. Onboarding friction: New team members do not learn one system. They learn eight. And the seventh was already deprecated by the second week. These patterns do not look broken on day one. They accumulate.\nContext switching is not free. Every window is a tax on mental bandwidth.\nThe Cost Of Tool Overload Tool overload does not fail loudly. It erodes quietly.\nCognitive load compounds: Each tool brings its own syntax, domain model, error messages, and failure modes. Switching between Terraform HCL, Helm templates, Prometheus recording rules, and Concourse pipeline YAML costs mental energy that never goes to problem-solving. Onboarding slows to a crawl: A new engineer does not need to learn a stack. They need to learn a constellation. The undocumented quirks — that one pipeline only works with a specific image tag, this dashboard breaks if the namespace is too long — live in institutional memory, not documentation. Incident response degrades: When everything breaks at 2 AM, the on-call engineer must decide which of five monitoring tools to trust, which of three log sources to query, and which runbook applies to a stack that has not been touched in six months. Security surface area expands: Every integration point between tools is a potential misconfiguration. Every unused tool with default credentials is a finding waiting to be discovered. License and maintenance costs multiply: Teams pay for overlapping SaaS seats, maintain bespoke integration scripts, and carry the operational burden of software they barely use. Analysis paralysis is not laziness. It is the system asking for a decision nobody made.\nA Real-World Example: The Microservices Monitoring Spiral Consider a team that starts with Prometheus and Grafana for monitoring. A solid choice. Then a team member attends a conference and discovers Datadog. It looks simpler. Management approves a trial. Now there are two monitoring systems.\nThe Prometheus setup has custom alerting rules tuned over two years. The Datadog trial has a cleaner UI but does not cover everything. Neither gets fully decommissioned. Both require maintenance. Some dashboards are in Grafana, some in Datadog. The on-call rotation checks both during incidents, because nobody is sure which one has the complete picture.\nA year later, a third tool appears for tracing. A fourth for logs. Each justified on its own merits. The team now maintains four observability systems, none of which are authoritative.\nThis is not a tool problem. It is a governance problem that looks like a tool problem.\nWhy Tool Overload Occurs Tool overload does not happen because teams are undisciplined. It happens because the incentives favor addition over subtraction.\nNo decision is a decision: When there is no clear standard for tool selection, every team makes its own choice. The result is fragmentation without anyone intending it. Addition is rewarded: Adding a tool feels proactive. It shows initiative. Removing a tool feels risky and generates no visibility. Vendor gravity pulls hard: Sales cycles, conference demos, and proof-of-concept trials create momentum that is easier to start than to stop. Fear of missing out: The ecosystem evolves fast. Teams adopt tools to stay current, even when the current stack works well enough. Sunk cost fallacy: Once a tool has been integrated, removing it requires admitting the initial investment was not worth it. Most teams avoid that conversation. If you don\u0026rsquo;t know where you are going, any tool will get you there.\nTaming The Toolchain Recovering from tool overload requires intentional subtraction, not just better selection criteria.\nInventory everything: Before deciding what to keep, document what exists. List every tool, who owns it, what problem it solves, and what would break if it disappeared. Define the paved road: Choose one tool per category for the standard path. CI/CD. Monitoring. Secrets. Logging. Teams can deviate, but the default should be clear. Retire in parallel: Removing a tool does not mean an overnight migration. It means freezing new usage, documenting the sunset, and setting a removal date within a reasonable window. Tie ownership to removal: Every tool should have an owner who is also responsible for its deprecation plan. If nobody owns the sunset, the tool never leaves. Measure by outcomes, not tool count: The goal is not zero tools. It is coherence. If the team can ship, debug, and recover without heroic effort, the toolchain is probably right-sized. Applying Chesterton\u0026rsquo;s Fence Before removing any tool, understand why it was added. Chesterton\u0026rsquo;s Fence principle says: do not remove a fence until you understand why it was put there in the first place.\nAsk why the tool exists: Was it solving a real problem, or was it a trial that never ended? Check what depends on it: Integration points, scripts, dashboards, and documentation may rely on a tool that nobody thinks about. Understand the removal cost: Sunsetting a tool is not free. The cost of migration, retraining, and runbook updates must be included in the decision. This is not an argument against change. It is an argument against removal by neglect — treating a tool as dead because nobody remembers why it was added, even when it still serves a purpose.\nA platform is not a restriction. It is a decision already made so the team can move faster.\nMoving Forward Tool overload is not solved by finding the perfect tool. It is solved by deciding what matters and standardizing around it. The teams that ship fastest are not the ones with the most tools. They are the ones whose tools fade into the background, letting the work itself take center stage.\nWhat is your experience with tool overload? Which tool in your stack would you remove today if you could?\nReferences The Three Ways: Principles Underpinning DevOps Team Topologies: Cognitive Load And Team-First Thinking Platform engineering: Gartner\u0026rsquo;s definition and market guide Barry Schwartz, The Paradox of Choice DORA: Generative organizational culture Chesterton\u0026rsquo;s Fence: A Principle Of Thinking ","permalink":"https://trinidadmarroquin.com/posts/devops-dirty-dozen/drowning-in-choices-tool-overload/","section":"posts","summary":"Part 2 of the DevOps Dirty Dozen Series: Non multa, sed multum — not many, but much.\nInsight: An excess of tools fragments focus. Depth and coherence matter more than choice.\nDevOps was supposed to simplify delivery. Instead, it has delivered a paradox: the more tools we adopt, the less we seem to accomplish. Teams spend more time evaluating, configuring, and integrating tools than shipping value. The promise of \u0026ldquo;best in class\u0026rdquo; per category has produced stacks that no single person fully understands.\n","tags":["devops","sre","tools","platform-engineering","cognitive-load"],"title":"Drowning in Choices: The DevOps Tool Overload Anti-Pattern"},{"categories":["DevOps Dirty Dozen"],"content":"Part 1 of the DevOps Dirty Dozen Series: Divide et impera — divide and conquer.\nInsight: Creating silos fragments unity and purpose.\nDevOps was supposed to unite us, but too often, it divides us into silos. Silos isolate teams, stifle communication, and undermine the very principles DevOps was built upon. Why do these silos exist? What makes them so destructive? And how can we tear them down to deliver on DevOps\u0026rsquo; promise of collaboration and innovation?\nIn this first article of the DevOps Dirty Dozen, we delve into DevOps silos, their origins, their impact, and strategies to break free.\nOriginally published on LinkedIn.\nSilos leave teams close enough to see each other, but too disconnected to move together.\nThe Anatomy Of Silos Silos are more than physical barriers. They are ingrained mindsets and practices that divide teams and departments. Here is what they look like:\nOffice silos: Physical separation between teams creates barriers to collaboration. For example, when the development team is on one floor and operations is on another, communication naturally dwindles. Functional silos: Teams work in isolation based on their roles. Development, Security, and Operations often act as independent entities rather than cohesive parts of a whole. Generational silos: Misunderstandings arise between younger and more seasoned staff, with each group perceiving the other as less effective. These silos create invisible walls that hinder the flow of information, innovation, and progress.\nSiloed work creates extra paths, duplicate effort, and slow feedback.\nThe Cost Of Silos Silos do not just inconvenience teams. They actively harm organizations.\nWasted time and redundancy: Teams solve the same problems differently, duplicating effort and wasting resources. Communication dead zones make this worse. Missed opportunities: Without cross-team collaboration, work in one department that could expedite solutions elsewhere often goes unnoticed. Unhealthy competition: Teams compete for resources and recognition, prioritizing their success over the organization\u0026rsquo;s goals. Poor customer experience: Disconnected teams result in fragmented solutions, leaving clients to navigate internal inefficiencies. Lower morale and trust: Tribalism fosters disengagement. Teams feel isolated and disconnected from the organization\u0026rsquo;s greater mission. A Real-World Example: The Fallout Of Siloed Teams Consider the infamous Healthcare.gov launch in 2013. The platform\u0026rsquo;s development involved multiple contractors and government agencies, each working in isolation without proper communication or integration. These silos led to catastrophic results: system crashes, a poor user experience, and political backlash.\nThe lack of collaboration between teams meant critical integration testing was not completed, leaving the system vulnerable to failure under real-world conditions. This high-profile failure highlighted how silos can derail even the most critical initiatives.\nLeadership blind spots can preserve silos even when everyone is trying to improve delivery.\nWhy Silos Develop In The First Place Silos are not built overnight. They emerge from systemic issues. Here are seven common causes:\nLeadership blind spots: Leaders unknowingly manage silos instead of dismantling them, focusing on isolated metrics rather than holistic collaboration. Incentive structures: Misaligned goals encourage teams to prioritize their success over the organization\u0026rsquo;s objectives. Lack of communication systems: Without shared channels for regular interaction, teams resort to information hoarding. Fear: Teams isolate themselves out of fear: fear of failure, criticism, or overstepping boundaries. Cultural inertia: \u0026ldquo;We have always done it this way\u0026rdquo; perpetuates division. Resource scarcity: Competing for limited resources reinforces isolation. Over-specialization: Highly specialized teams focus narrowly on their domain, neglecting the broader organizational picture. Collaboration turns isolated expertise into shared learning and better outcomes.\nBreaking Down Silos To eliminate silos, organizations need a structured approach:\nFoster open communication: Create shared channels and forums for regular cross-team interaction. Align incentives: Design goals that reward collaboration and shared success. Empower leadership: Equip leaders to recognize and address silo behaviors. Encourage transparency: Share successes and failures openly to build trust. Invest in cross-functional projects: Encourage teams to collaborate on initiatives that span multiple disciplines. Treat organizational improvement as an experiment: ask, test, measure, and adapt.\nApplying The Scientific Method Breaking silos requires experimentation and critical thinking, much like the scientific method:\nAsk questions: Why do silos exist in your organization? What is their impact? Form hypotheses: What specific actions could reduce or eliminate silos? Test solutions: Pilot changes like cross-team meetings or shared goals. Analyze results: Did these changes improve collaboration and outcomes? Refine as needed. Clear evidence helps teams challenge assumptions and move through uncertainty.\nCarl Sagan\u0026rsquo;s Baloney Detection Kit To challenge assumptions about silos, apply Carl Sagan\u0026rsquo;s Baloney Detection Kit:\nEncourage debate: Are silos truly necessary, or are they remnants of outdated thinking? Demand evidence: What data supports the effectiveness of silos or their removal? Test assumptions: Experiment with breaking silos and measure the results. Breaking silos is not a one-time fix. It is a commitment to moving forward together.\nMoving Forward Together Silos are not inevitable. With intentional leadership, structured communication, and a commitment to collaboration, DevOps teams can achieve the unity that drives innovation.\nWhat is your experience with DevOps silos? How has your team overcome them? Let us share insights and solutions to break down barriers and deliver on the promise of DevOps.\nReferences Issues with silos in architecture and engineering firms The Three Ways: The Principles Underpinning DevOps Team Topologies key concepts DORA: Generative organizational culture ","permalink":"https://trinidadmarroquin.com/posts/devops-dirty-dozen/breaking-barriers-devops-silos/","section":"posts","summary":"Part 1 of the DevOps Dirty Dozen Series: Divide et impera — divide and conquer.\nInsight: Creating silos fragments unity and purpose.\nDevOps was supposed to unite us, but too often, it divides us into silos. Silos isolate teams, stifle communication, and undermine the very principles DevOps was built upon. Why do these silos exist? What makes them so destructive? And how can we tear them down to deliver on DevOps\u0026rsquo; promise of collaboration and innovation?\n","tags":["devops","sre","systems-thinking","collaboration"],"title":"Breaking Barriers: The DevOps Silos Anti-Pattern"}]