Blog | Trinidad Marroquin

Practical notes on infrastructure and operations.

The goal of this blog is to document how systems are built, changed, operated, and recovered. Posts should include enough context to be useful later: constraints, tradeoffs, failure modes, and the operational lesson.

Series

The DevOps Dirty Dozen: Unmasking the Triple D’s of Anti-Patterns — a twelve-part series on DevOps anti-patterns, systems thinking, and practical improvement habits.
Practical Data Science For DevOps And SRE — a hands-on series on using simple, explainable analysis methods for reliability reviews, alerts, incidents, metrics, and operational decisions.

Common Topics

Kubernetes, Rancher, and vCenter operations.
Terraform infrastructure changes.
Packer image build workflows.
CI/CD pipeline design and troubleshooting.
Observability, alerting, dashboards, and incident response.
Systems thinking for production operations.

RKE2 Node Reboots When Longhorn Is Already Degraded

Node reboots are easy to underestimate in Kubernetes. They look smaller than upgrades because the target version does not change, but the risk profile can be just as high when storage is already unhealthy. In one RKE2 maintenance window, the cluster needed operating-system reboots while Longhorn was not fully healthy. That changed the plan. The work could not be treated as a generic rolling reboot across every node. It needed role-based batches, explicit health gates, and a hard boundary around storage nodes until the Longhorn state was understood. It also needed retained evidence; see Kubernetes Maintenance Evidence Bundles Need A Redaction Plan for the artifact-handling side of this procedure. ...

Ubuntu Unattended Upgrades Are Kubernetes Node Changes

Ubuntu unattended upgrades are useful on ordinary servers and workstations. On Kubernetes nodes, they are production changes. That distinction matters because a Kubernetes worker is not just a Linux host. It is part of a larger control loop that includes kubelet, the container runtime, CNI, CSI, storage paths, systemd units, udev device handling, and sometimes iSCSI or multipath. A package update that is safe in isolation can become disruptive when it lands outside a maintenance window. ...

Rancher RKE2 Minor Hops When UI Metadata And GitOps Plans Disagree

The hard part of a Rancher-managed RKE2 upgrade is not always clicking the target version. The hard part is keeping every control plane that thinks it owns the upgrade aligned: Kubernetes version-skew rules. Rancher UI version metadata. Rancher’s desired cluster version. system-upgrade-controller Plans. GitOps reconciliation for those Plans. node-level runtime cleanup after the upgrade. In one management-cluster upgrade, the safe Kubernetes path was clear: v1.31 -> v1.32 -> v1.33 But Rancher only advertised newer versions in the UI after the Rancher application itself was upgraded. The UI could offer v1.33 while the cluster still needed the intermediate v1.32 hop. That is where an otherwise normal upgrade becomes an ownership problem. ...

Rancher Management Cluster Upgrades Need More Than A Version Target

Rancher management-cluster upgrades are not just a Helm command. They are a sequence of readiness gates. The version target matters, but it is not the first question. The first question is whether the management cluster is stable enough to survive the change. In one upgrade run, the desired path was straightforward on paper: Rancher first, then Kubernetes/RKE2. The actual work exposed the operational details that make or break the window: stale workers, version skew, node join scripts, root filesystem pressure, backup transfer permissions, Rancher pre-upgrade hooks, Fleet health, and downstream cluster impersonation. ...

Turning Packer Template Builds Into A Concourse Workflow

Packer template builds are a good fit for CI/CD because they are repeatable, expensive enough to benefit from automation, and easy to forget after the first successful run. They also expose every hidden dependency in the platform: repository access, vCenter permissions, ISO availability, DNS, routing, worker placement, and secret handling. The goal was not just to make one template build pass. The goal was to make template builds runnable per site from Concourse, with enough visibility to debug failures and enough guardrails to avoid leaking credentials. ...

Installing And Configuring Calico On RKE2 From The Ground Up

Calico is not just a YAML install. It becomes part of the cluster’s trust model, routing behavior, NetworkPolicy enforcement, and incident response surface. The important decisions happen before the first manifest is applied: who owns CNI installation. which node interface Calico should use. which pod CIDR the cluster will allocate from. whether encapsulation is VXLAN, VXLAN cross-subnet, IP-in-IP, or BGP. how operators will prove the install is healthy after upgrades and node replacement. This guide is written for RKE2 clusters where Calico is installed and managed intentionally, not discovered later as a side effect of another installer. ...

When A Latent Rancher Worker Upgrade Becomes An Outage

Not every outage starts with a new deployment. Sometimes the trigger is old desired state that never finished converging. A controller keeps watching. A GitOps reconciler keeps restoring. The cluster looks stable until capacity drops, and then the old desired state becomes active at the worst possible time. That is the failure pattern I want to remember from this Rancher/RKE2 investigation. The Shape Of The Problem The cluster had a Rancher-managed RKE2 system-upgrade workflow with separate plans: ...

Operating A NetBox Virtualization Migration Without Losing Control

Inventory migrations are easy to underestimate. Changing a NetBox object from a DCIM device to a virtualization VM sounds like a data cleanup task. In practice, it touches ownership boundaries: IP assignments, primary IPs, cluster tags, Terraform imports, and the difference between audit-only work and infrastructure mutation. The safe way to run this kind of migration is to treat it like an operational change, not a CSV edit. The Problem Some VMware-backed Kubernetes nodes were represented in NetBox as DCIM devices. That worked for basic inventory lookup, but it did not match the automation model. ...

Drift Audit Report: Bringing Existing Kubernetes VMs Under Terraform And NetBox

This work was not a greenfield deployment. The cluster already existed. The VMs already existed in vSphere. NetBox already had inventory, but some of it was modeled as DCIM devices rather than virtualization virtual machines. Terraform had a reusable module that could manage VM lifecycle state, NetBox VM records, interfaces, IP addresses, and primary IPv4 relationships. The problem was getting from “existing infrastructure” to “managed state” without accidentally changing the running cluster. ...

Managing vSphere VM Lifecycle State With Terraform And NetBox

Provisioning a VM is only the first part of ownership. The more interesting operational question is what happens after the VM exists. Can Terraform continue to manage CPU, memory, disk sizing, NetBox metadata, idempotency, and teardown without turning every follow-up change into a manual vCenter edit? That was the next step after adding VM/IP guardrails. The automation already knew how to prevent duplicate VM names, duplicate IPs, stale DNS collisions, active network IP reuse, and NetBox/vCenter drift before create. The next improvement was to keep the VM lifecycle managed after creation. ...

Series#

Common Topics#

Series

Common Topics