Observability And Incidents Field Notes

An SLO is not useful because it exists in a document. It becomes useful when it changes operational behavior. Burn-rate alerting is one way to make that happen. Instead of paging because an error rate crossed a random threshold, the alert asks a better question: How quickly are we consuming the error budget? That question is easier to defend during an incident. It connects the alert to user impact, time window, and reliability policy instead of dashboard aesthetics. ...

An SLO dashboard should help the on-call engineer make the first decision during an incident. It is not a decoration layer for metrics. It is the operational view that answers: Is this user-impacting? How fast are we burning error budget? What changed? Who owns the response? If the dashboard cannot answer those questions quickly, the alert may be technically correct but operationally incomplete. This note pairs with SLO Burn-Rate Alerting With Prometheus and Burn-Rate Alerting Concepts For Operators. ...

An observability namespace can look like a platform outage even when customer workloads are healthy. In one downstream-cluster investigation, the noisy symptoms were all in monitoring and alerting: logging resources disappearing, an agent operator stuck during deletion, alert delivery failures, external secret authentication errors, and exporter noise. The first useful move was to split the question in two: Is the cluster currently unhealthy? Is the observability stack being removed, broken, or reconciled by another owner? Those are different incidents. ...

Service Level Indicators (SLIs) are easier to understand when they are tied to a working service. Abstract definitions are useful, but a small lab makes the tradeoffs visible: what counts as success, what counts as failure, how latency should be measured, and how trends can reveal degradation before a full incident. This field note uses a companion lab in GitHub: https://github.com/trinidadgithub/IaC/tree/main/sli_app The lab runs a small Flask application, exposes Prometheus metrics, provisions Prometheus and Grafana with Terraform, and includes a basic SLI dashboard. It also introduces lightweight data science habits: percentiles, rolling windows, error-rate comparison, and avoiding misleading averages. ...

When a flow collector reports a huge byte count, resist the urge to start with the loudest application log. Start by proving whether the node, interface, and time window support the story. This checklist is useful when Kubernetes or Rancher appears to be involved in a network spike. Identify The Conversation Ask for the flow detail, not only the top-talker summary: source IP destination IP source port destination port protocol bytes sessions start time end time Important ports in Rancher/RKE2 environments include: ...

A local SLI lab is useful when the goal is to understand the signal path before introducing production platform complexity. For a live local demo, see the sli_app lab in the IaC repository. If the lab does not run as expected, open a bug fix request against that repository with the failing command, host OS, Docker version, Terraform version, and relevant logs. The important part is not that everything runs on one machine. The important part is that the lab has the same basic observability chain operators rely on later: ...

AI-assisted operations tools are entering the market rapidly. Their value depends entirely on the quality of the observability data they consume. If the data is noisy, incomplete, or unstructured, AI tools amplify the noise instead of reducing it. This field note covers the foundations every operator should have in place before adding AI to the observability stack. Required Foundations Structured Metrics Metrics must have consistent labels across services. The same label (service, environment, region) should mean the same thing in every metric. If one service uses env and another uses environment, any tool consuming both will produce unreliable correlations. ...

Local infrastructure labs often start with hardcoded passwords, localhost endpoints, and convenience tokens. That is normal for learning, but dangerous when the lab pattern becomes a production pattern without review. The useful distinction is not “lab bad, production good.” The useful distinction is knowing which shortcuts are temporary and what must change before the pattern is reused. Common Lab Shortcuts Terraform-managed Docker labs often include: Grafana admin credentials in container environment variables. Concourse local users such as admin:admin. Vault dev server tokens in shell environment files. database passwords pulled into Terraform state. generated private keys written to local files. privileged containers for CI workers or system exporters. localhost endpoints that assume a single operator workstation. Each shortcut may be acceptable in a disposable lab. None should cross into shared infrastructure by accident. ...