Observability And Incidents Field Notes

Burn-rate alerting is easier to operate when the team understands the concepts before touching Prometheus rules. The implementation details matter, but the operational idea is simple: alert when a service is consuming its allowed failure budget too quickly. That is different from alerting because a metric crossed a convenient threshold. This note explains the mental model behind burn-rate alerts. For Prometheus examples and alert rule structure, see SLO Burn-Rate Alerting With Prometheus. For dashboard design after the alert fires, see What To Put On An SLO Dashboard. ...

An incident review should improve the system, not assign blame. The useful output is a better service, a better alert, a better runbook, or a clearer ownership boundary. If the review only produces a timeline and a vague action item, it is documentation theater. This template is designed for platform and SRE teams running production services, Kubernetes platforms, observability stacks, and shared infrastructure. Related notes: SLO Burn-Rate Alerting With Prometheus What To Put On An SLO Dashboard On-Call Escalation Policy For Platform Teams Incident Summary Incident title: Incident date: Severity: Status: Services affected: Primary owner: Incident commander: Review date: Write a short summary in plain language. ...

An escalation policy should tell the on-call engineer what to do when ownership, severity, or impact is unclear. It is not only a paging schedule. A schedule says who gets woken up. A policy says when to escalate, who should join, and what decisions are expected. For platform teams, this matters because incidents often cross boundaries: Kubernetes, networking, storage, CI/CD, observability, identity, and application ownership. Related notes: SLO Burn-Rate Alerting With Prometheus What To Put On An SLO Dashboard Incident Review Template For SRE Teams Policy Goals The policy should optimize for clear ownership and fast mitigation. ...

An SLO is not useful because it exists in a document. It becomes useful when it changes operational behavior. Burn-rate alerting is one way to make that happen. Instead of paging because an error rate crossed a random threshold, the alert asks a better question: How quickly are we consuming the error budget? That question is easier to defend during an incident. It connects the alert to user impact, time window, and reliability policy instead of dashboard aesthetics. ...

An SLO dashboard should help the on-call engineer make the first decision during an incident. It is not a decoration layer for metrics. It is the operational view that answers: Is this user-impacting? How fast are we burning error budget? What changed? Who owns the response? If the dashboard cannot answer those questions quickly, the alert may be technically correct but operationally incomplete. This note pairs with SLO Burn-Rate Alerting With Prometheus and Burn-Rate Alerting Concepts For Operators. ...

An observability namespace can look like a platform outage even when customer workloads are healthy. In one downstream-cluster investigation, the noisy symptoms were all in monitoring and alerting: logging resources disappearing, an agent operator stuck during deletion, alert delivery failures, external secret authentication errors, and exporter noise. The first useful move was to split the question in two: Is the cluster currently unhealthy? Is the observability stack being removed, broken, or reconciled by another owner? Those are different incidents. ...

Service Level Indicators (SLIs) are easier to understand when they are tied to a working service. Abstract definitions are useful, but a small lab makes the tradeoffs visible: what counts as success, what counts as failure, how latency should be measured, and how trends can reveal degradation before a full incident. This field note uses a companion lab in GitHub: https://github.com/trinidadgithub/IaC/tree/main/sli_app The lab runs a small Flask application, exposes Prometheus metrics, provisions Prometheus and Grafana with Terraform, and includes a basic SLI dashboard. It also introduces lightweight data science habits: percentiles, rolling windows, error-rate comparison, and avoiding misleading averages. ...

When a flow collector reports a huge byte count, resist the urge to start with the loudest application log. Start by proving whether the node, interface, and time window support the story. This checklist is useful when Kubernetes or Rancher appears to be involved in a network spike. Identify The Conversation Ask for the flow detail, not only the top-talker summary: source IP destination IP source port destination port protocol bytes sessions start time end time Important ports in Rancher/RKE2 environments include: ...

A local SLI lab is useful when the goal is to understand the signal path before introducing production platform complexity. For a live local demo, see the sli_app lab in the IaC repository. If the lab does not run as expected, open a bug fix request against that repository with the failing command, host OS, Docker version, Terraform version, and relevant logs. The important part is not that everything runs on one machine. The important part is that the lab has the same basic observability chain operators rely on later: ...

AI-assisted operations tools are entering the market rapidly. Their value depends entirely on the quality of the observability data they consume. If the data is noisy, incomplete, or unstructured, AI tools amplify the noise instead of reducing it. This field note covers the foundations every operator should have in place before adding AI to the observability stack. Required Foundations Structured Metrics Metrics must have consistent labels across services. The same label (service, environment, region) should mean the same thing in every metric. If one service uses env and another uses environment, any tool consuming both will produce unreliable correlations. ...