Prometheus is easy to start and easy to overload.

Most production problems are not caused by one bad alert. They come from unclear scrape ownership, high-cardinality labels, expensive dashboard queries, or rule changes that nobody can explain during an incident.

The same ownership rule applies to observability agents. If Grafana Alloy collects logs, give it resource limits and backend-readiness checks too. See Grafana Alloy Log Collector CPU Guardrails.

Own The Scrape Path

Every scrape target should have an owner and a reason to exist.

Track:

  • ServiceMonitor or PodMonitor name.
  • namespace and selector.
  • owning team.
  • scrape interval.
  • expected sample volume.
  • alerting or dashboard dependency.
  • retirement path when the workload leaves.

If no one can explain why a target is scraped, it will eventually become invisible cost and noisy failure.

Watch Cardinality

High cardinality usually arrives through labels that look harmless:

  • user ID.
  • request ID.
  • session ID.
  • raw URL path.
  • pod UID.
  • container hash.
  • unbounded tenant or customer labels.

Before adding a label, ask whether operators need to group by it during a real incident. If the answer is no, keep it out of metrics and use logs or traces for that dimension.

Recording Rule Boundaries

Recording rules should make common operational questions cheaper and safer.

Good candidates:

  • service-level request rates.
  • error ratios.
  • latency quantiles from known histograms.
  • saturation signals.
  • expensive joins used by dashboards.

Bad candidates:

  • rules with unclear ownership.
  • rules that hide label meaning.
  • rules that duplicate application dashboards without review.
  • rules that turn bad cardinality into retained bad cardinality.

Keep rule names boring and searchable. During an outage, job:http_requests:rate5m is more useful than a clever abbreviation.

Query Safety

Dashboard and ad-hoc queries need guardrails.

Review:

  • default time range.
  • max data points.
  • use of regex selectors.
  • joins across high-cardinality labels.
  • rate() window relative to scrape interval.
  • unbounded sum by (...) groupings.
  • queries used by alerts versus dashboards.

A query that works for one hour of data may hurt the system when someone opens a thirty-day dashboard during an incident.

Alert Rule Evidence

For alert changes, keep:

  • rule expression.
  • intended owner and route.
  • expected severity.
  • sample firing case.
  • dashboard or runbook link.
  • silence policy.
  • review date.

Prometheus rule evaluation should be treated as production logic. A green deployment is not enough if the expression pages the wrong team or cannot fire under real traffic.

Failure Model

The common failure is unmanaged growth:

teams add scrape targets -> cardinality grows -> dashboards slow down
-> alerts evaluate late -> operators blame Prometheus during an incident

The operating rule: Prometheus needs ownership for targets, limits for labels, and review for expensive queries before the incident depends on them.