Building A Small SLI Lab With Flask, Prometheus, And Grafana

Service Level Indicators (SLIs) are easier to understand when they are tied to a working service. Abstract definitions are useful, but a small lab makes the tradeoffs visible: what counts as success, what counts as failure, how latency should be measured, and how trends can reveal degradation before a full incident.

This field note uses a companion lab in GitHub:

https://github.com/trinidadgithub/IaC/tree/main/sli_app

The lab runs a small Flask application, exposes Prometheus metrics, provisions Prometheus and Grafana with Terraform, and includes a basic SLI dashboard. It also introduces lightweight data science habits: percentiles, rolling windows, error-rate comparison, and avoiding misleading averages.

What This Lab Demonstrates

The application intentionally creates variable latency and occasional failures. A perfectly reliable demo app is not useful for learning reliability measurement.

The lab helps answer practical questions:

• What percentage of requests are successful?
• How slow is the service for typical users and tail users?
• Is the error rate stable, improving, or getting worse?
• Are 4xx responses client errors, service errors, or excluded from the SLI?
• Which metrics belong on a Grafana dashboard?
• Which measurements could support an SLO later?

This is not a production architecture. It is a small observability lab for making SLI design concrete.

Repository Layout

The lab lives under sli_app in the IaC repository.

sli_app/
├── app.py
├── Dockerfile
├── requirements.txt
├── prometheus.yml
├── scripts/
│   └── generate_traffic.sh
└── terraform/
    ├── main.tf
    ├── outputs.tf
    ├── prometheus.yml
    └── grafana/
        ├── dashboards/
        │   ├── system-docker-monitoring.json
        │   └── sli-lab-dashboard.json
        └── provisioning/
            └── datasources/
                └── datasources.yml

Terraform creates a local Docker network and runs:

Prometheus scrapes the Flask app through the shared Docker network at sli_app:8000.

Service Shape

The Flask application exposes a few endpoints:

The distinction between /api/data and /api/submit is useful because read and write paths often have different latency and reliability expectations.

Metrics Exposed

The application uses prometheus_client and exposes metrics from a separate port, 8000.

The explicit labels make it possible to separate endpoint behavior and status classes. That matters when distinguishing expected 400 responses from service-side 500 failures.

Running The Lab

Clone the repository and apply Terraform:

git clone https://github.com/trinidadgithub/IaC.git
cd IaC/sli_app/terraform

terraform init
terraform validate
terraform apply

Grafana credentials:

username: admin
password: admin01

Generate traffic from the sli_app directory:

chmod +x scripts/generate_traffic.sh
./scripts/generate_traffic.sh

For a longer run:

ITERATIONS=100 SLEEP_SECONDS=0 ./scripts/generate_traffic.sh

The traffic script sends valid read/write requests and occasional invalid submit requests to produce expected 400 responses.

Defining The SLIs

SLIs should represent user-visible service behavior, not just container health.

Container uptime is not the same as availability. A container can be running while every request fails.

PromQL Examples

Request Rate

sum by (endpoint, method) (
  rate(sli_http_requests_total[5m])
)

Availability

This version treats 5xx responses as service failures:

1 - (
  sum(rate(sli_http_requests_total{status=~"5.."}[5m]))
  /
  sum(rate(sli_http_requests_total[5m]))
)

If the SLI should exclude expected client errors from the denominator:

1 - (
  sum(rate(sli_http_requests_total{status=~"5.."}[5m]))
  /
  sum(rate(sli_http_requests_total{status!~"4.."}[5m]))
)

The policy decision matters. A bad client request may not indicate service unreliability, but a broken API contract might.

Error Rate

sum(rate(sli_http_requests_total{status=~"5.."}[5m]))
/
sum(rate(sli_http_requests_total[5m]))

p95 Latency

histogram_quantile(
  0.95,
  sum by (le, endpoint, method) (
    rate(sli_http_request_duration_seconds_bucket[5m])
  )
)

p99 Latency

histogram_quantile(
  0.99,
  sum by (le, endpoint, method) (
    rate(sli_http_request_duration_seconds_bucket[5m])
  )
)

Practical Data Science Layer

This lab becomes more valuable when metrics are treated as time-series data rather than isolated values.

Percentiles Instead Of Averages

Average latency hides tail pain. If most users receive a response quickly but a meaningful minority wait several seconds, the average may look acceptable while real users suffer.

Use:

• p50 for typical behavior
• p95 for most-user experience
• p99 for tail pain

Average latency is still useful as a supporting signal, but it should not be the primary user-experience SLI.

Error-Rate Analysis

Error rate should be reviewed by endpoint, method, and status class:

sum by (endpoint, method, status) (
  rate(sli_http_requests_total[5m])
)

This helps separate:

• expected 400 responses from bad input
• service-side 500 responses
• endpoint-specific failure patterns
• read-path versus write-path behavior

Define what counts against the SLI before reviewing the graph. Otherwise, teams are tempted to redefine reliability after the fact.

Trend Awareness

A single five-minute window can be noisy. Compare short and longer windows to detect direction.

Short-window error rate:

sum(rate(sli_http_requests_total{status=~"5.."}[5m]))
/
sum(rate(sli_http_requests_total[5m]))

Longer-window error rate:

sum(rate(sli_http_requests_total{status=~"5.."}[30m]))
/
sum(rate(sli_http_requests_total[30m]))

If the five-minute rate is much higher than the thirty-minute rate, the service may be entering a failure window. If both are rising, the issue may be sustained.

Simple Baseline Comparison

PromQL offset can compare current behavior with a prior window:

histogram_quantile(
  0.95,
  sum by (le) (rate(sli_http_request_duration_seconds_bucket[5m]))
)

histogram_quantile(
  0.95,
  sum by (le) (rate(sli_http_request_duration_seconds_bucket[5m] offset 1h))
)

This is not advanced forecasting. It is operational awareness: is the service behaving materially differently from a recent stable period?

Grafana Dashboard

The lab provisions an SLI Lab - Flask Service dashboard with panels for:

• Request rate by endpoint
• Availability
• 5xx error rate
• p95 and p99 latency
• Status code breakdown
• Short-window versus long-window error-rate comparison

The dashboard is intentionally small. It focuses on operational questions rather than every metric available.

What This Teaches

This lab reinforces several SRE lessons:

• SLIs must be user-centered.
• Availability is not container uptime.
• Averages hide tail latency.
• 4xx and 5xx responses should not be blindly grouped together.
• Short windows detect fast changes but are noisy.
• Longer windows show sustained behavior but can hide spikes.
• Dashboards should support decisions.
• SLOs should be based on carefully chosen SLIs, not whatever metric is easiest to graph.

Gaps Addressed From The Original Notes

The original notes had the right general direction but needed cleanup before becoming a repeatable lab.

Key improvements:

• Updated Python and Flask dependency guidance.
• Removed unrelated outbound network traffic from the app.
• Used explicit Prometheus counters and histograms.
• Added labels for endpoint, method, and status.
• Kept the existing Terraform-managed Docker lab pattern.
• Pinned Prometheus and Grafana image versions.
• Added a repeatable traffic generation script.
• Added an SLI-specific Grafana dashboard.
• Defined availability from request success ratio, not uptime.
• Added explicit 4xx versus 5xx discussion.
• Used histogram percentiles for latency instead of averages.
• Added trend and baseline comparison examples.
• Removed unrelated GitHub branch-protection and C-programming notes.

Where To Take This Next

This lab is intentionally small, but it creates enough signal to support more realistic reliability conversations. The next useful layer would be to turn the observed SLIs into a simple SLO, such as 99% successful eligible requests over 30 days, and then calculate how quickly different failure rates consume that error budget.

Alerting can also be added once the SLO is clear. A high 5xx rate or sustained p95 latency spike is more meaningful when it is tied to a user-facing objective instead of an arbitrary threshold.

The same data can support a follow-up field note on percentiles. Comparing average latency with p95 and p99 latency is a practical way to show why averages often hide the user experience that reliability reviews are supposed to protect.

References

• Google SRE Book — Service Level Objectives
• Google SRE Workbook — Implementing SLOs
• Prometheus Querying Basics
• Prometheus Histograms And Summaries
• Grafana Prometheus Data Source
• Flask Documentation