Alerting & Dashboard Builder

Build effective alerts and dashboards based on SLOs.

SLO Definition

slos:

• name: api_availability objective: 99.9% window: 30d sli: | sum(rate(http_requests_total{status_code!~"5.."}[5m])) / sum(rate(http_requests_total[5m]))

• name: api_latency objective: 95% # 95% of requests under 500ms window: 30d sli: | histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]) ) < 0.5

Alert Rules

groups:

• name: slo_alerts rules:

  Fast burn (1% budget in 1h)

  • alert: AvailabilitySLOFastBurn expr: | (1 - (sum(rate(http_requests_total{status_code!~"5.."}[1h])) / sum(rate(http_requests_total[1h])))) > 0.01 for: 5m labels: severity: critical annotations: summary: "Burning 1% error budget per hour" runbook: "https://runbooks.example.com/availability-fast-burn"

  Slow burn (10% budget in 24h)

  • alert: AvailabilitySLOSlowBurn expr: | (1 - (sum(rate(http_requests_total{status_code!~"5.."}[24h])) / sum(rate(http_requests_total[24h])))) > 0.001 for: 1h labels: severity: warning annotations: summary: "Burning error budget slowly"

Dashboard Template

{ "title": "Service Health Dashboard", "rows": [ { "title": "Golden Signals", "panels": [ { "title": "Request Rate", "query": "sum(rate(http_requests_total[5m]))", "type": "graph" }, { "title": "Error Rate", "query": "sum(rate(http_requests_total{status_code=~"5.."}[5m]))", "type": "graph" }, { "title": "Latency (p50, p95, p99)", "queries": [ "histogram_quantile(0.50, rate(http_request_duration_seconds_bucket[5m]))", "histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))", "histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))" ] }, { "title": "Saturation (CPU, Memory)", "queries": [ "rate(process_cpu_seconds_total[5m])", "process_resident_memory_bytes" ] } ] }, { "title": "SLO Tracking", "panels": [ { "title": "Error Budget Remaining", "query": "1 - ((1 - 0.999) - (1 - slo_availability))" } ] } ] }

What to Do When Alert Fires

Alert Response Guide

HighErrorRate

What it means: More than 5% of requests are failing

First steps:

1. Check recent deployments (rollback if needed)
2. Review error logs for patterns
3. Check dependent services health
4. Verify database connectivity

Escalation: If not resolved in 15 min, page on-call lead

HighLatency

What it means: p95 latency above 2 seconds

First steps:

1. Check database query performance
2. Review recent code changes
3. Check cache hit rates
4. Look for slow external API calls

Temporary mitigation:

• Scale up instances
• Enable aggressive caching

LowAvailability

What it means: Availability below 99.5%

First steps:

1. Check infrastructure (AWS status page)
2. Review load balancer health checks
3. Check for DDoS activity
4. Verify auto-scaling functioning

Output Checklist

• SLOs defined

• Alert rules configured

• Dashboards created

• Runbooks linked

• Response guides documented ENDFILE