Configure Alerting Rules

Set up Prometheus alerting rules and Alertmanager for reliable, actionable incident notifications.

See Extended Examples for complete configuration files and templates.

When to Use

Implementing proactive monitoring with automated incident detection
Routing alerts to appropriate teams based on severity and service ownership
Reducing alert fatigue through intelligent grouping and deduplication
Integrating monitoring with on-call systems (PagerDuty, Opsgenie)
Establishing escalation policies for critical production issues
Migrating from legacy monitoring systems to Prometheus-based alerting
Creating actionable alerts that guide responders to resolution

Inputs

Required: Prometheus metrics to alert on (error rates, latency, saturation)
Required: On-call rotation and escalation policies
Optional: Existing alert definitions to migrate
Optional: Notification channels (Slack, email, PagerDuty)
Optional: Runbook documentation for common alerts

Procedure

Step 1: Deploy Alertmanager

Install and configure Alertmanager to receive alerts from Prometheus.

Docker Compose deployment (basic structure):

version: '3.8'
services:
  alertmanager:
    image: prom/alertmanager:v0.26.0
    ports:
      - "9093:9093"
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml
    # ... (see EXAMPLES.md for complete configuration)

Basic Alertmanager configuration (alertmanager.yml excerpt):

global:
  resolve_timeout: 5m
  slack_api_url: 'https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK'

route:
  receiver: 'default-receiver'
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

  routes:
    - match:
        severity: critical
      receiver: pagerduty-critical

# ... (see EXAMPLES.md for complete routing, inhibition rules, and receivers)

Configure Prometheus to use Alertmanager (prometheus.yml):

alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - alertmanager:9093
      timeout: 10s
      api_version: v2

Expected: Alertmanager UI accessible at http://localhost:9093, Prometheus "Status > Alertmanagers" shows UP status.

On failure:

Check Alertmanager logs: docker logs alertmanager
Verify Prometheus can reach Alertmanager: curl http://alertmanager:9093/api/v2/status
Test webhook URLs: curl -X POST <SLACK_WEBHOOK_URL> -d '{"text":"test"}'
Validate YAML syntax: amtool check-config alertmanager.yml

Step 2: Define Alerting Rules in Prometheus

Create alerting rules that fire when conditions are met.

Create alerting rules file (/etc/prometheus/rules/alerts.yml excerpt):

groups:
  - name: instance_alerts
    interval: 30s
    rules:
      - alert: InstanceDown
        expr: up == 0
        for: 5m
        labels:
          severity: critical
          team: infrastructure
        annotations:
          summary: "Instance {{ $labels.instance }} is down"
          description: "{{ $labels.instance }} has been down for >5min."
          runbook_url: "https://wiki.example.com/runbooks/instance-down"

      - alert: HighCPUUsage
        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High CPU usage on {{ $labels.instance }}"
          # ... (see EXAMPLES.md for complete alerts)

Alert design best practices:

for duration: Prevents flapping alerts. Use 5-10 minutes for most alerts.
Descriptive annotations: Include current value, affected resource, and runbook link.
Severity levels: critical (pages on-call), warning (investigate), info (FYI)
Team labels: Enable routing to correct team/channel
Runbook links: Every alert should have a runbook URL

Load rules into Prometheus:

# prometheus.yml
rule_files:
  - "rules/*.yml"

Validate and reload:

promtool check rules /etc/prometheus/rules/alerts.yml
curl -X POST http://localhost:9090/-/reload

Expected: Alerts visible in Prometheus "Alerts" page, alerts fire when thresholds exceeded, Alertmanager receives fired alerts.

On failure:

Check Prometheus logs for rule evaluation errors
Verify rule syntax with promtool check rules
Test alert queries independently in Prometheus UI
Inspect alert state transitions: Inactive → Pending → Firing

Step 3: Create Notification Templates

Design readable, actionable notification messages.

Create template file (/etc/alertmanager/templates/default.tmpl excerpt):

{{ define "slack.default.title" }}
[{{ .Status | toUpper }}] {{ .GroupLabels.alertname }}
{{ end }}

{{ define "slack.default.text" }}
{{ range .Alerts }}
*Alert:* {{ .Labels.alertname }}
*Severity:* {{ .Labels.severity }}
*Summary:* {{ .Annotations.summary }}
{{ if .Annotations.runbook_url }}*Runbook:* {{ .Annotations.runbook_url }}{{ end }}
{{ end }}
{{ end }}

# ... (see EXAMPLES.md for complete email and PagerDuty templates)

Use templates in receivers:

receivers:
  - name: 'slack-custom'
    slack_configs:
      - channel: '#alerts'
        title: '{{ template "slack.default.title" . }}'
        text: '{{ template "slack.default.text" . }}'

Expected: Notifications formatted consistently, include all relevant context, actionable with runbook links.

On failure:

Test template rendering: amtool template test --config.file=alertmanager.yml
Check template syntax errors in Alertmanager logs
Use {{ . | json }} to debug template data structure

Step 4: Configure Routing and Grouping

Optimize alert delivery with intelligent routing rules.

Advanced routing configuration (excerpt):

route:
  receiver: 'default-receiver'
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 30s

  routes:
    - match:
        team: platform
      receiver: 'team-platform'
      routes:
        - match:
            severity: critical
          receiver: 'pagerduty-platform'
          group_wait: 10s
          repeat_interval: 15m
          continue: true   # Also send to Slack

# ... (see EXAMPLES.md for complete routing with time intervals)

Grouping strategies:

# Group by alertname: All HighCPU alerts bundled together
group_by: ['alertname']

# Group by alertname AND cluster: Separate notifications per cluster
group_by: ['alertname', 'cluster']

Expected: Alerts routed to correct teams, grouped logically, timing appropriate for severity.

On failure:

Test routing: amtool config routes test --config.file=alertmanager.yml --alertname=HighCPU --label=severity=critical
Check routing tree: amtool config routes show --config.file=alertmanager.yml
Verify continue: true if alert should match multiple routes

Step 5: Implement Inhibition and Silencing

Reduce alert noise with inhibition rules and temporary silences.

Inhibition rules (suppress dependent alerts):

inhibit_rules:
  # Cluster down suppresses all node alerts in that cluster
  - source_match:
      alertname: 'ClusterDown'
      severity: 'critical'
    target_match_re:
      alertname: '(InstanceDown|HighCPU|HighMemory)'
    equal: ['cluster']

  # Service down suppresses latency and error alerts
  - source_match:
      alertname: 'ServiceDown'
    target_match_re:
      alertname: '(HighLatency|HighErrorRate)'
    equal: ['service', 'namespace']

# ... (see EXAMPLES.md for more inhibition patterns)

Create silences programmatically:

# Silence during maintenance
amtool silence add \
  instance=app-server-1 \
  --author="ops-team" \
  --comment="Scheduled maintenance" \
  --duration=2h

# List and manage silences
amtool silence query
amtool silence expire <SILENCE_ID>

Expected: Inhibition reduces cascade alerts automatically, silences prevent notifications during planned maintenance.

On failure:

Test inhibition logic with live alerts
Check Alertmanager UI "Silences" tab
Verify silence matchers are exact (labels must match perfectly)

Step 6: Integrate with External Systems

Connect Alertmanager to PagerDuty, Opsgenie, Jira, etc.

PagerDuty integration (excerpt):

receivers:
  - name: 'pagerduty'
    pagerduty_configs:
      - routing_key: 'YOUR_INTEGRATION_KEY'
        severity: '{{ .CommonLabels.severity }}'
        description: '{{ range .Alerts.Firing }}{{ .Annotations.summary }}{{ end }}'
        details:
          firing: '{{ .Alerts.Firing | len }}'
          alertname: '{{ .GroupLabels.alertname }}'
        # ... (see EXAMPLES.md for complete integration examples)

Webhook for custom integrations:

receivers:
  - name: 'webhook-custom'
    webhook_configs:
      - url: 'https://your-webhook-endpoint.com/alerts'
        send_resolved: true

Expected: Alerts create incidents in PagerDuty, appear in team communication channels, trigger on-call escalations.

On failure:

Verify API keys/tokens are valid
Check network connectivity to external services
Test webhook endpoints independently with curl
Enable debug mode: --log.level=debug

Validation

Common Pitfalls

Alert fatigue: Too many low-priority alerts cause responders to ignore critical ones. Set strict thresholds, use inhibition.
Missing for duration: Alerts without for fire on transient spikes. Always use 5-10 minute windows.
Overly broad grouping: Grouping by ['...'] sends individual notifications. Use specific label grouping.
No runbook links: Alerts without runbooks leave responders guessing. Every alert needs a runbook URL.
Incorrect severity: Mislabeling warnings as critical desensitizes team. Reserve critical for emergencies.
Forgotten silences: Silences without expiration can hide real issues. Always set end times.
Single route: All alerts to one channel loses context. Use team-specific routing.
No inhibition: Cascade alerts during outages create noise. Implement inhibition rules.

Related Skills

setup-prometheus-monitoring - Define metrics and recording rules that feed alerting rules
define-slo-sli-sla - Generate SLO burn rate alerts for error budget management
write-incident-runbook - Create runbooks linked from alert annotations
build-grafana-dashboards - Visualize alert firing history and silence patterns