root-cause-analysis

Analyze telemetry data for root cause analysis using Kopai CLI. Use when debugging errors, investigating latency issues, tracing request flows across services, or correlating logs with traces. Also use when users report production issues like "why is my API slow", "getting 500 errors", "service is down", "requests are timing out", or any symptom that needs telemetry-based investigation — even if they don't mention traces or observability explicitly.

Root Cause Analysis with Kopai

Guide for debugging production issues using telemetry data (traces, logs, metrics) via Kopai CLI.

Prerequisites

Ensure access to Kopai app backend. Make sure the services are set up to send their OpenTelemetry data to Kopai. See otel-instrumentation skill for setup.

RCA Workflow Summary

Find error traces
Get full trace context
Correlate logs with trace
Check related metrics
Identify root cause

Rules

1. Workflow (CRITICAL)

workflow-find-errors - Find Error Traces
workflow-get-context - Get Full Trace Context
workflow-correlate-logs - Correlate Logs with Trace
workflow-check-metrics - Check Related Metrics
workflow-identify-cause - Identify Root Cause & Present Findings

2. Patterns (HIGH)

pattern-http-errors - HTTP Error Debugging
pattern-slow-requests - Slow Request Analysis
pattern-distributed - Distributed Failure Tracing
pattern-log-driven - Log-Driven Investigation

Read rules/<rule-name>.md for details.

Tips

Always use --json for programmatic analysis
Pipe to jq for filtering/aggregation
Start with errors, then trace backwards
Check span Duration to find bottlenecks
Correlate TraceId across traces, logs, metrics
Use --severity-min 17 instead of --severity-text ERROR to catch all error-level logs regardless of text casing. Fall back to --body "error" for errors logged at INFO or with no severity.

References

trace-filters - Trace search filter options
log-filters - Log search filter options
metric-filters - Metric search filter options