Production Troubleshooting

Overview

Diagnose performance issues and errors in production/test environments using systematic investigation workflows with Sentry, kubectl, and Helm configuration analysis.

When to Use This Skill

Use this skill when:

• User reports performance issues on test/production (not localhost)

• Need to investigate slow queries or high latency

• Debugging pod crashes or resource throttling

• Analyzing Sentry traces for errors

• Checking Kubernetes resource limits and configurations

Investigation Workflow

Follow these steps in order when troubleshooting production issues:

Step 1: Check Sentry Traces

Start with Sentry to identify slow queries and external API latency patterns.

Using Sentry MCP:

• Search for traces related to the reported issue

• Look for slow database queries (>500ms)

• Check external API call latency

• Identify error patterns and stack traces

What to look for:

• Database query times exceeding 500ms

• External API calls with high latency

• Repeated error patterns

• Performance degradation trends

Step 2: Review Application Logs

Examine kubectl logs for timing information and error patterns.

Using agent-tools-k8s:

agent-tools-k8s logs --pod <pod-name> --env <env> --tail 200

Key log patterns to search for:

• [Server]

• Server startup and initialization timing

• [SSR]

• Server-side rendering timing

• [tRPC]

• TRPC query execution timing

• [DB Pool]

• Database connection pool status

• ERROR or WARN

• Application errors and warnings

Common issues:

• Sequential API calls instead of parallel (Promise.all)

• Long DB connection acquisition times

• Slow SSR rendering

Step 3: Check Pod Resource Usage

Verify CPU and memory usage to detect throttling.

Using agent-tools-k8s:

agent-tools-k8s top --env <env>

Warning signs:

• CPU usage >70% indicates potential throttling

• Memory usage >80% indicates potential OOM issues

• Consistent high utilization suggests under-provisioning

Step 4: Review Pod Configuration

Check resource limits and Helm values to identify misconfigurations.

Using kubectl:

kubectl get pod <pod-name> -n <namespace> -o yaml

Key sections to check:

• resources.limits.cpu and resources.limits.memory

• resources.requests.cpu and resources.requests.memory

• Environment variables configuration

• Image version and tags

Helm values locations:

• web-app: /kubernetes/helm/web-app/values.{test,prod}.yaml

Reference references/helm-values-locations.md for detailed Helm configuration structure.

Common Causes & Solutions

CPU/Memory Throttling

• Symptom: High CPU/memory usage (>70-80%)

• Solution: Increase resource limits in Helm values

Network Latency

• Symptom: Slow external API calls, DNS resolution delays

• Solution: Check network policies, verify DNS configuration, consider retry logic

Database Connection Pool Issues

• Symptom: [DB Pool] errors, slow connection acquisition

• Solution: Review idleTimeoutMillis and pool size configuration

Sequential API Calls

• Symptom: Multiple API calls taking cumulative time

• Solution: Refactor to use Promise.all() for parallel execution

Resources

kubectl commands

Common kubectl operations (use via agent-tools-k8s ):

• agent-tools-k8s logs --pod <pod> --env <env> --tail 200

• Extract and filter pod logs

• agent-tools-k8s top --env <env>

• Show CPU/memory usage for pods

• agent-tools-k8s describe --resource pod --name <pod> --env <env>

• Check resource limits and pod configuration

• agent-tools-k8s kubectl --env <env> --cmd "get pods"

• Raw kubectl for anything else

references/

• helm-values-locations.md

• Detailed guide to Helm values file structure and locations

• common-issues.md

• Catalog of common production issues and solutions