Incident Management

Incident Severity

Level Impact Response Time

SEV1 Complete outage Immediate

SEV2 Major degradation < 15 min

SEV3 Minor degradation < 1 hour

SEV4 Low impact Next business day

Incident Response

1. Detect

• Monitoring alerts

• Customer reports

• Error logs

2. Triage

• Assess severity

• Assign incident commander

• Create communication channel

3. Investigate

• Check recent changes

• Review logs and metrics

• Identify root cause

4. Mitigate

• Apply quick fix

• Rollback if needed

• Communicate status

5. Resolve

• Confirm fix

• Monitor for recurrence

• Close incident

6. Learn

• Post-mortem meeting

• Document findings

• Create action items

Post-Mortem Template

Post-Mortem: [Incident Title]

Summary

[Brief description of what happened]

Timeline

• HH:MM - [Event]
• HH:MM - [Event]
• HH:MM - [Resolution]

Impact

• Duration: [X hours]
• Users affected: [X]
• Revenue impact: [if applicable]

Root Cause

[What caused this incident]

Contributing Factors

• [Factor 1]
• [Factor 2]

What Went Well

• [Positive 1]
• [Positive 2]

What Could Be Improved

• [Improvement 1]
• [Improvement 2]

Action Items

• [Action 1] - Owner: [Name]
• [Action 2] - Owner: [Name]

Blameless Culture

• Focus on systems, not people

• "What failed?" not "Who failed?"

• Share learnings openly

• Celebrate near-misses