Incident Response Plan

Procedures for classifying, responding to, and learning from production incidents.

Severity Classification

Severity	Description	Response Time	Examples
SEV1	Critical outage, revenue impact, data loss	5 minutes	Payment flow down, database unreachable, auth broken
SEV2	Major degradation, partial outage	15 minutes	Search unavailable, chat delayed, admin dashboard down
SEV3	Minor degradation, non-critical feature broken	60 minutes	Email notifications delayed, analytics lagging
SEV4	Cosmetic, minimal user impact	Next business day	UI glitch, typo, non-critical page error

Response Team Roles

Role	Responsibility
Incident Commander	Coordinates response, makes go/no-go decisions, communicates with stakeholders
On-call Engineer	Investigates root cause, implements fix, verifies recovery
Communications Lead	Posts status updates, handles customer inquiries
Scribe	Records timeline, decisions, and actions for post-mortem

Response Procedure

1. Detect and Acknowledge

• Alert triggered via Slack, email, or PagerDuty
• On-call engineer acknowledges within response time target
• Create incident channel: #incident-YYYY-MM-DD-short-description

2. Classify and Communicate

• Assign severity based on impact
• Post initial message to #incidents channel:
  • Severity
  • Brief description
  • Current impact
  • Investigator assigned
• Update status page if SEV1 or SEV2

3. Investigate and Mitigate

• Check Sentry for error spikes
• Check Cloudflare dashboard for traffic anomalies
• Check health endpoints: /.health on each Worker
• Check database connectivity and slow queries
• Apply mitigation: rollback, feature flag off, rate limit increase, or hotfix

4. Resolve and Verify

• Confirm root cause identified and fixed
• Verify health endpoints return healthy
• Verify critical flows functional
• Close incident channel
• Post resolution to #incidents

5. Post-Mortem

Within 48 hours of resolution:

• Schedule post-mortem meeting
• Create document covering:
  • Timeline of events
  • Root cause analysis
  • Impact assessment (users affected, revenue impact, duration)
  • What went well
  • What went wrong
  • Action items with owners and deadlines
• Share post-mortem in #incidents
• Track action items to completion

Communication Templates

Initial Incident Message

[SEVX] Incident started at <timestamp>
Description: <brief description>
Impact: <users/features affected>
Investigator: <name>
Status: Investigating
Next update: <time>

Resolution Message

[SEVX] Resolved at <timestamp>
Duration: <minutes>
Root cause: <brief>
Fix: <brief>
Post-mortem: <link or "scheduled for <date>">

Escalation Path

1. On-call engineer (5 min)
2. Engineering lead (15 min if SEV1)
3. CTO (30 min if SEV1 unresolved)
4. External support (Cloudflare, Supabase, Stripe) as needed