Incident Management: On-Call Culture, Blameless Postmortems, and Runbooks

Every engineering team will have production incidents. The difference between high-performing and average teams isn't that high-performing teams have fewer incidents — it's that they recover faster, learn more from them, and systematically prevent recurrence.

This guide covers the process and culture that makes incident management a competitive advantage rather than a recurring crisis.

---

Incident Severity Levels

Consistent severity definitions let everyone understand impact and set response expectations:

Define these once. Put them in your engineering handbook. Make sure on-call engineers can apply them consistently without debate at 2am.

---

On-Call Rotation Design

What makes on-call sustainable:

• Maximum 1 week on-call per month per engineer
• Incidents during nights/weekends earn compensatory time off
• Clear escalation path (primary → secondary → engineering lead)
• New engineers shadow for 4 weeks before going primary
• On-call budget: time to fix what you're paged for, including root cause

What kills on-call culture:

• Alert fatigue (> 3 alerts per shift that aren't actionable)
• No escalation path (you're on your own at 3am)
• No time to fix problems encountered during on-call (technical debt compounds)
• No postmortems (same incidents recur)

# PagerDuty schedule via Terraform
resource "pagerduty_schedule" "engineering_primary" {
  name      = "Engineering Primary On-Call"
  time_zone = "America/New_York"

  layer {
    name                         = "Weekly Rotation"
    start                        = "2026-06-01T00:00:00-05:00"
    rotation_turn_length_seconds = 604800  # 1 week
    rotation_virtual_start       = "2026-06-01T00:00:00-05:00"

    users = [
      pagerduty_user.alice.id,
      pagerduty_user.bob.id,
      pagerduty_user.carol.id,
      pagerduty_user.david.id,
    ]
  }
}

resource "pagerduty_escalation_policy" "engineering" {
  name = "Engineering Escalation"

  rule {
    escalation_delay_in_minutes = 10
    target {
      type = "schedule_reference"
      id   = pagerduty_schedule.engineering_primary.id
    }
  }

  rule {
    escalation_delay_in_minutes = 10
    target {
      type = "schedule_reference"
      id   = pagerduty_schedule.engineering_secondary.id
    }
  }

  rule {
    escalation_delay_in_minutes = 10
    target {
      type = "user_reference"
      id   = pagerduty_user.engineering_lead.id
    }
  }
}

---

Incident Response Process

During an incident, clarity of roles prevents chaotic pile-ons where everyone is debugging simultaneously but no one is coordinating:

For small teams: IC + Technical Lead can be the same person for P2/P3. Always separate for P0/P1.

Incident Slack channel naming convention:

#inc-2026-0527-api-checkout-500
    ^year ^date  ^system ^symptom

---

Runbook Template

Runbooks are pre-written response guides for known failure modes. The time to write a runbook is not during an incident.

# Runbook: High API Error Rate (5xx)

**Last Updated:** 2026-05-20 | **Owner:** @alice | **Review Cycle:** Quarterly

When to Use This Runbook

Alert: api_error_rate_5xx > 1% for 5 minutes Or: Customer reports widespread errors

Severity Assessment

• 5% error rate, P0/P1 users affected → P0

• 1-5% error rate, some users affected → P1
• < 1% error rate, specific endpoints → P2

Step 1: Confirm the Alert (2 min)

# Check current error rate
curl -s "https://api.datadoghq.com/api/v1/query?query=sum:api.errors.5xx%7B*%7D.as_rate()" \
  -H "DD-API-KEY: $DATADOG_API_KEY" | jq '.series[0].pointlist[-1][1]'

# Check which endpoints are erroring
open "https://app.datadoghq.com/apm/services/api"
# Filter by: error rate, last 15 minutes

Step 2: Check Recent Deployments (2 min)

# Recent ECS deploys
aws ecs describe-services --cluster production --services api \
  --query 'services[0].deployments[*].{status:status,taskDef:taskDefinition,updated:updatedAt}'

# Recent GitHub merges
gh pr list --state merged --limit 5 --json title,mergedAt,author

If a deploy happened in the last 30 minutes: rollback first, investigate later.

Step 3: Rollback (if deploy is suspected cause) (5 min)

# Roll back to previous ECS task definition
PREVIOUS_TASK_DEF=$(aws ecs describe-services \
  --cluster production --services api \
  --query 'services[0].deployments[-1].taskDefinition' --output text)

aws ecs update-service \
  --cluster production \
  --service api \
  --task-definition $PREVIOUS_TASK_DEF \
  --force-new-deployment

Step 4: Check Database (5 min)

# Check for long-running queries
psql $DATABASE_URL -c "
SELECT pid, now() - pg_stat_activity.query_start AS duration, query, state
FROM pg_stat_activity
WHERE (now() - pg_stat_activity.query_start) > interval '30 seconds'
ORDER BY duration DESC;"

# Check connection count
psql $DATABASE_URL -c "SELECT count(*), state FROM pg_stat_activity GROUP BY state;"

# If connection count > 80% of max_connections: restart PgBouncer
ssh bastion "sudo systemctl restart pgbouncer"

Step 5: Check External Dependencies (5 min)

• Stripe status: https://status.stripe.com
• AWS status: https://health.aws.amazon.com
• Redis: redis-cli -u $REDIS_URL ping

Escalation

• P0: Page engineering lead immediately
• If database issue: page DBA on-call
• If Stripe issue: customer-facing comms + monitor their status page

Communication Templates

Status Page Update (initial):

We are investigating reports of errors affecting [feature]. Our team is actively working on a resolution. Updates every 10 minutes.

Status Page Update (resolved):

The issue affecting [feature] has been resolved. [X]% of requests were affected between [time] and [time]. A postmortem will be published within 48 hours.

---

## Blameless Postmortem Template

Blameless means: no individual is blamed for the incident. Systems failed; we improve systems. This is not naive — it's what makes engineers willing to be honest about mistakes.

```markdown
# Postmortem: Checkout Outage — 2026-05-15

**Severity:** P1 | **Duration:** 47 minutes (14:03–14:50 UTC)
**Impact:** ~30% of checkout attempts failed | **Revenue Impact:** ~$8,400
**Incident Commander:** @alice | **Author:** @bob | **Reviewed by:** @carol

Summary

A database migration that added an index to the orders table without CONCURRENTLY locked the table for 4 minutes. During that time, checkout requests timed out and returned 500 errors.

Timeline

Root Cause

CREATE INDEX orders_user_id_idx ON orders(user_id); without CONCURRENTLY acquired an AccessExclusiveLock. The orders table has 8M rows; the index took 4 minutes to build, during which all DML on the table was blocked.

Why It Wasn't Caught

1. Migration was reviewed but reviewer was not familiar with PostgreSQL lock implications
2. No automated check in CI for blocking DDL statements
3. The staging database had only 10K rows; the migration ran in < 1 second

Action Items

What Went Well

• Alert fired within 2 minutes of incident start
• Root cause identified quickly (pg_stat_activity clearly showed the lock)
• Communication to customers was clear and timely
• Escalation was smooth; IC role was clear

What Didn't Go Well

• Migration review process didn't catch the locking issue
• Staging environment didn't reflect production data volume
• MTTR could have been shorter if rollback procedure was attempted earlier

Lessons

We did not know the CONCURRENTLY keyword was required for production-safe index creation. This knowledge is now documented in our migration runbook, and the CI check ensures it will be automatically caught in future.

---

Reducing Alert Fatigue

If your on-call engineers receive > 5 actionable alerts per week, alert quality is poor. Audit alerts:

# scripts/alert_audit.py — analyze PagerDuty alert patterns
import pdpyras

session = pdpyras.APISession(api_key=PAGERDUTY_API_KEY)

incidents = session.list_all('incidents', params={
    'since': '2026-04-01T00:00:00Z',
    'until': '2026-05-01T00:00:00Z',
    'statuses[]': ['resolved'],
})

from collections import Counter
alert_titles = Counter(inc['title'] for inc in incidents)

print("Top 10 most frequent alerts (candidates for tuning):")
for title, count in alert_titles.most_common(10):
    print(f"  {count}x — {title}")

# Any alert firing > 5 times in a month without leading to a P0/P1:
# Raise threshold, add more context, or delete if not actionable

Alert quality criteria:

• Every alert is actionable (there's something to do when it fires)
• Every alert has a runbook linked
• False positive rate < 10% (9 out of 10 alerts represent real problems)
• Alert fatigue score: < 3 wakeups per on-call shift

---

Working With Viprasol

We help engineering teams build incident management processes — on-call rotation design, runbook libraries, postmortem templates, alert tuning, and PagerDuty/OpsGenie configuration. A mature incident process is a product reliability investment.

→ Talk to our team about reliability and incident management.

---

See Also

• Observability and Monitoring — the alerting foundation
• DevOps Best Practices — CI/CD and deployment practices that prevent incidents
• Engineering Metrics — MTTR as a DORA metric
• Database Migrations — preventing the common class of migration incidents
• Cloud Solutions — reliability and SRE engineering