Security Incident Response in 2026: Runbooks, Detection, Containment, and Post-Mortems

Every company eventually has a security incident. The difference between one that causes lasting damage and one you recover from cleanly is entirely preparation. A team that has never discussed incident response will spend the first hour of a breach arguing about who's in charge; a team with runbooks and rehearsed procedures will have the attacker evicted and notifications drafted in the same time.

This post covers the full incident response lifecycle: detection, triage, containment, eradication, recovery, and the post-mortem that prevents recurrence.

---

Incident Response Framework (NIST SP 800-61)

1. Preparation     → Runbooks, tooling, training — before incidents happen
2. Detection       → Something is wrong; what exactly?
3. Containment     → Stop the bleeding; limit damage
4. Eradication     → Remove the threat; close the attack vector
5. Recovery        → Restore normal operations safely
6. Post-Incident   → Learn, document, prevent recurrence

---

Severity Classification

Incident Severity Levels

P0 — Critical (page immediately, all-hands)

• Active data breach: customer PII or credentials exposed
• Ransomware or destructive attack
• Complete service outage with data loss risk
• Payment system compromise Response time: 15 minutes. Notify executive team + legal immediately.

P1 — High (page on-call, escalate if unresolved in 1 hour)

• Suspected unauthorized access to production systems
• Credential compromise (employee or service account)
• Service outage affecting >25% of customers
• DDoS attack affecting availability Response time: 30 minutes. Notify engineering leadership.

P2 — Medium (acknowledge in 2 hours, resolve in 24 hours)

• Vulnerability discovered in production (not yet exploited)
• Anomalous access patterns detected (possible reconnaissance)
• Partial service degradation Response time: 2 hours. Ticket created; assigned to security engineer.

P3 — Low (resolve within 1 week)

• Low-severity CVE in dependency (no active exploit)
• Security misconfiguration discovered (non-critical)
• Suspicious but low-confidence activity Response time: Next business day. Add to security backlog.

---

## Detection: What to Monitor

```yaml
# datadog-security-monitors.yaml — example alert configurations

# 1. Authentication anomalies
- name: "Brute Force Login Attempts"
  query: "sum(last_5m):sum:auth.login.failures{*} by {ip_address} > 20"
  message: |
    IP {{ip_address.name}} has failed login 20+ times in 5 minutes.
    Possible brute force attack.
    Runbook: https://runbooks.internal/security/brute-force
  priority: P1

# 2. Privilege escalation
- name: "Unexpected Admin Role Assignment"
  query: "sum(last_5m):sum:audit.role_change{role:admin} > 0"
  message: |
    Admin role assigned. Verify this was intentional.
    If unexpected: follow credential compromise runbook immediately.
  priority: P0

# 3. Data exfiltration indicators
- name: "Unusually Large Data Export"
  query: "sum(last_1h):sum:api.data_export.rows{*} by {user_id} > 100000"
  message: |
    User {{user_id.name}} exported >100K rows in 1 hour.
    This may indicate data exfiltration. Verify with account owner.
  priority: P1

# 4. Database access from unexpected source
- name: "Database Connection from Unknown IP"
  query: "sum(last_5m):count:db.connection.new{env:prod,!ip_address:known_range} > 0"
  priority: P0

# 5. New IAM credentials or access keys
- name: "New AWS Access Key Created"
  query: "sum(last_5m):count:aws.cloudtrail{event_name:CreateAccessKey} > 0"
  priority: P1

// src/security/audit-logger.ts — Structured audit logging for security events
import { logger } from '@/lib/logger';

export type SecurityEventType =
  | 'auth.login.success'
  | 'auth.login.failure'
  | 'auth.mfa.bypass_attempted'
  | 'auth.password.reset'
  | 'auth.session.revoked_all'
  | 'data.export.initiated'
  | 'data.bulk_delete'
  | 'admin.role.assigned'
  | 'admin.user.impersonated'
  | 'api_key.created'
  | 'api_key.deleted';

export function auditLog(
  event: SecurityEventType,
  context: {
    userId?: string;
    actorId?: string;   // Who performed the action (may differ for impersonation)
    ipAddress: string;
    userAgent?: string;
    resourceId?: string;
    metadata?: Record<string, unknown>;
  },
): void {
  logger.info({
    type: 'security_audit',
    event,
    ...context,
    timestamp: new Date().toISOString(),
  });

  // For high-severity events: also write to immutable audit store
  if (isHighSeverityEvent(event)) {
    writeToImmutableAuditLog({ event, ...context });
  }
}

function isHighSeverityEvent(event: SecurityEventType): boolean {
  return [
    'auth.mfa.bypass_attempted',
    'admin.role.assigned',
    'admin.user.impersonated',
    'data.bulk_delete',
  ].includes(event);
}

// Write to append-only S3 bucket (bucket policy: no deletes)
async function writeToImmutableAuditLog(entry: object): Promise<void> {
  const { S3Client, PutObjectCommand } = await import('@aws-sdk/client-s3');
  const s3 = new S3Client({ region: 'us-east-1' });

  await s3.send(new PutObjectCommand({
    Bucket: process.env.AUDIT_LOG_BUCKET!,
    Key: `audit/${new Date().toISOString().split('T')[0]}/${Date.now()}-${crypto.randomUUID()}.json`,
    Body: JSON.stringify(entry),
    ContentType: 'application/json',
  }));
}

---

Incident Response Runbook: Credential Compromise

# Runbook: Credential Compromise
**Severity**: P0–P1 | **Owner**: Security On-Call
**Last updated**: 2026-08-30

## Indicators
- Employee reports phishing click or credential theft
- Impossible travel (login from two countries within 1 hour)
- AWS CloudTrail shows access from unknown IP
- GitHub shows commit from unrecognized device

Immediate Actions (0–15 minutes)

Step 1: Identify compromised accounts

# Find all active sessions for the user
psql $DATABASE_URL -c "
  SELECT id, user_agent, ip_address, created_at, last_active
  FROM sessions WHERE user_id = '<user_id>'
  ORDER BY last_active DESC;"

# Check AWS CloudTrail for recent API calls
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=Username,AttributeValue=<username> \
  --start-time $(date -d '24 hours ago' -u +%Y-%m-%dT%H:%M:%SZ) \
  --region us-east-1

Step 2: Revoke all sessions (kills attacker access)

# Revoke all refresh tokens for the user
psql $DATABASE_URL -c "
  UPDATE refresh_tokens SET revoked_at = now()
  WHERE user_id = '<user_id>' AND revoked_at IS NULL;
  DELETE FROM sessions WHERE user_id = '<user_id>';"

# Rotate AWS credentials immediately
aws iam delete-access-key --access-key-id <key_id> --user-name <username>
aws iam create-access-key --user-name <username>

Step 3: Assess blast radius

• What data did the attacker have access to?
• Did they create new credentials (IAM keys, API keys)?
• Did they modify any infrastructure?
• Did they access customer data?

# Check for new IAM keys created during compromise window
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=CreateAccessKey \
  --start-time <compromise_start> --end-time <revocation_time>

# Check for new admin users created
psql $DATABASE_URL -c "
  SELECT id, email, created_at FROM users
  WHERE role = 'admin'
  AND created_at BETWEEN '<compromise_start>' AND '<revocation_time>';"

Step 4: Notify (if customer data accessed)

• If any customer PII accessed: escalate to P0, notify legal within 1 hour
• GDPR requires 72-hour notification to DPA if personal data breached
• CCPA requires "expedient" notification to affected California residents

Recovery

• Force password reset for compromised account
• Enable MFA if not already required
• Review and rotate any secrets the compromised account could access
• Verify no backdoors left (new admin accounts, IAM policies, webhooks)

---

Containment Automation

// src/security/containment.ts — Automated incident response actions
export async function containCompromisedUser(
  userId: string,
  reason: string,
  actorId: string,
): Promise<ContainmentReport> {
  const actions: string[] = [];

  // 1. Revoke all active sessions
  await db.query(
    `UPDATE refresh_tokens SET revoked_at = now(), revocation_reason = $1
     WHERE user_id = $2 AND revoked_at IS NULL`,
    [reason, userId],
  );
  actions.push('sessions_revoked');

  // 2. Suspend account (prevent new logins)
  await db.query(
    `UPDATE users SET status = 'suspended', suspended_at = now(), suspended_reason = $1
     WHERE id = $2`,
    [reason, userId],
  );
  actions.push('account_suspended');

  // 3. Revoke all API keys
  await db.query(
    `UPDATE api_keys SET revoked_at = now() WHERE user_id = $1 AND revoked_at IS NULL`,
    [userId],
  );
  actions.push('api_keys_revoked');

  // 4. Invalidate cache entries for this user
  const cacheKeys = await redis.keys(`user:${userId}:*`);
  if (cacheKeys.length > 0) await redis.del(...cacheKeys);
  actions.push('cache_invalidated');

  // 5. Audit log
  auditLog('auth.session.revoked_all', {
    userId,
    actorId,
    ipAddress: 'system',
    metadata: { reason, actionsToken: actions },
  });

  return { userId, actions, containedAt: new Date().toISOString() };
}

---

Post-Mortem Template

# Security Incident Post-Mortem
**Incident ID**: SEC-2026-042
**Severity**: P1
**Date**: 2026-08-30
**Duration**: 2h 15m (detected 14:23 → contained 16:38 UTC)
**Author**: [Security Engineer]
**Status**: Draft → Review → Published

---

Summary

[2-3 sentences: what happened, impact, resolution] A phishing email compromised one employee's credentials. The attacker accessed the admin dashboard for ~45 minutes before anomalous access patterns triggered an alert. No customer data was exfiltrated. The employee's account was contained and all credentials rotated.

Timeline

Impact

• Systems affected: Admin dashboard (read-only access confirmed via audit logs)
• Customer data accessed: None confirmed
• Duration of unauthorized access: ~45 minutes (13:52 – 15:10)
• Services disrupted: None

Root Cause

Phishing email bypassed email security filters. Employee was not enrolled in MFA. Admin dashboard did not require MFA even for admin accounts.

What Went Well

• Anomaly detection alert fired within 36 minutes of compromise
• On-call response was within target SLA (15 min)
• Containment playbook worked as documented
• Blast radius was limited by least-privilege access controls

What Went Poorly

• MFA not enforced for admin accounts despite policy requiring it
• Alert-to-page latency was 2 minutes (should be < 1 min)
• Forensic log retention was only 30 days (needs 90 days minimum)

Action Items

Blameless Principle

This post-mortem focuses on system failures, not individual failures. The employee who clicked the phishing link is not at fault — the system allowed admin access without MFA. Fix the system.

---

Working With Viprasol

We design security incident response programs and implement the detection, runbook, and containment infrastructure for engineering teams.

What we deliver:

• Security monitoring and alerting setup (DataDog, CloudWatch, SIEM)
• Incident response runbook library (credential compromise, data breach, DDoS, ransomware)
• Containment automation scripts and account lockdown procedures
• Breach notification workflow (GDPR 72h, CCPA, state notification laws)
• Post-mortem facilitation and action item tracking

→ Request a security incident response review → Cloud and security services

---

See Also

• Zero Trust Security Architecture
• SaaS Security Checklist
• Data Privacy Engineering and GDPR
• Observability: SLI, SLO, and Error Budgets
• Cloud Native Security