Webhook Design Patterns: Reliability, Security, and Retry Strategies

Webhooks seem simple — a third party calls your URL when something happens. But production webhook processing is where naive implementations fall apart. Duplicate deliveries crash idempotent-free handlers. Signature skipping lets attackers forge events. Missing retry logic means lost data when your server is briefly down.

This guide covers the patterns that make webhooks reliable enough for financial transactions, subscription billing, and payment processing — the places where dropped events have real business consequences.

---

Webhooks vs Polling: When Each Makes Sense

For third-party integrations (Stripe, GitHub, Shopify, Twilio), webhooks are almost always the right choice. For internal service communication, consider a message queue (SQS, Kafka) instead — you get replay, backpressure, and better observability.

---

Pattern 1: Signature Verification (HMAC-SHA256)

Every production webhook must verify the signature before processing. Without this, any attacker who knows your endpoint URL can forge events — including "payment succeeded" events that trigger fulfillment.

Stripe-style HMAC verification in Node.js:

// webhooks/verifySignature.ts
import crypto from 'crypto';

interface WebhookVerificationResult {
  valid: boolean;
  timestamp?: number;
  error?: string;
}

export function verifyStripeSignature(
  rawBody: Buffer,
  signatureHeader: string,
  webhookSecret: string,
  toleranceSeconds = 300  // Reject events older than 5 minutes
): WebhookVerificationResult {
  // Signature format: t=timestamp,v1=signature1,v1=signature2
  const parts = signatureHeader.split(',');
  const timestampPart = parts.find(p => p.startsWith('t='));
  const v1Signatures = parts
    .filter(p => p.startsWith('v1='))
    .map(p => p.slice(3));

  if (!timestampPart || v1Signatures.length === 0) {
    return { valid: false, error: 'Malformed signature header' };
  }

  const timestamp = parseInt(timestampPart.slice(2), 10);
  const age = Math.floor(Date.now() / 1000) - timestamp;

  if (age > toleranceSeconds) {
    return { valid: false, error: `Event too old: ${age}s` };
  }

  // Compute expected signature
  const signedPayload = `${timestamp}.${rawBody.toString('utf8')}`;
  const expected = crypto
    .createHmac('sha256', webhookSecret)
    .update(signedPayload, 'utf8')
    .digest('hex');

  // Constant-time comparison to prevent timing attacks
  const isValid = v1Signatures.some(sig => {
    try {
      return crypto.timingSafeEqual(
        Buffer.from(sig, 'hex'),
        Buffer.from(expected, 'hex')
      );
    } catch {
      return false;
    }
  });

  return isValid
    ? { valid: true, timestamp }
    : { valid: false, error: 'Signature mismatch' };
}

Critical: You must read the raw body before any JSON parsing. Express's json() middleware destroys the raw body. Use this instead:

// app.ts
app.post('/webhooks/stripe', 
  express.raw({ type: 'application/json' }),  // Raw body preserved
  async (req, res) => {
    const result = verifyStripeSignature(
      req.body,                                // Buffer, not parsed object
      req.headers['stripe-signature'] as string,
      process.env.STRIPE_WEBHOOK_SECRET!
    );

    if (!result.valid) {
      console.warn('Webhook signature invalid:', result.error);
      return res.status(400).json({ error: result.error });
    }

    const event = JSON.parse(req.body.toString());
    await processWebhookEvent(event);
    res.json({ received: true });
  }
);

---

Pattern 2: Idempotent Event Processing

Webhook providers guarantee at-least-once delivery — not exactly-once. Stripe may deliver the same event twice if your server returned a 500 or timed out. Your handler must be safe to call multiple times with the same event.

// webhooks/idempotency.ts
import { PrismaClient } from '@prisma/client';

const prisma = new PrismaClient();

export async function processOnce<T>(
  eventId: string,
  handler: () => Promise<T>
): Promise<{ result: T | null; duplicate: boolean }> {
  // Try to claim the event ID
  try {
    await prisma.processedWebhookEvent.create({
      data: {
        eventId,
        processedAt: new Date(),
      },
    });
  } catch (err: any) {
    // Unique constraint violation = already processed
    if (err.code === 'P2002') {
      console.info(`Duplicate webhook event ${eventId} — skipping`);
      return { result: null, duplicate: true };
    }
    throw err;
  }

  // First time seeing this event — process it
  try {
    const result = await handler();
    return { result, duplicate: false };
  } catch (err) {
    // Remove the lock so it can be retried
    await prisma.processedWebhookEvent.delete({ where: { eventId } });
    throw err;
  }
}

// Usage
async function handlePaymentSucceeded(event: Stripe.Event) {
  await processOnce(event.id, async () => {
    const session = event.data.object as Stripe.PaymentIntent;
    await prisma.order.update({
      where: { stripePaymentIntentId: session.id },
      data: { status: 'PAID', paidAt: new Date() },
    });
    await sendOrderConfirmationEmail(session.metadata.orderId);
  });
}

The Prisma schema for the idempotency table:

model ProcessedWebhookEvent {
  eventId     String   @id
  processedAt DateTime
  
  @@index([processedAt])  // For cleanup jobs
}

Clean up old records with a nightly job: DELETE FROM processed_webhook_events WHERE processed_at < NOW() - INTERVAL '30 days'.

---

Pattern 3: Async Processing with a Queue

Never do heavy work synchronously in a webhook handler. Webhook providers have short timeout windows (Stripe: 30s, GitHub: 10s). If your handler takes too long, the provider retries — triggering duplicate processing.

The correct pattern:

Webhook arrives → validate signature → enqueue job → respond 200
                                          ↓
                                    Worker processes job asynchronously

With BullMQ (Redis-backed):

// webhooks/queue.ts
import { Queue, Worker } from 'bullmq';
import Redis from 'ioredis';

const connection = new Redis(process.env.REDIS_URL!);

export const webhookQueue = new Queue('webhooks', {
  connection,
  defaultJobOptions: {
    attempts: 5,
    backoff: {
      type: 'exponential',
      delay: 1000,  // 1s, 2s, 4s, 8s, 16s
    },
    removeOnComplete: { age: 86400 },   // Keep 24h of completed jobs
    removeOnFail: { count: 100 },        // Keep last 100 failed jobs
  },
});

// Webhook handler — just enqueue
app.post('/webhooks/stripe', express.raw({ type: 'application/json' }), async (req, res) => {
  const result = verifyStripeSignature(req.body, req.headers['stripe-signature'] as string, process.env.STRIPE_WEBHOOK_SECRET!);
  if (!result.valid) return res.status(400).json({ error: result.error });

  await webhookQueue.add('stripe-event', {
    eventId: req.headers['stripe-signature']?.split(',')[0],
    payload: JSON.parse(req.body.toString()),
  });

  res.json({ received: true });  // Respond immediately
});

// Worker — process asynchronously
const worker = new Worker('webhooks', async (job) => {
  const { eventId, payload } = job.data;

  switch (payload.type) {
    case 'payment_intent.succeeded':
      await handlePaymentSucceeded(payload);
      break;
    case 'customer.subscription.deleted':
      await handleSubscriptionCancelled(payload);
      break;
    case 'invoice.payment_failed':
      await handlePaymentFailed(payload);
      break;
    default:
      console.info(`Unhandled event type: ${payload.type}`);
  }
}, { connection });

worker.on('failed', (job, err) => {
  console.error(`Webhook job ${job?.id} failed after ${job?.attemptsMade} attempts:`, err);
});

---

Pattern 4: Sending Webhooks (Outbound)

If your platform sends webhooks to customers, you need the same reliability guarantees on the sending side.

// webhooks/sender.ts
interface WebhookDelivery {
  id: string;
  endpoint: string;
  secret: string;
  payload: object;
  attemptNumber: number;
}

async function deliverWebhook(delivery: WebhookDelivery): Promise<boolean> {
  const timestamp = Math.floor(Date.now() / 1000);
  const body = JSON.stringify(delivery.payload);
  const signature = crypto
    .createHmac('sha256', delivery.secret)
    .update(`${timestamp}.${body}`)
    .digest('hex');

  try {
    const res = await fetch(delivery.endpoint, {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        'X-Webhook-Timestamp': timestamp.toString(),
        'X-Webhook-Signature': `v1=${signature}`,
        'X-Webhook-Delivery-Id': delivery.id,
        'User-Agent': 'YourApp-Webhook/1.0',
      },
      body,
      signal: AbortSignal.timeout(10_000),  // 10s timeout
    });

    await logDeliveryAttempt(delivery.id, res.status, delivery.attemptNumber);
    return res.status >= 200 && res.status < 300;
  } catch (err) {
    await logDeliveryAttempt(delivery.id, 0, delivery.attemptNumber, String(err));
    return false;
  }
}

// Retry schedule: immediately, +1m, +5m, +30m, +2h, +8h, +24h
const RETRY_DELAYS_MS = [0, 60_000, 300_000, 1_800_000, 7_200_000, 28_800_000, 86_400_000];

Expose a delivery log in your dashboard so customers can see webhook history, retry manually, and debug failures. This alone cuts support tickets significantly.

---

Security Checklist

---

Implementation Cost

Senior backend developer rates: $120–150/hr (US). Offshore: $40–70/hr.

---

Working With Viprasol

We've built webhook infrastructure for payment platforms, e-commerce integrations, and SaaS products handling millions of events per month. Our implementations include signature verification, idempotency, async queuing, retry logic, and delivery dashboards — built to handle the edge cases that production traffic exposes.

→ Discuss your integration requirements with our backend team.

---

See Also

• API Security Best Practices — securing the full API surface
• Event-Driven Architecture — Kafka and SQS for internal event processing
• Redis Use Cases — BullMQ, rate limiting, and pub/sub patterns
• Payment Gateway Integration — Stripe webhook implementation end-to-end
• Web Development Services — backend API and integration work