Distributed Systems Patterns: Saga, Outbox, Idempotency Keys, and Two-Phase Commit

Distributed systems fail in ways that single-process systems don't. A network call succeeds on one side but times out on the other. A message is delivered twice. A service crashes after writing to its database but before emitting an event. These failure modes require specific patterns to handle correctly.

---

The Core Problem: Dual Writes

The root cause of most distributed consistency bugs is the dual write problem: you need to update your database AND publish an event (or call another service), but you can't do both atomically.

// ❌ Dual write — this breaks in production
async function placeOrder(userId: string, items: CartItem[]): Promise<Order> {
  const order = await db.orders.create({ userId, items, status: 'PENDING' });

  // If the service crashes here, the order is in the DB but no event is published.
  // Downstream services never know about this order.
  await eventBus.publish('order.created', { orderId: order.id });

  return order;
}

Three patterns to solve this:

1. Transactional Outbox — write the event to the database in the same transaction, publish separately
2. Saga — coordinate multi-step distributed transactions with compensating actions
3. Idempotent consumers — handle duplicate delivery safely

---

The Transactional Outbox Pattern

Write events to an outbox table in the same database transaction as your data change. A separate process polls the outbox and publishes events to the message bus.

-- Outbox table
CREATE TABLE outbox_events (
  id          UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  topic       TEXT NOT NULL,           -- 'order.created', 'payment.processed'
  payload     JSONB NOT NULL,
  created_at  TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  published_at TIMESTAMPTZ,            -- NULL = not yet published
  attempts    INTEGER NOT NULL DEFAULT 0
);

CREATE INDEX idx_outbox_unpublished ON outbox_events(created_at)
  WHERE published_at IS NULL;

// Atomic: write to orders AND outbox in one transaction
async function placeOrder(userId: string, items: CartItem[]): Promise<Order> {
  return db.$transaction(async (tx) => {
    const order = await tx.orders.create({
      data: { userId, status: 'PENDING', items: { create: items } },
    });

    // Write event to outbox — same transaction, guaranteed consistency
    await tx.outboxEvents.create({
      data: {
        topic: 'order.created',
        payload: { orderId: order.id, userId, items },
      },
    });

    return order;
  });
}

// Outbox relay: runs as a background job, polls and publishes
async function relayOutboxEvents(): Promise<void> {
  while (true) {
    const events = await db.outboxEvents.findMany({
      where: { publishedAt: null, attempts: { lt: 5 } },
      orderBy: { createdAt: 'asc' },
      take: 100,
    });

    for (const event of events) {
      try {
        await messageBus.publish(event.topic, event.payload);

        await db.outboxEvents.update({
          where: { id: event.id },
          data: { publishedAt: new Date() },
        });
      } catch (err) {
        await db.outboxEvents.update({
          where: { id: event.id },
          data: { attempts: { increment: 1 } },
        });
        logger.error({ event: event.id, err }, 'Failed to publish outbox event');
      }
    }

    await sleep(1_000);  // Poll every second
  }
}

Optimizations:

• Use pg_notify + LISTEN to trigger relay immediately instead of polling
• Use FOR UPDATE SKIP LOCKED for multiple relay workers (no duplicate publishing)

-- Multiple relay workers: claim events without blocking each other
SELECT * FROM outbox_events
WHERE published_at IS NULL AND attempts < 5
ORDER BY created_at
LIMIT 10
FOR UPDATE SKIP LOCKED;

---

Idempotency Keys

Idempotency means: if you receive the same request twice, you produce the same result without side effects the second time.

Critical for: payment processing, email sending, any write operation that clients retry.

// Idempotency key: client generates a unique key per operation
// Server stores the result; if key seen again, return stored result

async function processPayment(
  idempotencyKey: string,
  params: PaymentParams,
): Promise<PaymentResult> {
  // Check if we've already processed this request
  const existing = await db.idempotencyKeys.findUnique({
    where: { key: idempotencyKey },
  });

  if (existing) {
    if (existing.status === 'PROCESSING') {
      throw new ConflictError('Request is already being processed');
    }
    // Return the cached result — don't charge twice
    return existing.result as PaymentResult;
  }

  // Claim the key (with a unique constraint preventing races)
  try {
    await db.idempotencyKeys.create({
      data: {
        key: idempotencyKey,
        status: 'PROCESSING',
        createdAt: new Date(),
      },
    });
  } catch (e) {
    if (isUniqueConstraintError(e)) {
      throw new ConflictError('Duplicate request');
    }
    throw e;
  }

  // Process the payment
  try {
    const result = await stripe.paymentIntents.create({
      amount: params.amountCents,
      currency: params.currency,
      customer: params.customerId,
      idempotency_key: idempotencyKey,  // Stripe also supports idempotency keys
    });

    const paymentResult: PaymentResult = {
      status: 'success',
      transactionId: result.id,
      amountCents: result.amount,
    };

    // Store result for future duplicate requests
    await db.idempotencyKeys.update({
      where: { key: idempotencyKey },
      data: { status: 'COMPLETE', result: paymentResult },
    });

    return paymentResult;
  } catch (err) {
    await db.idempotencyKeys.update({
      where: { key: idempotencyKey },
      data: { status: 'FAILED', error: String(err) },
    });
    throw err;
  }
}

// Client: always send idempotency key on retries
async function checkout(cartId: string): Promise<void> {
  // Generate once, persist, reuse on retry
  const idempotencyKey = `checkout-${cartId}-${Date.now()}`;
  localStorage.setItem(`checkout-key-${cartId}`, idempotencyKey);

  const key = localStorage.getItem(`checkout-key-${cartId}`) ?? idempotencyKey;

  await api.post('/payments', { cartId }, {
    headers: { 'Idempotency-Key': key },
  });
}

---

The Saga Pattern

A Saga is a sequence of local transactions. Each step publishes an event that triggers the next step. If any step fails, compensating transactions undo the previous steps.

Choreography-based Saga (event-driven, decentralized):

// Order Saga — choreography style
// Each service listens to events and reacts

// 1. Order Service: creates order, emits event
async function handlePlaceOrder(cmd: PlaceOrderCommand) {
  const order = await db.orders.create({ ...cmd, status: 'PENDING' });
  await eventBus.publish('order.placed', { orderId: order.id, userId: cmd.userId });
}

// 2. Inventory Service: listens, reserves inventory
eventBus.subscribe('order.placed', async (event) => {
  const reserved = await tryReserveInventory(event.orderId);
  if (reserved) {
    await eventBus.publish('inventory.reserved', { orderId: event.orderId });
  } else {
    await eventBus.publish('inventory.reservation.failed', { orderId: event.orderId });
  }
});

// 3. Payment Service: listens to inventory.reserved, charges
eventBus.subscribe('inventory.reserved', async (event) => {
  const order = await getOrder(event.orderId);
  const charged = await tryChargePayment(order);
  if (charged) {
    await eventBus.publish('payment.processed', { orderId: event.orderId });
  } else {
    await eventBus.publish('payment.failed', { orderId: event.orderId });
  }
});

// 4a. Order Service: payment processed — mark order confirmed
eventBus.subscribe('payment.processed', async (event) => {
  await db.orders.update({ id: event.orderId, status: 'CONFIRMED' });
  await eventBus.publish('order.confirmed', { orderId: event.orderId });
});

// 4b. Compensation: payment failed — release inventory
eventBus.subscribe('payment.failed', async (event) => {
  await releaseInventory(event.orderId);  // Compensating transaction
  await db.orders.update({ id: event.orderId, status: 'PAYMENT_FAILED' });
  await eventBus.publish('order.failed', { orderId: event.orderId });
});

Orchestration-based Saga (centralized coordinator, easier to understand):

// Saga Orchestrator: owns the entire workflow
class OrderSagaOrchestrator {
  async execute(orderId: string): Promise<void> {
    const saga = await db.sagas.create({
      data: { orderId, state: 'STARTED', steps: [] }
    });

    try {
      // Step 1: Reserve inventory
      await this.step(saga.id, 'RESERVE_INVENTORY', async () => {
        await inventoryService.reserve(orderId);
      }, async () => {
        await inventoryService.release(orderId);  // Compensate
      });

      // Step 2: Process payment
      await this.step(saga.id, 'PROCESS_PAYMENT', async () => {
        await paymentService.charge(orderId);
      }, async () => {
        await paymentService.refund(orderId);  // Compensate
      });

      // Step 3: Fulfill order
      await this.step(saga.id, 'FULFILL_ORDER', async () => {
        await fulfillmentService.schedule(orderId);
      }, null);  // No compensation for fulfillment start

      await db.sagas.update({ id: saga.id, state: 'COMPLETE' });
    } catch (err) {
      await this.compensate(saga.id);
      throw err;
    }
  }

  private async step(
    sagaId: string,
    name: string,
    action: () => Promise<void>,
    compensate: (() => Promise<void>) | null,
  ): Promise<void> {
    await action();
    await db.sagaSteps.create({ data: { sagaId, name, compensate: !!compensate } });
  }

  private async compensate(sagaId: string): Promise<void> {
    const steps = await db.sagaSteps.findMany({
      where: { sagaId },
      orderBy: { createdAt: 'desc' },  // Reverse order
    });

    for (const step of steps) {
      if (step.compensate) {
        await this.runCompensation(step.name);
      }
    }

    await db.sagas.update({ id: sagaId, state: 'COMPENSATED' });
  }
}

---

Two-Phase Commit: Why to Avoid It

2PC coordinates a transaction across multiple databases. Both parties must vote "commit" for the transaction to succeed.

The problem: 2PC requires all participants to be available. If the coordinator crashes after participants vote "yes" but before sending "commit", participants are locked in uncertainty indefinitely. This defeats the distributed system's availability and can cause cascading locks.

Use 2PC only when: you control both databases, they both support XA transactions, and you accept the availability tradeoff (rare in SaaS).

Use Saga when: you have microservices with independent databases, which is the common case.

---

Working With Viprasol

We design and implement reliable distributed systems — transactional outbox patterns, saga orchestrators, idempotent payment processing, and event-driven architectures that handle failures gracefully.

→ Talk to our team about distributed systems architecture.

---

See Also

• Event-Driven Architecture — event bus patterns and message queues
• Event Sourcing — append-only event log as the source of truth
• API Rate Limiting — idempotency in rate-limited APIs
• Microservices Architecture — when distributed patterns become necessary
• Cloud Solutions — distributed systems architecture and reliability