Background Job Processing: BullMQ, Temporal, and Reliable Job Queues

Any work that takes more than a second doesn't belong in an HTTP request handler. Sending emails, processing images, generating reports, triggering webhooks, syncing external data — these belong in background jobs that run asynchronously, can be retried on failure, and don't affect your API's response time.

The question is which system to use. BullMQ is the right choice for most Node.js workloads. Temporal is the right choice when you need durable, long-running workflows that survive process restarts.

---

When to Use What

---

BullMQ: Production Setup

BullMQ uses Redis as a backing store. Jobs are durable (survive worker restarts), can be retried, prioritized, and rate-limited.

// lib/queues/index.ts
import { Queue, Worker, QueueEvents, Job } from 'bullmq';
import { redis } from '../redis';

// Queue configuration
const queueOpts = {
  connection: redis,
  defaultJobOptions: {
    attempts: 3,
    backoff: {
      type: 'exponential',
      delay: 5000,  // 5s, 10s, 20s
    },
    removeOnComplete: { count: 1000 },  // Keep last 1000 completed jobs
    removeOnFail: { count: 5000 },      // Keep last 5000 failed jobs
  },
};

// Define queues
export const emailQueue = new Queue('email', queueOpts);
export const webhookQueue = new Queue('webhook', queueOpts);
export const reportQueue = new Queue('report', {
  ...queueOpts,
  defaultJobOptions: {
    ...queueOpts.defaultJobOptions,
    attempts: 1,       // Reports: fail fast, don't retry
    timeout: 120_000,  // 2-minute timeout
  },
});

// Priority queue for urgent work
export const notificationQueue = new Queue('notification', {
  connection: redis,
});

// lib/queues/emailWorker.ts
import { Worker, Job } from 'bullmq';
import { sendEmail } from '../email';
import { redis } from '../redis';

interface EmailJobData {
  to: string;
  subject: string;
  template: string;
  data: Record<string, unknown>;
  tenantId: string;
}

// Worker processes jobs — one at a time (or concurrently if configured)
const emailWorker = new Worker<EmailJobData>(
  'email',
  async (job: Job<EmailJobData>) => {
    const { to, subject, template, data, tenantId } = job.data;

    // Update job progress (visible in Bull Board UI)
    await job.updateProgress(10);

    // Log with job context
    job.log(`Sending ${template} email to ${to}`);

    await sendEmail({ to, subject, template, data });

    await job.updateProgress(100);
    return { sentAt: new Date().toISOString() };
  },
  {
    connection: redis,
    concurrency: 10,       // Process 10 emails simultaneously
    limiter: {
      max: 100,            // Max 100 emails per 10 seconds (rate limit)
      duration: 10_000,
    },
  }
);

// Event handlers for monitoring
emailWorker.on('completed', (job, result) => {
  console.info({ jobId: job.id, queue: 'email', event: 'completed', result });
});

emailWorker.on('failed', (job, error) => {
  console.error({
    jobId: job?.id,
    queue: 'email',
    event: 'failed',
    error: error.message,
    attemptsMade: job?.attemptsMade,
    data: job?.data,
  });
});

emailWorker.on('stalled', (jobId) => {
  // Job was taken by a worker that died — BullMQ auto-retries stalled jobs
  console.warn({ jobId, event: 'stalled' });
});

---

Job Enqueueing Patterns

// Enqueue a job
await emailQueue.add('welcome-email', {
  to: user.email,
  subject: 'Welcome to Acme!',
  template: 'welcome',
  data: { name: user.name },
  tenantId: user.tenantId,
});

// Delayed job (send after 1 hour)
await emailQueue.add(
  'trial-reminder',
  { to: user.email, template: 'trial-day-7', data: {}, tenantId: user.tenantId },
  { delay: 60 * 60 * 1000 }  // 1 hour in ms
);

// Repeating job (cron)
await reportQueue.add(
  'monthly-billing-report',
  { type: 'billing', period: 'current-month' },
  { repeat: { cron: '0 9 1 * *' } }  // 9am on 1st of every month
);

// Priority job (0 = highest priority, higher number = lower priority)
await notificationQueue.add(
  'security-alert',
  { userId, message: 'Unusual login detected' },
  { priority: 1 }  // Process before standard notifications (priority: 10)
);

// Bulk enqueue (efficient for fan-out)
const jobs = users.map(user => ({
  name: 'weekly-digest',
  data: { userId: user.id, tenantId: user.tenantId },
  opts: { delay: Math.random() * 60_000 },  // Spread over 1 minute
}));
await emailQueue.addBulk(jobs);

---

Dead Letter Queue Pattern

Jobs that exhaust all retries need to land somewhere for investigation — not just disappear:

// lib/queues/deadLetterWorker.ts
const dlqWorker = new Worker(
  'dead-letter',
  async (job: Job) => {
    // Dead-letter jobs are for visibility and manual retry, not processing
    console.error({
      event: 'dead_letter',
      originalQueue: job.data.originalQueue,
      jobId: job.data.originalJobId,
      error: job.data.lastError,
      data: job.data.originalData,
    });

    // Alert engineering team
    await sendSlackAlert({
      channel: '#engineering-alerts',
      text: `Dead letter job: queue=${job.data.originalQueue} error=${job.data.lastError}`,
    });

    // Save to DB for investigation
    await db.deadLetterJobs.create({
      originalQueue: job.data.originalQueue,
      originalJobId: job.data.originalJobId,
      error: job.data.lastError,
      data: job.data.originalData,
      deadAt: new Date(),
    });
  },
  { connection: redis }
);

// Catch failed jobs and route to DLQ
const queueEvents = new QueueEvents('email', { connection: redis });
queueEvents.on('failed', async ({ jobId }) => {
  const job = await emailQueue.getJob(jobId);
  if (!job) return;

  // If all retries exhausted
  if (job.attemptsMade >= (job.opts.attempts ?? 1)) {
    await deadLetterQueue.add('failed-email', {
      originalQueue: 'email',
      originalJobId: jobId,
      lastError: job.failedReason,
      originalData: job.data,
    });
  }
});

---

Temporal: Durable Workflows

Temporal is the right tool when a workflow spans hours/days, involves external services that may be unavailable, or requires human approval steps.

// workflows/subscriptionRenewal.ts
import { proxyActivities, sleep, condition } from '@temporalio/workflow';
import type * as activities from './activities';

const { chargeCard, sendRenewalEmail, updateSubscriptionStatus, notifyFinanceTeam } =
  proxyActivities<typeof activities>({
    startToCloseTimeout: '30 seconds',
    retry: {
      maximumAttempts: 3,
      initialInterval: '5 seconds',
      backoffCoefficient: 2,
    },
  });

// Workflow: durable subscription renewal
// Survives worker restarts — state is persisted by Temporal server
export async function subscriptionRenewalWorkflow(params: {
  tenantId: string;
  amountCents: number;
  paymentMethodId: string;
}): Promise<{ success: boolean; invoiceId?: string }> {

  await sendRenewalEmail({ tenantId: params.tenantId, type: 'upcoming' });

  // Wait 24 hours before charging (Temporal handles this durably)
  await sleep('24 hours');

  try {
    // Attempt charge
    const result = await chargeCard({
      tenantId: params.tenantId,
      amountCents: params.amountCents,
      paymentMethodId: params.paymentMethodId,
    });

    await updateSubscriptionStatus({ tenantId: params.tenantId, status: 'active' });
    return { success: true, invoiceId: result.invoiceId };

  } catch (error) {
    // Charge failed — retry with exponential backoff over 3 days
    let attempts = 0;
    while (attempts < 3) {
      attempts++;
      await sleep(`${attempts * 24} hours`);

      try {
        const result = await chargeCard({
          tenantId: params.tenantId,
          amountCents: params.amountCents,
          paymentMethodId: params.paymentMethodId,
        });

        await updateSubscriptionStatus({ tenantId: params.tenantId, status: 'active' });
        return { success: true, invoiceId: result.invoiceId };
      } catch {
        await sendRenewalEmail({ tenantId: params.tenantId, type: 'payment-failed' });
      }
    }

    // All attempts failed — suspend account
    await updateSubscriptionStatus({ tenantId: params.tenantId, status: 'suspended' });
    await notifyFinanceTeam({ tenantId: params.tenantId, reason: 'payment-failure' });
    return { success: false };
  }
}

// Start the workflow from your API
import { Client, Connection } from '@temporalio/client';

const connection = await Connection.connect({ address: 'temporal:7233' });
const client = new Client({ connection });

// Kick off renewal workflow — Temporal handles durability
await client.workflow.start(subscriptionRenewalWorkflow, {
  taskQueue: 'subscriptions',
  workflowId: `renewal-${tenantId}-${billingPeriod}`,
  args: [{ tenantId, amountCents, paymentMethodId }],
});

---

Monitoring: Bull Board UI

// app.ts — expose Bull Board for job monitoring
import { createBullBoard } from '@bull-board/api';
import { BullMQAdapter } from '@bull-board/api/bullMQAdapter';
import { FastifyAdapter } from '@bull-board/fastify';

const serverAdapter = new FastifyAdapter();
serverAdapter.setBasePath('/admin/queues');

createBullBoard({
  queues: [
    new BullMQAdapter(emailQueue),
    new BullMQAdapter(webhookQueue),
    new BullMQAdapter(reportQueue),
  ],
  serverAdapter,
});

app.register(serverAdapter.registerPlugin(), { prefix: '/admin/queues' });
// Access at http://localhost:3000/admin/queues
// Protect with auth middleware in production

---

Working With Viprasol

We design and implement background job systems for production applications — BullMQ for standard async processing, Temporal for complex durable workflows, monitoring and alerting, and migration from ad-hoc processing to reliable queues.

→ Talk to our team about background job infrastructure.

---

See Also

• Redis Use Cases — Redis as BullMQ backing store
• Webhook Design Patterns — webhooks as a background job use case
• Event-Driven Architecture — event-driven alternatives to job queues
• Observability and Monitoring — monitoring job queues in production
• Web Development Services — backend architecture and async processing