Message Queues with AWS SQS

AWS Simple Queue Service (SQS) is a managed message queuing service. It lets different parts of your application communicate asynchronously by passing messages through a queue.

Why Use a Queue?

Some operations are too slow or unreliable to run inside an HTTP request handler:

Operation	Problem	Solution
Sending email	SMTP can be slow (100–500ms+)	Push to queue, respond immediately
Image resizing	CPU-intensive, blocks event loop	Queue the job, process in background
Calling a third-party API	Can fail or timeout	Queue with retry support
Generating a PDF report	Seconds of processing	Queue it, notify user when done

Instead of doing the work in the route handler, you push a message to SQS and immediately return a response. A separate worker process picks up the message and does the actual work.

SQS Queue Types

Type	When to use
Standard	High throughput, at-least-once delivery, possible out-of-order
FIFO	Exactly-once delivery, strict ordering (e.g., payment processing)

For most use cases (sending emails, background jobs), a Standard queue is sufficient.

Set Up an SQS Queue

1. Go to AWS Console → SQS → Create queue
2. Choose Standard
3. Give it a name like my-app-jobs
4. Under Configuration, set a reasonable Visibility timeout (how long a message is hidden after a worker picks it up — set to longer than your job takes, e.g., 60 seconds)
5. Click Create queue and copy the Queue URL

Install

npm

npm install @aws-sdk/client-sqs dotenv

Yarn

yarn add @aws-sdk/client-sqs dotenv

pnpm

pnpm add @aws-sdk/client-sqs dotenv

Bun

bun add @aws-sdk/client-sqs dotenv

Configure the SQS Client

config/sqs.js

import { SQSClient } from "@aws-sdk/client-sqs";

const sqs = new SQSClient({
  region: process.env.AWS_REGION,
  credentials: {
    accessKeyId: process.env.AWS_ACCESS_KEY_ID,
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
  },
});

export default sqs;

Add the queue URL to .env:

.env

AWS_REGION=us-east-1
AWS_ACCESS_KEY_ID=your_access_key_id
AWS_SECRET_ACCESS_KEY=your_secret_access_key
SQS_QUEUE_URL=https://sqs.us-east-1.amazonaws.com/123456789/my-app-jobs

Sending a Message (Producer)

The Express route is the producer — it pushes a job to the queue and immediately responds to the client.

utils/sendToQueue.js

import { SendMessageCommand } from "@aws-sdk/client-sqs";
import sqs from "../config/sqs.js";

export async function sendToQueue(jobType, payload) {
  const command = new SendMessageCommand({
    QueueUrl: process.env.SQS_QUEUE_URL,
    MessageBody: JSON.stringify({ jobType, payload }),
    MessageAttributes: {
      JobType: {
        DataType: "String",
        StringValue: jobType,
      },
    },
  });

  return sqs.send(command);
}

Express route that queues a job

routes/reports.js

import express from "express";
import { sendToQueue } from "../utils/sendToQueue.js";

const router = express.Router();

// Client requests a report — we queue the work and respond immediately
router.post("/generate", async (req, res) => {
  const { userId, reportType, dateRange } = req.body;

  if (!userId || !reportType) {
    return res.status(400).json({ message: "userId and reportType are required" });
  }

  try {
    await sendToQueue("GENERATE_REPORT", { userId, reportType, dateRange });

    res.status(202).json({
      message: "Report generation started. You'll receive an email when it's ready.",
    });
  } catch (error) {
    res.status(500).json({ message: "Failed to queue report job" });
  }
});

export default router;

HTTP 202 Accepted

Return 202 Accepted (not 200 OK) when the work has been queued but not yet completed. This correctly signals to the client that the request was received and accepted but the result isn't available yet.

Receiving and Processing Messages (Consumer)

The consumer is a separate process (or a background loop in your app) that polls the queue and processes jobs. This is not an HTTP route — it runs independently.

workers/jobWorker.js

import { ReceiveMessageCommand, DeleteMessageCommand } from "@aws-sdk/client-sqs";
import sqs from "../config/sqs.js";
import { sendEmail } from "../utils/sendEmail.js";

async function processMessage(message) {
  const { jobType, payload } = JSON.parse(message.Body);

  console.log(`Processing job: ${jobType}`, payload);

  switch (jobType) {
    case "SEND_WELCOME_EMAIL":
      await sendEmail({
        to: payload.email,
        subject: `Welcome, ${payload.name}!`,
        html: `<p>Welcome to the platform, ${payload.name}!</p>`,
        text: `Welcome, ${payload.name}!`,
      });
      break;

    case "GENERATE_REPORT":
      // Long-running task: generate the report, then email it
      console.log(`Generating ${payload.reportType} report for user ${payload.userId}`);
      // ... report generation logic ...
      break;

    default:
      console.warn(`Unknown job type: ${jobType}`);
  }
}

async function deleteMessage(receiptHandle) {
  const command = new DeleteMessageCommand({
    QueueUrl: process.env.SQS_QUEUE_URL,
    ReceiptHandle: receiptHandle,
  });
  await sqs.send(command);
}

async function pollQueue() {
  console.log("Worker started. Polling for messages...");

  while (true) {
    try {
      const command = new ReceiveMessageCommand({
        QueueUrl: process.env.SQS_QUEUE_URL,
        MaxNumberOfMessages: 5,
        WaitTimeSeconds: 20, // Long polling — waits up to 20s if queue is empty
        MessageAttributeNames: ["All"],
      });

      const response = await sqs.send(command);

      if (!response.Messages || response.Messages.length === 0) {
        continue; // No messages, poll again
      }

      for (const message of response.Messages) {
        try {
          await processMessage(message);
          await deleteMessage(message.ReceiptHandle);
          console.log("Message processed and deleted");
        } catch (err) {
          console.error("Failed to process message:", err);
          // Don't delete — let SQS make it visible again after the visibility timeout
          // so it can be retried (or moved to a Dead Letter Queue)
        }
      }
    } catch (err) {
      console.error("Polling error:", err);
      await new Promise((resolve) => setTimeout(resolve, 5000)); // Wait before retrying
    }
  }
}

pollQueue();

Delete only on success

Only delete a message from SQS after successfully processing it. If processing throws an error, don't delete — SQS will make the message visible again after the visibility timeout, allowing another worker to retry it.

Running the Worker

Run the worker as a separate process alongside your Express server:

package.json

{
  "scripts": {
    "start": "node index.js",
    "worker": "node workers/jobWorker.js",
    "dev": "nodemon index.js",
    "dev:worker": "nodemon workers/jobWorker.js"
  }
}

In development, open two terminals:

# Terminal 1 — Express server
npm run dev

# Terminal 2 — Worker
npm run dev:worker

Dead Letter Queue (DLQ)

A Dead Letter Queue is a separate queue where SQS automatically moves messages that fail to process after a configured number of attempts (the Maximum receives setting).

To set it up:

1. Create a second SQS queue named my-app-jobs-dlq
2. Edit your main queue → Dead-letter queue → select the DLQ
3. Set Maximum receives to 3 (after 3 failed attempts, move to DLQ)

Then monitor your DLQ for messages — these represent jobs your worker consistently failed to process and need manual investigation.

IAM Permissions for SQS

{
  "Effect": "Allow",
  "Action": [
    "sqs:SendMessage",
    "sqs:ReceiveMessage",
    "sqs:DeleteMessage",
    "sqs:GetQueueAttributes"
  ],
  "Resource": "arn:aws:sqs:us-east-1:123456789:my-app-jobs"
}

Key Takeaways

• SQS decouples slow or unreliable work from your HTTP response cycle
• The Express route is the producer (pushes messages); the worker is the consumer (processes them)
• Use long polling (WaitTimeSeconds: 20) in the consumer to reduce empty API calls and cost
• Only delete a message after successful processing — failed messages are retried automatically
• Set up a Dead Letter Queue to catch persistently failing jobs
• Return 202 Accepted when work has been queued but not yet completed