Never send email from inside a web request

The claim

Any call to a third-party service inside a request handler makes that service a hard dependency of your application. Sending mail synchronously means your order confirmation page has the availability of your mail provider multiplied by the availability of your own stack, and it means a customer's order can be lost because an SMTP connection timed out after the payment succeeded.

What actually goes wrong

Consider the usual controller: charge the card, write the order, send the confirmation, render the page. If the mail call hangs for 30 seconds:

• The customer stares at a spinner and refreshes, sometimes producing a second charge.
• One of your worker processes is occupied doing nothing for 30 seconds. At 8 workers, 20 simultaneous checkouts exhaust the pool and the whole site stops responding.
• If the request times out after the charge and before the commit, you have taken money for an order that does not exist.

The last one is not hypothetical. It is the single most common serious bug we find in small commerce applications, and it is caused entirely by ordering.

The correct ordering

Commit the state change first, enqueue the side effect second, return immediately.

BEGIN;
  INSERT INTO orders (...) VALUES (...) RETURNING id;
  INSERT INTO outbox (topic, payload, run_after)
    VALUES ('order.confirmation', '{"order_id": 4821}', now());
COMMIT;

Writing the job into the same database transaction as the order is the point. If the transaction rolls back, the email is not queued. If it commits, the email is guaranteed to be queued. Enqueuing to an external broker before committing gives you the opposite: emails about orders that were never created.

A worker polls the table:

SELECT * FROM outbox
WHERE processed_at IS NULL AND run_after <= now()
ORDER BY id
FOR UPDATE SKIP LOCKED
LIMIT 20;

FOR UPDATE SKIP LOCKED is what makes this safe with multiple workers — each worker claims rows nobody else holds, with no coordination and no separate queue server. For anything under a few hundred jobs per second, Postgres is a perfectly good queue and one fewer thing to operate.

Retries need a schedule and a ceiling

Mail providers return transient failures routinely. Retry with exponential backoff and stop eventually:

attempt 1: immediate
attempt 2: +1 min
attempt 3: +5 min
attempt 4: +25 min
attempt 5: +2 h
then: dead letter, alert a human

The dead-letter step is not optional. A job that retries forever is how you discover, three weeks later, that one malformed address has been generating 40,000 failed attempts per day.

At-least-once means duplicates

Any queue that survives a crash will occasionally deliver a job twice — the worker sent the mail and died before marking the row processed. Design for it rather than trying to prevent it. Pass an idempotency key to the provider:

X-Entity-Ref-ID: order-4821-confirmation

Most transactional mail APIs deduplicate on this header within a time window. For jobs that hit your own systems, make the handler idempotent: check whether the state you are about to set is already set, and return successfully if it is.

The objection about immediate delivery

Someone will point out that customers expect the confirmation to arrive right away, and that a queue introduces delay. In practice the delay is the polling interval, which you control: a worker checking every two seconds delivers mail faster than most mail providers accept and relay it. Nobody has ever complained that a receipt took three seconds instead of one. What people do complain about is a checkout page that hangs, which is the thing you are removing.

What else belongs in the queue

The same reasoning applies to every slow or unreliable operation triggered by a user action: PDF and invoice generation, image processing, webhook delivery to a customer's endpoint, syncing to a CRM, pushing an order to a shipping provider. If it calls something you do not operate, or takes more than about 200 ms, it belongs behind the queue.

The one thing to monitor

Queue depth alone is not enough — a depth of 400 is fine if it is draining. Alert on the age of the oldest unprocessed job:

SELECT extract(epoch from now() - min(run_after))
FROM outbox WHERE processed_at IS NULL;

Over ten minutes means your workers have stopped, and the failure is otherwise entirely silent: the site is up, orders are being taken, and no confirmation has gone out since lunch.