Message Queues: Delivery Semantics, Idempotency, and the Outbox Pattern

Problem Context

Direct synchronous calls between services are the simplest thing that can possibly work — until one downstream service slows down and the failure cascades upstream. A message queue (or stream / event bus) breaks that coupling: the producer hands off a message and moves on; the consumer processes when it's ready, at the rate it can sustain. The system absorbs traffic spikes, survives downstream outages, and gives you replayability for free.

But messaging introduces its own failure modes — duplicate delivery, out-of-order messages, poison messages, lost messages, and the existential question of whether you actually want exactly-once (you almost never do; you want idempotent at-least-once). This guide is the messaging mental model in 2026 — when to queue vs stream, the delivery guarantees you actually get, and the patterns that keep the consumer code sane.

Concept Explanation

Three shapes of asynchronous messaging:

• Queue (work distribution) — one message, one consumer. Multiple competing consumers share the load. Azure Service Bus queues, RabbitMQ.
• Stream (event log) — append-only ordered log; many consumers, each tracks its own offset; messages persist for a retention window. Event Hubs, Kafka, Kinesis.
• Pub/sub topic — one message, many subscribers (each with its own queue). Service Bus topics, Event Grid.

flowchart LR
    P["Producer"] --> Q["Service Bus Queue<br/>(work distribution)"]
    Q --> C1["Consumer 1"]
    Q --> C2["Consumer 2"]
    Q -.->|max-deliveries exceeded| DLQ["Dead-letter Queue"]

    P2["Producer"] --> T["Topic"]
    T --> S1["Sub: billing"]
    T --> S2["Sub: analytics"]
    T --> S3["Sub: notifications"]

    P3["Producer"] --> EH["Event Hubs<br/>(append-only stream)"]
    EH --> RD1["Reader: realtime ETL<br/>offset 12345"]
    EH --> RD2["Reader: ML pipeline<br/>offset 8000 (replay)"]

    style DLQ fill:#dc2626,color:#fff,stroke:#b91c1c
    style EH fill:#0078D4,color:#fff,stroke:#005a9e

Implementation

Step 1: Pick the right primitive

Service Bus QUEUE / TOPIC
  Use when: enterprise messaging, ordered per-session FIFO, transactions,
            DLQ, scheduled delivery, dedup window
  Throughput: tens of thousands msg/s (Premium)
  Retention: TTL per message, default 14 days

Event Hubs (or Kafka)
  Use when: telemetry, clickstreams, ETL, replay, fan-out to many readers
  Throughput: millions of events/s
  Retention: hours to days; consumers track offsets

Event Grid
  Use when: lightweight pub/sub for state-change events (push, not pull)
  Throughput: high; handler invocation per event
  Retention: 24h delivery retry, then DLQ

Storage Queue
  Use when: simple, cheap, < 80 GB total, polling-based, no DLQ
  Throughput: 2k msg/s/queue; not for new high-scale apps

Step 2: Producer — send with idempotency keys

var sender = client.CreateSender("orders");

var msg = new ServiceBusMessage(JsonSerializer.SerializeToUtf8Bytes(order))
{
    ContentType = "application/json",
    MessageId   = order.Id.ToString(),    // ← dedup window uses this
    SessionId   = order.TenantId.ToString(),  // ← optional: FIFO per tenant
    ApplicationProperties =
    {
        ["correlation-id"]     = correlationId,
        ["schema-version"]     = "v2",
        ["idempotency-key"]    = order.IdempotencyKey
    }
};
await sender.SendMessageAsync(msg);

Step 3: Consumer — peek-lock + explicit Complete + Defer

var processor = client.CreateProcessor("orders", new ServiceBusProcessorOptions
{
    AutoCompleteMessages = false,            // we control completion
    MaxConcurrentCalls   = 16,
    PrefetchCount        = 32,
    ReceiveMode          = ServiceBusReceiveMode.PeekLock
});

processor.ProcessMessageAsync += async args =>
{
    var msgId = args.Message.MessageId;
    if (await store.AlreadyProcessedAsync(msgId))      // idempotency check
    { await args.CompleteMessageAsync(args.Message); return; }

    try
    {
        await ProcessAsync(args.Message);
        await store.MarkProcessedAsync(msgId);
        await args.CompleteMessageAsync(args.Message);
    }
    catch (PoisonMessageException ex)
    {
        await args.DeadLetterMessageAsync(args.Message, "Poison", ex.Message);
    }
    catch (TransientException)
    {
        await args.AbandonMessageAsync(args.Message); // delivery count++
    }
};

processor.ProcessErrorAsync += err => { log.LogError(err.Exception, "ProcessError"); return Task.CompletedTask; };
await processor.StartProcessingAsync();

Step 4: Idempotency — the cure for at-least-once

// "Exactly-once" doesn't really exist across networks. Aim for at-least-once
// delivery + idempotent handlers. Options:

// (a) Idempotency table — atomic insert of message_id; conflict = already processed
INSERT INTO processed_messages (message_id) VALUES (@id);  -- PK conflict = skip

// (b) Natural idempotency — the operation is intrinsically safe to retry
UPDATE accounts SET balance = 100 WHERE id = @id;          -- SET = idempotent
                                                            -- (vs += which isn't)

// (c) Outbox pattern — publish message in same DB tx as the state change
//     A relay job moves rows from "outbox" table to the broker once,
//     even if the tx commits and the publish fails

Step 5: Ordering — sessions for per-key FIFO

// Service Bus: only sessions guarantee FIFO; a single session is processed by
// one consumer at a time. Throughput per session = single consumer.
var sessionProcessor = client.CreateSessionProcessor("orders", new ServiceBusSessionProcessorOptions
{
    MaxConcurrentSessions  = 32,
    AutoCompleteMessages   = false
});
// Set SessionId on producer to "tenant-123" or "order-id-42" — all messages
// with that SessionId go through one consumer in send order.

Step 6: Dead-letter handling — replay, don't reprocess blindly

// Periodically drain the DLQ, fix the root cause, then resubmit
var dlqReceiver = client.CreateReceiver("orders",
    new ServiceBusReceiverOptions { SubQueue = SubQueue.DeadLetter });

await foreach (var dead in dlqReceiver.ReceiveMessagesAsync(50))
{
    log.LogWarning("DLQ {Id}: reason={R}", dead.MessageId,
        dead.DeadLetterReason);
    if (await CanReplayAsync(dead))
        await sender.SendMessageAsync(new ServiceBusMessage(dead.Body));
    await dlqReceiver.CompleteMessageAsync(dead);
}

Pitfalls

1. Believing in exactly-once. Network partitions and crashes mean producers may publish twice and consumers may process twice. Design for at-least-once + idempotency. Service Bus dedup window helps for short windows; idempotency tables are the long-term fix.

2. AutoComplete + thrown exception = lost message. If the broker is configured to auto-complete on handler return, an exception still completes the message and you lose it. Always disable auto-complete and call CompleteMessageAsyncexplicitly on success.

3. Poison messages stuck in a loop. A bad message is delivered, fails, retried, fails... up toMaxDeliveryCount, then dead-lettered. Without a DLQ-monitoring pipeline you don't know it's happening. Alert on DLQ depth.

4. Treating ordering as default. Out of the box, Service Bus and Event Hubs both fan out to multiple consumers and order is best-effort. If you need ordering, use sessions (Service Bus) or partition by key (Event Hubs). Both reduce throughput.

5. Long handlers + lock expiry. Default lock is 30s. A slow handler crosses 30s, the lock expires, the message is redelivered, you process it twice. Either renew the lock (RenewMessageLockAsync) or chunk the work.

6. Mixing transactional state with publish without an outbox. Save to DB, then publish. The publish fails. State diverges. The transactional outbox pattern (publish via a row in the same DB tx) is the clean fix.

Practical Takeaways

• Queue for work distribution, stream for telemetry/replay, topic for fan-out, Event Grid for lightweight state-change events.
• Always at-least-once + idempotent handlers. "Exactly-once" is marketing.
• Peek-lock + explicit Complete. Auto-complete loses messages on exceptions.
• Use the outbox pattern when you publish as part of a DB transaction.
• Use sessions (or partition keys) only when you actually need ordering — they cap throughput.
• Monitor DLQ depth and lock-renewal events. They're your early warning.
• For replayable analytics workloads, prefer Event Hubs/Kafka — consumer-managed offsets are the killer feature.