Start With a Direct Call

A direct call is the simplest way for one service to ask another for something, and while the work is quick and the other side is reliable, it's the right way. The trouble starts when the caller waits on work it doesn't need the answer to: a confirmation email, an invoice, a warehouse update. Now a slow email provider makes checkout slow, and a broken one makes checkout fail.

A queue cuts that link. The caller records what needs doing and moves on, and a consumer does the work at its own pace. That fixes the first problem and brings new ones, each with its own signal: a message that never arrives, one that arrives twice, one that can never succeed. The table above lists them in order. This article covers the kinds of messaging, then the answer to each situation and a way to check it.

If one slow service takes others down
  • Your "async" service is actually synchronous over HTTP and falls over together
  • You can't explain when to use Service Bus vs Event Hubs vs Event Grid
  • Your consumer crashes mid-handler and the message replays forever
  • You believed the "exactly-once" marketing line and don't handle duplicates

This is the message-queue playbook — queues vs streams, delivery semantics that actually exist, idempotency, and the failure modes that bite in production.

Queues, Streams and Delivery Guarantees

Three shapes of asynchronous messaging:

  • Queue (work distribution) — one message, one consumer. Multiple competing consumers share the load. Azure Service Bus queues, RabbitMQ.
  • Stream (event log) — append-only ordered log; many consumers, each tracks its own offset; messages persist for a retention window. Event Hubs, Kafka, Kinesis.
  • Pub/sub topic — one message, many subscribers (each with its own queue). Service Bus topics, Event Grid.

flowchart LR
    P["Producer"] --> Q["Service Bus Queue<br/>(work distribution)"]
    Q --> C1["Consumer 1"]
    Q --> C2["Consumer 2"]
    Q -.->|max-deliveries exceeded| DLQ["Dead-letter Queue"]

    P2["Producer"] --> T["Topic"]
    T --> S1["Sub: billing"]
    T --> S2["Sub: analytics"]
    T --> S3["Sub: notifications"]

    P3["Producer"] --> EH["Event Hubs<br/>(append-only stream)"]
    EH --> RD1["Reader: realtime ETL<br/>offset 12345"]
    EH --> RD2["Reader: ML pipeline<br/>offset 8000 (replay)"]

    style DLQ fill:#dc2626,color:#fff,stroke:#b91c1c
    style EH fill:#0078D4,color:#fff,stroke:#005a9e

Building a Reliable Pipeline

The examples use Azure Service Bus. The situations are the ones every broker puts you in.

Step 1: When you need to hand work off, pick the kind of messaging that fits

The names are confusingly close, and the choice decides what you can do later. Ask whether each message should be handled once by one consumer, or read by many consumers who each keep their own place.

Service Bus QUEUE / TOPIC
  Use when: enterprise messaging, ordered per-session FIFO, transactions,
            DLQ, scheduled delivery, dedup window
  Throughput: depends on tier and messaging units
  Retention: TTL per message or queue

Event Hubs (or Kafka)
  Use when: telemetry, clickstreams, ETL, replay, fan-out to many readers
  Throughput: millions of events/s
  Retention: up to 7 days on Standard, 90 on Premium and Dedicated; consumers track offsets

Event Grid
  Use when: lightweight pub/sub for state-change events (push, not pull)
  Throughput: high; handler invocation per event
  Retention: 24h delivery retry, then DLQ

Storage Queue
  Use when: simple, cheap, polling-based; no automatic dead-lettering
  Throughput: up to 2,000 msg/s per queue
✅

Check it: For each kind of message you plan to send, write down who consumes it and whether anyone will need to replay it later. One consumer and no replay points to a queue. Many readers, or replay, points to a stream.

Step 2: When the same message might be sent twice, give it an identity

A producer that times out can't tell whether its message arrived, so it sends again. A stable id on every message is what lets the broker, and later the consumer, recognise the repeat.

var sender = client.CreateSender("orders");

var msg = new ServiceBusMessage(JsonSerializer.SerializeToUtf8Bytes(order))
{
    ContentType = "application/json",
    MessageId   = order.Id.ToString(),    // ← dedup window uses this
    SessionId   = order.TenantId.ToString(),  // ← optional: FIFO per tenant
    ApplicationProperties =
    {
        ["correlation-id"]     = correlationId,
        ["schema-version"]     = "v2",
        ["idempotency-key"]    = order.IdempotencyKey
    }
};
await sender.SendMessageAsync(msg);
✅

Check it: Send the same order twice in quick succession. With duplicate detection turned on for the queue, only one copy should be delivered. Then send it again after the detection window has passed. It arrives, which is why the consumer needs its own check in Step 4.

Step 3: When a handler can fail halfway, complete the message yourself

The dangerous default is letting the client mark a message done as soon as your handler returns. Take the message under a lock, do the work, and only then say it's finished. A failure in between hands the message back.

var processor = client.CreateProcessor("orders", new ServiceBusProcessorOptions
{
    AutoCompleteMessages = false,            // we control completion
    MaxConcurrentCalls   = 16,
    PrefetchCount        = 32,
    ReceiveMode          = ServiceBusReceiveMode.PeekLock
});

processor.ProcessMessageAsync += async args =>
{
    var msgId = args.Message.MessageId;
    if (await store.AlreadyProcessedAsync(msgId))      // idempotency check
    { await args.CompleteMessageAsync(args.Message); return; }

    try
    {
        await ProcessAsync(args.Message);
        await store.MarkProcessedAsync(msgId);
        await args.CompleteMessageAsync(args.Message);
    }
    catch (PoisonMessageException ex)
    {
        await args.DeadLetterMessageAsync(args.Message, "Poison", ex.Message);
    }
    catch (TransientException)
    {
        await args.AbandonMessageAsync(args.Message); // delivery count++
    }
};

processor.ProcessErrorAsync += err => { log.LogError(err.Exception, "ProcessError"); return Task.CompletedTask; };
await processor.StartProcessingAsync();
✅

Check it: Throw a transient error in the handler. The message should come back with its delivery count one higher. Then catch an error and return normally without completing. The message should come back again when its lock runs out, not vanish.

Step 4: When a message arrives twice, make handling it twice harmless

At-least-once delivery means exactly that. After a retry, an expired lock or a restart, your handler will see a message it has already handled, so handling it again must change nothing.

// "Exactly-once" doesn't really exist across networks. Aim for at-least-once
// delivery + idempotent handlers. Options:

// (a) Idempotency table — atomic insert of message_id; conflict = already processed
INSERT INTO processed_messages (message_id) VALUES (@id);  -- PK conflict = skip

// (b) Natural idempotency — the operation is intrinsically safe to retry
UPDATE accounts SET balance = 100 WHERE id = @id;          -- SET = idempotent
                                                            -- (vs += which isn't)

// (c) Outbox pattern — publish message in same DB tx as the state change
//     A relay job moves rows from the "outbox" table to the broker and retries
//     until the publish succeeds, so delivery is at least once
✅

Check it: Deliver the same message to the handler twice in a test. The side effect should happen once: one email, one row, one charge. The second run should hit the primary-key conflict and stop there.

Step 5: When order matters for one customer, use sessions

Several consumers pulling from one queue finish messages in whatever order they happen to. When order matters, it almost always matters per customer or per order, not across everything, and that narrower promise is one you can afford.

// Service Bus: only sessions guarantee FIFO; a single session is processed by
// one consumer at a time. Throughput per session = single consumer.
var sessionProcessor = client.CreateSessionProcessor("orders", new ServiceBusSessionProcessorOptions
{
    MaxConcurrentSessions  = 32,
    AutoCompleteMessages   = false
});
// Set SessionId on producer to "tenant-123" or "order-id-42" — all messages
// with that SessionId go through one consumer in send order.
✅

Check it: Send created, updated and cancelled for one order in quick succession, with several consumers running. With the same session id they must be handled in that order. Remove the session id and run it again to see the difference.

Step 6: When a message can never succeed, set it aside and look at it

Some messages are simply wrong, and no number of retries will fix them. After a set number of deliveries the broker moves such a message to a dead-letter queue. That only helps if somebody reads it.

// Periodically drain the DLQ, fix the root cause, then resubmit
var dlqReceiver = client.CreateReceiver("orders",
    new ServiceBusReceiverOptions { SubQueue = SubQueue.DeadLetter });

var seen = new HashSet<long>();
while (true)
{
    var batch = await dlqReceiver.ReceiveMessagesAsync(maxMessages: 50, maxWaitTime: TimeSpan.FromSeconds(5));
    if (batch.Count == 0 || batch.Count(m => seen.Add(m.SequenceNumber)) == 0) break;

    foreach (var dead in batch)
    {
        log.LogWarning("DLQ {Id}: reason={R}", dead.MessageId, dead.DeadLetterReason);
        if (await CanReplayAsync(dead))
        {
            await sender.SendMessageAsync(new ServiceBusMessage(dead));
            await dlqReceiver.CompleteMessageAsync(dead);
        }
        // Anything else stays in the DLQ when its lock expires
    }
}
✅

Check it: Send a message your handler always rejects. It should reach the dead-letter queue once the delivery limit is hit, with a reason attached, and the messages behind it should keep flowing. Then check that an alert fires on dead-letter depth, because a queue nobody watches is a place where work disappears.

How Queues Lose or Duplicate Work

1. Believing in exactly-once. Network partitions and crashes mean producers may publish twice and consumers may process twice. Design for at-least-once + idempotency. Service Bus dedup window helps for short windows; idempotency tables are the long-term fix.

2. Auto-complete + swallowed exception = lost message. With auto-complete on, the processor completes a message when your handler returns, and abandons it if the handler throws. Catch an error and return normally, and the message is gone. Disable auto-complete and call CompleteMessageAsyncexplicitly on success.

3. Poison messages stuck in a loop. A bad message is delivered, fails, retried, fails... up toMaxDeliveryCount, then dead-lettered. Without a DLQ-monitoring pipeline you don't know it's happening. Alert on DLQ depth.

4. Treating ordering as default. Competing consumers on a Service Bus queue don't give you ordering, and Event Hubs keeps order only within a partition. If you need ordering, use sessions (Service Bus) or partition by key (Event Hubs). Both reduce throughput.

5. Long handlers + lock expiry. The default lock is 1 minute, and the processor renews it for up to 5 minutes. A handler that runs longer loses the lock, the message is redelivered, you process it twice. Either renew the lock (RenewMessageLockAsync) or chunk the work.

6. Mixing transactional state with publish without an outbox. Save to DB, then publish. The publish fails. State diverges. The transactional outbox pattern (publish via a row in the same DB tx) is the clean fix.

When I Would Use a Queue

  • Queue for work distribution, stream for telemetry/replay, topic for fan-out, Event Grid for lightweight state-change events.
  • Always at-least-once + idempotent handlers. "Exactly-once" claims only hold inside one system.
  • Peek-lock + explicit Complete, so a swallowed exception can't complete a message by accident.
  • Use the outbox pattern when you publish as part of a DB transaction.
  • Use sessions (or partition keys) only when you actually need ordering — they cap throughput.
  • Monitor DLQ depth and lock-renewal events. They're your early warning.
  • For replayable analytics workloads, prefer Event Hubs/Kafka — consumer-managed offsets are the killer feature.

Try It on Your Own System This Week

Three checks on a system that already uses messaging, about an hour in total:

  1. Look at the dead-letter queue. If it's empty, good. If you don't know where it is, or nothing alerts on it, that is your first job.
  2. Find the handler with the most valuable side effect. A charge, an email, a stock change. Ask what happens if its message is delivered twice.
  3. Find every place that saves to a database and then publishes. Ask what happens if the process stops between the two. If the answer is a missing message, you need the outbox described in Step 4.

Queues for AI Workloads

LLM work suits queues well. Calls are slow, expensive and rate limited, and users often don't need the answer right away. Accept the request, return a job ID, process it in the background, and notify on completion. This protects your web tier from long-held connections, and absorbs bursts that would otherwise hit provider limits.

A few things differ from a typical queue. Jobs vary enormously in length, so set the visibility timeout or lock duration for your slowest case, and extend it while working. Otherwise, a long job reappears in the queue, and a second worker starts it, and you pay twice. Control concurrency to stay within your provider's rate limits. If you get 429s, slow the consumers rather than retrying quickly. Put big payloads in object storage, and pass a reference. Most brokers have message size limits, and prompts with documents get large.

Idempotency, again

Most brokers deliver at least once, so any job can run twice. For a model call, that wastes money. For a side effect, such as sending an email, it's a bug. Use an idempotency key, and record completed work, so that a retry finds the result rather than redoing the job. Be careful with exactly-once claims across systems: what you can build is at-least-once delivery with idempotent processing, which gives you the same effect.

Poison messages and ordering

A message that always fails can retry forever and hold up whatever is behind it. Limit retries, then move it to a dead-letter queue with the error attached. Then alert on that queue, and have a tool to replay messages after a fix. Distinguish a temporary failure, such as a timeout, from a permanent one, such as a malformed input, and don't retry the second kind.

Kafka and similar logs preserve order only within a partition, so choose a key that groups what must stay in order, such as a customer ID. A traditional queue with competing consumers gives you no order at all. If order really matters, you'll also find that one slow message holds up everything behind it. Before you build around ordering, ask if a version number or timestamp on each event would be enough, because consumers that tolerate out-of-order messages are much easier to scale.

Monitor queue age as well as depth. Depth says how much is waiting, and age says how long users have been waiting, which is what matters.