Idempotent Consumers Design

Idempotent consumers design for safe retries

Retries are inevitable in distributed systems, so your consumers must tolerate duplicates. Idempotency is the simplest way to do that because it converts retries into no-ops instead of data corruption.

Prerequisites

• .NET 8 SDK
• PostgreSQL or SQL Server
• A message broker such as Kafka or RabbitMQ

Choose a deduplication strategy

1. Unique constraint on a natural business key.
2. Processed message table keyed by message ID.
3. Upsert operations that converge to the same state.

Example: processed message table

CREATE TABLE processed_messages (
    message_id UUID PRIMARY KEY,
    processed_at TIMESTAMPTZ NOT NULL
);

public async Task HandleAsync(OrderPaidMessage message, CancellationToken token)
{
    var inserted = await _db.Database.ExecuteSqlInterpolatedAsync($@"
        INSERT INTO processed_messages (message_id, processed_at)
        VALUES ({message.MessageId}, NOW())
        ON CONFLICT DO NOTHING",
        token);

    if (inserted == 0)
    {
        return;
    }

    await _billingService.ApplyPaymentAsync(message.OrderId, message.Amount, token);
}

Protect side effects

• Use transactional outboxes for downstream events.
• Wrap external calls in idempotency keys.
• Store the last processed sequence per aggregate when ordering matters.

Operational checks

• Monitor duplicate rates to detect upstream retries.
• Alert on dead-letter counts after idempotency failures.
• Keep message IDs stable across retries.

Things to remember

• Idempotency is a contract, not an optimization.
• Deduplication tables must be indexed and cleaned with retention policies.
• Handle external side effects with idempotency keys.