Designing Event-Driven Systems Correctly

Designing event-driven systems correctly for reliability

Event-driven architecture simplifies scaling and integration, but it is easy to make it brittle with the wrong coupling patterns. This guide focuses on designing for long-term evolvability, correctness, and operational confidence.

Prerequisites

• .NET 8 SDK
• Broker such as Kafka or RabbitMQ
• Relational database for the outbox pattern

Core design principles

• Explicit contracts: events are versioned contracts, not private DTOs.
• Asynchronous boundaries: events represent facts, not synchronous requests.
• Idempotency everywhere: consumers must treat duplicates as normal.

Use the outbox pattern for atomicity

Publish events from the same transaction that mutates state. This avoids the dual-write problem.

public sealed class OrderCreatedOutboxMessage
{
    public Guid Id { get; init; }
    public string Type { get; init; } = "order.created.v1";
    public string Payload { get; init; } = string.Empty;
    public DateTimeOffset OccurredAt { get; init; }
}

public async Task CreateOrderAsync(Order order)
{
    await using var tx = await _db.Database.BeginTransactionAsync();

    _db.Orders.Add(order);
    _db.OutboxMessages.Add(new OrderCreatedOutboxMessage
    {
        Id = Guid.NewGuid(),
        Payload = JsonSerializer.Serialize(order),
        OccurredAt = DateTimeOffset.UtcNow
    });

    await _db.SaveChangesAsync();
    await tx.CommitAsync();
}

Dispatch outbox messages reliably

Use a background worker to read and publish messages. Mark records as sent only when the broker confirms delivery.

protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
    while (!stoppingToken.IsCancellationRequested)
    {
        var batch = await _db.OutboxMessages
            .Where(x => x.SentAt == null)
            .OrderBy(x => x.OccurredAt)
            .Take(100)
            .ToListAsync(stoppingToken);

        foreach (var message in batch)
        {
            await _publisher.PublishAsync(message.Type, message.Payload, stoppingToken);
            message.SentAt = DateTimeOffset.UtcNow;
        }

        await _db.SaveChangesAsync(stoppingToken);
        await Task.Delay(TimeSpan.FromSeconds(1), stoppingToken);
    }
}

Design consumers for idempotency

Store a deduplication key in your business table or maintain a processed-events table with a unique index. This allows you to reprocess safely after failures or retries.

Event choreography vs orchestration

• Choreography works when each service owns its own state and reacts independently.
• Orchestration is better for long-running workflows where you need a single place to enforce consistency.

Observability requirements

• Correlate events with trace IDs and partition keys.
• Emit metrics for processing lag and error rate.
• Capture schema versions in logs for post-incident analysis.

Things to remember

• Use the outbox pattern to guarantee atomic state changes and event publication.
• Design for retries and duplicates from the start.
• Keep event payloads stable and versioned.