Backpressure in Event-Driven Systems

Backpressure in Event-Driven Systems

Backpressure is the ability of a consumer to signal that it cannot keep up with the producer. Without backpressure, queues grow unbounded, latency spikes, and services crash under load.

Why Backpressure Matters

Event-driven systems are inherently asynchronous. Producers can outpace consumers for many reasons:

• Sudden traffic spikes.
• Slow downstream dependencies.
• Hot partitions in a stream.

Without backpressure, the system fails in unpredictable ways, often through memory exhaustion or timeouts.

Backpressure Strategies

1. Bounded Queues

Use bounded queues to impose explicit limits. When the queue is full, either drop messages, apply load shedding, or slow the producer.

2. Pull-Based Consumption

Kafka and other log-based systems provide backpressure naturally via consumer offsets. Consumers only pull when ready.

3. Reactive Streams

Reactive Streams provide a standardized request(n) mechanism that lets consumers control the flow.

4. Adaptive Concurrency

Use feedback loops to shrink concurrency when error rate or latency increases.

Spring Boot + Reactor Example

@Component
public class OrderEventProcessor {
    public Flux<OrderEvent> process(Flux<OrderEvent> events) {
        return events
                .onBackpressureBuffer(10_000)
                .limitRate(500)
                .flatMap(this::handleEvent, 64)
                .timeout(Duration.ofSeconds(2));
    }

    private Mono<OrderEvent> handleEvent(OrderEvent event) {
        return Mono.just(event);
    }
}

Kafka-Specific Backpressure

Kafka consumers can throttle by:

• Reducing max.poll.records.
• Increasing max.poll.interval.ms to allow longer processing.
• Pausing partitions dynamically when backlog builds.

Observability Signals

Track these metrics to detect backpressure issues:

• Consumer lag per partition.
• Queue depth and drain rate.
• JVM heap pressure and GC pauses.

Summary

Backpressure is essential for stability in event-driven systems. Combining bounded buffers, pull-based protocols, and reactive flow control prevents overload and protects downstream services.