Backpressure in Event-Driven Systems

Backpressure in Event-Driven Systems#

Backpressure is the ability of a consumer to signal that it cannot keep up with the producer. Without backpressure, queues grow unbounded, latency spikes, and services crash under load.

Why Backpressure Matters#

Event-driven systems are inherently asynchronous. Producers can outpace consumers for many reasons:

Sudden traffic spikes.
Slow downstream dependencies.
Hot partitions in a stream.

Without backpressure, the system fails in unpredictable ways, often through memory exhaustion or timeouts.

Backpressure Strategies#

1. Bounded Queues#

Use bounded queues to impose explicit limits. When the queue is full, either drop messages, apply load shedding, or slow the producer.

2. Pull-Based Consumption#

Kafka and other log-based systems provide backpressure naturally via consumer offsets. Consumers only pull when ready.

3. Reactive Streams#

Reactive Streams provide a standardized request(n) mechanism that lets consumers control the flow.

4. Adaptive Concurrency#

Use feedback loops to shrink concurrency when error rate or latency increases.

Spring Boot + Reactor Example#

@Component
public class OrderEventProcessor {
    public Flux<OrderEvent> process(Flux<OrderEvent> events) {
        return events
                .onBackpressureBuffer(10_000)
                .limitRate(500)
                .flatMap(this::handleEvent, 64)
                .timeout(Duration.ofSeconds(2));
    }

    private Mono<OrderEvent> handleEvent(OrderEvent event) {
        return Mono.just(event);
    }
}

Kafka-Specific Backpressure#

Kafka consumers can throttle by:

Reducing max.poll.records.
Increasing max.poll.interval.ms to allow longer processing.
Pausing partitions dynamically when backlog builds.

Observability Signals#

Track these metrics to detect backpressure issues:

Consumer lag per partition.
Queue depth and drain rate.
JVM heap pressure and GC pauses.

Summary#

Backpressure is essential for stability in event-driven systems. Combining bounded buffers, pull-based protocols, and reactive flow control prevents overload and protects downstream services.