Handling Backpressure in Streaming Systems

Handling backpressure in streaming systems

Backpressure is the mechanism that prevents fast producers from overwhelming slow consumers. Without it, your system either drops data or collapses under load. This post shows how to implement backpressure in Python streaming consumers.

Prerequisites

• Kafka cluster
• Python 3.11+
• aiokafka client

Understand the pressure points

• Network IO: fetches can return more data than you can process.
• CPU-bound handlers: serialization and enrichment can fall behind.
• Downstream systems: databases and APIs often throttle.

Use bounded queues and pause/resume

With aiokafka, you can pause partitions when a local queue is full.

import asyncio
from aiokafka import AIOKafkaConsumer

async def process_queue(queue):
    while True:
        message = await queue.get()
        await handle_event(message)
        queue.task_done()

async def consume():
    consumer = AIOKafkaConsumer(
        "metrics",
        bootstrap_servers="localhost:9092",
        enable_auto_commit=False,
        auto_offset_reset="earliest",
    )
    await consumer.start()

    queue = asyncio.Queue(maxsize=500)
    worker = asyncio.create_task(process_queue(queue))

    try:
        async for msg in consumer:
            if queue.full():
                consumer.pause(*consumer.assignment())
                await queue.put(msg)
                consumer.resume(*consumer.assignment())
            else:
                await queue.put(msg)

            await consumer.commit()
    finally:
        worker.cancel()
        await consumer.stop()

asyncio.run(consume())

Apply downstream rate limits

• Use connection pools with bounded size.
• Add circuit breakers around third-party calls.
• Batch updates to reduce per-event overhead.

Things to remember

• Backpressure protects the entire pipeline, not just consumers.
• Pausing partitions is more predictable than increasing thread counts.
• Always commit offsets after processing to avoid losing data.