Exactly-Once vs At-Least-Once Delivery

Exactly-once vs at-least-once delivery in practice

Delivery semantics are not marketing terms. They are contracts between your producer, broker, and consumer that define which failures you tolerate and which duplicates you must handle. This post compares the two semantics and shows how to implement them using Python and Kafka.

Prerequisites

• Kafka cluster with transactions enabled
• Python 3.11+
• confluent-kafka client

Semantics in one sentence

• At-least-once: every record is delivered one or more times.
• Exactly-once: every record is delivered once and only once in the presence of retries and restarts.

Failure modes and consequences

Failure	At-least-once	Exactly-once
Producer retry	Duplicate records	Deduplicated via producer id and sequence
Consumer crash before commit	Records replayed	Records replayed but processed once if transactional
Broker leader failover	Potential duplicates	Duplicates prevented with idempotent producer

Exactly-once requires coordination across producer and consumer, typically with transactions and a consumer that reads and writes within the same transaction boundary.

At-least-once consumer example

This pattern commits offsets after processing, so failures can replay data.

from confluent_kafka import Consumer

config = {
    "bootstrap.servers": "localhost:9092",
    "group.id": "billing-service",
    "auto.offset.reset": "earliest",
    "enable.auto.commit": False,
}

consumer = Consumer(config)
consumer.subscribe(["billing-events"])

try:
    while True:
        msg = consumer.poll(1.0)
        if msg is None:
            continue
        if msg.error():
            raise RuntimeError(msg.error())

        process_event(msg.value())
        consumer.commit(message=msg)
finally:
    consumer.close()

Exactly-once with transactions

To achieve exactly-once processing, the producer must be idempotent and transactional, and the consumer must send offsets as part of the same transaction.

from confluent_kafka import Consumer, Producer, TopicPartition

producer = Producer({
    "bootstrap.servers": "localhost:9092",
    "enable.idempotence": True,
    "transactional.id": "billing-transformer-1",
})
producer.init_transactions()

consumer = Consumer({
    "bootstrap.servers": "localhost:9092",
    "group.id": "billing-transformer",
    "enable.auto.commit": False,
    "isolation.level": "read_committed",
})
consumer.subscribe(["billing-events"])

while True:
    msg = consumer.poll(1.0)
    if msg is None:
        continue
    if msg.error():
        raise RuntimeError(msg.error())

    producer.begin_transaction()
    transform_and_publish(msg.value(), producer)
    offsets = [TopicPartition(msg.topic(), msg.partition(), msg.offset() + 1)]
    producer.send_offsets_to_transaction(offsets, consumer.consumer_group_metadata())
    producer.commit_transaction()

Tradeoffs to understand

• Transactions reduce throughput because they add coordination and fencing checks.
• Exactly-once guarantees only apply within a single Kafka cluster unless you add external transactional storage.
• At-least-once is still appropriate when downstream processing is idempotent or the cost of duplicates is small.

Things to remember

• Exactly-once requires idempotent producers, transactions, and read-committed consumers.
• At-least-once needs idempotent business logic to be safe.
• Tune transaction.timeout.ms and monitor aborted transactions to avoid silent data loss.