Writing

Exactly-Once Processing in Kafka: What It Actually Means

May 15, 2025 ·6 min
kafkadistributed-systemsengineering

“Exactly-once” is the phrase that gets a distributed systems engineer’s eyebrow up, because in the general case it is provably impossible. You cannot guarantee that a message crosses an unreliable network exactly one time. What Kafka actually gives you is narrower, more precise, and genuinely useful: exactly-once processing semantics within the boundary of the Kafka system. Understanding where that boundary sits is the whole job.

The three delivery guarantees

Every messaging system offers one of three contracts:

  • At-most-once — fire and forget. Fast, but messages can be lost on failure.
  • At-least-once — retry until acknowledged. Nothing is lost, but a retry after a dropped ack means duplicates.
  • Exactly-once — the holy grail: every message takes effect once, no loss, no duplication.

Most systems quietly run at-least-once and push deduplication onto the consumer. Kafka’s contribution is making exactly-once achievable without that manual bookkeeping — as long as your side effects live inside Kafka.

The idempotent producer

The first building block is the idempotent producer. Each producer is assigned a Producer ID (PID), and every message in a partition carries a monotonic sequence number. The broker remembers the last sequence number it accepted per (PID, partition). If a network hiccup causes the producer to retry a send, the broker sees a sequence number it has already committed and silently drops the duplicate.

This eliminates duplicates from producer retries — the single most common source of double-writes — with effectively zero application code. Turn it on with enable.idempotence=true and you’ve removed an entire class of bug.

The transactional API

Idempotence handles one producer to one partition. Real pipelines do consume → process → produce: read from an input topic, transform, write to an output topic, and commit the input offset. For that to be atomic, Kafka exposes a transactional API:

producer.initTransactions();
producer.beginTransaction();
// produce to output topic(s)
producer.sendOffsetsToTransaction(offsets, consumerGroupId);
producer.commitTransaction();   // or abortTransaction() on failure

The crucial move is sendOffsetsToTransaction. The consumer’s offset commit becomes part of the same transaction as the output writes. Either both happen or neither does. Downstream consumers set isolation.level=read_committed so they never see records from aborted or in-flight transactions. The result: an input message is processed and its effect produced exactly once, atomically.

Where the boundary breaks

The honesty in “exactly-once” is in its scope. The guarantee holds for Kafka-to-Kafka flows. The moment your processing reaches outside Kafka — charging a card, writing to a SQL row, calling a third-party API — you are back to at-least-once unless that external system is itself idempotent.

How this shaped the billing pipeline

On the event-driven billing and reconciliation system I work on, the contract is exactly this: Kafka guarantees no duplicate or dropped events inside the pipeline, and every external write is made idempotent by key — a deterministic event ID that the database uses as a unique constraint. Kafka handles the in-system half; idempotency keys handle the boundary. That division of labour is what “exactly-once” means in practice: not magic, but a precise contract you design the rest of the system to honour.