Data Delivery Semantics for Message Queues: At-Most-Once, At-Least-Once, and Exactly-Once Explained

Data delivery semantics for message queues define the guarantees for message transfer between producers and consumers, with three primary levels: at-most-once (may lose messages), at-least-once (may duplicate but never lose), and exactly-once (processes each message precisely one time).

In distributed systems, choosing the correct delivery guarantee is critical for balancing reliability against performance. The liquidslr/system-design-notes repository provides a comprehensive breakdown of these patterns in 19. Distributed Message Queue/README.md, detailing how modern message brokers like Apache Kafka and RabbitMQ implement these guarantees. Understanding these semantics ensures your application handles message loss, duplication, and ordering correctly based on business requirements.

At-Most-Once Delivery

At-most-once guarantees that a message is delivered no more than once; however, it may be lost entirely. This semantic prioritizes low latency and throughput over durability.

The producer sends messages asynchronously without waiting for broker acknowledgment. In the reference implementation described in 19. Distributed Message Queue/README.md, the producer sets acks=0 and disables retries, firing messages without confirmation. Concurrently, the consumer commits its offset immediately after fetching the message, before processing the payload. If the consumer crashes after committing but before processing, that message is never processed again.

This approach suits use cases where occasional data loss is acceptable, such as high-frequency telemetry or non-critical UI events where missing a single event does not impact business logic.

At-Least-Once Delivery

At-least-once ensures that every message is delivered one or more times with no loss, though duplicates may occur. This is the most common default for production message queues.

According to the source documentation, the producer implements this by attaching acknowledgment levels (ack=1 or ack=all) and retrying until the broker confirms receipt. The consumer processes the payload first, then commits the offset only after successful processing. If the consumer crashes after processing but before committing, the message will be redelivered upon restart, potentially creating duplicates.

Applications using this semantic must implement idempotent processing or deduplication logic to handle duplicate messages gracefully. This pattern fits general-purpose scenarios like order processing, log aggregation, and event sourcing where data loss is unacceptable but duplicates can be managed.

Exactly-Once Delivery

Exactly-once guarantees that each message is processed precisely one time, eliminating both loss and duplication. This requires the most complex coordination between producers, brokers, and consumers.

As detailed in 19. Distributed Message Queue/README.md, this semantic combines at-least-once mechanics with transactional writes and coordinated commits. The system tracks in-sync replicas (ISR) and utilizes a two-phase commit protocol to ensure atomicity. Kafka implements this through the transactional API, where producers use a transactional.id and consumers read with isolation.level set to read_committed.

The trade-off is considerable complexity and overhead, including extra network round-trips and state tracking. This semantic is reserved for critical domains such as financial transactions, billing systems, and inventory updates where duplicates or losses are strictly prohibited.

Implementation Patterns and Code Examples

The following examples demonstrate how to configure each semantic using Kafka client libraries. The patterns apply conceptually to other message queue systems like RabbitMQ or Pulsar.

Configuring At-Most-Once (Fire-and-Forget)

Set acks=0 on the producer to disable broker acknowledgments, and commit offsets immediately in the consumer:

# Producer: no ack, async fire-and-forget

producer = KafkaProducer(
    bootstrap_servers='broker:9092',
    acks=0,                     # No acknowledgment from broker

    retries=0,                  # No retries

)

producer.send('events', b'payload')
producer.flush()   # optional, ensures network buffer is drained

// Consumer: commit offset immediately after fetch
msg, _ := consumer.ReadMessage(ctx)
process(msg.Value)                # optional, may be a no-op

consumer.CommitOffsets()          # commit before processing → at-most-once

Configuring At-Least-Once (Default Safe Mode)

Configure the producer to wait for all in-sync replicas and retry on failure. The consumer processes the message before committing:

producer = KafkaProducer(
    bootstrap_servers='broker:9092',
    acks='all',                  # Wait for all ISR replicas

    retries=5,                   # Retry on transient errors

    retry_backoff_ms=100,
)

producer.send('orders', serialize(order))
producer.flush()

// Consumer: process first, then commit
msg, _ := consumer.ReadMessage(ctx)
if err := handleOrder(msg.Value); err != nil {
    // do not commit, message will be redelivered
    continue
}
consumer.CommitMessage(msg)    // commit only after successful handling

Configuring Exactly-Once (Transactional API)

Enable transactions on the producer and set the consumer isolation level to read_committed:

Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "broker:9092");
props.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "tx-producer-1");
KafkaProducer<String, String> producer = new KafkaProducer<>(props);
producer.initTransactions();

producer.beginTransaction();
producer.send(new ProducerRecord<>("payments", key, value));
producer.commitTransaction();   // atomic commit across producer & broker

// Consumer with read-process-write transaction
props.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
    for (ConsumerRecord<String, String> record : records) {
        // process record
    }
    consumer.commitSync();   // commits offsets only for committed messages
}

Choosing the Right Delivery Semantic

Select your delivery guarantee based on data criticality and tolerance for duplication:

• At-most-once – Choose for high-throughput, non-critical data like analytics events or UI interactions where latency matters more than completeness.

• At-least-once – Use for most general-purpose applications where durability is required but duplicates can be handled through idempotent operations or deduplication layers.

• Exactly-once – Implement only for financial transactions, inventory management, or compliance scenarios where the cost of duplicates outweighs the performance overhead.

Summary

• At-most-once provides the fastest throughput but risks message loss by avoiding retries and committing offsets immediately.
• At-least-once prevents data loss through producer retries and post-processing offset commits, requiring consumer-side idempotency to handle duplicates.
• Exactly-once guarantees single processing through transactional coordination and two-phase commits, adding significant latency and complexity.
• The 19. Distributed Message Queue/README.md file in liquidslr/system-design-notes details these patterns with specific reference to Kafka's ISR and transactional mechanisms.
• Most production systems use at-least-once semantics with idempotent consumers, reserving exactly-once for critical financial or inventory pipelines.

Frequently Asked Questions

What is the default delivery semantic for most message queues?

Most open-source message queues like Apache Kafka and RabbitMQ default to at-least-once delivery when configured with acknowledgments enabled. As implemented in the liquidslr/system-design-notes examples, this requires the producer to wait for broker confirmation (acks=1 or acks=all) and the consumer to manage offset commits carefully to prevent data loss while accepting potential duplicates.

How does exactly-once delivery impact performance?

Exactly-once delivery introduces considerable overhead due to additional network round-trips for two-phase commits and state tracking across distributed components. According to the source analysis, transactional producers require coordination with the transaction coordinator, and consumers must filter uncommitted messages using read_committed isolation levels, which increases latency and reduces throughput compared to at-least-once configurations.

Can I combine different delivery semantics in the same system?

Yes, different topics or queues within the same cluster can use different semantics based on data criticality. For example, you might configure at-most-once for high-volume telemetry topics while using exactly-once for payment processing topics. The broker configuration and client application code determine the semantic per producer-consumer pair, as detailed in 19. Distributed Message Queue/README.md.

What is the difference between at-least-once and exactly-once?

At-least-once guarantees no message loss but allows duplicates if the consumer crashes after processing but before committing the offset. Exactly-once extends this guarantee by ensuring that even if retries occur, the message is processed exactly one time through transactional coordination and idempotent semantics enforced at the broker level. The key distinction is that exactly-once requires atomic commits across both the message broker and any external state stores involved in the transaction.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}