At-least-once Delivery
Also known as: at-least-once, at-least-once semantics.
A delivery guarantee under which every message or job arrives one or more times, never zero — so duplicates are possible and must be tolerated.
Synthetic dataExamples on this page use synthetic data created for illustration.
Definition
At-least-once delivery means a message or job is guaranteed to arrive but may arrive more than once, for example when an acknowledgement is lost and the sender tries again. Consumers must therefore tolerate duplicates, usually by making their work idempotent.
In plain terms
A sender cannot tell “the message was lost” from “the message arrived but the reply was lost”. To be safe it sends again. That choice trades the risk of losing a message for the risk of delivering it twice, and most queues and schedulers make this trade on purpose.
The rest of the system then has two honest options: accept that duplicates happen and make processing idempotent, or add bookkeeping that detects and discards repeats.
Example
A worker takes job J from a queue with a 10-minute lease (a time limit after which the job is offered again). The worker is slow; at minute 10 the lease expires and a second worker takes J. Both finish and write a result.
| Result table | Rows for job J |
|---|---|
| No unique key on the job id | 2 |
| Unique key on the job id (second write replaces the first) | 1 |
Three common guarantees, side by side:
| Guarantee | Can a message be lost? | Can it arrive twice? |
|---|---|---|
| At-most-once | Yes | No |
| At-least-once | No | Yes |
| Effectively-once (at-least-once plus idempotent processing) | No | Yes, but with no extra effect |
Common misunderstandings
- “Duplicates mean the queue is broken.” They are the designed cost of never losing a message.
- “Exactly-once delivery is just a setting.” Across a network it is achieved by deduplication or idempotent writes, not by the network itself.
- “Retries fix lost data, so nothing else is needed.” Without idempotent writes, the retries themselves create bad data.
References
- Martin Kleppmann. Designing Data-Intensive Applications . O'Reilly Media, 2017. ISBN 978-1449373320.
- Tyler Akidau, Slava Chernyak, Reuven Lax. Streaming Systems . O'Reilly Media, 2018. ISBN 978-1491983874.