At-Least-Once Delivery and Exactly-Once Effects: Understanding the Difference

At-least-once delivery and exactly-once effects describe two different reliability guarantees. A message may arrive more than once, but one logical event should produce only one business outcome. Delivery guarantees, exactly-once effects, and idempotency are central to this distinction.

Consider a billing webhook that times out. The provider retries the event, and the same subscription upgrade reaches your backend twice. If the handler cannot recognize that it has already processed the event, it may grant the same entitlement twice.

This is a common problem in payment, subscription, webhook, and event-driven systems. This OTTclouds article explains why retries create duplicates, how idempotency protects business state, and where exactly-once guarantees stop.

This article continues our discussion of the Transactional Outbox Pattern. The Outbox helps reduce the risk of losing an event after a database update. This article focuses on the other side of the problem: how to accept duplicate deliveries without creating duplicate business effects. Together, these mechanisms form the foundation of a more reliable webhook pipeline.

What Are At-Least-Once Delivery and Exactly-Once Effects?

At-least-once delivery means that a system may deliver the same message multiple times to reduce the risk of losing it. Exactly-once effects mean that one logical operation produces only one business outcome within a defined scope.

These guarantees answer two different questions:

• Delivery guarantee: How many times can the receiver see the message?
• Effect guarantee: How many times does the charge, entitlement update, or ledger entry actually happen?

Duplicate delivery does not have to produce a duplicate effect. The same event may reach a handler three times, while only the first valid attempt changes business state.

The key is to make the handler idempotent. Once the event has been processed, later deliveries of the same logical event should not create any additional changes.

How Is Delivery Different from Effect?

Suppose a customer upgrades a subscription. The billing provider sends a subscription.upgraded event to your backend.

The backend updates the customer’s entitlement successfully. However, the response is lost before it reaches the provider. The provider cannot tell whether the request succeeded, so it sends the same event again.

A simple handler might perform the business action every time it receives the event:

receive event
grant premium entitlement
return success

In this design, the same upgrade may be applied more than once.

A safer handler identifies the logical event before creating the effect:

receive event
identify stable event_id
atomically record event_id and update entitlement
return success

The event may still be delivered several times. The entitlement transition, however, is committed only once.

At-Most-Once vs At-Least-Once vs Exactly-Once

At-most-once prioritizes avoiding duplicate processing, at-least-once prioritizes avoiding message loss, and exactly-once aims to apply each logical operation once within a controlled boundary.

The main difference is how each model balances message loss, duplicate delivery, and coordination complexity.

In practice, many billing and webhook systems use at-least-once delivery with idempotency to achieve exactly-once effects at the application layer.

Whenever a system claims to provide “exactly once,” engineers should ask three questions:

• Exactly once for which effect?
• Across which systems?
• Inside which transaction boundary?

The Problem Begins When an Acknowledgement Is Lost

A sender normally transmits a message and waits for an acknowledgement, often called an ACK, from the receiver.

The difficult failure case looks like this:

1. The sender transmits the message.
2. The receiver accepts and processes it.
3. The receiver sends an acknowledgement.
4. The acknowledgement is lost, or the connection closes.
5. The sender cannot tell whether the receiver completed the operation.

The sender now has two imperfect choices:

• Retry the message: This may create a duplicate if the receiver already processed the first attempt.
• Do not retry: This may lose the operation if the receiver never received it.

This is not a defect specific to webhook providers. It is a normal uncertainty when two systems communicate over a network.

For example, a provider may resend a webhook when it does not receive a successful response. The receiving integration is expected to check whether the event has already been processed before running the business logic again.

The practical design goal is therefore not to eliminate every duplicate message in transit. It is to make duplicate delivery safe.

Why Exactly-Once Delivery Has a Limited Scope

Exactly-once delivery is only meaningful within a boundary where the platform controls message delivery, state storage, processing, and acknowledgement.

Once an event crosses a network boundary into an external system, uncertainty returns. The external system may have completed the operation, while its acknowledgement was lost on the way back.

The sender cannot know whether the previous call succeeded. Retrying may create a duplicate. Stopping may leave the operation incomplete.

Some platforms provide exactly-once semantics within their own controlled environment. However, that guarantee always has a scope.

Kafka Streams, for example, can coordinate consumer offsets, state stores, and producer transactions so that events are processed once within a Kafka workflow. Apache Flink can restore operator state from checkpoints, allowing processing to resume in a state that is close to a failure-free execution.

These guarantees do not automatically extend to an external HTTP endpoint, email service, or database that does not participate in the same transaction protocol.

Consider the following flow:

Kafka transaction commits
        ↓
Consumer calls an external HTTP API
        ↓
The external API succeeds
        ↓
Consumer crashes before saving local completion state

When the consumer restarts, it may call the external API again because it has no persisted evidence that the first call completed.

Kafka cannot roll back a side effect that has already happened in the external system.

For this reason, exactly-once delivery should not be presented as an end-to-end guarantee once the workflow leaves the transaction boundary controlled by the platform. The receiving system still needs idempotency and deduplication.

How Does Idempotency Create Exactly-Once Effects?

Idempotency allows the same logical operation to be submitted or processed multiple times without producing additional business effects after the first successful execution.

The system assigns each event or request a stable identifier called an idempotency key. Whenever the message arrives, the service uses this key to determine whether it represents a new operation or a retry of an operation that has already been processed.

A safe idempotent flow usually includes the following steps:

1. Receive the event and extract its idempotency key.
2. Store the key in durable storage.
3. Use a unique constraint so that only one worker can claim the key.
4. Update business state in the same database transaction.
5. If the key already exists, skip the business logic or return the previously stored result.

The most important requirement is atomicity. Recording the idempotency key and changing business state must succeed or fail together.

If these steps happen separately, two workers may both see that the event does not exist and then create the same effect.

An Idempotency Key Alone Is Not Enough

The following code appears reasonable, but it contains a race condition:

if event_id not in processed_events:
    grant_entitlement()
    insert processed_events(event_id)

Two workers may perform the check before either one inserts the event:

Worker A: event does not exist
Worker B: event does not exist
Worker A: grants entitlement
Worker B: grants entitlement

Both workers believe they are processing a new event. The result is two entitlement changes from one logical operation.

To prevent this, the system must coordinate event ownership and the business update through an atomic operation.

When the deduplication record and business state are stored in the same database, they can be handled in one transaction:

BEGIN;

INSERT INTO processed_events (
    event_id,
    event_type,
    payload_hash,
    status
)
VALUES (
    :event_id,
    :event_type,
    :payload_hash,
    'processing'
)
ON CONFLICT (event_id) DO NOTHING;

-- If no row was inserted, the event already exists.
-- Validate the payload if required, then stop processing.

UPDATE subscriptions
SET plan = :new_plan,
    updated_at = NOW()
WHERE subscription_id = :subscription_id;

UPDATE processed_events
SET status = 'completed',
    completed_at = NOW()
WHERE event_id = :event_id;

COMMIT;

The implementation must stop before the business update when the insert does not claim a new event.

The unique constraint on event_id ensures that only one worker can create the processing record. The transaction then keeps the deduplication record and subscription update aligned. Either both changes are committed, or neither one is.

The system should also validate the request content. If the same idempotency key arrives with a different payload, it may not be a valid retry. It may indicate that the key has been reused incorrectly.

A payload_hash or a set of immutable operation fields can help detect this conflict.

An idempotency key therefore does not create exactly-once effects by itself. The guarantee depends on durable storage, concurrency control, atomic business updates, and a clear policy for key reuse.

What Does the Transactional Outbox Pattern Solve?

The Transactional Outbox Pattern prevents a service from updating business data without also recording the event that must be published. It does not eliminate duplicate messages or guarantee exactly-once effects across downstream systems.

The pattern addresses the dual-write problem, where a service must perform two independent operations:

Update data in the local database
        ↓
Publish an event to the message broker

If these operations are not part of the same transaction, the service can fail between them.

For example:

1. The database update commits.
2. The service crashes before publishing the event.
3. Internal state has changed.
4. Downstream services never receive the update.

The Transactional Outbox Pattern changes the flow:

BEGIN

Update business state
Insert event into the outbox table

COMMIT

Both database changes happen inside one local transaction. They either succeed together or fail together.

After the transaction commits, a separate process called a relay reads pending events from the outbox table and publishes them to the message broker.

This reduces the gap between changing business state and recording the intention to publish an event.

However, the Outbox does not remove duplicate publication.

The following sequence can still occur:

1. The relay publishes the event successfully.
2. The message broker confirms that it received the event.
3. The relay crashes before updating the outbox row to sent.
4. The row remains pending.
5. After restarting, the relay publishes the event again.

The broker may therefore receive the same event more than once.

This is not necessarily a failure of the Outbox Pattern. It is a consequence of prioritizing event durability when the final publish state has not been saved.

Transactional Outbox commonly provides at-least-once publication. It solves the dual-write gap, but it does not ensure that a consumer processes the event once or that the entire downstream workflow creates only one business outcome.

Consumers still need stable event IDs, idempotency, and deduplication.

Deduplicate at Every Hop in the Event Pipeline

Deduplication must be applied at every meaningful hop because duplicates can appear at any system boundary, not only at the original webhook endpoint.

A provider may retry a webhook. An outbox relay may publish again after a crash. A broker may redeliver a message. A delivery worker may repeat a job whose completion was never confirmed.

Each hop owns a different effect. It may update business state, create a delivery record, or call an external service. Each component therefore needs a stable identifier and a deduplication rule for the work it controls.

Hop 1: Inbound Event

The provider event ID should be stored under a unique constraint.

Within one transaction, the service can:

1. Insert the provider event ID.
2. Update payment or subscription state.
3. Create a canonical outbox event.
4. Commit the transaction.

If the event ID already exists, the handler checks the stored processing state and returns success without repeating the business transition.

Hop 2: Outbox Relay

Multiple relay workers may process the outbox table in parallel.

In PostgreSQL, FOR UPDATE SKIP LOCKED allows a worker to skip rows currently locked by another worker:

SELECT id
FROM outbox_events
WHERE status = 'pending'
ORDER BY id
LIMIT 20
FOR UPDATE SKIP LOCKED;

This reduces the chance that two active workers claim the same row at the same time.

It does not eliminate duplicates caused by a crash after publishing.

Publisher confirms to allow the relay to know whether the broker has accepted responsibility for the message:

publish(event)

if broker did not confirm:
    keep event pending
    retry later

if broker confirmed:
    mark event sent

A failure boundary still exists between the broker confirmation and the mark sent update.

The relay may publish again after a crash. The canonical event ID must therefore remain unchanged across all retries.

Hop 3: Fan-Out

Suppose one canonical event must be delivered to five registered endpoints.

The fan-out service should create each logical delivery under a unique key:

(event_id, endpoint_id)

If the canonical event is replayed:

• Three completed deliveries should remain completed.
• Two pending deliveries may be queued again.
• The service should not create five new delivery records.

Idempotency here means more than avoiding an insert error. It means avoiding work that has already been completed.

Hop 4: HTTP Delivery and the Receiving Consumer

The delivery worker should check the saved delivery state before sending:

delivery = load(delivery_id)
if delivery.status is succeeded or dead_lettered:
    return
send_http_request(delivery)

This check is necessary, but it does not cover every failure case.

The following sequence can still happen:

HTTP endpoint processes the request successfully
        ↓
The response is lost or the worker crashes
        ↓
The local delivery remains pending
        ↓
The worker retries the request

The sender cannot safely conclude that the first request had no effect. The receiving system may have completed the operation even though its response never arrived.

Every retry should therefore use the same idempotency key or canonical event ID. This allows the receiving consumer to recognize the request as another attempt at the same operation.

Exactly-once effects at the final receiver still depend on how that receiver implements idempotency and atomic deduplication.

A Reference Billing and Subscription Flow for MonetKit

MonetKit supports subscription, product, and customer-state management across the App Store, Google Play, and Stripe.

A relevant architecture can normalize provider-specific events into a common subscription event format before distributing them to downstream services.

The following flow is an illustrative reference based on the MonetKit product use case. It is not a verified description of MonetKit’s current production architecture.

Stage 1: Receive and Process the Provider Event

When a webhook arrives from Stripe, the App Store, or Google Play, the system can:

1. Verify the provider signature.
2. Extract the stable provider event ID.
3. Validate the event type and payload.
4. Insert a deduplication record using a unique constraint.
5. Update transaction or subscription state.
6. Determine whether an entitlement must be granted, changed, or revoked.
7. Create a canonical event and write it to the outbox.
8. Commit all changes in one database transaction.

A canonical event may look like this:

{
  "event_id": "canonical-subscription-event-id",
  "event_type": "subscription.entitlement_changed",
  "provider": "stripe",
  "provider_event_id": "provider-event-id",
  "subscription_id": "subscription-id",
  "entitlement_state": "active",
  "occurred_at": "provider-event-time",
  "schema_version": 1
}

The actual fields must be verified against MonetKit’s official event contract.

Stage 2: Publish the Canonical Event

After the database transaction commits, the relay reads the canonical event from the outbox and publishes it to the message broker.

If the publish attempt times out or the relay does not receive a broker confirmation, the system should:

• Keep the record eligible for retry.
• Reuse the same event_id.
• Apply retry with backoff.
• Store the attempt count and latest error.
• Expose enough information for monitoring and incident handling.

The broker may have accepted the event even when the relay failed to save the sent state. After restarting, the relay may publish the event again.

A stable event_id allows downstream services to recognize it as a replay of the same logical event.

Stage 3: Fan-Out and Consumer Processing

The canonical event may be distributed to several downstream components:

• An entitlement service that grants or revokes access.
• An analytics pipeline that records usage and revenue data.
• A notification service that informs the customer.
• A customer webhook service that forwards the event.
• An audit or reconciliation process that checks system consistency.

Each consumer must define idempotency around the effect it owns.

For example:

• The entitlement service may use (event_id, entitlement_type) to avoid granting the same access twice.
• The customer webhook service may use (event_id, endpoint_id) to avoid creating duplicate deliveries to one endpoint.
• The analytics pipeline may retain duplicate raw data but deduplicate before aggregation when reporting requires accurate counts.

The exact implementation depends on each consumer’s data contract. The general rule remains the same: preserve a stable event identity across the pipeline and deduplicate at every stage that creates a real effect.

Benefits of At-Least-Once Delivery with Idempotency

At-least-once delivery with idempotency allows a system to retry uncertain operations without repeating their business effects. A message may arrive more than once, but each logical operation should produce one result within the defined boundary.

Safer Retries

When a request times out, the sender cannot tell whether the receiver failed to process it or completed it without returning a response.

Using the same idempotency key or event ID on every retry allows the receiver to recognize the operation and skip work that has already completed.

Protection for Critical Business State

Idempotency reduces the risk that webhook redelivery or broker replay creates multiple payment records, subscription updates, entitlements, refunds, or ledger entries.

This is especially important for operations that are expensive or difficult to reverse.

Better Recovery

Pending work can be stored durably and resumed after a service restart or a temporary dependency failure.

Recovery remains safe when processing state is persisted, and all retries use the same logical identifier.

Clearer Incident Investigation

Event IDs, processing states, attempt counts, recent errors, and timestamps provide a traceable history for each operation.

Operations teams can determine how far an event progressed, why it was retried, and whether the business effect completed.

Lower Coupling Between Services

The producer does not need every downstream service to finish within the original request.

Entitlement, analytics, notification, and customer webhook services can process and retry independently. A local failure does not have to block the entire workflow.

Trade-Offs and Limitations

This design improves reliability, but it adds database work, storage, latency, and operational complexity.

More Database Reads, Writes, and Storage

Each hop may require unique constraints, processed-event records, delivery rows, and retry states.

At high volumes, these tables require careful indexing, partitioning, retention, and cleanup.

Eventual Consistency

Outbox relays, queues, and retries mean that downstream state may update later than the source state.

The architecture prioritizes durability and recovery over immediate consistency.

Duplicate Messages Still Exist

The design does not remove duplicates from the network. It makes them identifiable and prevents them from creating additional effects when idempotency is implemented correctly.

Deduplication Records Need a Retention Policy

Deleting records too early may cause an old event to be treated as new. Keeping every record forever increases storage requirements.

Retention should account for provider retry windows, manual replay policies, broker retention, reconciliation, and audit requirements.

Idempotency Does Not Solve Event Ordering

An older subscription.updated event may arrive after subscription.canceled.

Both events may be unique, so deduplication alone cannot prevent the older event from overwriting newer state.

The system may also need:

• A provider sequence number.
• An aggregate version.
• Event timestamps.
• A source-of-truth lookup.
• Per-key partitioning.
• State-transition validation.

External Consumers Remain Outside Your Control

The sender can provide a stable idempotency key, but it cannot force the receiving system to use that key correctly.

Exactly-once effects at the final destination depend on how the consumer stores the key, handles concurrency, and applies business updates.

Some Side Effects Cannot Be Rolled Back

An email, SMS, push notification, or external API call may already have completed and cannot be undone through a local database rollback.

These cases may require an idempotent provider API, an operation ledger, reconciliation, or a compensation strategy.

At-least-once delivery with idempotency is therefore most valuable when losing an operation and repeating it are both costly. The trade-off is additional state, monitoring, retry logic, and deduplication at each service boundary.

When Should You Use At-Least-Once Delivery with Idempotency?

Use at-least-once delivery with idempotency when an operation must not be lost, but processing it more than once could also cause significant damage.

It is particularly useful for workflows involving money, access rights, balances, inventory, or audit-sensitive data.

Common use cases include:

• Payments and refunds: Prevent one event from recording or executing the same transaction more than once.
• Subscription activation, renewal, and cancellation: Avoid applying the same provider update repeatedly.
• Entitlement grants and revocations: Prevent duplicate access changes.
• Ledger and invoice processing: Avoid duplicate accounting entries or invoices.
• Inventory reservation: Prevent one order from reserving or deducting the same stock more than once.
• Order fulfilment: Avoid duplicate shipments, payment records, or fulfilment requests.
• Customer webhook delivery: Retry failed endpoints while preserving the event identity.
• Cross-service state synchronisation: Prevent the same event from changing downstream state repeatedly.
• Audit-sensitive operations: Maintain a clear history of processing attempts, status changes, and outcomes.

This approach is also appropriate whenever the system communicates with independent components over a network, including payment providers, message brokers, external APIs, and customer systems.

At these boundaries, a timeout or lost response makes it impossible for the sender to know with certainty whether the previous operation succeeded.

When Is the Full Pattern Unnecessary?

Not every event needs an outbox, relay, retry queue, deduplication table, and dead-letter process. A simpler design may be more appropriate when the cost of message loss or duplicate processing is low.

A full implementation may not be necessary when:

• The data is low-value telemetry.
• Duplicates can be removed during aggregation.
• The operation is naturally idempotent, such as using PUT to set an absolute value.
• At-most-once delivery is acceptable because duplicate notifications are more harmful than missing ones.
• Every update fits inside one local database transaction.
• The system can periodically rebuild or reconcile state from an authoritative source.
• Traffic is low and rare inconsistencies can be corrected manually.

The decision should consider three costs:

1. The cost of losing the event.
2. The cost of applying the effect more than once.
3. The cost of building and operating the reliability mechanism.

The strongest guarantee is not always the right guarantee. The right design provides a level of reliability that matches the value of the business effect and the consequences of losing or repeating it.

Conclusion

The main difference between at-least-once and exactly-once lies in what is guaranteed and where that guarantee ends. At-least-once delivery allows a message to be retried when the previous result is uncertain. Exactly-once effects aim to apply one logical business operation once within a controlled boundary.

The Transactional Outbox Pattern reduces the risk of losing an event between the database and the message broker, but it does not remove duplicates. Idempotency identifies retries and prevents them from creating additional business effects.

For payment and subscription systems such as MonetKit, the practical approach is to preserve a stable event identity, deduplicate at every meaningful hop, and define the scope of each guarantee clearly. The goal is not to make every message appear only once. It is to ensure that retries do not corrupt business state.

FAQs