The Transactional Outbox Pattern: How We Stop Webhooks from Disappearing

The transactional outbox pattern is a distributed systems pattern that makes event publishing more reliable without relying on distributed transactions. It solves the dual-write problem by saving both the business data and the event in the same atomic database transaction. A separate relay process then reads the stored event and publishes it to a message broker, with retries if delivery fails.

I have seen this failure happen in the simplest possible way. The database commit succeeds, but the service crashes before it sends the HTTP request. Our system knows that the customer has paid. The customer’s backend does not.

That gap is why the backend teams behind OTTclouds and MonetKit use the transactional outbox pattern to build more reliable event and webhook pipelines for payment, subscription, and monetization systems.

This article explains where webhooks get lost, how the Outbox Pattern protects events, and how it supports eventual consistency across microservices.

What Is the Transactional Outbox Pattern?

The transactional outbox pattern is a distributed systems pattern that stores a business change and the event produced by that change in the same local database transaction.

Instead of sending a webhook or publishing a message directly inside the request, the service first records the event in an Outbox table. A background process then reads that event and delivers it to a message broker, another service, or an external webhook endpoint.

The pattern is closely connected to three concepts:

• Dual-write problem: A service must update its database and write to another system, but the two operations cannot share one reliable transaction.
• At-least-once delivery: An event is delivered one or more times rather than guaranteed to arrive exactly once.
• Idempotent consumer: A consumer can safely receive the same event more than once without repeating the business effect.

When the transaction succeeds, both the business data and the Outbox event are stored. When it fails, both changes are rolled back.

The delivery happens later, from a durable record that can be retried.

That is the core idea:

Do not send the event first. Store the fact that the event must be sent.

Why Can a Webhook Disappear?

A webhook can “disappear” when the database update succeeds, but the service crashes before the HTTP request is sent. The internal system may already mark a subscription as active, while the customer’s backend never receives the corresponding event. This happens because a database transaction cannot also protect an external network call performed outside that transaction.

At first, webhooks look straightforward.

An event occurs. The application builds a payload and sends an HTTP request to the customer’s endpoint. The endpoint receives the request and continues the corresponding business process.

For many applications, this approach may appear good enough.

In a billing or subscription system, however, a webhook is rarely just a helpful notification. It may be the only way the customer’s backend learns that:

• A user completed a payment.
• A subscription was renewed.
• An invoice was paid.
• Access to content should be unlocked.
• A subscription expired or was canceled.

The problem begins when one request must perform two separate operations:

1. Update the database.
2. Send a webhook or publish an event.

Suppose the database commit succeeds, but the service crashes just before sending the webhook.

Your system now shows the subscription as active, but the customer’s backend never receives the update.

The webhook has disappeared.

Reversing the order does not solve the problem. If the service sends the webhook first and the database transaction later fails, the customer receives an event for a change that never actually happened.

This is the dual-write problem.

The application is writing to two separate systems, but it wants both operations to behave as though they belong to one atomic transaction.

Without an Outbox, there is no safe ordering:

• Commit the database first, and the event may be lost.
• Publish the event first, and the event may describe data that was never committed.

Changing the order does not remove the risk. It only changes which failure you are willing to accept.

The Transactional Outbox Pattern in Microservices

In a microservices architecture, a large system is divided into smaller services. Each service owns a specific business capability and usually manages its own database.

This separation allows teams to develop, deploy, and scale services independently. It also makes consistency more difficult.

Consider a subscription platform with three services:

• Billing Service records the payment.
• Subscription Service updates the subscription state.
• Entitlement Service grants access to the content.

When a payment succeeds, the Billing Service must update its own database and publish an event so the other services can continue the workflow.

The database transaction inside the Billing Service only protects data in that service’s database. It cannot also guarantee that a message has been written to Kafka, RabbitMQ, or another message broker.

If the database commit succeeds but the service crashes before publishing the event, the Billing Service records the payment while the Entitlement Service never unlocks access.

Publish the event first, and the reverse problem appears. Downstream services may process the event even though the original payment transaction later rolls back.

This is the consistency problem at the center of many microservices systems.

Each service can keep its own data internally consistent, but there is no single transaction spanning:

• The producer’s database.
• The message broker.
• The databases of downstream services.
• External webhook endpoints.

Distributed systems therefore often rely on eventual consistency.

The services may not become consistent at the same moment. However, the architecture must make sure they can eventually reach the correct state.

The transactional outbox pattern handles the first and most important step: it ensures the producer does not lose the event that other services depend on.

Transaction:
  updateSubscription()
  insertOutboxEvent()

If the transaction commits, both the subscription change and the event are stored. If it rolls back, neither exists.

A background Outbox Processor later publishes the event and retries when delivery fails.

The pattern does not force every service to participate in one large transaction. It protects the event at its source, then relies on message delivery, retries, and idempotent consumers to bring the rest of the system into alignment.

The Trap of Calling an External API Inside the Request

A database transaction only protects changes inside the database.

The moment the application performs an external side effect, such as publishing a message, calling another service, sending an email, or posting a webhook, it has crossed the transaction boundary.

Consider this simplified payment handler:

async handleInvoicePaid(event):
  transaction:
    writeBillingTransaction()
    updateSubscription()
  # Database commit completes here

  publish(invoicePaidEvent)  # Outside the transaction

The handler now has two states that should remain aligned:

• The payment has been recorded.
• The payment event has been published.

Only the first state is protected by the database transaction.

If the Billing Service stops between those two steps, the system becomes inconsistent. Billing knows that the payment succeeded, but Entitlement has not unlocked access, and Notification has not informed the user.

The direct API call creates a fragile gap between the local business transaction and the external side effect.

That gap may only last for a few milliseconds.

In production, a few milliseconds is enough.

Why Not Use a Distributed Transaction?

When teams first encounter the dual-write problem, distributed transactions often sound like the obvious solution.

The best-known approach is Two-Phase Commit, or 2PC.

A coordinator manages the transaction in two stages.

During the preparation phase, the coordinator asks every participant whether it is ready to commit. Only after all participants agree does the coordinator start the commit phase. If one participant cannot continue, the entire transaction is rolled back.

In theory, this provides exactly what we want: everything succeeds together, or everything fails together.

In real microservices systems, 2PC is often difficult to implement and operate.

Not every database, broker, service, or external dependency supports the same distributed transaction protocol. A customer’s webhook endpoint certainly cannot accept a request that means, “prepare this operation, but do not process it until I send a separate commit command.”

Even where 2PC is technically possible, it introduces high costs:

• A coordinator must be deployed and maintained.
• Locks may remain open for longer.
• A slow participant can block the others.
• Availability and scalability may suffer.
• New and difficult failure modes appear.
• Services become more tightly coupled.

Microservices are designed to give services a degree of operational independence. A distributed transaction pulls them back into a tightly coordinated commit process.

The transactional outbox pattern takes another route.

Instead of creating one transaction across several systems, it moves everything that must be atomic into a single resource: the producer service’s database.

The service stores its business change and Outbox event in one local transaction. Publishing happens afterward through a retryable background process.

This gives the system:

• Local atomicity inside each service.
• Eventual consistency between services.

Outbox Is Not a Pattern Like Singleton

When developers hear the word “pattern,” they may think of Singleton, Factory, or Observer.

Transactional Outbox belongs to a different category.

Patterns such as Singleton and Factory help organize objects and classes inside an application process. Transactional Outbox is a distributed systems pattern. It manages a side effect that crosses boundaries between databases, services, message brokers, HTTP endpoints, and external systems.

A simple way to separate them is:

Singleton organizes objects in code. Outbox organizes side effects in a distributed system.

The database transaction of one microservice cannot directly protect a network call or another service’s database. The Outbox becomes a durable boundary between an internal business change and the event that must leave the service.

It also rarely works alone.

A production Outbox implementation usually needs several supporting patterns and mechanisms.

Idempotent Consumer

Transactional Outbox normally provides at-least-once delivery. The same event may therefore be delivered more than once.

An idempotent consumer stores or recognizes the event IDs it has already processed. When the same event arrives again, it returns a successful result without applying the business operation twice.

Retry with Backoff

A broker, downstream service, or webhook endpoint may be temporarily unavailable.

Retry with Backoff increases the delay between attempts. This prevents the system from repeatedly flooding a failing dependency.

Dead Letter Queue

Some events continue to fail after multiple retries.

A Dead Letter Queue stores these events so engineers can inspect, repair, or replay them without blocking the rest of the pipeline.

Saga Pattern

Outbox ensures that the initiating event is not lost. It does not coordinate every subsequent step in a long-running business workflow.

For processes such as payment, inventory reservation, order creation, and fulfillment, a Saga can coordinate the later steps and trigger compensating actions when necessary.

A reliable event-driven system will often combine Outbox with idempotency, retry, dead-letter handling, and sometimes Saga orchestration.

Read more: Idempotency Keys in Payment Systems: Why a Request Should Never Be Processed Twice

How Does the Transactional Outbox Pattern Work?

The transactional outbox pattern writes the business data and the event that must be delivered into the same database transaction.

If the transaction succeeds, both the business change and the Outbox record are committed. If it fails, both are rolled back.

A background process then reads unprocessed events from the Outbox table, publishes them to a message broker or webhook pipeline, and marks them as completed.

If the network or a downstream service is unavailable, the event remains in the database and can be retried.

The mental model is simple:

Instead of publishing an event immediately, record that the event must be published in the same transaction that created it.

For example, the Billing Service may process a successful payment like this:

transaction:
  writeBillingTransaction()
  updateSubscription()
  insertOutboxEvent(invoicePaidEvent)

# Either all three records are committed,
# or none of them are.

After a successful commit, the system has two important pieces of data:

• The new business state.
• A durable event describing that state change.

The Outbox Processor later publishes the event. Services such as Entitlement, Notification, and Analytics can subscribe and update their own data.

Those services may not process the event at the same time. But because the event is durable and retryable, the system has a path toward eventual consistency.

Designing the Outbox Table

Each event-producing service can maintain an Outbox table inside the database it already owns.

A basic schema may look like this:

CREATE TABLE outbox_events (
  id              TEXT PRIMARY KEY,
  aggregate_type  TEXT NOT NULL,
  aggregate_id    TEXT NOT NULL,
  event_type      TEXT NOT NULL,
  payload         JSONB NOT NULL,
  status          TEXT NOT NULL,
  attempts        INT NOT NULL DEFAULT 0,
  next_attempt_at TIMESTAMPTZ,
  created_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
  published_at    TIMESTAMPTZ
);

The exact names are less important than the responsibilities of the stored data.

Use a Stable Event ID

Every event should have a unique, stable ID.

Consumers can use this ID to determine whether an event has already been processed. If the relay publishes the event again, the consumer can detect the duplicate and avoid repeating the business action.

A stable ID also improves traceability. Engineers can follow one event from the Billing Service through the broker, into the Entitlement Service, and eventually to the customer’s webhook endpoint.

Store Enough Payload to Replay the Event

The Outbox should contain enough information to publish the event again without rebuilding its entire payload from several business tables.

This becomes useful when engineers need to:

• Retry an event after an outage.
• Debug communication between services.
• Backfill events for a new consumer.
• Inspect the payload created at a particular time.
• Replay events after fixing a pipeline issue.

The payload does not always need to contain the entire entity. It should contain enough context for consumers to understand what happened and what data they need.

Track Publishing and Retry State

The Outbox Processor needs to distinguish between events that are pending, published, retrying, or permanently failed.

Fields such as status, attempts, next_attempt_at, and published_at make that lifecycle visible.

Without clear state, engineers cannot easily tell where an event is stuck, how many attempts have been made, or whether the event should move to a dead-letter queue.

How Does the Outbox Processor Publish Events?

Once the business transaction commits, event delivery becomes a separate process.

A background worker, often called an Outbox Processor, Message Relay, or simply relay, performs several tasks:

1. Find unpublished events.
2. Claim a batch so multiple workers do not process the same records simultaneously.
3. Publish each event to a message broker or webhook pipeline.
4. Mark successful events as published.
5. Schedule retries for failed events.

A simplified relay may look like this:

loop:
  events = selectUnpublishedOutboxEvents(limit)

  for event in events:
    result = publish(event)

    if result is successful:
      markAsPublished(event)
    else:
      scheduleRetry(event)

  sleep(interval)

If the broker is down, the event remains in the database. The relay can try again on the next cycle.

This is the key strength of the pattern.

It replaces one unsafe sequence:

Update database → Send network request

with two controlled steps.

The first is protected by a local transaction:

Update business data + Store Outbox event

The second operates from durable data:

Read Outbox event → Publish with retry

In a microservices system, the event may be published to Kafka, RabbitMQ, Amazon SNS/SQS, or another broker.

For MonetKit, the event may also move through a fan-out pipeline and eventually become a webhook delivered to the customer’s backend.

The transport can change. The principle does not:

Persist the event safely before attempting to send it.

Why Is Transactional Outbox a Good Fit for Microservices?

Transactional Outbox works well when one service changes its state and other services need to react.

It does not create a system-wide transaction. Instead, it gives each producer a safe local transaction and a reliable event that downstream services can process over time.

Local Atomicity Without Distributed Transactions

Business data and the Outbox event are stored in the same database transaction.

This means both exist, or neither exists. The service cannot commit the business change while forgetting to record the corresponding event.

The scope of this guarantee is important.

Transactional Outbox only provides atomicity between the business data and the Outbox event inside the producer’s database. It does not make several service databases commit together.

Consistency between services still follows an eventual consistency model.

This is a deliberate trade-off. Instead of making every service wait inside a distributed transaction, each service manages its own local state and communicates through events.

Events Survive Service and Network Failures

Networks fail. Brokers become unavailable. Services restart during deployments or crash during execution.

These failures do not erase an event that has already been written to the Outbox.

When the dependency becomes available again, the relay continues publishing the remaining events.

The system no longer depends on one HTTP request succeeding at the right moment. Event delivery becomes a durable, observable job that can be retried.

Producers and Consumers Become Less Temporally Coupled

If Billing calls Entitlement directly, the two services depend on each other being available at the same time.

When Entitlement becomes slow or unavailable, the payment request may also slow down or fail.

With an Outbox and message broker, Billing only needs to finish its own local transaction. Entitlement can process the event later when it is ready.

This reduces temporal coupling. The producer and consumer do not have to be online simultaneously.

The System Gains an Audit Trail and Replay Path

If Outbox records are retained or archived correctly, they can become an audit trail of the events produced by the service.

During an incident, engineers can investigate questions such as:

• Was the event created?
• When was it created?
• What payload did it contain?
• How many delivery attempts were made?
• Was it accepted by the message broker?
• Which endpoint received it?
• Why did the latest attempt fail?

When an event must be replayed or backfilled, the source data already exists. The team does not have to reconstruct the history from application logs.

How MonetKit Uses the Transactional Outbox Pattern

MonetKit is a monetization platform that receives events from Stripe, the Apple App Store, and Google Play. It normalizes those events, updates billing state, and sends webhooks to customer systems.

This is a natural use case for the transactional outbox pattern.

A single provider event may lead to several changes:

• Record a billing transaction.
• Update a subscription.
• Grant or revoke an entitlement.
• Update an invoice.
• Publish internal events.
• Send a webhook to the customer.

These actions may be handled by different services. MonetKit therefore needs to ensure that the initiating event is not lost after the billing state has changed.

In billing infrastructure, a webhook is not an optional notification.

For many customers, it is how their backend learns which user should receive access, which subscription has expired, or which invoice has been paid.

Losing one webhook can cause MonetKit and the customer’s system to drift apart.

In billing, that drift may result in lost revenue, incorrect access, and bugs that are extremely difficult to trace.

Stage 1: Receive and Process the Provider Event

An event arrives from Stripe, Apple, or Google.

The handler performs the required business operations, which may include:

• Verify the provider event.
• Record the billing transaction.
• Update the subscription.
• Grant or revoke internal entitlements.
• Create a canonical event in the Outbox.

The canonical event is written in the same transaction as the billing data.

If the transaction commits, MonetKit has both the updated billing state and the event that must be delivered. If it fails, both changes are rolled back.

The billing state and the intent to publish remain atomic.

Stage 2: Read Events from the Outbox

A background process scans the Outbox and claims unpublished events.

As volume grows, multiple relay workers can run in parallel. The workers must avoid claiming the same event at the same time.

A common approach is to use database row-level locks. Each worker receives a separate batch without requiring an additional distributed lock.

The relay then publishes the event into a broker or the next delivery stage.

The overall system still uses at-least-once delivery. An event may be published more than once, so downstream consumers and webhook receivers must remain idempotent.

Stage 3: Fan Out and Deliver

After the event is published, it may be distributed to several consumers.

For example, a subscription.renewed event may be used by:

• Entitlement Service to extend access.
• Notification Service to send a confirmation.
• Analytics Service to update reporting.
• Webhook Delivery Service to notify the customer’s backend.

For webhooks, the fan-out stage determines which configured endpoints should receive the event. A customer may have different endpoints for different products or environments.

The delivery stage may then:

• Create and sign the payload.
• Send the HTTP POST request.
• Record the response and status code.
• Retry with backoff.
• Apply timeouts and circuit breakers.
• Move failed deliveries to a dead-letter queue.

All this delivery complexity sits behind the durable Outbox boundary.

Once the event has been recorded in the business transaction, a network failure, broker outage, or service restart cannot silently erase it.

That is the reason the pattern exists.

Trade-Offs of the Transactional Outbox Pattern

Outbox makes event delivery more reliable, but it is not free. A production implementation introduces operational and architectural costs that must be planned for.

Polling Adds Latency

A common implementation uses a relay that polls the Outbox table every few seconds.

This creates a delay between the database commit and the moment the event is published or the webhook is delivered.

For asynchronous webhook flows, this delay is usually acceptable.

If the system requires millisecond-level delivery, polling may not be the right implementation. The team may also need to reconsider whether a webhook is the correct communication mechanism for that requirement.

One optimization is to wake the relay immediately after the transaction commits. Polling can remain as a safety net in case the wake-up signal is lost.

Another option is Change Data Capture, which reads changes from the database transaction log and can reduce delivery latency.

At-Least-Once Is Not Exactly-Once

Transactional Outbox does not guarantee that every event is delivered exactly once.

Consider this sequence:

1. The relay publishes an event successfully.
2. The broker or webhook endpoint processes it.
3. The relay crashes before marking the Outbox record as published.
4. The relay restarts and sees the record as incomplete.
5. The event is published again.

This is not necessarily an implementation bug. It is a natural consequence of at-least-once delivery.

Rather than chasing exactly-once delivery across the entire distributed system, teams usually target exactly-once effects.

An event may arrive more than once, but the business result is applied only once.

This requires a stable event ID and an idempotent consumer.

For example, the Entitlement Service should not extend the subscription twice when it receives the same subscription.renewed event again.

Outbox Does Not Solve Every Consistency Problem

Outbox ensures that the producer does not lose the initiating event.

It does not guarantee that every consumer processes the event successfully or immediately.

A consumer may still encounter invalid data, internal failures, or unavailable dependencies. The system therefore needs retries, dead-letter handling, monitoring, and sometimes compensation logic.

For workflows spanning several business steps, Outbox is often paired with a Saga.

Outbox protects the event that starts the workflow. Saga manages what happens when later steps succeed, fail, or need to be reversed.

The Outbox Table Keeps Growing

Every event creates a new row.

Without a retention strategy, the table grows indefinitely. Queries for unpublished events may become slower, while database storage, indexes, and maintenance costs increase.

The system needs a data lifecycle policy, which may include:

• Delete old published events after a retention period.
• Archive events to another storage system.
• Partition the table by day or month.
• Add indexes designed for replay queries.
• Separate short-term operational data from long-term audit data.

The correct retention period depends on the product’s replay, audit, and compliance requirements.

The System Gains Another Operational Component

Outbox requires at least one background processor.

That component must be:

• Deployed.
• Monitored.
• Scaled.
• Alerted.
• Checked for backlogs.
• Integrated with retry handling.
• Connected to dead-letter workflows.

If the relay becomes stuck, events are not lost, but they are delayed. The services may remain inconsistent for longer than expected.

Useful production metrics include:

• Number of pending events.
• Age of the oldest pending event.
• Time from database commit to publish.
• Publish success rate.
• Average number of retries.
• Number of events in dead-letter storage.
• Rate of Outbox event creation and processing.

Outbox is valuable when an event must not disappear.

When a side effect is purely best-effort, such as a noncritical analytics log, the added infrastructure may cost more than the event is worth.

Conclusion

The Transactional Outbox Pattern does not make networks reliable. It does not turn an entire microservices architecture into one transaction, and it does not guarantee that an event is delivered only once.

What it provides is a durable boundary between a local business transaction and asynchronous communication.

Instead of updating the database and hoping the event is published successfully, the service stores the business data and Outbox event in one transaction. A background relay then keeps trying to publish the event until it succeeds or enters a controlled failure workflow.

The service no longer has to choose between losing an event and publishing an event for data that was never committed.

In a microservices architecture, Outbox preserves atomicity inside the producer service and creates the foundation for eventual consistency across the system.

For billing, payment, subscription, and monetization infrastructure, this is more than a cleaner way to organize code.

It is how event delivery continues safely when services crash, networks fail, and external systems become unavailable.

Frequently Asked Questions About the Transactional Outbox Pattern