Problem statement
Your company runs a distributed system that handles user data. When a user is created or updated, two things are supposed to happen:
- The system writes the changes to the user database (the source of truth).
- It then publishes an event to Kafka, notifying downstream services so they can update their read-optimized tables.
The system successfully writes to the database. Immediately after, before it can send the Kafka event, the service crashes due to a bug, an out-of-memory error, or a sudden shutdown. 😢😢
Your write database has the correct user information but the read-optimized table has no idea this change happened.
This failure results in inconsistent data. This, right here, is what we call the Dual Write Operations Issue.
There are several ways to solve this issue, one of them is using
Transactional outbox pattern
What is transactional outbox pattern
A design pattern used in distributed systems to reliably publish events/messages to other systems only if the corresponding database transaction succeeds.
In a normal database transaction, you can’t directly publish an event to Kafka because if the transaction fails, rollback is impossible. Database transactions can be undone if something goes wrong, but Kafka messages are fire-and-forget.
So Instead of sending the event directly to Kafka, you insert the event into an outbox table in the same database transaction , This ensures that either both the user data and the event is saved, or neither is (thanks to ACID transactions).
This event is later published to Kafka by a relay process. This ensures our read-optimized tables stay in sync with the main database.
A nodejs example that insert user in the user table and inserts an event in the outbox
try {
// Start transaction
await client.query('BEGIN');
// 1. Insert user
const userResult = await client.query(
'INSERT INTO users (name, email) VALUES ($1, $2) RETURNING id',
["test", "[email protected]"]
);
const userId = userResult.rows[0].id;
// 2. Insert into outbox (same transaction)
const eventPayload = {
userId,
email: "[email protected]",
timestamp: new Date().toISOString()
};
await client.query(
'INSERT INTO outbox (event_type, payload) VALUES ($1, $2)',
['UserCreated', eventPayload]
);
// 3. Commit if both succeed
await client.query('COMMIT');
console.log('User and event saved!');
return userId;
} catch (error) {
// Rollback on any error
await client.query('ROLLBACK');
console.error('Transaction failed:', error);
throw error;
}
The Relay Process
The message relay runs in the background, constantly checking the outbox table for unpublished events. When it finds this unpublished records , it sends them to Kafka and are marked as processed or deleted from the outbox.
Ways to implement the Relay Process
Polling Service is a background process that constantly checks the outbox table for new or unpublished events. While it gets the job done, it can add unnecessary load to your database, especially if you're scanning large parts of the table without proper indexing or batching, and it gets worse if it runs too frequently.
Change Data Capture (CDC) can be used in conjunction with the Transactional Outbox pattern to efficiently capture changes made to the outbox table and publish them to kafka.
Few Drawbacks
Kafka might deliver the same message multiple times. The receiving service needs to handle this gracefully by implementing idempotency, which is basically making sure that processing the same event twice doesn't break anything.
Event ordering is something you really need to watch out for too. If you're doing event sourcing, those events hitting Kafka better be in the exact same sequence as your database updates, otherwise you're going to reconstruct the wrong state.
When Should you use transactional outbox pattern
- You're building an event-driven application where a database update initiates an event notification.
- You want to ensure atomicity in operations that involve two services.
- Mostly valid for microservice architecture, for monolith it adds unnecessary complexity and performance overhead.
Additional articles
Top comments (0)