Microservices

Chapter 06 of 27 · Part 4: Transactional messaging and domain events

Transactional messaging and idempotent consumers

How to update the database and publish a message as one step, and how consumers survive duplicates and reordering.

key idea / coordinatorservice / good pathin progress / watch outproblem / failureneutral

Sending a message with a database change

If you update the DB and then crash before sending, the system is inconsistent. The outbox pattern makes both writes one local transaction.

Transactional outboxThe service writes data and the message in one local transaction. A relay then publishes it.Order serviceDatabase transactionORDER tableINSERT / UPDATEOUTBOX tableINSERT messageMessage relayreads outboxBroker
One local ACID transaction. No distributed transaction needed.
Two ways to run the relayPolling publisherQuery OUTBOX on a timerSimple, fine at low scalePolling costs add upTransaction log tailingRead the DB commit logFast and scalableMore work, e.g. Debezium

Delivery guarantees and the outbox

A broker also balances load. If an instance dies, Kafka reassigns its partitions to the others. You need no external load balancer, and no rate limiter either: a burst becomes consumer lag, not rejected requests. That lag is the signal to scale out (see lesson 22 on scaling). From Practical Event-Driven Microservices Architecture, ch. 7.

Four places where delivery can failThe publish, its acknowledgement, the delivery and its acknowledgement can each fail.Order svcBrokerInventory svc1 publish2 ack (can be lost)3 deliver4 ack (can be lost)A lost ack makes the sender retry, so the same message can arrive twice.
SemanticsBehaviourFit
At-most-onceNo retry. Never duplicated, may be lost (for example on deploy).Telemetry where only fresh data matters.
At-least-onceRetry until acknowledged. May duplicate. Small performance cost.The practical default, with idempotent consumers.
Exactly-onceKafka uses sequence numbers and transactions across topics. Reported cost: about 3 percent against at-least-once.Only inside the broker flow. A database or external call still needs its own coordination.

Saving state and publishing an event are two separate operations. If you save the order and fail to publish, the stream no longer matches your database and every consumer inherits the problem. The outbox pattern fixes it.

Transactional outboxThe order and an outbox row are saved in one transaction; a relay publishes outbox rows to the broker.Order svchandles requestone DB transactionOrder tableOutbox tableOrderCreated rowRelaypoll or CDCBrokerPoll the outbox on startup and after failures, or use CDC such as Debezium. Delete rows afterpublishing.

Two alternatives. First, publish the event and build state by consuming it, so the stream is the only source of truth. This fits stateless processors, but a service that validates against its own state would read stale data. Second, without cross-table transactions, make writes idempotent upserts and acknowledge the command only after both writes succeed; a retry then fills the missing one. Wrapping both in a transaction or using compensating actions only narrows the gap.

Change data capture when you cannot edit the monolith

The book tells of an inventory migration where calling the new service from every flow looked perfect in test, then drifted in production. Stored procedures with hundreds of lines, data fixes by support teams and "mushroom applications" such as spreadsheets all changed the data. CDC captures changes at the database, so none of them can be missed. It also works for vendor monoliths. From Practical Event-Driven Microservices Architecture, ch. 2.

CDC with SQL Server and KafkaThe transaction log fills a CDC table; an ETL component polls it by log sequence number and publishes stock events.Stock tablewritesTx logevery changeCDC tableLSN, op codeETL pollerbatchesEventsin KafkaInventorynew serviceOperation codes: 1 delete, 2 insert, 3 row before update, 4 row after update.Poll after the last LSN, cap the batch (for example 1000 rows), map 2 to StockCreated and 4 toStockChanged.
ApproachGoodWatch out
Query with a watermark (timestamp, id or version)Works on any database. Flexible filters and bulk load of existing data.Heavy queries load the DB. Several updates collapse into one. Deletes need soft delete.
Read the database logEvery change including deletes, with before and after. Lighter on the DB.Depends on the vendor. Exposes the table model, so adapt it to domain events. High volume needs retention rules.
Framework (Kafka Connect, NiFi)Configuration instead of code.Hard to debug locally, limited customization, leaks the DB model.
Custom ETL componentEvents with domain meaning (a stock update may mean a reservation).More code to write and run. One per bounded context for many teams.

The ETL must also bootstrap the current state into the stream so a new service starts complete. Eventually the new service becomes the source of data and the ETL is retired. SQL Server also offers Change Tracking, which compares versions and is lighter when you need only the latest state.

Duplicates, retries and dead letters

Failures happen between a service and the broker. A lost acknowledgment makes the sender retry, and the broker then holds a duplicate.

How a duplicate is bornOrder svcBrokerInventory svcpublish eventack lost in networkretry publisheventevent (duplicate)The producer cannot tell if the publish or the ack failed.
DeliveryPromiseCatch
At most onceNo duplicatesMessages can be lost. Fine for telemetry
At least onceNo lossDuplicates. Most common choice
Exactly onceNeitherVery hard. Needs every component in the flow to take part

So assume at-least-once and make handlers safe to repeat. Also acknowledge only after processing, or a crash loses the message.

When to acknowledgeAck firstAck, then processCrash: message leaksProcess firstProcess, then ackCrash: reprocessedAuto-commit of offsets is the usual cause of leaks. Use manual acks.
Making a handler idempotentChange the eventStockOut qty 1Replay twice: minus 2StockChanged now 5Replay twice: still 5Remember the eventStore processed idsOr processed versionsSkip any repeatReplay is harmlessStockChanged is safe but says less about intent, so weigh the trade-off.
Retry then dead-letterConsumeevent arrivesRetrywait + jitterMax retriesstop loopingDead-letterpark the eventReplayafter the fixRetry only errors that can succeed. A poison event must not block the stream.

Events can also arrive out of order, for example when two instances consume at different speeds. Give each event the entity version, and ignore an event older than the stored state when only the latest state matters.

Ignoring an old event by versionEvent v21 arrivesstored state is v22Compare versions21 is below 22Skip itkeep the newer stateAnother cure: route by entity id so one partition keeps order.
Rewind or dead-letter? With a durable broker such as Kafka, moving the offset back after a fix is often simpler than a dead-letter queue. A dead-letter queue pays off when many good events follow the bad one. To save state and publish an event together, use the outbox table (read by polling or by change data capture such as Debezium).

From Practical Event-Driven Microservices Architecture (ch. 5 to 7).

Concurrency and out-of-order events

In a monolith you lock the product in memory. An event-driven service runs many instances, each with many threads, so a local lock protects nothing across instances. Unlikely races stop being unlikely at thousands of events per second. From Practical Event-Driven Microservices Architecture, ch. 6.

Two instances sell the last unitBoth instances read stock 1 and both write 0, so two orders are accepted for one unit.Instance 1DatabaseInstance 2read stock = 1read stock = 1write stock = 0write stock = 0two orders, one unit

Three families of fixes exist. Pessimistic means ask permission first. Optimistic means apologize afterwards. By design means arranging events so the clash cannot happen (next section).

Optimistic update with a version columnThe update only applies if the timestamp is unchanged since the read; zero affected rows means retry.Readstock + timestampCalculatepick warehouseGuarded updateWHERE ts = old0 rows: retrystock changedA SQL UPDATE ... WHERE LatestChangeTimestamp = ? is enough. Elasticsearch, Cassandra and event storesuse the same idea.
PessimisticOptimistic
HowTake a distributed lock (Consul, ZooKeeper, Redis) or a DB transaction, then work alone.No lock. Check at write time that nothing changed.
Best whenConflicts are likely, for example a hot product.Conflicts are rare, for example a house for sale.
RisksStatic lock key makes the service single-threaded; use the product id. A crashed or partitioned instance can lose its lock yet still write. Long transactions hurt.Under contention ten buyers mean one success and nine retries, costlier than a lock. Retries must be idempotent.
Aggregate size matters: locking or versioning per product lets different products run in parallel. Per category would serialize far more work.
Out-of-order events and versionsThe consumer stores version 22, then receives version 21 late and ignores it.Publishedv21 stock 2v22 stock 1Arrivesv22 firstv21 laterConsumerstate is v22ignores v21Causes: publisher clock skew, broker or network issues, retries, replays, concurrent consumers atdifferent speeds.

Give events an aggregate version or a timestamp. Timestamps are easy but not sequential; truly sequential ids are hard to generate in a distributed system. Partial events such as StockIn and StockOut cannot skip versions and are not idempotent, which makes them fragile. Even with versions, two threads can still process 21 and 22 together, so combine with a concurrency strategy.

Partitioning: concurrency solved by design

The better answer is to make the clash impossible. A Kafka topic is a distributed append log split into partitions. Order is guaranteed only inside one partition, and each partition has exactly one consumer per consumer group. Consumers are cheap: a consumer is just an offset pointer. From Practical Event-Driven Microservices Architecture, ch. 6.

Topic, partitions and consumer groupsProducers write to partitions, and one instance per consumer group reads each partition at its own offset.Order service instances publish with a routing keyPartition 0offset 0 1 2 3 ...Partition 1offset 0 1 2 3 ...Partition 2offset 0 1 2 3 ...Inventory 1group: inventoryInventory 2group: inventoryInventory 3group: inventoryA fourth instance would sit idle. Another group, such as product, reads the same data at its ownoffsets.

Use the product id as the routing key. Every order for one product then lands in the same partition, is read by the same instance and in order. The distributed race becomes a local one. Inside the instance, route messages to threads with the same key so the same product is never handled by two threads. The service is parallel across keys and sequential within one.

End-to-end partitioningOne routing key sends all events for a product through one partition, one instance and one thread.Producerkey = product idPartitionsame keysame partitionInstanceone consumerper partitionThreadroute by keyinside serviceResultno locksin orderDifferent products still run in parallel on other partitions and threads.
LimitWhat to do
Hotspots: a key with most events (category instead of product) overloads one partitionChoose a key with enough granularity and diversity, several times the number of partitions.
Momentary hotspots: a hot product launchAccept rare spikes. Avoid recovery jobs that dump many events with one key.
Number of partitions caps scaling; adding partitions does not re-route old dataCreate topics with more partitions than needed today.
A service that mixes events and a synchronous APITurn the request into a command on the queue, or fall back to locking.

Recap

Quick check

You publish an event right after saving to the database and the broker is down. What pattern prevents the mismatch?
Show answer

The transactional outbox. Save the entity and an outbox row in one transaction, then publish the row later through a relay or CDC.

A consumer acknowledges messages on receipt and crashes during a deploy. What happens?
Show answer

The messages in memory are lost (message leak). Acknowledge after processing so unfinished work is redelivered, and make handling idempotent for the duplicates.

A consumer gets the same StockOut (qty 1) event twice after a lost ack. How do you make that safe?
Show answer

Make the handler idempotent: store the processed event id or version and skip repeats, or publish a StockChanged event that carries the current quantity.

Two instances of a service read the same stock and both write. You route events by product id to one partition. Is the race gone?
Show answer

The distributed race is gone because one instance owns that partition. Threads inside the instance can still race, so route by key to threads or use a local lock.