Microservices

Chapter 05 of 27 · Part 3: Communication style

Asynchronous messaging and brokers

Messaging as a communication style: channels, brokers, commands versus events, and the pitfalls to avoid.

key idea / coordinatorservice / good pathin progress / watch outproblem / failureneutral

Asynchronous messaging

Messages travel over channels. A broker sits in the middle, buffers messages and hides service locations.

Message channelsA point-to-point channel delivers to one consumer. A publish-subscribe channel delivers to all.Point-to-point: one consumer gets itSenderChannelABCPublish-subscribe: all consumersSenderChannelABC
Commands usually use point-to-point. Events usually use publish-subscribe.
Async request and replyClientRequest channelServiceReply channelmsgId + reply channeldelivercorrelationId = msgIdmatch reply to requestThe correlation id ties the reply back to the request.
Broker-based messagingBroker: gainsLoose couplingBuffers messagesAll interaction stylesIPC is explicitBroker: costsCan be a bottleneckSingle point of failureOne more thing to run

Brokers in depth

A broker holds messages until they can be delivered. How it groups consumers and counts retries decides how it behaves.

Queues and topicsA queue shares work among instances of one consumer group. A topic gives each consumer group its own copy of every message.Queue: one groupTopic: many groupsSenderQueueInst 1Inst 2Inst 3each message goes to one instanceSenderTopicWarehouseone instanceNotificationsone instanceeach group gets a copy
Delivery guaranteesAt-most-once may lose messages. At-least-once may repeat them. Exactly-once is very hard, so write handlers that tolerate duplicates.At most onceMay lose messages.Never duplicates.At least onceNever lost, may repeat.Make handlers idempotent.Exactly onceVery hard in a cluster.Ask how the broker does it.Order: usually guaranteed only within one partition (Kafka).Kafka also keeps messages for a set time, so consumers can replay them.Telemetry may accept losses. Business events usually cannot.
Poison messages and the dead letter queueA message that crashes every worker keeps returning to the queue. A retry limit moves it to a dead letter queue where people can inspect and replay it.QueueWorker crashesDead letter queuelock expires, message returnsafter the retry limitpeople view, fix and replaySet a maximum retry count on every queue.
From Practical Event-Driven Microservices: at-least-once is arguably the most flexible choice. Failures are rare, so the performance hit is small, and duplicates cost little when handlers are idempotent.

Direct and indirect messaging in practice

Bootstrapping Microservices prefers the words direct and indirect to synchronous and asynchronous.

Direct and indirect messagingDirect: an HTTP POST with an immediate reply. Indirect: a message to a queue the receiver reads later. Broadcast: an exchange copies each message to every bound queue.Direct: HTTP POST to a named serviceVideo-streamingHistoryPOST http://history/vieweddirect responseIndirect: queue, one receiverVideo-streamingQueueHistorypublishpullno reply goes back to the senderIndirect: exchange, broadcastVideo-streamingExchangefanoutQueue AQueue BHistoryAnother svc
Bootstrapping Microservices uses a video-streaming service sending a "viewed" message to a history service.
Direct (HTTP)Indirect (RabbitMQ)
ReplyImmediate success or failureNone to the sender
CouplingSender names the receiverSender does not know who listens
ReceiverCannot ignore itHandles it when ready
Best forCommands needing an answerNotifications and broadcasts
CostTight couplingFlow is harder to trace

With indirect messages no service is in charge. Behavior emerges from services reacting to messages. An unacknowledged message is not lost when a service crashes, because another replica handles it.

From Bootstrapping Microservices: plain DNS names are enough to target a service. Docker Compose uses the container name as the host name, so http://history/viewed works with no extra setup.

Durable or ephemeral broker, and the right to be forgotten

A broker either deletes a message once it is consumed (ephemeral, such as classic RabbitMQ queues) or keeps it in a log (durable, such as Kafka). Durable brokers make replay, new consumers and shared data streams possible. From Practical Event-Driven Microservices Architecture, ch. 3.

Ephemeral queue versus durable logAn ephemeral broker removes messages after acknowledgement. A durable broker keeps them and consumers read by offset.EphemeralProducerQueueremoved after ackConsumerone passDurableProducerEvent logkept, read by offsetService A (offset 120)New service B (offset 0)
EphemeralDurable
Message after handlingRemoved by the brokerStays in the log
Best atPoint-to-point routing, fast short-lived messagesStreaming, replay, sharing data with many consumers
WeaknessStruggles when messages pile up, can hurt the whole clusterReplication cost; erasure and security are harder
Right to be forgottenNaturally satisfiedNeeds a strategy such as crypto shredding
Crypto shreddingEvents are encrypted with a per-user key from a key store. Destroying the key makes the user events unreadable.Order serviceencrypts per userBrokerstores ciphertextInventorydecryptsKey storeone key per userDestroy a user key and that user data is unusable, even though the events stay in the log.

Only the services that handle sensitive data need this. It can also be done at infrastructure level with a sidecar that encrypts before publishing, so that each service does not implement it. Deleting an entity is normally just another event (a tombstone), but history should stay history, which is why the key approach matters.

Message types, patterns and events versus documents

Messages have a header (correlation id, message id, version or timestamp) and a body. The body is one of four kinds. Keep logic-driving flags out of headers. From Practical Event-Driven Microservices Architecture, ch. 3.

Command, event, document and queryFour kinds of message with their naming, meaning and receivers.Commandimperative verbmay be rejectedone receiverEventpast tensea factmany receiversDocumentfull entity stateno change meaningmany receiversQueryread onlyusually sync HTTPnot a messageCreateOrderCommand OrderCreatedEvent OrderDocument GET /stock/15251212
Send/receive, publish/subscribe and request/responseThe UI sends a command, the order service publishes an event to subscribers, and the UI matches the event by correlation id.UIOrder svcInventoryNotifyCreateOrder commandvalidate and saveOrderCreated eventOrderCreated eventevent + correlation idSend/receive: one target. Pub/sub: any number. The response is the event with the same correlationid.

Events and documents both announce that an entity changed. Choose by what the consumer needs. From Practical Event-Driven Microservices Architecture, ch. 3.

One change as an event and as a documentOrderAddressChanged holds the change only. OrderDocument holds the whole order.OrderAddressChangedOrderId, Address, UserIdkeeps the meaning of the changegood for the billing serviceOrderDocumentOrderId, Address, OrderLinesOrderedAt, UserIdmeaning must be inferredgood when most fields are neededorWith Kafka log compaction, documents keyed by OrderId act like a table of current orders.
Consumer situationPrefer
Reacts only to one kind of change (update the invoice address)Event
Needs most of the entity (notify the user with all order data)Document
Needs the meaning or user intent for its logicEvent
Two fields changed together; you want one messageDocument
New consumers must load the current state of everythingDocument on a compacted topic

Default to events that capture user intent, then adapt to the consumers. Lesson 21 returns to event size.

Service topologies and common pitfalls

Not every service is event-driven. The book lists four shapes you will meet. From Practical Event-Driven Microservices Architecture, ch. 3.

Four service topologiesCRUD service, CRUD service that publishes, command-driven service, and process manager.1 CRUD serviceClientServiceDB2 CRUD that also publishesDBServiceEvents3 Commands in, events outCommandsServiceEvents4 Process managerEventsManagerCommands

Topology 1 suits simple, low-scale domains and gives immediate feedback. Topology 2 is a cheap way to bring an existing service into an event-driven system. Topology 3 is fully asynchronous but makes validation feedback hard to return. Topology 4, the process manager, reacts to events in one domain and sends a command to each affected aggregate, for example one per order after a user changes an address.

Three common anti-patternsFaking synchronous answers, publishing a command to many services, and passive-aggressive events.Faking syncAPI returns 202then blocks and pollsCommand to manyCreateOrder commandtwo services obey itPassive-aggressiveStockLow eventhidden "reorder!"
PitfallWhy it hurtsInstead
Faking synchronous responsesBlocking until the change shows up breaks under load and times out.Return 202 and expose status; accept eventual consistency.
Publishing one command to many servicesA command targets one aggregate; others may even reject it.One service owns the command; others react to its event.
Passive-aggressive eventsAn event that secretly asks another service to act is a hidden command.Use a saga or process manager so the flow is visible.

Messaging gotchas

Scaling consumers must not break ordering, and duplicates will happen.

Sharded channel keeps orderMessages with the same shard key go to the same shard and the same consumer instance.Senderkey = orderIdChannel with shardsShard 0Shard 1Shard 2Consumer instance 1Consumer instance 2
All events for one order go to one shard, so they are processed in order.
Handling duplicate messagesMessage arrivesid: xyzInsert id in tableprocessed-ids tableInsert fails?it is a duplicateProcess oncesame transactionBrokers promise at-least-once delivery, so handlers must be idempotent or track message ids.

Messaging improves availability

Every synchronous hop multiplies the failure odds. Cut the hops.

Synchronous calls reduce availabilitySync call chain (all must be up)AvailabilityOrder svc 99.5%alone: 99.5%+ Consumer svc 99.5%two in a row: 99.0%+ Restaurant svc 99.5%three in a row: 98.5%Availability multiplies: 0.995 x 0.995 x 0.995 = 98.5%.
Three ways to cut sync callsAvoid sync callsUse async interaction stylesReplicate data via eventsReply first, finish later
Reply first, finish laterClientOrder svcConsumer svcRestaurant svccreateOrdersave order: PENDINGreturn order idValidateConsumerValidateOrderConsumerValidatedOrderValidatedstate: VALIDATEDOrder service answers right away. Validation finishes through messages.

Recap

Quick check

The billing service must update the invoice address whenever the order address changes. Event or document?
Show answer

An event (OrderAddressChanged). The consumer only cares about that change and the event keeps its meaning. A document would force billing to compare with stored state.

A user asks to be forgotten, but their events sit in a Kafka topic. What can you do?
Show answer

If events were encrypted with a per-user key from a key store, destroy that key. The data in the log becomes unreadable without rewriting the log.

Why is publishing one CreateOrder command to inventory and notification services a smell?
Show answer

A command targets one aggregate and can be rejected. Others should react to the OrderCreated event the order service publishes after it accepts the command.