Kafka Outbox Duplicates: Dedupe by Event ID, Not Offset

In this chapter

The relay sent it twice. Apply it once.

What breaks

The outbox relay publishes OrderPaid, Kafka acks, then the relay crashes before marking the row published. On restart it sends OrderPaid again. Loyalty adds the points twice.

Three ways to respond

• Skip records whose offset was already processed: The resend is a new record with a new offset (42 ≠ 41).
• Turn on enable.idempotence in the relay’s producer: Dedupes retries inside one producer session. A restarted relay is a brand-new send.
• Dedupe on the outbox event id, in the same DB transaction: Insert evt-7 into a unique table with the change. A repeat fails the insert and is skipped. (the fix)

The fix: Dedupe on the outbox event id, in the same DB transaction

The duplicate is created upstream, so Kafka offsets and producer idempotence cannot see it. Give every outbox row a stable id (a UUID written with the business change) and send it in the event. The consumer stores that id in a processed_events table with a unique key, in the same transaction as its own update, then commits the offset. If the effect is naturally idempotent (an upsert, “set status = PAID”, a version check) you can skip the table. Expire old ids after your retention window.

Remember: Dedupe on the event id, not the offset, in the same transaction.

More on Reliability problems

• Transactional Outbox Pattern with Kafka Explained
• Kafka acks=all and min.insync.replicas Explained
• Idempotent Kafka Consumers: Handle Duplicate Events

Keep learning

• Previous: Idempotent Kafka Consumers: Handle Duplicate Events
• Next: Kafka Message Ordering: Global vs Per-Key Order
• All Kafka tutorial chapters

More courses

• Jobs, linked to lessons — Browse jobs where every requirement links to the chapter that teaches it. See how ready you are, how long the gap takes to close, and which listings look stale.
• Match a job post to courses — Paste a job post. Each skill is matched to a track here and put in learning order, foundations first.
• React from open source with patterns — Patterns extracted from Material UI, Radix, React Aria, React Hook Form, Redux Toolkit, TanStack, and Next.js — not slides about hooks.
• Database in depth — How a query becomes work. Indexes, isolation, locks, and the plans that explain why a page is slow — not an ORM tour.
• System Design — Core concepts first (SQL vs NoSQL, CAP, indexing, replication, sharding), then designs grouped by what carries them: the database, Redis, Kafka, or many services.
• Microservices — Microservice patterns one diagram at a time: how to split a monolith, how services talk, and how data stays consistent without distributed transactions.
• Frontend system design — How a UI is sliced when it is a product, not a page: ownership of state, cache, routes, and what happens when the network lies.
• TypeScript in depth — The type system as a design tool. Inference, narrowing, generics, and the errors that mean the model is wrong — not a catalog of utility types.
• CSS in depth — See the box, then the page. Visual chapters from the box model through Flexbox surprises, Grid, selectors, and units, then backgrounds, shapes, and visual effects, each with a playground.
• Accessibility — Foundations first (WCAG in plain words, semantic HTML, lists, alt text), then hands-on HTML labs, a full pass of WAI-ARIA 1.3, and a final fix-the-page challenge.
• Docker — What an image is, why layers cache, and how containers find each other on a network.
• Redis — Fifteen Redis patterns, one picture each: cache, session, lock, counter, rate limiter, queue, and when Redis is the wrong tool.